[ 147.203822] Lustre: soaked-MDT0000: Connection restored to 880972f7-970d-b37a-1a61-2887fd2e2d91 (at 192.168.1.142@o2ib100) [ 147.203825] Lustre: Skipped 6 previous similar messages [ 156.268838] Lustre: soaked-MDT0000: Connection restored to 192.168.1.110@o2ib10 (at 192.168.1.110@o2ib10) [ 156.279660] Lustre: Skipped 24 previous similar messages [ 173.233671] Lustre: MGS: Connection restored to 9665679f-f79b-1bf9-6fd6-abc25509729a (at 192.168.1.131@o2ib100) [ 173.245048] Lustre: Skipped 28 previous similar messages [ 190.236882] Lustre: soaked-MDT0000: Recovery over after 0:43, of 32 clients 32 recovered and 0 were evicted. [ 218.357311] Lustre: MGS: Connection restored to 192.168.1.105@o2ib10 (at 192.168.1.105@o2ib10) [ 218.368494] Lustre: Skipped 12 previous similar messages [ 806.665594] Lustre: soaked-MDT0000: Client 91eacee7-b83c-19ec-9326-47c5ebda930d (at 192.168.1.116@o2ib100) reconnecting [ 806.678906] Lustre: soaked-MDT0000: Connection restored to 91eacee7-b83c-19ec-9326-47c5ebda930d (at 192.168.1.116@o2ib100) [ 806.692353] Lustre: Skipped 9 previous similar messages [ 876.738766] Lustre: soaked-MDT0000: Client 60b5ead5-442a-039e-5d32-f9759a392209 (at 192.168.1.128@o2ib100) reconnecting [ 1020.276376] Lustre: soaked-MDT0000: Client e211cd09-6b0c-e43b-059c-f657101042a0 (at 192.168.1.117@o2ib100) reconnecting [ 1020.291053] Lustre: soaked-MDT0000: Connection restored to e211cd09-6b0c-e43b-059c-f657101042a0 (at 192.168.1.117@o2ib100) [ 1020.305862] Lustre: Skipped 1 previous similar message [ 1216.697961] Lustre: soaked-MDT0000: Client e211cd09-6b0c-e43b-059c-f657101042a0 (at 192.168.1.117@o2ib100) reconnecting [ 1239.749007] Lustre: soaked-MDT0000: Client 31859d53-53e2-75a2-da2d-75f6bdca7375 (at 192.168.1.118@o2ib100) reconnecting [ 1300.750770] Lustre: soaked-MDT0000: Client 6f6036ce-2e3a-7e95-1c07-11e5d5a627fe (at 192.168.1.124@o2ib100) reconnecting [ 1300.765260] Lustre: soaked-MDT0000: Connection restored to 5a0e6555-4c2d-39c9-90c4-a332f62c8353 (at 192.168.1.124@o2ib100) [ 1300.779749] Lustre: Skipped 2 previous similar messages [ 1385.754795] Lustre: soaked-MDT0000: Client d8128f9f-f4c7-b311-df64-40bd3b032886 (at 192.168.1.127@o2ib100) reconnecting [ 1469.707717] Lustre: soaked-MDT0000: Client e211cd09-6b0c-e43b-059c-f657101042a0 (at 192.168.1.117@o2ib100) reconnecting [ 1542.764014] Lustre: soaked-MDT0000: Client e211cd09-6b0c-e43b-059c-f657101042a0 (at 192.168.1.117@o2ib100) reconnecting [ 1742.697552] Lustre: soaked-MDT0000: Client e211cd09-6b0c-e43b-059c-f657101042a0 (at 192.168.1.117@o2ib100) reconnecting [ 1742.712158] Lustre: Skipped 2 previous similar messages [ 1818.700200] Lustre: soaked-MDT0000: Connection restored to e211cd09-6b0c-e43b-059c-f657101042a0 (at 192.168.1.117@o2ib100) [ 1818.714597] Lustre: Skipped 7 previous similar messages [ 1960.700811] Lustre: 3698:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1494355306/real 1494355306] req@ffff8808053c2d00 x1566943337184128/t0(0) o104->soaked-MDT0000@192.168.1.123@o2ib100:15/16 lens 296/224 e 0 to 1 dl 1494355313 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [ 2035.771561] Lustre: soaked-MDT0000: Client 54aab9cb-1b41-13ac-e1b9-f63d1e78e154 (at 192.168.1.137@o2ib100) reconnecting [ 2035.785133] Lustre: Skipped 4 previous similar messages [ 2540.719596] Lustre: soaked-MDT0000: Connection restored to e211cd09-6b0c-e43b-059c-f657101042a0 (at 192.168.1.117@o2ib100) [ 2540.735126] Lustre: Skipped 5 previous similar messages [ 2588.807156] Lustre: soaked-MDT0000: Client e81e92b7-aede-6d47-4552-8475e879b939 (at 192.168.1.134@o2ib100) reconnecting [ 2588.822296] Lustre: Skipped 3 previous similar messages [ 3168.746656] Lustre: soaked-MDT0000: Connection restored to e211cd09-6b0c-e43b-059c-f657101042a0 (at 192.168.1.117@o2ib100) [ 3168.761959] Lustre: Skipped 8 previous similar messages [ 3271.274472] Lustre: soaked-MDT0000: Client a60139c3-b36a-37dd-9360-e306d02df1da (at 192.168.1.131@o2ib100) reconnecting [ 3271.289131] Lustre: Skipped 8 previous similar messages [ 4071.250600] Lustre: soaked-MDT0000: Client e211cd09-6b0c-e43b-059c-f657101042a0 (at 192.168.1.117@o2ib100) reconnecting [ 4071.265568] Lustre: Skipped 3 previous similar messages [ 4071.274511] Lustre: soaked-MDT0000: Connection restored to e211cd09-6b0c-e43b-059c-f657101042a0 (at 192.168.1.117@o2ib100) [ 4071.289924] Lustre: Skipped 4 previous similar messages [ 4107.234610] Lustre: 3702:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1494357452/real 1494357452] req@ffff8807c13c2a00 x1566943434075904/t0(0) o104->soaked-MDT0000@192.168.1.133@o2ib100:15/16 lens 296/224 e 0 to 1 dl 1494357459 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [ 4203.237982] Lustre: 3690:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1494357548/real 1494357548] req@ffff8807e682d700 x1566943439034896/t0(0) o104->soaked-MDT0000@192.168.1.138@o2ib100:15/16 lens 296/224 e 0 to 1 dl 1494357555 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [ 4731.411273] Lustre: soaked-MDT0000: Client dc7bce1e-8cc2-256d-8a24-aa9176826892 (at 192.168.1.125@o2ib100) reconnecting [ 4731.426572] Lustre: Skipped 2 previous similar messages [ 4731.435634] Lustre: soaked-MDT0000: Connection restored to eee4fa14-7423-c90d-6b47-20743657a666 (at 192.168.1.125@o2ib100) [ 4731.451200] Lustre: Skipped 2 previous similar messages [ 4852.341464] Lustre: MGS: Received new LWP connection from 192.168.1.117@o2ib100, removing former export from same NID [ 5679.798406] Lustre: MGS: Received new LWP connection from 192.168.1.125@o2ib100, removing former export from same NID [ 5679.813587] Lustre: MGS: Connection restored to eee4fa14-7423-c90d-6b47-20743657a666 (at 192.168.1.125@o2ib100) [ 5679.827819] Lustre: Skipped 5 previous similar messages [ 5815.372459] Lustre: soaked-MDT0000: Client f75498d6-bc9c-d0bf-fdad-54dda94a52b2 (at 192.168.1.123@o2ib100) reconnecting [ 5815.387540] Lustre: Skipped 4 previous similar messages [ 6707.068214] Lustre: 3465:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1494360052/real 0] req@ffff88009c88a700 x1566943440356288/t0(0) o13->soaked-OST0009-osc-MDT0000@192.168.1.105@o2ib10:7/4 lens 224/368 e 0 to 1 dl 1494360059 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 [ 6707.109009] Lustre: soaked-OST0009-osc-MDT0000: Connection to soaked-OST0009 (at 192.168.1.105@o2ib10) was lost; in progress operations using this service will wait for recovery to complete [ 6708.204232] Lustre: 3468:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1494360053/real 0] req@ffff8803f36fda00 x1566943440356384/t0(0) o13->soaked-OST0003-osc-MDT0000@192.168.1.105@o2ib10:7/4 lens 224/368 e 0 to 1 dl 1494360060 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 [ 6708.245384] Lustre: soaked-OST0003-osc-MDT0000: Connection to soaked-OST0003 (at 192.168.1.105@o2ib10) was lost; in progress operations using this service will wait for recovery to complete [ 6709.700307] Lustre: 3459:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1494360055/real 0] req@ffff8803d9a4b300 x1566943440356528/t0(0) o13->soaked-OST000f-osc-MDT0000@192.168.1.105@o2ib10:7/4 lens 224/368 e 0 to 1 dl 1494360062 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 [ 6709.742155] Lustre: 3459:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 6709.756469] Lustre: soaked-OST000f-osc-MDT0000: Connection to soaked-OST000f (at 192.168.1.105@o2ib10) was lost; in progress operations using this service will wait for recovery to complete [ 6709.781932] Lustre: Skipped 1 previous similar message [ 6712.790413] Lustre: 3457:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1494360059/real 0] req@ffff8803f36f8000 x1566943440356832/t0(0) o8->soaked-OST0009-osc-MDT0000@192.168.1.105@o2ib10:28/4 lens 520/544 e 0 to 1 dl 1494360065 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [ 6770.792424] Lustre: 3457:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1494360112/real 0] req@ffff880360faec00 x1566943440361664/t0(0) o8->soaked-OST000f-osc-MDT0000@192.168.1.105@o2ib10:28/4 lens 520/544 e 0 to 1 dl 1494360123 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [ 6770.834398] Lustre: 3457:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 6792.429530] Lustre: soaked-MDT0000: Client 6985bb7d-8a06-d23c-3360-00e6c4a7e369 (at 192.168.1.143@o2ib100) reconnecting [ 6792.445246] Lustre: Skipped 1 previous similar message [ 6792.454636] Lustre: soaked-MDT0000: Connection restored to 7ea3a484-c027-1559-8163-dc85944c882c (at 192.168.1.143@o2ib100) [ 6792.470642] Lustre: Skipped 2 previous similar messages [ 6797.703361] LNet: 3447:0:(o2iblnd_cb.c:3198:kiblnd_check_conns()) Timed out tx for 192.168.1.105@o2ib10: 2 seconds [ 6850.795225] Lustre: 3457:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1494360187/real 0] req@ffff880360faef00 x1566943440366816/t0(0) o8->soaked-OST0015-osc-MDT0000@192.168.1.105@o2ib10:28/4 lens 520/544 e 0 to 1 dl 1494360203 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [ 6850.837297] Lustre: 3457:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 6873.706004] LNet: 3447:0:(o2iblnd_cb.c:3198:kiblnd_check_conns()) Timed out tx for 192.168.1.105@o2ib10: 3 seconds [ 6873.721194] LNet: 3447:0:(o2iblnd_cb.c:3198:kiblnd_check_conns()) Skipped 1 previous similar message [ 6875.796087] Lustre: 3457:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1494360212/real 0] req@ffff880360faa700 x1566943440368512/t0(0) o8->soaked-OST0003-osc-MDT0000@192.168.1.105@o2ib10:28/4 lens 520/544 e 0 to 1 dl 1494360228 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [ 6875.837956] Lustre: 3457:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 6924.482528] Lustre: MGS: haven't heard from client 6d59f9e1-ce44-7010-80ac-34a863a225a8 (at 192.168.1.105@o2ib10) in 230 seconds. I think it's dead, and I am evicting it. exp ffff8803ef392c00, cur 1494360276 expire 1494360126 last 1494360046 [ 6930.797983] Lustre: 3457:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1494360262/real 0] req@ffff880360fabf00 x1566943440371952/t0(0) o8->soaked-OST000f-osc-MDT0000@192.168.1.105@o2ib10:28/4 lens 520/544 e 0 to 1 dl 1494360283 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [ 6930.840157] Lustre: 3457:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 6950.903056] Lustre: soaked-MDT0000: Received LWP connection from 192.168.1.104@o2ib10, removing former export from 192.168.1.105@o2ib10 [ 6956.871962] Lustre: soaked-MDT0000: Received LWP connection from 192.168.1.104@o2ib10, removing former export from 192.168.1.105@o2ib10 [ 6961.975059] Lustre: soaked-MDT0000: Received LWP connection from 192.168.1.104@o2ib10, removing former export from 192.168.1.105@o2ib10 [ 6967.630431] Lustre: soaked-MDT0000: Received LWP connection from 192.168.1.104@o2ib10, removing former export from 192.168.1.105@o2ib10 [ 7015.073630] LustreError: 11-0: soaked-OST0009-osc-MDT0000: operation ost_create to node 192.168.1.104@o2ib10 failed: rc = -19 [ 7015.089895] Lustre: soaked-OST0009-osc-MDT0000: Connection to soaked-OST0009 (at 192.168.1.104@o2ib10) was lost; in progress operations using this service will wait for recovery to complete [ 7015.115197] LustreError: 3577:0:(osp_precreate.c:602:osp_precreate_send()) soaked-OST0009-osc-MDT0000: can't precreate: rc = -19 [ 7015.134131] LustreError: 3577:0:(osp_precreate.c:1257:osp_precreate_thread()) soaked-OST0009-osc-MDT0000: cannot precreate objects: rc = -19 [ 7065.802724] Lustre: 3457:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1494360412/real 0] req@ffff8803f037d700 x1566943440507120/t0(0) o8->soaked-OST0009-osc-MDT0000@192.168.1.105@o2ib10:28/4 lens 520/544 e 0 to 1 dl 1494360418 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [ 7065.844563] Lustre: 3457:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 7104.026447] LustreError: 11-0: soaked-OST0015-osc-MDT0000: operation ost_statfs to node 192.168.1.104@o2ib10 failed: rc = -107 [ 7104.046785] Lustre: soaked-OST0015-osc-MDT0000: Connection to soaked-OST0015 (at 192.168.1.104@o2ib10) was lost; in progress operations using this service will wait for recovery to complete [ 7113.338839] LustreError: 11-0: soaked-OST0003-osc-MDT0000: operation ost_statfs to node 192.168.1.104@o2ib10 failed: rc = -107 [ 7113.359126] Lustre: soaked-OST0003-osc-MDT0000: Connection to soaked-OST0003 (at 192.168.1.104@o2ib10) was lost; in progress operations using this service will wait for recovery to complete [ 7149.548919] LustreError: 11-0: soaked-OST000f-osc-MDT0000: operation ost_statfs to node 192.168.1.104@o2ib10 failed: rc = -107 [ 7149.565481] Lustre: soaked-OST000f-osc-MDT0000: Connection to soaked-OST000f (at 192.168.1.104@o2ib10) was lost; in progress operations using this service will wait for recovery to complete [ 7166.616620] LustreError: 5550:0:(lfsck_lib.c:2680:lfsck_load_one_trace_file()) soaked-MDT0000-osd: unlink lfsck sub trace file lfsck_namespace_00: rc = 0 [ 7408.971576] NMI watchdog: BUG: soft lockup - CPU#6 stuck for 23s! [OI_scrub:5551] [ 7408.981437] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 7409.118481] CPU: 6 PID: 5551 Comm: OI_scrub Tainted: P OE ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 7409.132130] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 7409.149309] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 7409.159160] RIP: 0010:[] [] osd_inode_iteration+0x489/0xcc0 [osd_ldiskfs] [ 7409.171858] RSP: 0018:ffff880703603d18 EFLAGS: 00000293 [ 7409.179266] RAX: 0000000000000004 RBX: 0000000023f30a01 RCX: 0000000000000000 [ 7409.188702] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 7409.198119] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [ 7409.207516] R10: 0000000023f30a01 R11: ffffea000c8fcc00 R12: 0000000023f30a01 [ 7409.216888] R13: ffff880703603d08 R14: 0000000023f30a01 R15: ffff880703603d08 [ 7409.226244] FS: 0000000000000000(0000) GS:ffff88042e180000(0000) knlGS:0000000000000000 [ 7409.236663] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7409.244443] CR2: 00007f64d55202e0 CR3: 00000000019be000 CR4: 00000000000407e0 [ 7409.253766] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 7409.263065] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 7409.272335] Stack: [ 7409.275859] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [ 7409.285442] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [ 7409.294988] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [ 7409.304513] Call Trace: [ 7409.308465] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 7409.317858] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 7409.326938] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 7409.335799] [] ? wake_up_state+0x20/0x20 [ 7409.343188] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 7409.352696] [] kthread+0xcf/0xe0 [ 7409.359268] [] ? kthread_create_on_node+0x140/0x140 [ 7409.367669] [] ret_from_fork+0x58/0x90 [ 7409.374792] [] ? kthread_create_on_node+0x140/0x140 [ 7409.383151] Code: 00 e8 7c eb 97 ff e9 0f fc ff ff 0f 1f 80 00 00 00 00 45 89 e9 4c 8d 85 67 ff ff ff 48 8b 4d a8 48 8d 55 88 48 8b b5 38 ff ff ff <48> 8b bd 48 ff ff ff 48 8b 85 28 ff ff ff ff d0 85 c0 41 89 c5 [ 7449.101981] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [ 7449.111641] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 7449.244273] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 7449.257593] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 7449.273896] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 7449.283300] RIP: 0010:[] [] osd_inode_iteration+0x299/0xcc0 [osd_ldiskfs] [ 7449.295582] RSP: 0018:ffff880703603d18 EFLAGS: 00000287 [ 7449.302543] RAX: ffff88042bd78800 RBX: ffff880703603d57 RCX: ffff880703603d98 [ 7449.311540] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff88042bd79800 [ 7449.320527] RBP: ffff880703603df0 R08: 0000000000000001 R09: 0000000000000004 [ 7449.329498] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [ 7449.338460] R13: ffffea00027e3c80 R14: 000000009f8f2001 R15: ffff880703603d08 [ 7449.347419] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 7449.357453] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7449.364869] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 7449.373845] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 7449.382821] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 7449.391781] Stack: [ 7449.394999] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [ 7449.404292] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [ 7449.413581] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [ 7449.422887] Call Trace: [ 7449.426631] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 7449.435796] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 7449.444675] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 7449.453365] [] ? wake_up_state+0x20/0x20 [ 7449.460583] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 7449.469952] [] kthread+0xcf/0xe0 [ 7449.476389] [] ? kthread_create_on_node+0x140/0x140 [ 7449.484672] [] ret_from_fork+0x58/0x90 [ 7449.491690] [] ? kthread_create_on_node+0x140/0x140 [ 7449.499965] Code: 8b 45 88 48 8b 80 a8 03 00 00 48 3b 50 20 0f 83 57 01 00 00 44 0f b6 85 67 ff ff ff 48 8d 4d a8 48 8d 55 88 48 8b b5 38 ff ff ff <48> 8b bd 48 ff ff ff 48 8b 85 30 ff ff ff ff d0 83 f8 03 41 89 [ 7477.102955] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [ 7477.112593] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 7477.244786] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 7477.258109] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 7477.274414] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 7477.283822] RIP: 0010:[] [] osd_scrub_exec+0x1c/0x5b0 [osd_ldiskfs] [ 7477.295506] RSP: 0018:ffff880703603c78 EFLAGS: 00000296 [ 7477.302474] RAX: ffffffffa121b990 RBX: ffff880703603c37 RCX: 0000000000000000 [ 7477.311475] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 7477.320458] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [ 7477.329426] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [ 7477.338386] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [ 7477.347339] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 7477.357380] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7477.364792] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 7477.373764] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 7477.382733] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 7477.391685] Stack: [ 7477.394896] ffff880703603d08 ffff880703603d57 ffff880703603df0 000000009f8f2001 [ 7477.404190] ffff880703603d57 000000009f8f2001 0000000000000004 ffff880703603d57 [ 7477.413486] ffff88042bd78800 ffff88042bd78800 00000000000072cf ffff880419dc3a00 [ 7477.422765] Call Trace: [ 7477.426502] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [ 7477.435674] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 7477.444848] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 7477.453722] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 7477.462408] [] ? wake_up_state+0x20/0x20 [ 7477.469617] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 7477.478970] [] kthread+0xcf/0xe0 [ 7477.485408] [] ? kthread_create_on_node+0x140/0x140 [ 7477.493700] [] ret_from_fork+0x58/0x90 [ 7477.500712] [] ? kthread_create_on_node+0x140/0x140 [ 7477.508990] Code: 89 04 24 31 c0 e8 a5 03 98 ff e9 c7 fd ff ff 66 66 66 66 90 55 48 89 e5 41 57 49 89 f7 41 56 41 55 41 54 45 31 e4 53 48 83 ec 68 <48> 8b 9e 08 10 00 00 4c 89 45 90 65 48 8b 04 25 28 00 00 00 48 [ 7483.585176] INFO: rcu_sched self-detected stall on CPU { 24} (t=60000 jiffies g=907112 c=907111 q=32519) [ 7483.597097] Task dump for CPU 24: [ 7483.601835] OI_scrub R running task 0 5551 2 0x00000088 [ 7483.610750] INFO: rcu_sched detected stalls on CPUs/tasks: { [ 7483.610750] ffff88083fde6dd0 00000000fb6db880 ffff88082da03db0 ffffffff810c47e8 [ 7483.627368] 0000000000000018 ffffffff81a1e7c0 ffff88082da03dc8 ffffffff810c8079 [ 7483.636644] 0000000000000009 ffff88082da03df8 ffffffff811378e0 ffff88082da101c0 [ 7483.645897] Call Trace: [ 7483.649561] [] sched_show_task+0xa8/0x110 [ 7483.657516] [] dump_cpu_task+0x39/0x70 [ 7483.664455] [] rcu_dump_cpu_stacks+0x90/0xd0 [ 7483.671954] [] rcu_check_callbacks+0x442/0x720 [ 7483.679627] [] ? tick_sched_handle.isra.13+0x60/0x60 [ 7483.687875] [] update_process_times+0x47/0x80 [ 7483.695408] [] tick_sched_handle.isra.13+0x25/0x60 [ 7483.703412] [] tick_sched_timer+0x41/0x70 [ 7483.710514] [] __hrtimer_run_queues+0xd2/0x260 [ 7483.718081] [] hrtimer_interrupt+0xb0/0x1e0 [ 7483.725353] [] ? call_softirq+0x1c/0x30 [ 7483.732218] [] local_apic_timer_interrupt+0x37/0x60 [ 7483.740259] [] smp_apic_timer_interrupt+0x3f/0x60 [ 7483.748085] [] apic_timer_interrupt+0x6d/0x80 [ 7483.755511] [] ? osd_inode_iteration+0x4ac/0xcc0 [osd_ldiskfs] [ 7483.765316] [] ? osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [ 7483.774416] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 7483.783334] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 7483.791961] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 7483.800373] [] ? wake_up_state+0x20/0x20 [ 7483.807324] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 7483.816412] [] kthread+0xcf/0xe0 [ 7483.822603] [] ? kthread_create_on_node+0x140/0x140 [ 7483.830628] [] ret_from_fork+0x58/0x90 [ 7483.837381] [] ? kthread_create_on_node+0x140/0x140 [ 7483.845401] 24} (detected by 9, t=60262 jiffies, g=907112, c=907111, q=32606) [ 7483.854251] Task dump for CPU 24: [ 7483.858683] OI_scrub R running task 0 5551 2 0x00000088 [ 7483.867293] ffff880703603640 ffffffff8168bdc0 ffff88083fde6dd0 ffff880703603fd8 [ 7483.876334] ffff880703603fd8 ffff880703603fd8 ffff88083fde6dd0 ffff88082da16c40 [ 7483.885368] 0000000000000000 7fffffffffffffff ffff88043ffac170 ffffffff8168a440 [ 7483.894398] Call Trace: [ 7483.897867] [] ? __schedule+0x3b0/0x990 [ 7483.904734] [] ? bit_wait+0x50/0x50 [ 7483.911197] [] schedule+0x29/0x70 [ 7483.917460] [] schedule_timeout+0x239/0x2c0 [ 7483.924705] [] ? ktime_get_ts64+0x4c/0xf0 [ 7483.931744] [] ? delayacct_end+0x8f/0xb0 [ 7483.938663] [] ? __delayacct_blkio_end+0x34/0x60 [ 7483.946341] [] io_schedule_timeout+0xe8/0x130 [ 7483.953708] [] ? __wait_on_bit+0x7e/0x90 [ 7483.960586] [] ? bit_wait+0x50/0x50 [ 7483.966974] [] ? iam_lfix_split+0x140/0x140 [osd_ldiskfs] [ 7483.975489] [] ? out_of_line_wait_on_bit+0x81/0xb0 [ 7483.983318] [] ? wake_bit_function+0x40/0x40 [ 7483.990568] [] ? __wait_on_buffer+0x2a/0x30 [ 7483.997734] [] ? ldiskfs_bread+0x7c/0xc0 [ldiskfs] [ 7484.005577] [] ? ldiskfs_es_lookup_extent+0x2a/0x180 [ldiskfs] [ 7484.014572] [] ? ldiskfs_es_lookup_extent+0x2a/0x180 [ldiskfs] [ 7484.023547] [] ? mark_page_accessed+0x59/0xa0 [ 7484.030866] [] ? __find_get_block+0xbc/0x120 [ 7484.038089] [] ? __getblk+0x2d/0x300 [ 7484.044541] [] ? ldiskfs_getblk+0xa6/0x200 [ldiskfs] [ 7484.052550] [] ? ldiskfs_bread+0x27/0xc0 [ldiskfs] [ 7484.060360] [] ? iam_lfix_lookup+0x19d/0x3b0 [osd_ldiskfs] [ 7484.068996] [] ? lprocfs_counter_sub+0xc1/0x130 [obdclass] [ 7484.077593] [] ? __brelse+0x3d/0x50 [ 7484.083960] [] ? __wake_up+0x44/0x50 [ 7484.090422] [] ? jbd2_journal_stop+0x1ea/0x3d0 [jbd2] [ 7484.098538] [] ? osd_oi_lookup+0xa4/0x160 [osd_ldiskfs] [ 7484.106846] [] ? __slab_free+0x81/0x2f0 [ 7484.113602] [] ? __ldiskfs_journal_stop+0x3c/0xb0 [ldiskfs] [ 7484.122299] [] ? osd_scrub_refresh_mapping+0x1b5/0x440 [osd_ldiskfs] [ 7484.131845] [] ? osd_scrub_check_update+0x4d1/0x10c0 [osd_ldiskfs] [ 7484.141208] [] ? kfree+0x103/0x140 [ 7484.147472] [] ? osd_scrub_check_update+0x4d1/0x10c0 [osd_ldiskfs] [ 7484.156840] [] ? osd_scrub_exec+0x79/0x5b0 [osd_ldiskfs] [ 7484.165246] [] ? osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [ 7484.174236] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 7484.183014] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 7484.191503] [] ? osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 7484.199983] [] ? wake_up_state+0x20/0x20 [ 7484.206821] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 7484.215791] [] ? kthread+0xcf/0xe0 [ 7484.222021] [] ? kthread_create_on_node+0x140/0x140 [ 7484.229908] [] ? ret_from_fork+0x58/0x90 [ 7484.236718] [] ? kthread_create_on_node+0x140/0x140 [ 7509.104065] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [ 7509.113369] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 7509.242314] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 7509.255409] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 7509.271320] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 7509.280561] RIP: 0010:[] [] osd_scrub_exec+0x8d/0x5b0 [osd_ldiskfs] [ 7509.292090] RSP: 0018:ffff880703603ce8 EFLAGS: 00000292 [ 7509.298925] RAX: 0000000000000000 RBX: ffff880703603c78 RCX: 0000000000000000 [ 7509.307803] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 7509.316693] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [ 7509.325591] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000010 [ 7509.334485] R13: ffffffffa121b9c4 R14: ffffffffffffff10 R15: ffff8803162f6000 [ 7509.343378] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 7509.353346] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7509.360693] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 7509.369618] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 7509.378521] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 7509.387406] Stack: [ 7509.390565] ffff8800b2a3746c 0000000000000004 00000000ffffffff 0000000000000001 [ 7509.399800] ffff880703603df0 ffffffffa121d1e9 ffffffffa121b990 ffffffffa1217a20 [ 7509.409057] ffff8800b2a36000 00000000810d354f ffff8803162f6000 ffff8800b2a37468 [ 7509.418311] Call Trace: [ 7509.421999] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [ 7509.431126] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 7509.440249] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 7509.449073] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 7509.457701] [] ? wake_up_state+0x20/0x20 [ 7509.464870] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 7509.474188] [] kthread+0xcf/0xe0 [ 7509.480577] [] ? kthread_create_on_node+0x140/0x140 [ 7509.488817] [] ret_from_fork+0x58/0x90 [ 7509.495804] [] ? kthread_create_on_node+0x140/0x140 [ 7509.504051] Code: 85 c0 41 89 c1 0f 84 a3 01 00 00 41 80 a7 74 14 00 00 fe 48 8b 4d d0 65 48 33 0c 25 28 00 00 00 0f 85 b1 03 00 00 48 83 c4 68 5b <41> 5c 41 5d 41 5e 41 5f 5d c3 66 0f 1f 84 00 00 00 00 00 4c 8d [ 7537.105047] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [ 7537.114663] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 7537.247015] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 7537.260328] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 7537.276595] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 7537.286000] RIP: 0010:[] [] osd_scrub_exec+0x8d/0x5b0 [osd_ldiskfs] [ 7537.297674] RSP: 0018:ffff880703603ce8 EFLAGS: 00000292 [ 7537.304628] RAX: 0000000000000000 RBX: ffff880703603c78 RCX: 0000000000000000 [ 7537.313620] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 7537.322600] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [ 7537.331561] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000010 [ 7537.340513] R13: ffffffffa121ba87 R14: ffffffffffffff10 R15: ffff8803162f6000 [ 7537.349467] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 7537.359492] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7537.366905] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 7537.375875] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 7537.384843] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 7537.393790] Stack: [ 7537.397004] ffff8800b2a3746c 0000000000000004 00000000ffffffff 0000000000000001 [ 7537.406306] ffff880703603df0 ffffffffa121d1e9 ffffffffa121b990 ffffffffa1217a20 [ 7537.415609] ffff8800b2a36000 00000000810d354f ffff8803162f6000 ffff8800b2a37468 [ 7537.424905] Call Trace: [ 7537.428632] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [ 7537.437797] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 7537.446967] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 7537.455835] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 7537.464502] [] ? wake_up_state+0x20/0x20 [ 7537.471716] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 7537.481075] [] kthread+0xcf/0xe0 [ 7537.487515] [] ? kthread_create_on_node+0x140/0x140 [ 7537.495819] [] ret_from_fork+0x58/0x90 [ 7537.502832] [] ? kthread_create_on_node+0x140/0x140 [ 7537.511119] Code: 85 c0 41 89 c1 0f 84 a3 01 00 00 41 80 a7 74 14 00 00 fe 48 8b 4d d0 65 48 33 0c 25 28 00 00 00 0f 85 b1 03 00 00 48 83 c4 68 5b <41> 5c 41 5d 41 5e 41 5f 5d c3 66 0f 1f 84 00 00 00 00 00 4c 8d [ 7565.106022] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [ 7565.115664] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 7565.247902] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 7565.261189] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 7565.277449] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 7565.286832] RIP: 0010:[] [] osd_scrub_next+0x5/0x3c0 [osd_ldiskfs] [ 7565.298396] RSP: 0018:ffff880703603d10 EFLAGS: 00000287 [ 7565.305342] RAX: ffffffffa1217a20 RBX: ffff880703603d57 RCX: ffff880703603d98 [ 7565.314325] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 7565.323293] RBP: ffff880703603df0 R08: 0000000000000001 R09: 0000000000000004 [ 7565.332256] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [ 7565.341209] R13: ffffea00027e3c80 R14: 000000009f8f2001 R15: ffff880703603d08 [ 7565.350164] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 7565.360200] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7565.367615] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 7565.376596] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 7565.385560] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 7565.394512] Stack: [ 7565.397729] ffffffffa121cff9 ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 [ 7565.407010] 00000000810d354f ffff8803162f6000 ffff8800b2a37468 0000000020000000 [ 7565.416320] 010000000000000c 0000000000000000 0000000000000000 ffff8800b2a36000 [ 7565.425624] Call Trace: [ 7565.429352] [] ? osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [ 7565.438720] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 7565.447892] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 7565.456767] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 7565.465434] [] ? wake_up_state+0x20/0x20 [ 7565.472655] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 7565.482018] [] kthread+0xcf/0xe0 [ 7565.488453] [] ? kthread_create_on_node+0x140/0x140 [ 7565.496748] [] ret_from_fork+0x58/0x90 [ 7565.503768] [] ? kthread_create_on_node+0x140/0x140 [ 7565.512041] Code: 48 c1 e0 03 48 8d 4c 02 10 48 01 c2 e8 25 fc ff ff 48 83 c4 08 5b 41 5c 41 5d 41 5e 5d c3 0f 1f 84 00 00 00 00 00 66 66 66 66 90 <55> 48 89 e5 41 57 49 89 f7 41 56 41 55 49 89 fd 41 54 49 89 d4 [ 7593.106996] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [ 7593.116636] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 7593.248385] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 7593.261703] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 7593.277982] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 7593.287378] RIP: 0010:[] [] osd_inode_iteration+0x251/0xcc0 [osd_ldiskfs] [ 7593.299615] RSP: 0018:ffff880703603d18 EFLAGS: 00000286 [ 7593.306576] RAX: ffff8800b2a37468 RBX: 000000009f8f2001 RCX: 0000000000000000 [ 7593.315559] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 7593.324539] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [ 7593.333514] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffff880703603d57 [ 7593.342481] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [ 7593.351449] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 7593.361494] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7593.368919] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 7593.377902] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 7593.386867] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 7593.395820] Stack: [ 7593.399038] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [ 7593.408348] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [ 7593.417638] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [ 7593.426928] Call Trace: [ 7593.430651] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 7593.439824] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 7593.448702] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 7593.457373] [] ? wake_up_state+0x20/0x20 [ 7593.464586] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 7593.473935] [] kthread+0xcf/0xe0 [ 7593.480371] [] ? kthread_create_on_node+0x140/0x140 [ 7593.488652] [] ret_from_fork+0x58/0x90 [ 7593.495668] [] ? kthread_create_on_node+0x140/0x140 [ 7593.503944] Code: b0 00 00 00 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 00 8b 75 98 48 8b 7d 88 e8 dc 2d f5 ff 48 85 c0 48 89 45 90 0f 84 b3 08 00 00 <44> 8b 7d a0 48 8b 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 e8 77 [ 7621.107972] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [ 7621.117616] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 7621.250206] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 7621.263532] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 7621.279855] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 7621.289275] RIP: 0010:[] [] osd_inode_iteration+0x473/0xcc0 [osd_ldiskfs] [ 7621.301566] RSP: 0018:ffff880703603d18 EFLAGS: 00000293 [ 7621.308545] RAX: 0000000000000004 RBX: 000000009f8f2001 RCX: ffff880703603d98 [ 7621.317539] RDX: ffff880703603d78 RSI: 0000000000000000 RDI: ffff8803162f6000 [ 7621.326542] RBP: ffff880703603df0 R08: 0000000000000001 R09: 0000000000000004 [ 7621.335515] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [ 7621.344468] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [ 7621.353425] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 7621.363457] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7621.370871] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 7621.379849] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 7621.388817] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 7621.397769] Stack: [ 7621.400987] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [ 7621.410288] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [ 7621.419586] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [ 7621.428890] Call Trace: [ 7621.432623] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 7621.441778] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 7621.450655] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 7621.459344] [] ? wake_up_state+0x20/0x20 [ 7621.466561] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 7621.475914] [] kthread+0xcf/0xe0 [ 7621.482350] [] ? kthread_create_on_node+0x140/0x140 [ 7621.490652] [] ret_from_fork+0x58/0x90 [ 7621.497669] [] ? kthread_create_on_node+0x140/0x140 [ 7621.505952] Code: 00 00 48 c7 05 9b 86 02 00 00 00 00 00 c7 05 89 86 02 00 01 00 00 00 e8 7c eb 97 ff e9 0f fc ff ff 0f 1f 80 00 00 00 00 45 89 e9 <4c> 8d 85 67 ff ff ff 48 8b 4d a8 48 8d 55 88 48 8b b5 38 ff ff [ 7649.108955] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [ 7649.118578] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 7649.250555] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 7649.263860] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 7649.280148] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 7649.289550] RIP: 0010:[] [] osd_scrub_exec+0x5/0x5b0 [osd_ldiskfs] [ 7649.301104] RSP: 0018:ffff880703603d10 EFLAGS: 00000293 [ 7649.308063] RAX: ffffffffa121b990 RBX: 000000009f8f2001 RCX: 0000000000000000 [ 7649.317046] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 7649.326028] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [ 7649.334988] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffff880703603d57 [ 7649.343936] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [ 7649.352885] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 7649.362911] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7649.370319] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 7649.379293] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 7649.388258] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 7649.397215] Stack: [ 7649.400438] ffffffffa121d1e9 ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 [ 7649.409730] 00000000810d354f ffff8803162f6000 ffff8800b2a37468 0000000020000000 [ 7649.419029] 010000000000000c 0000000000000000 0000000000000000 ffff8800b2a36000 [ 7649.428334] Call Trace: [ 7649.432069] [] ? osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [ 7649.441434] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 7649.450606] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 7649.459492] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 7649.468169] [] ? wake_up_state+0x20/0x20 [ 7649.475390] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 7649.484744] [] kthread+0xcf/0xe0 [ 7649.491187] [] ? kthread_create_on_node+0x140/0x140 [ 7649.499478] [] ret_from_fork+0x58/0x90 [ 7649.506495] [] ? kthread_create_on_node+0x140/0x140 [ 7649.514770] Code: 05 85 a5 02 00 00 00 02 00 89 54 24 08 48 8b 40 40 ba 07 00 00 00 48 89 04 24 31 c0 e8 a5 03 98 ff e9 c7 fd ff ff 66 66 66 66 90 <55> 48 89 e5 41 57 49 89 f7 41 56 41 55 41 54 45 31 e4 53 48 83 [ 7663.615454] INFO: rcu_sched self-detected stall on CPU { 24} (t=240024 jiffies g=907112 c=907111 q=317538) [ 7663.627634] Task dump for CPU 24: [ 7663.632386] OI_scrub R running task 0 5551 2 0x00000088 [ 7663.641323] ffff88083fde6dd0 00000000fb6db880 ffff88082da03db0 ffffffff810c47e8 [ 7663.650653] 0000000000000018 ffffffff81a1e7c0 ffff88082da03dc8 ffffffff810c8079 [ 7663.659955] 0000000000000009 ffff88082da03df8 ffffffff811378e0 ffff88082da101c0 [ 7663.669240] Call Trace: [ 7663.672935] [] sched_show_task+0xa8/0x110 [ 7663.680934] [] dump_cpu_task+0x39/0x70 [ 7663.687912] [] rcu_dump_cpu_stacks+0x90/0xd0 [ 7663.695447] [] rcu_check_callbacks+0x442/0x720 [ 7663.703157] [] ? tick_sched_handle.isra.13+0x60/0x60 [ 7663.711421] [] update_process_times+0x47/0x80 [ 7663.718983] [] tick_sched_handle.isra.13+0x25/0x60 [ 7663.727007] [] tick_sched_timer+0x41/0x70 [ 7663.734156] [] __hrtimer_run_queues+0xd2/0x260 [ 7663.741759] [] hrtimer_interrupt+0xb0/0x1e0 [ 7663.749052] [] ? call_softirq+0x1c/0x30 [ 7663.755938] [] local_apic_timer_interrupt+0x37/0x60 [ 7663.763976] [] smp_apic_timer_interrupt+0x3f/0x60 [ 7663.771824] [] apic_timer_interrupt+0x6d/0x80 [ 7663.779260] [] ? osd_scrub_next+0x149/0x3c0 [osd_ldiskfs] [ 7663.788604] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [ 7663.797522] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 7663.806424] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 7663.815043] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 7663.823465] [] ? wake_up_state+0x20/0x20 [ 7663.830419] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 7663.839515] [] kthread+0xcf/0xe0 [ 7663.845691] [] ? kthread_create_on_node+0x140/0x140 [ 7663.853721] [] ret_from_fork+0x58/0x90 [ 7663.860482] [] ? kthread_create_on_node+0x140/0x140 [ 7689.110349] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [ 7689.119772] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 7689.249982] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 7689.263158] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 7689.279206] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 7689.288508] RIP: 0010:[] [] osd_inode_iteration+0x2be/0xcc0 [osd_ldiskfs] [ 7689.300660] RSP: 0018:ffff880703603d18 EFLAGS: 00000297 [ 7689.307516] RAX: 0000000000000004 RBX: 000000009f8f2001 RCX: ffff880703603d98 [ 7689.316415] RDX: ffff880703603d78 RSI: 0000000000000000 RDI: ffff8803162f6000 [ 7689.325307] RBP: ffff880703603df0 R08: 0000000000000001 R09: 0000000000000004 [ 7689.334197] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffff880703603d57 [ 7689.343091] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [ 7689.351979] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 7689.361941] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7689.369286] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 7689.378188] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 7689.387095] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 7689.395988] Stack: [ 7689.399152] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [ 7689.408394] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [ 7689.417632] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [ 7689.426898] Call Trace: [ 7689.430592] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 7689.439729] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 7689.448569] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 7689.457213] [] ? wake_up_state+0x20/0x20 [ 7689.464402] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 7689.473722] [] kthread+0xcf/0xe0 [ 7689.480118] [] ? kthread_create_on_node+0x140/0x140 [ 7689.488366] [] ret_from_fork+0x58/0x90 [ 7689.495347] [] ? kthread_create_on_node+0x140/0x140 [ 7689.503597] Code: 8b b5 38 ff ff ff 48 8b bd 48 ff ff ff 48 8b 85 30 ff ff ff ff d0 83 f8 03 41 89 c5 0f 84 4b 02 00 00 0f 8e 15 01 00 00 83 f8 05 <66> 90 0f 84 8a 02 00 00 83 f8 06 0f 85 a1 01 00 00 48 8b 7d 90 [ 7717.111318] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [ 7717.120935] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 7717.253424] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 7717.266748] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 7717.283101] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 7717.292534] RIP: 0010:[] [] ldiskfs_itable_unused_count+0x6/0x30 [ldiskfs] [ 7717.304894] RSP: 0018:ffff880703603d08 EFLAGS: 00000286 [ 7717.311879] RAX: ffff8800b2a37468 RBX: ffff880703603d57 RCX: 0000000000000000 [ 7717.320879] RDX: ffff880703603d78 RSI: ffff880419dc3a00 RDI: ffff88042bd79800 [ 7717.329874] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [ 7717.338848] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [ 7717.347809] R13: 0000000000000018 R14: ffff880703603c78 R15: 0000000000000296 [ 7717.356756] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 7717.366776] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7717.374181] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 7717.383155] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 7717.392119] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 7717.401065] Stack: [ 7717.404272] ffff880703603df0 ffffffffa121cfb9 ffffffffa121b990 ffffffffa1217a20 [ 7717.413548] ffff8800b2a36000 00000000810d354f ffff8803162f6000 ffff8800b2a37468 [ 7717.422845] 0000000020000000 010000000000000c 0000000000000000 0000000000000000 [ 7717.432137] Call Trace: [ 7717.435883] [] ? osd_inode_iteration+0x269/0xcc0 [osd_ldiskfs] [ 7717.445239] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 7717.454414] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 7717.463280] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 7717.471962] [] ? wake_up_state+0x20/0x20 [ 7717.479173] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 7717.488536] [] kthread+0xcf/0xe0 [ 7717.494963] [] ? kthread_create_on_node+0x140/0x140 [ 7717.503238] [] ret_from_fork+0x58/0x90 [ 7717.510245] [] ? kthread_create_on_node+0x140/0x140 [ 7717.518512] Code: 00 00 31 c0 0f b7 56 10 48 89 e5 48 83 39 3f 76 07 0f b7 46 30 c1 e0 10 09 d0 5d c3 66 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 <48> 8b 8f a8 03 00 00 31 c0 0f b7 56 1c 48 89 e5 48 83 39 3f 76 [ 7745.112287] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [ 7745.121947] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 7745.254065] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 7745.267351] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 7745.283624] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 7745.293014] RIP: 0010:[] [] ldiskfs_itable_unused_count+0x25/0x30 [ldiskfs] [ 7745.305452] RSP: 0018:ffff880703603d08 EFLAGS: 00000202 [ 7745.312393] RAX: 00000000000072ce RBX: 0000000000000001 RCX: ffff88042bd78800 [ 7745.321391] RDX: 00000000000072ce RSI: ffff880419dc3a00 RDI: ffff88042bd79800 [ 7745.330368] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [ 7745.339313] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [ 7745.348262] R13: 0000000000000018 R14: ffff880703603c78 R15: 0000000000000296 [ 7745.357211] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 7745.367235] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7745.374639] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 7745.383605] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 7745.392568] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 7745.401514] Stack: [ 7745.404718] ffff880703603df0 ffffffffa121cfb9 ffffffffa121b990 ffffffffa1217a20 [ 7745.414005] ffff8800b2a36000 00000000810d354f ffff8803162f6000 ffff8800b2a37468 [ 7745.423313] 0000000020000000 010000000000000c 0000000000000000 0000000000000000 [ 7745.432596] Call Trace: [ 7745.436323] [] osd_inode_iteration+0x269/0xcc0 [osd_ldiskfs] [ 7745.445503] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 7745.454666] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 7745.463526] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 7745.472198] [] ? wake_up_state+0x20/0x20 [ 7745.479398] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 7745.488764] [] kthread+0xcf/0xe0 [ 7745.495186] [] ? kthread_create_on_node+0x140/0x140 [ 7745.503460] [] ret_from_fork+0x58/0x90 [ 7745.510467] [] ? kthread_create_on_node+0x140/0x140 [ 7745.518736] Code: 84 00 00 00 00 00 66 66 66 66 90 55 48 8b 8f a8 03 00 00 31 c0 0f b7 56 1c 48 89 e5 48 83 39 3f 76 07 0f b7 46 32 c1 e0 10 09 d0 <5d> c3 66 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 89 16 48 8b [ 7773.113267] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [ 7773.122905] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 7773.255132] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 7773.268412] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 7773.284667] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 7773.294051] RIP: 0010:[] [] osd_scrub_exec+0xf9/0x5b0 [osd_ldiskfs] [ 7773.305698] RSP: 0018:ffff880703603c78 EFLAGS: 00000246 [ 7773.312640] RAX: 0000000000000000 RBX: ffff880703603c37 RCX: 0000000000000000 [ 7773.321605] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 7773.330566] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [ 7773.339514] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [ 7773.348459] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [ 7773.357409] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 7773.367435] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7773.374842] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 7773.383816] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 7773.392779] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 7773.401735] Stack: [ 7773.404951] 0000000000000004 0000000000000001 ffff880703603df0 ffff880703603d98 [ 7773.414264] ffff880703603d57 000000009f8f2001 0000000000000004 ffff880703603d57 [ 7773.423561] 0000000000000004 ffff880703603d98 ffff880703603d78 0000000000000000 [ 7773.432850] Call Trace: [ 7773.436576] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [ 7773.445743] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 7773.454903] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 7773.463771] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 7773.472458] [] ? wake_up_state+0x20/0x20 [ 7773.479664] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 7773.489025] [] kthread+0xcf/0xe0 [ 7773.495453] [] ? kthread_create_on_node+0x140/0x140 [ 7773.503753] [] ret_from_fork+0x58/0x90 [ 7773.510765] [] ? kthread_create_on_node+0x140/0x140 [ 7773.519034] Code: 88 48 85 db 74 23 f6 83 18 0a 00 00 08 74 1a 4d 85 e4 74 15 41 8b 87 6c 14 00 00 41 39 84 24 0c 0a 00 00 0f 82 d9 00 00 00 31 c0 <41> f6 87 74 14 00 00 04 0f 85 6e ff ff ff 41 83 f9 02 0f 84 64 [ 7801.114237] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [ 7801.123938] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 7801.256247] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 7801.269533] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 7801.285798] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 7801.295184] RIP: 0010:[] [] osd_scrub_exec+0x1c/0x5b0 [osd_ldiskfs] [ 7801.306838] RSP: 0018:ffff880703603c78 EFLAGS: 00000296 [ 7801.313785] RAX: ffffffffa121b990 RBX: ffff880703603c37 RCX: 0000000000000000 [ 7801.322770] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 7801.331741] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [ 7801.340702] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [ 7801.349659] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [ 7801.358617] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 7801.368652] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7801.376065] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 7801.385029] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 7801.393991] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 7801.402940] Stack: [ 7801.406156] ffff880703603d08 0000000000000001 ffff880703603df0 000000009f8f2001 [ 7801.415463] ffff880703603d57 000000009f8f2001 0000000000000004 ffff880703603d57 [ 7801.424760] ffff88042bd78800 ffff88042bd78800 00000000000072cf ffff880419dc3a00 [ 7801.434050] Call Trace: [ 7801.437771] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [ 7801.446947] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 7801.456108] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 7801.464971] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 7801.473643] [] ? wake_up_state+0x20/0x20 [ 7801.480851] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 7801.490206] [] kthread+0xcf/0xe0 [ 7801.496636] [] ? kthread_create_on_node+0x140/0x140 [ 7801.504929] [] ret_from_fork+0x58/0x90 [ 7801.511939] [] ? kthread_create_on_node+0x140/0x140 [ 7801.520215] Code: 89 04 24 31 c0 e8 a5 03 98 ff e9 c7 fd ff ff 66 66 66 66 90 55 48 89 e5 41 57 49 89 f7 41 56 41 55 41 54 45 31 e4 53 48 83 ec 68 <48> 8b 9e 08 10 00 00 4c 89 45 90 65 48 8b 04 25 28 00 00 00 48 [ 7829.115219] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [ 7829.124851] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 7829.257016] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 7829.270298] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 7829.286560] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 7829.295930] RIP: 0010:[] [] ldiskfs_itable_unused_count+0x23/0x30 [ldiskfs] [ 7829.308364] RSP: 0018:ffff880703603d08 EFLAGS: 00000297 [ 7829.315308] RAX: 0000000000000000 RBX: 0000000000000001 RCX: ffff88042bd78800 [ 7829.324289] RDX: 00000000000072ce RSI: ffff880419dc3a00 RDI: ffff88042bd79800 [ 7829.333248] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [ 7829.342204] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [ 7829.351156] R13: 0000000000000018 R14: ffff880703603c78 R15: 0000000000000246 [ 7829.360105] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 7829.370132] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7829.377534] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 7829.386503] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 7829.395453] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 7829.404403] Stack: [ 7829.407607] ffff880703603df0 ffffffffa121cfb9 ffffffffa121b990 ffffffffa1217a20 [ 7829.416904] ffff8800b2a36000 00000000810d354f ffff8803162f6000 ffff8800b2a37468 [ 7829.426195] 0000000020000000 010000000000000c 0000000000000000 0000000000000000 [ 7829.435482] Call Trace: [ 7829.439215] [] osd_inode_iteration+0x269/0xcc0 [osd_ldiskfs] [ 7829.448378] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 7829.457538] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 7829.466399] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 7829.475074] [] ? wake_up_state+0x20/0x20 [ 7829.482282] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 7829.491637] [] kthread+0xcf/0xe0 [ 7829.498066] [] ? kthread_create_on_node+0x140/0x140 [ 7829.506354] [] ret_from_fork+0x58/0x90 [ 7829.513362] [] ? kthread_create_on_node+0x140/0x140 [ 7829.521629] Code: 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 8b 8f a8 03 00 00 31 c0 0f b7 56 1c 48 89 e5 48 83 39 3f 76 07 0f b7 46 32 c1 e0 10 <09> d0 5d c3 66 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 89 16 [ 7843.624724] INFO: rcu_sched self-detected stall on CPU { 24} (t=420027 jiffies g=907112 c=907111 q=736665) [ 7843.636869] Task dump for CPU 24: [ 7843.641610] OI_scrub R running task 0 5551 2 0x00000088 [ 7843.650505] ffff88083fde6dd0 00000000fb6db880 ffff88082da03db0 ffffffff810c47e8 [ 7843.659825] 0000000000000018 ffffffff81a1e7c0 ffff88082da03dc8 ffffffff810c8079 [ 7843.669131] 0000000000000009 ffff88082da03df8 ffffffff811378e0 ffff88082da101c0 [ 7843.678406] Call Trace: [ 7843.682093] [] sched_show_task+0xa8/0x110 [ 7843.690070] [] dump_cpu_task+0x39/0x70 [ 7843.697024] [] rcu_dump_cpu_stacks+0x90/0xd0 [ 7843.704546] [] rcu_check_callbacks+0x442/0x720 [ 7843.712237] [] ? tick_sched_handle.isra.13+0x60/0x60 [ 7843.720488] [] update_process_times+0x47/0x80 [ 7843.728035] [] tick_sched_handle.isra.13+0x25/0x60 [ 7843.736038] [] tick_sched_timer+0x41/0x70 [ 7843.743154] [] __hrtimer_run_queues+0xd2/0x260 [ 7843.750729] [] hrtimer_interrupt+0xb0/0x1e0 [ 7843.758002] [] ? call_softirq+0x1c/0x30 [ 7843.764881] [] local_apic_timer_interrupt+0x37/0x60 [ 7843.772900] [] smp_apic_timer_interrupt+0x3f/0x60 [ 7843.780721] [] apic_timer_interrupt+0x6d/0x80 [ 7843.788142] [] ? osd_inode_iteration+0x49b/0xcc0 [osd_ldiskfs] [ 7843.797973] [] ? osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [ 7843.807065] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 7843.815959] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 7843.824566] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 7843.832979] [] ? wake_up_state+0x20/0x20 [ 7843.839935] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 7843.849048] [] kthread+0xcf/0xe0 [ 7843.855225] [] ? kthread_create_on_node+0x140/0x140 [ 7843.863269] [] ret_from_fork+0x58/0x90 [ 7843.870032] [] ? kthread_create_on_node+0x140/0x140 [ 7869.116620] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [ 7869.126049] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 7869.256223] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 7869.269396] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 7869.285484] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 7869.294797] RIP: 0010:[] [] osd_scrub_exec+0xd4/0x5b0 [osd_ldiskfs] [ 7869.306380] RSP: 0018:ffff880703603c78 EFLAGS: 00000286 [ 7869.313247] RAX: ffff8803162f1008 RBX: ffff880703603c37 RCX: 0000000000000000 [ 7869.322158] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 7869.331063] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [ 7869.339966] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [ 7869.348863] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [ 7869.357763] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 7869.367738] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7869.375092] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 7869.384003] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 7869.392922] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 7869.401820] Stack: [ 7869.405003] 0000000000000004 0000000000000001 0000000000000004 ffff880703603d98 [ 7869.414269] ffff880703603d57 0000000000000000 ffff8803162f6000 ffffffffffffff10 [ 7869.423526] ffffffffa1217b80 0000000000000010 0000000000000282 ffff880703603ce0 [ 7869.432774] Call Trace: [ 7869.436482] [] ? osd_scrub_next+0x160/0x3c0 [osd_ldiskfs] [ 7869.445332] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [ 7869.454476] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 7869.463614] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 7869.472457] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 7869.481104] [] ? wake_up_state+0x20/0x20 [ 7869.488277] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 7869.497603] [] kthread+0xcf/0xe0 [ 7869.504005] [] ? kthread_create_on_node+0x140/0x140 [ 7869.512255] [] ret_from_fork+0x58/0x90 [ 7869.519240] [] ? kthread_create_on_node+0x140/0x140 [ 7869.527491] Code: 4d 88 4c 89 ef e8 1d fb 46 e0 41 83 87 68 14 00 00 01 49 83 87 88 11 00 00 01 4c 89 ef e8 f5 9f e9 df 44 8b 4d 88 48 85 db 74 23 83 18 0a 00 00 08 74 1a 4d 85 e4 74 15 41 8b 87 6c 14 00 00 [ 7888.786761] perf: interrupt took too long (2552 > 2500), lowering kernel.perf_event_max_sample_rate to 78000 [ 7913.118138] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [ 7913.127746] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 7913.259299] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 7913.272570] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 7913.288783] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 7913.298151] RIP: 0010:[] [] osd_inode_iteration+0x49b/0xcc0 [osd_ldiskfs] [ 7913.310371] RSP: 0018:ffff880703603d18 EFLAGS: 00000246 [ 7913.317309] RAX: 0000000000000000 RBX: 000000009f8f2001 RCX: 0000000000000000 [ 7913.326285] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 7913.335257] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [ 7913.344228] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffff880703603d57 [ 7913.353209] R13: ffff880703603d08 R14: 000000009f8f2001 R15: ffffea00027e3c80 [ 7913.362181] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 7913.372233] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7913.379655] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 7913.388634] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 7913.397619] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 7913.406591] Stack: [ 7913.409834] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [ 7913.419151] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [ 7913.428463] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [ 7913.437783] Call Trace: [ 7913.441527] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 7913.450708] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 7913.459594] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 7913.468289] [] ? wake_up_state+0x20/0x20 [ 7913.475524] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 7913.484895] [] kthread+0xcf/0xe0 [ 7913.491358] [] ? kthread_create_on_node+0x140/0x140 [ 7913.499671] [] ret_from_fork+0x58/0x90 [ 7913.506723] [] ? kthread_create_on_node+0x140/0x140 [ 7913.515022] Code: 45 89 e9 4c 8d 85 67 ff ff ff 48 8b 4d a8 48 8d 55 88 48 8b b5 38 ff ff ff 48 8b bd 48 ff ff ff 48 8b 85 28 ff ff ff ff d0 85 c0 <41> 89 c5 75 1c 8b 85 58 ff ff ff 41 3b 04 24 72 10 48 8b 85 50 [ 7941.119120] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [ 7941.128790] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 7941.261388] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 7941.274706] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 7941.291031] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 7941.300454] RIP: 0010:[] [] osd_scrub_exec+0x30/0x5b0 [osd_ldiskfs] [ 7941.312154] RSP: 0018:ffff880703603c78 EFLAGS: 00000296 [ 7941.319131] RAX: 00000000fb6db880 RBX: ffff880703603c37 RCX: 0000000000000000 [ 7941.328130] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 7941.337109] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [ 7941.346080] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [ 7941.355044] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [ 7941.363995] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 7941.374031] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7941.381447] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 7941.390422] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 7941.399400] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 7941.408369] Stack: [ 7941.411588] ffff880703603df0 000000009f8f2001 ffffea00027e3c80 000000009f8f2001 [ 7941.420880] ffff880703603d57 ffff880703603d57 ffff8800b2a37468 0000000000000000 [ 7941.430169] ffff880703603d78 ffff880419dc3a00 ffff88042bd79800 ffffffffffffff10 [ 7941.439465] Call Trace: [ 7941.443198] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [ 7941.452369] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 7941.461526] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 7941.470398] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 7941.479076] [] ? wake_up_state+0x20/0x20 [ 7941.486284] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 7941.495639] [] kthread+0xcf/0xe0 [ 7941.502069] [] ? kthread_create_on_node+0x140/0x140 [ 7941.510361] [] ret_from_fork+0x58/0x90 [ 7941.517353] [] ? kthread_create_on_node+0x140/0x140 [ 7941.525623] Code: 55 48 89 e5 41 57 49 89 f7 41 56 41 55 41 54 45 31 e4 53 48 83 ec 68 48 8b 9e 08 10 00 00 4c 89 45 90 65 48 8b 04 25 28 00 00 00 <48> 89 45 d0 31 c0 48 8d 43 08 48 85 db 4c 0f 45 e0 41 83 f9 04 [ 7969.120090] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [ 7969.129754] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 7969.261862] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 7969.275159] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 7969.291410] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 7969.300802] RIP: 0010:[] [] ldiskfs_itable_unused_count+0x23/0x30 [ldiskfs] [ 7969.313246] RSP: 0018:ffff880703603d08 EFLAGS: 00000297 [ 7969.320196] RAX: 0000000000000000 RBX: 000000009f8f2001 RCX: ffff88042bd78800 [ 7969.329171] RDX: 00000000000072ce RSI: ffff880419dc3a00 RDI: ffff88042bd79800 [ 7969.338135] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [ 7969.347081] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [ 7969.356025] R13: 0000000000000018 R14: ffff880703603c78 R15: 0000000000000246 [ 7969.364974] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 7969.375000] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7969.382402] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 7969.391364] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 7969.400320] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 7969.409266] Stack: [ 7969.412470] ffff880703603df0 ffffffffa121cfb9 ffffffffa121b990 ffffffffa1217a20 [ 7969.421746] ffff8800b2a36000 00000000810d354f ffff8803162f6000 ffff8800b2a37468 [ 7969.431025] 0000000020000000 010000000000000c 0000000000000000 0000000000000000 [ 7969.440317] Call Trace: [ 7969.444049] [] osd_inode_iteration+0x269/0xcc0 [osd_ldiskfs] [ 7969.453216] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 7969.462380] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 7969.471244] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 7969.479914] [] ? wake_up_state+0x20/0x20 [ 7969.487122] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 7969.496477] [] kthread+0xcf/0xe0 [ 7969.502915] [] ? kthread_create_on_node+0x140/0x140 [ 7969.511194] [] ret_from_fork+0x58/0x90 [ 7969.518206] [] ? kthread_create_on_node+0x140/0x140 [ 7969.526484] Code: 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 8b 8f a8 03 00 00 31 c0 0f b7 56 1c 48 89 e5 48 83 39 3f 76 07 0f b7 46 32 c1 e0 10 <09> d0 5d c3 66 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 89 16 [ 7997.121077] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [ 7997.130716] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 7997.262935] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 7997.276226] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 7997.292496] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 7997.301883] RIP: 0010:[] [] osd_scrub_exec+0x1c/0x5b0 [osd_ldiskfs] [ 7997.313547] RSP: 0018:ffff880703603c78 EFLAGS: 00000296 [ 7997.320483] RAX: ffffffffa121b990 RBX: ffff880703603c37 RCX: 0000000000000000 [ 7997.329454] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 7997.338418] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [ 7997.347370] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [ 7997.356319] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [ 7997.365272] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 7997.375303] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7997.382713] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 7997.391689] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 7997.400652] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 7997.409611] Stack: [ 7997.412828] ffff880703603d08 000000009f8f2001 ffff880703603df0 000000009f8f2001 [ 7997.422139] ffff880703603d57 000000009f8f2001 0000000000000004 ffff880703603d57 [ 7997.431433] ffff8800b2a37468 0000000000000000 ffff880703603d78 ffff8800b2a36000 [ 7997.440727] Call Trace: [ 7997.444457] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [ 7997.453626] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 7997.462781] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 7997.471647] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 7997.480318] [] ? wake_up_state+0x20/0x20 [ 7997.487523] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 7997.496879] [] kthread+0xcf/0xe0 [ 7997.503318] [] ? kthread_create_on_node+0x140/0x140 [ 7997.511605] [] ret_from_fork+0x58/0x90 [ 7997.518619] [] ? kthread_create_on_node+0x140/0x140 [ 7997.526891] Code: 89 04 24 31 c0 e8 a5 03 98 ff e9 c7 fd ff ff 66 66 66 66 90 55 48 89 e5 41 57 49 89 f7 41 56 41 55 41 54 45 31 e4 53 48 83 ec 68 <48> 8b 9e 08 10 00 00 4c 89 45 90 65 48 8b 04 25 28 00 00 00 48 [ 8023.633993] INFO: rcu_sched self-detected stall on CPU { 24} (t=600030 jiffies g=907112 c=907111 q=1108460) [ 8023.646258] Task dump for CPU 24: [ 8023.651005] OI_scrub R running task 0 5551 2 0x00000088 [ 8023.659926] ffff88083fde6dd0 00000000fb6db880 ffff88082da03db0 ffffffff810c47e8 [ 8023.669234] 0000000000000018 ffffffff81a1e7c0 ffff88082da03dc8 ffffffff810c8079 [ 8023.678525] 0000000000000009 ffff88082da03df8 ffffffff811378e0 ffff88082da101c0 [ 8023.687789] Call Trace: [ 8023.691478] [] sched_show_task+0xa8/0x110 [ 8023.699452] [] dump_cpu_task+0x39/0x70 [ 8023.706417] [] rcu_dump_cpu_stacks+0x90/0xd0 [ 8023.713942] [] rcu_check_callbacks+0x442/0x720 [ 8023.721633] [] ? tick_sched_handle.isra.13+0x60/0x60 [ 8023.729891] [] update_process_times+0x47/0x80 [ 8023.737443] [] tick_sched_handle.isra.13+0x25/0x60 [ 8023.745457] [] tick_sched_timer+0x41/0x70 [ 8023.752577] [] __hrtimer_run_queues+0xd2/0x260 [ 8023.760163] [] hrtimer_interrupt+0xb0/0x1e0 [ 8023.767445] [] ? call_softirq+0x1c/0x30 [ 8023.774323] [] local_apic_timer_interrupt+0x37/0x60 [ 8023.782349] [] smp_apic_timer_interrupt+0x3f/0x60 [ 8023.790177] [] apic_timer_interrupt+0x6d/0x80 [ 8023.797608] [] ? osd_inode_iteration+0x251/0xcc0 [osd_ldiskfs] [ 8023.807445] [] ? osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [ 8023.816543] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 8023.825451] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 8023.834068] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 8023.842480] [] ? wake_up_state+0x20/0x20 [ 8023.849433] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 8023.858526] [] kthread+0xcf/0xe0 [ 8023.864699] [] ? kthread_create_on_node+0x140/0x140 [ 8023.872740] [] ret_from_fork+0x58/0x90 [ 8023.879503] [] ? kthread_create_on_node+0x140/0x140 [ 8049.122881] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [ 8049.132267] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 8049.262514] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 8049.275697] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 8049.291782] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 8049.301086] RIP: 0010:[] [] osd_inode_iteration+0x4a6/0xcc0 [osd_ldiskfs] [ 8049.313241] RSP: 0018:ffff880703603d18 EFLAGS: 00000246 [ 8049.320113] RAX: 0000000020000000 RBX: 000000009f8f2001 RCX: 0000000000000000 [ 8049.329026] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 8049.337939] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [ 8049.346837] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [ 8049.355745] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [ 8049.364650] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 8049.374629] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8049.381983] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 8049.390898] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8049.399819] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 8049.408727] Stack: [ 8049.411904] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [ 8049.421167] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [ 8049.430440] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [ 8049.439683] Call Trace: [ 8049.443369] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 8049.452515] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 8049.461364] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 8049.470030] [] ? wake_up_state+0x20/0x20 [ 8049.477234] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 8049.486571] [] kthread+0xcf/0xe0 [ 8049.492991] [] ? kthread_create_on_node+0x140/0x140 [ 8049.501251] [] ret_from_fork+0x58/0x90 [ 8049.508242] [] ? kthread_create_on_node+0x140/0x140 [ 8049.516501] Code: 8b 4d a8 48 8d 55 88 48 8b b5 38 ff ff ff 48 8b bd 48 ff ff ff 48 8b 85 28 ff ff ff ff d0 85 c0 41 89 c5 75 1c 8b 85 58 ff ff ff <41> 3b 04 24 72 10 48 8b 85 50 ff ff ff 44 3b 30 0f 87 95 fd ff [ 8077.123851] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [ 8077.133459] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 8077.265336] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 8077.278659] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 8077.294975] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 8077.304377] RIP: 0010:[] [] osd_inode_iteration+0x251/0xcc0 [osd_ldiskfs] [ 8077.316632] RSP: 0018:ffff880703603d18 EFLAGS: 00000286 [ 8077.323601] RAX: ffff8800b2a37468 RBX: 000000009f8f2001 RCX: 0000000000000000 [ 8077.332597] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 8077.341584] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [ 8077.350554] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [ 8077.359514] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [ 8077.368469] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 8077.378489] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8077.385899] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 8077.394869] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8077.403834] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 8077.412792] Stack: [ 8077.416010] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [ 8077.425305] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [ 8077.434596] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [ 8077.443892] Call Trace: [ 8077.447602] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 8077.456762] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 8077.465637] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 8077.474315] [] ? wake_up_state+0x20/0x20 [ 8077.481537] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 8077.490884] [] kthread+0xcf/0xe0 [ 8077.497317] [] ? kthread_create_on_node+0x140/0x140 [ 8077.505602] [] ret_from_fork+0x58/0x90 [ 8077.512624] [] ? kthread_create_on_node+0x140/0x140 [ 8077.520909] Code: b0 00 00 00 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 00 8b 75 98 48 8b 7d 88 e8 dc 2d f5 ff 48 85 c0 48 89 45 90 0f 84 b3 08 00 00 <44> 8b 7d a0 48 8b 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 e8 77 [ 8098.494062] perf: interrupt took too long (3223 > 3190), lowering kernel.perf_event_max_sample_rate to 62000 [ 8125.125524] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [ 8125.135792] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 8125.268115] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 8125.281439] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 8125.297751] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 8125.307192] RIP: 0010:[] [] osd_scrub_next+0x3d/0x3c0 [osd_ldiskfs] [ 8125.318888] RSP: 0018:ffff880703603c70 EFLAGS: 00000246 [ 8125.325856] RAX: 0000000000000000 RBX: ffff880703603c37 RCX: ffff880703603d98 [ 8125.334860] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 8125.343854] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [ 8125.352832] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [ 8125.361800] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [ 8125.370776] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 8125.380831] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8125.388267] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 8125.397270] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8125.406258] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 8125.415236] Stack: [ 8125.418468] 000000009f8f2001 0000000000000004 ffff880703603d57 ffff880703603df0 [ 8125.427784] 0000000000000000 ffff880703603d57 000000009f8f2001 0000000000000004 [ 8125.437107] ffff880703603d57 ffff8800b2a37468 0000000000000000 ffff880703603d78 [ 8125.446420] Call Trace: [ 8125.450181] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [ 8125.459371] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 8125.468559] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 8125.477448] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 8125.486150] [] ? wake_up_state+0x20/0x20 [ 8125.493401] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 8125.502783] [] kthread+0xcf/0xe0 [ 8125.509234] [] ? kthread_create_on_node+0x140/0x140 [ 8125.517554] [] ret_from_fork+0x58/0x90 [ 8125.524591] [] ? kthread_create_on_node+0x140/0x140 [ 8125.532893] Code: 49 89 fd 41 54 49 89 d4 53 48 89 cb 48 83 ec 70 65 48 8b 04 25 28 00 00 00 48 89 45 d0 31 c0 48 8b 05 f8 59 9c ff 48 85 c0 75 7b <41> f6 87 58 10 00 00 08 0f 84 62 03 00 00 4d 8d b7 e0 10 00 00 [ 8153.126499] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [ 8153.136264] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 8153.268064] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 8153.281369] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 8153.297634] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 8153.307031] RIP: 0010:[] [] osd_inode_iteration+0x489/0xcc0 [osd_ldiskfs] [ 8153.319285] RSP: 0018:ffff880703603d18 EFLAGS: 00000293 [ 8153.326240] RAX: 0000000000000004 RBX: 000000009f8f2001 RCX: 0000000000000000 [ 8153.335674] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 8153.344653] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [ 8153.353629] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [ 8153.362589] R13: ffff880703603d08 R14: 000000009f8f2001 R15: ffff880703603d08 [ 8153.371569] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 8153.381612] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8153.389025] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 8153.398000] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8153.406970] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 8153.415932] Stack: [ 8153.419151] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [ 8153.428437] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [ 8153.438223] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [ 8153.447535] Call Trace: [ 8153.451257] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 8153.460445] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 8153.469329] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 8153.478003] [] ? wake_up_state+0x20/0x20 [ 8153.485218] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 8153.494578] [] kthread+0xcf/0xe0 [ 8153.501010] [] ? kthread_create_on_node+0x140/0x140 [ 8153.509315] [] ret_from_fork+0x58/0x90 [ 8153.516342] [] ? kthread_create_on_node+0x140/0x140 [ 8153.524626] Code: 00 e8 7c eb 97 ff e9 0f fc ff ff 0f 1f 80 00 00 00 00 45 89 e9 4c 8d 85 67 ff ff ff 48 8b 4d a8 48 8d 55 88 48 8b b5 38 ff ff ff <48> 8b bd 48 ff ff ff 48 8b 85 28 ff ff ff ff d0 85 c0 41 89 c5 [ 8181.127481] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [ 8181.137147] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 8181.269565] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 8181.282882] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 8181.299193] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 8181.308618] RIP: 0010:[] [] ldiskfs_itable_unused_count+0x23/0x30 [ldiskfs] [ 8181.321095] RSP: 0018:ffff880703603d08 EFLAGS: 00000297 [ 8181.328069] RAX: 0000000000000000 RBX: ffff880703603d57 RCX: ffff88042bd78800 [ 8181.337067] RDX: 00000000000072ce RSI: ffff880419dc3a00 RDI: ffff88042bd79800 [ 8181.346039] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [ 8181.355013] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [ 8181.363975] R13: 0000000000000018 R14: ffff880703603c78 R15: 0000000000000246 [ 8181.372924] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 8181.382952] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8181.390353] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 8181.399321] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8181.408282] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 8181.417236] Stack: [ 8181.420449] ffff880703603df0 ffffffffa121cfb9 ffffffffa121b990 ffffffffa1217a20 [ 8181.429733] ffff8800b2a36000 00000000810d354f ffff8803162f6000 ffff8800b2a37468 [ 8181.439010] 0000000020000000 010000000000000c 0000000000000000 0000000000000000 [ 8181.448294] Call Trace: [ 8181.452052] [] osd_inode_iteration+0x269/0xcc0 [osd_ldiskfs] [ 8181.461219] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 8181.470374] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 8181.479245] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 8181.487926] [] ? wake_up_state+0x20/0x20 [ 8181.495131] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 8181.504482] [] kthread+0xcf/0xe0 [ 8181.510896] [] ? kthread_create_on_node+0x140/0x140 [ 8181.519178] [] ret_from_fork+0x58/0x90 [ 8181.526186] [] ? kthread_create_on_node+0x140/0x140 [ 8181.534454] Code: 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 8b 8f a8 03 00 00 31 c0 0f b7 56 1c 48 89 e5 48 83 39 3f 76 07 0f b7 46 32 c1 e0 10 <09> d0 5d c3 66 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 89 16 [ 8203.643258] INFO: rcu_sched self-detected stall on CPU { 24} (t=780033 jiffies g=907112 c=907111 q=1271641) [ 8203.655527] Task dump for CPU 24: [ 8203.660271] OI_scrub R running task 0 5551 2 0x00000088 [ 8203.669205] ffff88083fde6dd0 00000000fb6db880 ffff88082da03db0 ffffffff810c47e8 [ 8203.678539] 0000000000000018 ffffffff81a1e7c0 ffff88082da03dc8 ffffffff810c8079 [ 8203.687824] 0000000000000009 ffff88082da03df8 ffffffff811378e0 ffff88082da101c0 [ 8203.697102] Call Trace: [ 8203.700774] [] sched_show_task+0xa8/0x110 [ 8203.708760] [] dump_cpu_task+0x39/0x70 [ 8203.715715] [] rcu_dump_cpu_stacks+0x90/0xd0 [ 8203.723229] [] rcu_check_callbacks+0x442/0x720 [ 8203.730917] [] ? tick_sched_handle.isra.13+0x60/0x60 [ 8203.739167] [] update_process_times+0x47/0x80 [ 8203.746742] [] tick_sched_handle.isra.13+0x25/0x60 [ 8203.754764] [] tick_sched_timer+0x41/0x70 [ 8203.761890] [] __hrtimer_run_queues+0xd2/0x260 [ 8203.769481] [] hrtimer_interrupt+0xb0/0x1e0 [ 8203.776750] [] ? call_softirq+0x1c/0x30 [ 8203.783629] [] local_apic_timer_interrupt+0x37/0x60 [ 8203.791651] [] smp_apic_timer_interrupt+0x3f/0x60 [ 8203.799484] [] apic_timer_interrupt+0x6d/0x80 [ 8203.806914] [] ? osd_scrub_next+0x2b/0x3c0 [osd_ldiskfs] [ 8203.816164] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [ 8203.825059] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 8203.833950] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 8203.842558] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 8203.850980] [] ? wake_up_state+0x20/0x20 [ 8203.857929] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 8203.867035] [] kthread+0xcf/0xe0 [ 8203.873205] [] ? kthread_create_on_node+0x140/0x140 [ 8203.881233] [] ret_from_fork+0x58/0x90 [ 8203.887979] [] ? kthread_create_on_node+0x140/0x140 [ 8229.129152] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [ 8229.138572] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 8229.268671] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 8229.281835] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 8229.297898] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 8229.307193] RIP: 0010:[] [] osd_inode_iteration+0x4ac/0xcc0 [osd_ldiskfs] [ 8229.319328] RSP: 0018:ffff880703603d18 EFLAGS: 00000216 [ 8229.326188] RAX: 0000000020000000 RBX: 000000009f8f2001 RCX: 0000000000000000 [ 8229.335085] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 8229.343987] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [ 8229.352880] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000001 [ 8229.361779] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [ 8229.370669] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 8229.380626] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8229.387968] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 8229.396870] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8229.405778] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 8229.414673] Stack: [ 8229.417837] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [ 8229.427082] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [ 8229.436319] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [ 8229.445556] Call Trace: [ 8229.449253] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 8229.458386] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 8229.467228] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 8229.475889] [] ? wake_up_state+0x20/0x20 [ 8229.483079] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 8229.492414] [] kthread+0xcf/0xe0 [ 8229.498813] [] ? kthread_create_on_node+0x140/0x140 [ 8229.507081] [] ret_from_fork+0x58/0x90 [ 8229.514072] [] ? kthread_create_on_node+0x140/0x140 [ 8229.522325] Code: 88 48 8b b5 38 ff ff ff 48 8b bd 48 ff ff ff 48 8b 85 28 ff ff ff ff d0 85 c0 41 89 c5 75 1c 8b 85 58 ff ff ff 41 3b 04 24 72 10 <48> 8b 85 50 ff ff ff 44 3b 30 0f 87 95 fd ff ff 45 31 ff e9 a2 [ 8257.130127] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [ 8257.139734] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 8257.271672] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 8257.284980] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 8257.301298] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 8257.310697] RIP: 0010:[] [] osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 8257.322765] RSP: 0018:ffff880703603d10 EFLAGS: 00000293 [ 8257.329727] RAX: ffffffffa121b990 RBX: 000000009f8f2001 RCX: 0000000000000000 [ 8257.338709] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 8257.347699] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [ 8257.356665] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffff880703603d57 [ 8257.365622] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [ 8257.374576] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 8257.384603] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8257.392014] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 8257.400984] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8257.409954] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 8257.418910] Stack: [ 8257.422129] ffffffffa121d1e9 ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 [ 8257.431420] 00000000810d354f ffff8803162f6000 ffff8800b2a37468 0000000020000000 [ 8257.440721] 010000000000000c 0000000000000000 0000000000000000 ffff8800b2a36000 [ 8257.450022] Call Trace: [ 8257.453748] [] ? osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [ 8257.463119] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 8257.472294] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 8257.481177] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 8257.489877] [] ? wake_up_state+0x20/0x20 [ 8257.497096] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 8257.506453] [] kthread+0xcf/0xe0 [ 8257.512895] [] ? kthread_create_on_node+0x140/0x140 [ 8257.521190] [] ret_from_fork+0x58/0x90 [ 8257.528218] [] ? kthread_create_on_node+0x140/0x140 [ 8257.536499] Code: b5 07 00 00 c7 05 85 a5 02 00 00 00 02 00 89 54 24 08 48 8b 40 40 ba 07 00 00 00 48 89 04 24 31 c0 e8 a5 03 98 ff e9 c7 fd ff ff <66> 66 66 66 90 55 48 89 e5 41 57 49 89 f7 41 56 41 55 41 54 45 [ 8285.131096] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [ 8285.140754] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 8285.272515] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 8285.285810] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 8285.302089] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 8285.311482] RIP: 0010:[] [] osd_scrub_next+0x2b/0x3c0 [osd_ldiskfs] [ 8285.323144] RSP: 0018:ffff880703603c70 EFLAGS: 00000282 [ 8285.330091] RAX: 00000000fb6db880 RBX: ffff880703603c37 RCX: ffff880703603d98 [ 8285.339082] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 8285.348055] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [ 8285.357011] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [ 8285.365975] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [ 8285.374940] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 8285.384982] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8285.392402] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 8285.401380] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8285.410348] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 8285.419303] Stack: [ 8285.422508] 000000009f8f2001 ffff880703603d08 000000009f8f2001 ffff880703603df0 [ 8285.431812] 000000009f8f2001 ffff880703603d57 000000009f8f2001 0000000000000004 [ 8285.441117] ffff880703603d57 0000000000000000 0000000000000000 ffff880703603d78 [ 8285.450392] Call Trace: [ 8285.454107] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [ 8285.463273] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 8285.472439] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 8285.481307] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 8285.489977] [] ? wake_up_state+0x20/0x20 [ 8285.497186] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 8285.506538] [] kthread+0xcf/0xe0 [ 8285.512969] [] ? kthread_create_on_node+0x140/0x140 [ 8285.521252] [] ret_from_fork+0x58/0x90 [ 8285.528270] [] ? kthread_create_on_node+0x140/0x140 [ 8285.536537] Code: 66 66 66 66 90 55 48 89 e5 41 57 49 89 f7 41 56 41 55 49 89 fd 41 54 49 89 d4 53 48 89 cb 48 83 ec 70 65 48 8b 04 25 28 00 00 00 <48> 89 45 d0 31 c0 48 8b 05 f8 59 9c ff 48 85 c0 75 7b 41 f6 87 [ 8313.132072] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [ 8313.141708] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 8313.273288] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 8313.286569] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 8313.302808] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 8313.312195] RIP: 0010:[] [] osd_scrub_next+0x160/0x3c0 [osd_ldiskfs] [ 8313.323920] RSP: 0018:ffff880703603ce0 EFLAGS: 00000282 [ 8313.330861] RAX: 0000000000000004 RBX: ffff880703603c78 RCX: ffff880703603d98 [ 8313.339832] RDX: ffff880703603d78 RSI: 0000000000000000 RDI: ffff8803162f6000 [ 8313.348792] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [ 8313.357738] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000010 [ 8313.366678] R13: ffffffffa121ba87 R14: ffffffffffffff10 R15: ffff8803162f6000 [ 8313.375619] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 8313.385639] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8313.393033] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 8313.401989] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8313.410933] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 8313.419864] Stack: [ 8313.423059] ffff880419dc3a00 ffff8800b2a3746c 0000000000000000 00000000ffffffff [ 8313.432342] 0000000000000001 ffff880703603df0 ffffffffa121cff9 ffffffffa121b990 [ 8313.441634] ffffffffa1217a20 ffff8800b2a36000 00000000810d354f ffff8803162f6000 [ 8313.450925] Call Trace: [ 8313.454636] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [ 8313.463801] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 8313.472961] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 8313.481821] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 8313.490478] [] ? wake_up_state+0x20/0x20 [ 8313.497684] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 8313.507026] [] kthread+0xcf/0xe0 [ 8313.513453] [] ? kthread_create_on_node+0x140/0x140 [ 8313.521734] [] ret_from_fork+0x58/0x90 [ 8313.528749] [] ? kthread_create_on_node+0x140/0x140 [ 8313.537015] Code: 10 00 00 02 00 00 00 48 89 df e8 fc 69 47 e0 b8 05 00 00 00 48 8b 75 d0 65 48 33 34 25 28 00 00 00 0f 85 4c 02 00 00 48 83 c4 70 <5b> 41 5c 41 5d 41 5e 41 5f 5d c3 48 8b 05 be 58 9c ff 48 85 c0 [ 8341.133049] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [ 8341.142675] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 8341.274370] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 8341.287660] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 8341.303922] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 8341.313306] RIP: 0010:[] [] osd_scrub_exec+0x30/0x5b0 [osd_ldiskfs] [ 8341.324941] RSP: 0018:ffff880703603c78 EFLAGS: 00000296 [ 8341.331881] RAX: 00000000fb6db880 RBX: ffff880703603c37 RCX: 0000000000000000 [ 8341.340857] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 8341.349821] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [ 8341.358772] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [ 8341.367723] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [ 8341.376673] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 8341.386710] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8341.394128] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 8341.403100] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8341.412060] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 8341.421008] Stack: [ 8341.424221] ffff880703603d08 ffff880703603d57 ffff880703603df0 000000009f8f2001 [ 8341.433506] ffff880703603d57 000000009f8f2001 0000000000000004 ffff880703603d57 [ 8341.442808] ffff88042bd78800 ffff88042bd78800 00000000000072cf ffff880419dc3a00 [ 8341.452106] Call Trace: [ 8341.455825] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [ 8341.464997] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 8341.474155] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 8341.483022] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 8341.491684] [] ? wake_up_state+0x20/0x20 [ 8341.498890] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 8341.508230] [] kthread+0xcf/0xe0 [ 8341.514664] [] ? kthread_create_on_node+0x140/0x140 [ 8341.522944] [] ret_from_fork+0x58/0x90 [ 8341.529952] [] ? kthread_create_on_node+0x140/0x140 [ 8341.538223] Code: 55 48 89 e5 41 57 49 89 f7 41 56 41 55 41 54 45 31 e4 53 48 83 ec 68 48 8b 9e 08 10 00 00 4c 89 45 90 65 48 8b 04 25 28 00 00 00 <48> 89 45 d0 31 c0 48 8d 43 08 48 85 db 4c 0f 45 e0 41 83 f9 04 [ 8369.134029] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [ 8369.143658] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 8369.275277] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 8369.288566] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 8369.304826] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 8369.314206] RIP: 0010:[] [] osd_inode_iteration+0x2c0/0xcc0 [osd_ldiskfs] [ 8369.326427] RSP: 0018:ffff880703603d18 EFLAGS: 00000297 [ 8369.333367] RAX: 0000000000000004 RBX: 000000009f8f2001 RCX: ffff880703603d98 [ 8369.342344] RDX: ffff880703603d78 RSI: 0000000000000000 RDI: ffff8803162f6000 [ 8369.351308] RBP: ffff880703603df0 R08: 0000000000000001 R09: 0000000000000004 [ 8369.360247] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000001 [ 8369.369191] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [ 8369.378140] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 8369.388160] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8369.395563] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 8369.404519] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8369.413465] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 8369.422398] Stack: [ 8369.425597] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [ 8369.434878] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [ 8369.444171] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [ 8369.453463] Call Trace: [ 8369.457161] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 8369.466321] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 8369.475183] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 8369.483840] [] ? wake_up_state+0x20/0x20 [ 8369.491036] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 8369.500381] [] kthread+0xcf/0xe0 [ 8369.506806] [] ? kthread_create_on_node+0x140/0x140 [ 8369.515074] [] ret_from_fork+0x58/0x90 [ 8369.522087] [] ? kthread_create_on_node+0x140/0x140 [ 8369.530354] Code: 38 ff ff ff 48 8b bd 48 ff ff ff 48 8b 85 30 ff ff ff ff d0 83 f8 03 41 89 c5 0f 84 4b 02 00 00 0f 8e 15 01 00 00 83 f8 05 66 90 <0f> 84 8a 02 00 00 83 f8 06 0f 85 a1 01 00 00 48 8b 7d 90 48 85 [ 8383.652544] INFO: rcu_sched self-detected stall on CPU { 24} (t=960036 jiffies g=907112 c=907111 q=1306848) [ 8383.664808] Task dump for CPU 24: [ 8383.669559] OI_scrub R running task 0 5551 2 0x00000088 [ 8383.678489] ffff88083fde6dd0 00000000fb6db880 ffff88082da03db0 ffffffff810c47e8 [ 8383.687814] 0000000000000018 ffffffff81a1e7c0 ffff88082da03dc8 ffffffff810c8079 [ 8383.697134] 0000000000000009 ffff88082da03df8 ffffffff811378e0 ffff88082da101c0 [ 8383.706441] Call Trace: [ 8383.710137] [] sched_show_task+0xa8/0x110 [ 8383.718137] [] dump_cpu_task+0x39/0x70 [ 8383.725105] [] rcu_dump_cpu_stacks+0x90/0xd0 [ 8383.732643] [] rcu_check_callbacks+0x442/0x720 [ 8383.740347] [] ? tick_sched_handle.isra.13+0x60/0x60 [ 8383.748618] [] update_process_times+0x47/0x80 [ 8383.756185] [] tick_sched_handle.isra.13+0x25/0x60 [ 8383.764209] [] tick_sched_timer+0x41/0x70 [ 8383.771340] [] __hrtimer_run_queues+0xd2/0x260 [ 8383.778940] [] hrtimer_interrupt+0xb0/0x1e0 [ 8383.786231] [] ? call_softirq+0x1c/0x30 [ 8383.793122] [] local_apic_timer_interrupt+0x37/0x60 [ 8383.801155] [] smp_apic_timer_interrupt+0x3f/0x60 [ 8383.808996] [] apic_timer_interrupt+0x6d/0x80 [ 8383.816429] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 8383.825750] [] ? osd_scrub_next+0x1e/0x3c0 [osd_ldiskfs] [ 8383.834272] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [ 8383.843176] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 8383.852080] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 8383.860695] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 8383.869103] [] ? wake_up_state+0x20/0x20 [ 8383.876054] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 8383.885138] [] kthread+0xcf/0xe0 [ 8383.891317] [] ? kthread_create_on_node+0x140/0x140 [ 8383.899340] [] ret_from_fork+0x58/0x90 [ 8383.906095] [] ? kthread_create_on_node+0x140/0x140 [ 8409.135416] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [ 8409.144794] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 8409.274556] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 8409.287727] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 8409.303769] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 8409.313060] RIP: 0010:[] [] osd_scrub_exec+0x96/0x5b0 [osd_ldiskfs] [ 8409.324621] RSP: 0018:ffff880703603d10 EFLAGS: 00000292 [ 8409.331490] RAX: 0000000000000000 RBX: 000000009f8f2001 RCX: 0000000000000000 [ 8409.340395] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 8409.349292] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [ 8409.358194] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000001 [ 8409.367089] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [ 8409.375967] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 8409.385933] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8409.393278] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 8409.402192] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8409.411097] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 8409.419988] Stack: [ 8409.423155] ffffffffa121d1e9 ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 [ 8409.432404] 00000000810d354f ffff8803162f6000 ffff8800b2a37468 0000000020000000 [ 8409.441649] 010000000000000c 0000000000000000 0000000000000000 ffff8800b2a36000 [ 8409.450901] Call Trace: [ 8409.454593] [] ? osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [ 8409.463919] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 8409.473049] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 8409.481890] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 8409.490538] [] ? wake_up_state+0x20/0x20 [ 8409.497714] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 8409.507046] [] kthread+0xcf/0xe0 [ 8409.513453] [] ? kthread_create_on_node+0x140/0x140 [ 8409.521700] [] ret_from_fork+0x58/0x90 [ 8409.528688] [] ? kthread_create_on_node+0x140/0x140 [ 8409.536937] Code: 00 00 41 80 a7 74 14 00 00 fe 48 8b 4d d0 65 48 33 0c 25 28 00 00 00 0f 85 b1 03 00 00 48 83 c4 68 5b 41 5c 41 5d 41 5e 41 5f 5d 66 0f 1f 84 00 00 00 00 00 4c 8d ae f0 10 00 00 44 89 4d 88 [ 8437.136398] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [ 8437.145991] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 8437.278124] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 8437.291409] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 8437.307676] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 8437.317060] RIP: 0010:[] [] osd_inode_iteration+0x251/0xcc0 [osd_ldiskfs] [ 8437.329319] RSP: 0018:ffff880703603d18 EFLAGS: 00000286 [ 8437.336270] RAX: ffff8800b2a37468 RBX: 000000009f8f2001 RCX: 0000000000000000 [ 8437.345259] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 8437.354236] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [ 8437.363191] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [ 8437.372151] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [ 8437.381107] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 8437.391143] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8437.398553] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 8437.407523] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8437.416502] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 8437.425446] Stack: [ 8437.428659] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [ 8437.437927] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [ 8437.447236] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [ 8437.456535] Call Trace: [ 8437.460272] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 8437.469440] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 8437.478320] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 8437.486998] [] ? wake_up_state+0x20/0x20 [ 8437.494212] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 8437.503577] [] kthread+0xcf/0xe0 [ 8437.510015] [] ? kthread_create_on_node+0x140/0x140 [ 8437.518320] [] ret_from_fork+0x58/0x90 [ 8437.525340] [] ? kthread_create_on_node+0x140/0x140 [ 8437.533624] Code: b0 00 00 00 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 00 8b 75 98 48 8b 7d 88 e8 dc 2d f5 ff 48 85 c0 48 89 45 90 0f 84 b3 08 00 00 <44> 8b 7d a0 48 8b 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 e8 77 [ 8460.520491] Lustre: MGS: Received new LWP connection from 192.168.1.124@o2ib100, removing former export from same NID [ 8460.533571] Lustre: MGS: Connection restored to 5a0e6555-4c2d-39c9-90c4-a332f62c8353 (at 192.168.1.124@o2ib100) [ 8460.545926] Lustre: Skipped 17 previous similar messages [ 8465.137373] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [ 8465.147021] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 8465.279251] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 8465.292541] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 8465.308810] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 8465.318211] RIP: 0010:[] [] osd_inode_iteration+0x4ac/0xcc0 [osd_ldiskfs] [ 8465.330469] RSP: 0018:ffff880703603d18 EFLAGS: 00000216 [ 8465.337436] RAX: 0000000020000000 RBX: ffff880703603d57 RCX: 0000000000000000 [ 8465.346429] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 8465.355425] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [ 8465.364417] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [ 8465.373406] R13: ffffea00027e3c80 R14: 000000009f8f2001 R15: 0000000000000018 [ 8465.382403] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 8465.392482] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8465.399926] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 8465.408925] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8465.417907] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 8465.426893] Stack: [ 8465.430158] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [ 8465.439494] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [ 8465.448822] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [ 8465.458147] Call Trace: [ 8465.461913] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 8465.471117] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 8465.480024] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 8465.488740] [] ? wake_up_state+0x20/0x20 [ 8465.495993] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 8465.505394] [] kthread+0xcf/0xe0 [ 8465.511868] [] ? kthread_create_on_node+0x140/0x140 [ 8465.520205] [] ret_from_fork+0x58/0x90 [ 8465.527257] [] ? kthread_create_on_node+0x140/0x140 [ 8465.535567] Code: 88 48 8b b5 38 ff ff ff 48 8b bd 48 ff ff ff 48 8b 85 28 ff ff ff ff d0 85 c0 41 89 c5 75 1c 8b 85 58 ff ff ff 41 3b 04 24 72 10 <48> 8b 85 50 ff ff ff 44 3b 30 0f 87 95 fd ff ff 45 31 ff e9 a2 [ 8493.138348] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [ 8493.148023] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 8493.280438] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 8493.293742] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 8493.310037] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 8493.319449] RIP: 0010:[] [] osd_inode_iteration+0x282/0xcc0 [osd_ldiskfs] [ 8493.331709] RSP: 0018:ffff880703603d18 EFLAGS: 00000287 [ 8493.338673] RAX: ffff88042bd78800 RBX: 000000009f8f2001 RCX: ffff88042bd78800 [ 8493.347664] RDX: 00000000000072cf RSI: ffff880419dc3a00 RDI: ffff88042bd79800 [ 8493.356647] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [ 8493.365616] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [ 8493.374574] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [ 8493.383528] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 8493.393557] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8493.400968] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 8493.409947] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8493.418935] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 8493.427912] Stack: [ 8493.431133] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [ 8493.440459] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [ 8493.449765] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [ 8493.459073] Call Trace: [ 8493.462835] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 8493.472036] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 8493.480919] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 8493.489610] [] ? wake_up_state+0x20/0x20 [ 8493.496832] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 8493.506200] [] kthread+0xcf/0xe0 [ 8493.512637] [] ? kthread_create_on_node+0x140/0x140 [ 8493.520925] [] ret_from_fork+0x58/0x90 [ 8493.527946] [] ? kthread_create_on_node+0x140/0x140 [ 8493.536233] Code: 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 e8 77 5b f4 ff 42 8d 14 38 48 8b 45 88 48 8b 80 a8 03 00 00 48 3b 50 20 0f 83 57 01 00 00 <44> 0f b6 85 67 ff ff ff 48 8d 4d a8 48 8d 55 88 48 8b b5 38 ff [ 8521.139318] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [ 8521.148963] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 8521.280892] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 8521.294204] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 8521.310529] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 8521.319945] RIP: 0010:[] [] osd_scrub_exec+0x79/0x5b0 [osd_ldiskfs] [ 8521.331617] RSP: 0018:ffff880703603c78 EFLAGS: 00000202 [ 8521.338595] RAX: 0000000000000000 RBX: ffff880703603c37 RCX: 00000000fb6db880 [ 8521.347592] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 8521.356581] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [ 8521.365561] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [ 8521.374513] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [ 8521.383472] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 8521.393509] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8521.400925] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 8521.409900] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8521.418870] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 8521.427827] Stack: [ 8521.431048] ffffea00027e3c80 000000009f8f2001 ffff880703603df0 0000000000000001 [ 8521.440342] ffff880703603d57 000000009f8f2001 0000000000000004 0000000000000001 [ 8521.449630] ffffffffa1217a20 ffff880703603d98 ffff880703603d78 ffff8800b2a36000 [ 8521.458928] Call Trace: [ 8521.462656] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 8521.471545] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [ 8521.480742] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 8521.489927] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 8521.498791] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 8521.507459] [] ? wake_up_state+0x20/0x20 [ 8521.514694] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 8521.524057] [] kthread+0xcf/0xe0 [ 8521.530488] [] ? kthread_create_on_node+0x140/0x140 [ 8521.538778] [] ret_from_fork+0x58/0x90 [ 8521.545792] [] ? kthread_create_on_node+0x140/0x140 [ 8521.554060] Code: 07 74 4f 41 83 f9 02 74 78 48 89 ca 44 89 c9 e8 0e c6 ff ff 85 c0 41 89 c1 0f 84 a3 01 00 00 41 80 a7 74 14 00 00 fe 48 8b 4d d0 <65> 48 33 0c 25 28 00 00 00 0f 85 b1 03 00 00 48 83 c4 68 5b 41 [ 8549.140299] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [ 8549.149924] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 8549.281524] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 8549.294789] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 8549.311005] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 8549.320370] RIP: 0010:[] [] ldiskfs_itable_unused_count+0x6/0x30 [ldiskfs] [ 8549.332691] RSP: 0018:ffff880703603d08 EFLAGS: 00000286 [ 8549.339637] RAX: ffff8800b2a37468 RBX: 000000009f8f2001 RCX: 0000000000000000 [ 8549.348605] RDX: ffff880703603d78 RSI: ffff880419dc3a00 RDI: ffff88042bd79800 [ 8549.357570] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [ 8549.366529] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [ 8549.375488] R13: 0000000000000018 R14: ffff880703603c78 R15: 0000000000000286 [ 8549.384440] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 8549.394468] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8549.401867] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 8549.410825] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8549.419776] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 8549.428715] Stack: [ 8549.431927] ffff880703603df0 ffffffffa121cfb9 ffffffffa121b990 ffffffffa1217a20 [ 8549.441230] ffff8800b2a36000 00000000810d354f ffff8803162f6000 ffff8800b2a37468 [ 8549.450513] 0000000020000000 010000000000000c 0000000000000000 0000000000000000 [ 8549.459807] Call Trace: [ 8549.463532] [] ? osd_inode_iteration+0x269/0xcc0 [osd_ldiskfs] [ 8549.472896] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 8549.482060] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 8549.490927] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 8549.499584] [] ? wake_up_state+0x20/0x20 [ 8549.506800] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 8549.516151] [] kthread+0xcf/0xe0 [ 8549.522580] [] ? kthread_create_on_node+0x140/0x140 [ 8549.530863] [] ret_from_fork+0x58/0x90 [ 8549.537874] [] ? kthread_create_on_node+0x140/0x140 [ 8549.546151] Code: 00 00 31 c0 0f b7 56 10 48 89 e5 48 83 39 3f 76 07 0f b7 46 30 c1 e0 10 09 d0 5d c3 66 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 <48> 8b 8f a8 03 00 00 31 c0 0f b7 56 1c 48 89 e5 48 83 39 3f 76 [ 8563.661807] INFO: rcu_sched self-detected stall on CPU { 24} (t=1140039 jiffies g=907112 c=907111 q=1391933) [ 8563.674140] Task dump for CPU 24: [ 8563.678865] OI_scrub R running task 0 5551 2 0x00000088 [ 8563.687782] ffff88083fde6dd0 00000000fb6db880 ffff88082da03db0 ffffffff810c47e8 [ 8563.697105] 0000000000000018 ffffffff81a1e7c0 ffff88082da03dc8 ffffffff810c8079 [ 8563.706407] 0000000000000009 ffff88082da03df8 ffffffff811378e0 ffff88082da101c0 [ 8563.715688] Call Trace: [ 8563.719368] [] sched_show_task+0xa8/0x110 [ 8563.727339] [] dump_cpu_task+0x39/0x70 [ 8563.734293] [] rcu_dump_cpu_stacks+0x90/0xd0 [ 8563.741813] [] rcu_check_callbacks+0x442/0x720 [ 8563.749507] [] ? tick_sched_handle.isra.13+0x60/0x60 [ 8563.757757] [] update_process_times+0x47/0x80 [ 8563.765305] [] tick_sched_handle.isra.13+0x25/0x60 [ 8563.773307] [] tick_sched_timer+0x41/0x70 [ 8563.780430] [] __hrtimer_run_queues+0xd2/0x260 [ 8563.788014] [] hrtimer_interrupt+0xb0/0x1e0 [ 8563.795284] [] ? call_softirq+0x1c/0x30 [ 8563.802157] [] local_apic_timer_interrupt+0x37/0x60 [ 8563.810179] [] smp_apic_timer_interrupt+0x3f/0x60 [ 8563.817996] [] apic_timer_interrupt+0x6d/0x80 [ 8563.825414] [] ? osd_scrub_next+0x2b/0x3c0 [osd_ldiskfs] [ 8563.834646] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [ 8563.843544] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 8563.852445] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 8563.861051] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 8563.869461] [] ? wake_up_state+0x20/0x20 [ 8563.876414] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 8563.885499] [] kthread+0xcf/0xe0 [ 8563.891664] [] ? kthread_create_on_node+0x140/0x140 [ 8563.899682] [] ret_from_fork+0x58/0x90 [ 8563.906430] [] ? kthread_create_on_node+0x140/0x140 [ 8589.141692] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [ 8589.151085] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 8589.280566] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 8589.293731] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 8589.309793] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 8589.319081] RIP: 0010:[] [] osd_scrub_next+0x3d/0x3c0 [osd_ldiskfs] [ 8589.330636] RSP: 0018:ffff880703603c70 EFLAGS: 00000246 [ 8589.337486] RAX: 0000000000000000 RBX: ffff880703603c37 RCX: ffff880703603d98 [ 8589.346389] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 8589.355279] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [ 8589.364167] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [ 8589.373057] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [ 8589.381943] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 8589.391904] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8589.399240] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 8589.408136] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8589.417036] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 8589.425927] Stack: [ 8589.429087] 000000009f8f2001 ffff880703603d08 0000000000000001 ffff880703603df0 [ 8589.438309] 000000009f8f2001 ffff880703603d57 000000009f8f2001 0000000000000004 [ 8589.447558] ffff880703603d57 ffff88042bd78800 ffff88042bd78800 00000000000072cf [ 8589.456798] Call Trace: [ 8589.460461] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [ 8589.469588] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 8589.478709] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 8589.487538] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 8589.496174] [] ? wake_up_state+0x20/0x20 [ 8589.503345] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 8589.512645] [] kthread+0xcf/0xe0 [ 8589.519037] [] ? kthread_create_on_node+0x140/0x140 [ 8589.527280] [] ret_from_fork+0x58/0x90 [ 8589.534251] [] ? kthread_create_on_node+0x140/0x140 [ 8589.542486] Code: 49 89 fd 41 54 49 89 d4 53 48 89 cb 48 83 ec 70 65 48 8b 04 25 28 00 00 00 48 89 45 d0 31 c0 48 8b 05 f8 59 9c ff 48 85 c0 75 7b <41> f6 87 58 10 00 00 08 0f 84 62 03 00 00 4d 8d b7 e0 10 00 00 [ 8617.142669] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [ 8617.152268] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 8617.283823] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 8617.297101] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 8617.313343] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 8617.322721] RIP: 0010:[] [] ldiskfs_itable_unused_count+0x16/0x30 [ldiskfs] [ 8617.335139] RSP: 0018:ffff880703603d08 EFLAGS: 00000246 [ 8617.342074] RAX: 0000000000000000 RBX: 0000000000000001 RCX: ffff88042bd78800 [ 8617.351049] RDX: 00000000000072ce RSI: ffff880419dc3a00 RDI: ffff88042bd79800 [ 8617.360020] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [ 8617.368968] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [ 8617.377912] R13: 0000000000000018 R14: ffff880703603c78 R15: 0000000000000286 [ 8617.386856] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 8617.396872] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8617.404266] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 8617.413217] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8617.422159] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 8617.431097] Stack: [ 8617.434291] ffff880703603df0 ffffffffa121cfb9 ffffffffa121b990 ffffffffa1217a20 [ 8617.443573] ffff8800b2a36000 00000000810d354f ffff8803162f6000 ffff8800b2a37468 [ 8617.452859] 0000000020000000 010000000000000c 0000000000000000 0000000000000000 [ 8617.462135] Call Trace: [ 8617.465853] [] osd_inode_iteration+0x269/0xcc0 [osd_ldiskfs] [ 8617.475022] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 8617.484177] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 8617.493039] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 8617.501691] [] ? wake_up_state+0x20/0x20 [ 8617.508898] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 8617.518241] [] kthread+0xcf/0xe0 [ 8617.524664] [] ? kthread_create_on_node+0x140/0x140 [ 8617.532939] [] ret_from_fork+0x58/0x90 [ 8617.539944] [] ? kthread_create_on_node+0x140/0x140 [ 8617.548206] Code: 07 0f b7 46 30 c1 e0 10 09 d0 5d c3 66 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 8b 8f a8 03 00 00 31 c0 0f b7 56 1c 48 89 e5 <48> 83 39 3f 76 07 0f b7 46 32 c1 e0 10 09 d0 5d c3 66 0f 1f 84 [ 8645.143643] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [ 8645.153277] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 8645.284989] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 8645.298261] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 8645.314518] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 8645.323898] RIP: 0010:[] [] osd_inode_iteration+0x282/0xcc0 [osd_ldiskfs] [ 8645.336119] RSP: 0018:ffff880703603d18 EFLAGS: 00000287 [ 8645.343054] RAX: ffff88042bd78800 RBX: ffff880703603d57 RCX: ffff88042bd78800 [ 8645.352028] RDX: 00000000000072cf RSI: ffff880419dc3a00 RDI: ffff88042bd79800 [ 8645.360988] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [ 8645.369934] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [ 8645.378882] R13: ffffea00027e3c80 R14: 000000009f8f2001 R15: ffff880703603d08 [ 8645.387832] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 8645.397862] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8645.405265] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 8645.414231] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8645.423190] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 8645.432145] Stack: [ 8645.435357] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [ 8645.444652] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [ 8645.453970] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [ 8645.463269] Call Trace: [ 8645.466989] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 8645.476157] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 8645.485030] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 8645.493711] [] ? wake_up_state+0x20/0x20 [ 8645.500918] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 8645.510282] [] kthread+0xcf/0xe0 [ 8645.516724] [] ? kthread_create_on_node+0x140/0x140 [ 8645.525015] [] ret_from_fork+0x58/0x90 [ 8645.532028] [] ? kthread_create_on_node+0x140/0x140 [ 8645.540303] Code: 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 e8 77 5b f4 ff 42 8d 14 38 48 8b 45 88 48 8b 80 a8 03 00 00 48 3b 50 20 0f 83 57 01 00 00 <44> 0f b6 85 67 ff ff ff 48 8d 4d a8 48 8d 55 88 48 8b b5 38 ff [ 8673.144612] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [ 8673.154237] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 8673.286111] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 8673.299421] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 8673.315735] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 8673.325134] RIP: 0010:[] [] osd_scrub_exec+0x96/0x5b0 [osd_ldiskfs] [ 8673.336804] RSP: 0018:ffff880703603d10 EFLAGS: 00000292 [ 8673.343773] RAX: 0000000000000000 RBX: ffff880703603d98 RCX: 0000000000000000 [ 8673.352763] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 8673.361735] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [ 8673.370691] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000001 [ 8673.379650] R13: 0000000000000004 R14: 000000009f8f2001 R15: ffffea00027e3c80 [ 8673.388597] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 8673.398617] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8673.406019] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 8673.414981] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8673.423937] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 8673.432884] Stack: [ 8673.436090] ffffffffa121d1e9 ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 [ 8673.445364] 00000000810d354f ffff8803162f6000 ffff8800b2a37468 0000000020000000 [ 8673.454641] 010000000000000c 0000000000000000 0000000000000000 ffff8800b2a36000 [ 8673.463952] Call Trace: [ 8673.467671] [] ? osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [ 8673.477020] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 8673.486189] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 8673.495059] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 8673.503730] [] ? wake_up_state+0x20/0x20 [ 8673.510927] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 8673.520270] [] kthread+0xcf/0xe0 [ 8673.526704] [] ? kthread_create_on_node+0x140/0x140 [ 8673.534975] [] ret_from_fork+0x58/0x90 [ 8673.541982] [] ? kthread_create_on_node+0x140/0x140 [ 8673.550240] Code: 00 00 41 80 a7 74 14 00 00 fe 48 8b 4d d0 65 48 33 0c 25 28 00 00 00 0f 85 b1 03 00 00 48 83 c4 68 5b 41 5c 41 5d 41 5e 41 5f 5d 66 0f 1f 84 00 00 00 00 00 4c 8d ae f0 10 00 00 44 89 4d 88 [ 8701.145595] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [ 8701.155222] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 8701.286903] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 8701.300188] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 8701.316428] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 8701.325810] RIP: 0010:[] [] osd_inode_iteration+0x4ac/0xcc0 [osd_ldiskfs] [ 8701.338043] RSP: 0018:ffff880703603d18 EFLAGS: 00000216 [ 8701.344982] RAX: 0000000020000000 RBX: 000000009f8f2001 RCX: 0000000000000000 [ 8701.353960] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 8701.362926] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [ 8701.371881] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [ 8701.380823] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [ 8701.389773] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 8701.399800] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8701.407216] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 8701.416194] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8701.425157] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 8701.434106] Stack: [ 8701.437328] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [ 8701.446615] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [ 8701.455930] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [ 8701.465223] Call Trace: [ 8701.468952] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 8701.478127] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 8701.487003] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 8701.495687] [] ? wake_up_state+0x20/0x20 [ 8701.502898] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 8701.512259] [] kthread+0xcf/0xe0 [ 8701.518692] [] ? kthread_create_on_node+0x140/0x140 [ 8701.526975] [] ret_from_fork+0x58/0x90 [ 8701.533979] [] ? kthread_create_on_node+0x140/0x140 [ 8701.542258] Code: 88 48 8b b5 38 ff ff ff 48 8b bd 48 ff ff ff 48 8b 85 28 ff ff ff ff d0 85 c0 41 89 c5 75 1c 8b 85 58 ff ff ff 41 3b 04 24 72 10 <48> 8b 85 50 ff ff ff 44 3b 30 0f 87 95 fd ff ff 45 31 ff e9 a2 [ 8729.146563] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [ 8729.156187] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 8729.287992] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 8729.301303] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 8729.317614] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 8729.327018] RIP: 0010:[] [] osd_scrub_exec+0xf9/0x5b0 [osd_ldiskfs] [ 8729.338675] RSP: 0018:ffff880703603c78 EFLAGS: 00000246 [ 8729.345634] RAX: 0000000000000000 RBX: ffff880703603c37 RCX: 0000000000000000 [ 8729.354627] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 8729.363604] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [ 8729.372563] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [ 8729.381513] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [ 8729.390458] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 8729.400481] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8729.407883] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 8729.416847] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8729.425799] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 8729.434745] Stack: [ 8729.437956] ffff880703603d08 000000009f8f2001 ffffea00027e3c80 000000009f8f2001 [ 8729.447233] ffff880703603d57 ffff880703603d57 ffffffffa121b990 0000000000000000 [ 8729.456519] ffff880703603d78 ffff8800b2a36000 ffff8803162f6000 ffffffffffffff10 [ 8729.465812] Call Trace: [ 8729.469535] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 8729.478694] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [ 8729.487848] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 8729.496999] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 8729.505861] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 8729.514510] [] ? wake_up_state+0x20/0x20 [ 8729.521705] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 8729.531051] [] kthread+0xcf/0xe0 [ 8729.537474] [] ? kthread_create_on_node+0x140/0x140 [ 8729.545742] [] ret_from_fork+0x58/0x90 [ 8729.552733] [] ? kthread_create_on_node+0x140/0x140 [ 8729.560992] Code: 88 48 85 db 74 23 f6 83 18 0a 00 00 08 74 1a 4d 85 e4 74 15 41 8b 87 6c 14 00 00 41 39 84 24 0c 0a 00 00 0f 82 d9 00 00 00 31 c0 <41> f6 87 74 14 00 00 04 0f 85 6e ff ff ff 41 83 f9 02 0f 84 64 [ 8743.671076] INFO: rcu_sched self-detected stall on CPU { 24} (t=1320042 jiffies g=907112 c=907111 q=1404478) [ 8743.683386] Task dump for CPU 24: [ 8743.688111] OI_scrub R running task 0 5551 2 0x00000088 [ 8743.697007] ffff88083fde6dd0 00000000fb6db880 ffff88082da03db0 ffffffff810c47e8 [ 8743.706310] 0000000000000018 ffffffff81a1e7c0 ffff88082da03dc8 ffffffff810c8079 [ 8743.715594] 0000000000000009 ffff88082da03df8 ffffffff811378e0 ffff88082da101c0 [ 8743.724850] Call Trace: [ 8743.728513] [] sched_show_task+0xa8/0x110 [ 8743.736459] [] dump_cpu_task+0x39/0x70 [ 8743.743391] [] rcu_dump_cpu_stacks+0x90/0xd0 [ 8743.750889] [] rcu_check_callbacks+0x442/0x720 [ 8743.758557] [] ? tick_sched_handle.isra.13+0x60/0x60 [ 8743.766788] [] update_process_times+0x47/0x80 [ 8743.774324] [] tick_sched_handle.isra.13+0x25/0x60 [ 8743.782310] [] tick_sched_timer+0x41/0x70 [ 8743.789407] [] __hrtimer_run_queues+0xd2/0x260 [ 8743.796975] [] hrtimer_interrupt+0xb0/0x1e0 [ 8743.804230] [] ? call_softirq+0x1c/0x30 [ 8743.811089] [] local_apic_timer_interrupt+0x37/0x60 [ 8743.819096] [] smp_apic_timer_interrupt+0x3f/0x60 [ 8743.826909] [] apic_timer_interrupt+0x6d/0x80 [ 8743.834316] [] ? osd_inode_iteration+0x255/0xcc0 [osd_ldiskfs] [ 8743.844118] [] ? osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [ 8743.853210] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 8743.862107] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 8743.870713] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 8743.879108] [] ? wake_up_state+0x20/0x20 [ 8743.886050] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 8743.895132] [] kthread+0xcf/0xe0 [ 8743.901308] [] ? kthread_create_on_node+0x140/0x140 [ 8743.909326] [] ret_from_fork+0x58/0x90 [ 8743.916071] [] ? kthread_create_on_node+0x140/0x140 [ 8769.147963] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [ 8769.157338] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 8769.286866] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 8769.300048] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 8769.316119] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 8769.325423] RIP: 0010:[] [] osd_inode_iteration+0x269/0xcc0 [osd_ldiskfs] [ 8769.337564] RSP: 0018:ffff880703603d18 EFLAGS: 00000202 [ 8769.344429] RAX: 00000000000072ce RBX: ffff880703603d57 RCX: ffff88042bd78800 [ 8769.353339] RDX: 00000000000072ce RSI: ffff880419dc3a00 RDI: ffff88042bd79800 [ 8769.362234] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [ 8769.371131] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [ 8769.380037] R13: ffffea00027e3c80 R14: 000000009f8f2001 R15: 0000000000000018 [ 8769.388934] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 8769.398897] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8769.406245] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 8769.415147] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8769.424053] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 8769.432948] Stack: [ 8769.436113] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [ 8769.445348] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [ 8769.454602] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [ 8769.463854] Call Trace: [ 8769.467528] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 8769.476663] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 8769.485505] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 8769.494149] [] ? wake_up_state+0x20/0x20 [ 8769.501340] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 8769.510680] [] kthread+0xcf/0xe0 [ 8769.517078] [] ? kthread_create_on_node+0x140/0x140 [ 8769.525321] [] ret_from_fork+0x58/0x90 [ 8769.532301] [] ? kthread_create_on_node+0x140/0x140 [ 8769.540547] Code: 88 e8 dc 2d f5 ff 48 85 c0 48 89 45 90 0f 84 b3 08 00 00 44 8b 7d a0 48 8b 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 e8 77 5b f4 ff <42> 8d 14 38 48 8b 45 88 48 8b 80 a8 03 00 00 48 3b 50 20 0f 83 [ 8797.148932] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [ 8797.158526] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 8797.290259] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 8797.303571] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 8797.319869] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 8797.329272] RIP: 0010:[] [] osd_scrub_exec+0x5/0x5b0 [osd_ldiskfs] [ 8797.340838] RSP: 0018:ffff880703603d10 EFLAGS: 00000293 [ 8797.347796] RAX: ffffffffa121b990 RBX: ffff880703603d98 RCX: 0000000000000000 [ 8797.356777] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 8797.365740] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [ 8797.374701] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000001 [ 8797.383645] R13: 0000000000000004 R14: 000000009f8f2001 R15: 0000000000000018 [ 8797.392581] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 8797.402603] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8797.410000] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 8797.418953] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8797.427902] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 8797.436845] Stack: [ 8797.440047] ffffffffa121d1e9 ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 [ 8797.449306] 00000000810d354f ffff8803162f6000 ffff8800b2a37468 0000000020000000 [ 8797.458587] 010000000000000c 0000000000000000 0000000000000000 ffff8800b2a36000 [ 8797.467876] Call Trace: [ 8797.471589] [] ? osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [ 8797.480943] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 8797.490106] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 8797.498962] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 8797.507620] [] ? wake_up_state+0x20/0x20 [ 8797.514820] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 8797.524158] [] kthread+0xcf/0xe0 [ 8797.530579] [] ? kthread_create_on_node+0x140/0x140 [ 8797.538854] [] ret_from_fork+0x58/0x90 [ 8797.545862] [] ? kthread_create_on_node+0x140/0x140 [ 8797.554121] Code: 05 85 a5 02 00 00 00 02 00 89 54 24 08 48 8b 40 40 ba 07 00 00 00 48 89 04 24 31 c0 e8 a5 03 98 ff e9 c7 fd ff ff 66 66 66 66 90 <55> 48 89 e5 41 57 49 89 f7 41 56 41 55 41 54 45 31 e4 53 48 83 [ 8825.149909] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [ 8825.159522] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 8825.291092] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 8825.304374] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 8825.320626] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 8825.330005] RIP: 0010:[] [] osd_scrub_next+0x169/0x3c0 [osd_ldiskfs] [ 8825.341741] RSP: 0018:ffff880703603d08 EFLAGS: 00000282 [ 8825.348674] RAX: 0000000000000004 RBX: 000000009f8f2001 RCX: ffff880703603d98 [ 8825.357646] RDX: ffff880703603d78 RSI: 0000000000000000 RDI: ffff8803162f6000 [ 8825.366605] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [ 8825.375549] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [ 8825.384488] R13: 0000000000000018 R14: ffff880703603c78 R15: 0000000000000202 [ 8825.393426] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 8825.403438] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8825.410831] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 8825.419789] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8825.428737] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 8825.437674] Stack: [ 8825.440875] ffff880703603df0 ffffffffa121cff9 ffffffffa121b990 ffffffffa1217a20 [ 8825.450176] ffff8800b2a36000 00000000810d354f ffff8803162f6000 ffff8800b2a37468 [ 8825.459471] 0000000020000000 010000000000000c 0000000000000000 0000000000000000 [ 8825.468742] Call Trace: [ 8825.472454] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [ 8825.481620] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 8825.490767] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 8825.499624] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 8825.508281] [] ? wake_up_state+0x20/0x20 [ 8825.515478] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 8825.524818] [] kthread+0xcf/0xe0 [ 8825.531236] [] ? kthread_create_on_node+0x140/0x140 [ 8825.539500] [] ret_from_fork+0x58/0x90 [ 8825.546503] [] ? kthread_create_on_node+0x140/0x140 [ 8825.554766] Code: df e8 fc 69 47 e0 b8 05 00 00 00 48 8b 75 d0 65 48 33 34 25 28 00 00 00 0f 85 4c 02 00 00 48 83 c4 70 5b 41 5c 41 5d 41 5e 41 5f <5d> c3 48 8b 05 be 58 9c ff 48 85 c0 0f 84 c2 fe ff ff 0f 1f 44 [ 8853.150884] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [ 8853.160498] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 8853.292053] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 8853.305334] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 8853.321569] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 8853.330944] RIP: 0010:[] [] osd_inode_iteration+0x269/0xcc0 [osd_ldiskfs] [ 8853.343160] RSP: 0018:ffff880703603d18 EFLAGS: 00000202 [ 8853.350092] RAX: 00000000000072ce RBX: ffff880703603d57 RCX: ffff88042bd78800 [ 8853.359062] RDX: 00000000000072ce RSI: ffff880419dc3a00 RDI: ffff88042bd79800 [ 8853.368020] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [ 8853.376959] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [ 8853.385902] R13: ffffea00027e3c80 R14: 000000009f8f2001 R15: ffff880703603d08 [ 8853.394841] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 8853.404861] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8853.412261] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 8853.421223] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8853.430177] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 8853.439119] Stack: [ 8853.442327] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [ 8853.451607] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [ 8853.460906] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [ 8853.470218] Call Trace: [ 8853.473937] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 8853.483101] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 8853.491973] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 8853.500639] [] ? wake_up_state+0x20/0x20 [ 8853.507844] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 8853.517186] [] kthread+0xcf/0xe0 [ 8853.523616] [] ? kthread_create_on_node+0x140/0x140 [ 8853.531897] [] ret_from_fork+0x58/0x90 [ 8853.538916] [] ? kthread_create_on_node+0x140/0x140 [ 8853.547193] Code: 88 e8 dc 2d f5 ff 48 85 c0 48 89 45 90 0f 84 b3 08 00 00 44 8b 7d a0 48 8b 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 e8 77 5b f4 ff <42> 8d 14 38 48 8b 45 88 48 8b 80 a8 03 00 00 48 3b 50 20 0f 83 [ 8881.151858] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [ 8881.161505] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 8881.293327] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 8881.306638] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 8881.322938] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 8881.332345] RIP: 0010:[] [] osd_inode_iteration+0x4ac/0xcc0 [osd_ldiskfs] [ 8881.344587] RSP: 0018:ffff880703603d18 EFLAGS: 00000216 [ 8881.351549] RAX: 0000000020000000 RBX: ffff880703603d98 RCX: 0000000000000000 [ 8881.360529] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 8881.369506] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [ 8881.378470] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000001 [ 8881.387420] R13: 0000000000000004 R14: 000000009f8f2001 R15: ffffea00027e3c80 [ 8881.396366] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 8881.406391] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8881.413794] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 8881.422757] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8881.431715] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 8881.440657] Stack: [ 8881.443862] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [ 8881.453137] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [ 8881.462426] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [ 8881.471719] Call Trace: [ 8881.475436] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 8881.484612] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 8881.493482] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 8881.502152] [] ? wake_up_state+0x20/0x20 [ 8881.509366] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 8881.518713] [] kthread+0xcf/0xe0 [ 8881.525141] [] ? kthread_create_on_node+0x140/0x140 [ 8881.533412] [] ret_from_fork+0x58/0x90 [ 8881.540419] [] ? kthread_create_on_node+0x140/0x140 [ 8881.548686] Code: 88 48 8b b5 38 ff ff ff 48 8b bd 48 ff ff ff 48 8b 85 28 ff ff ff ff d0 85 c0 41 89 c5 75 1c 8b 85 58 ff ff ff 41 3b 04 24 72 10 <48> 8b 85 50 ff ff ff 44 3b 30 0f 87 95 fd ff ff 45 31 ff e9 a2 [ 8909.152833] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [ 8909.162446] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 8909.294151] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 8909.307457] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 8909.323746] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 8909.333143] RIP: 0010:[] [] osd_inode_iteration+0x47a/0xcc0 [osd_ldiskfs] [ 8909.345374] RSP: 0018:ffff880703603d18 EFLAGS: 00000293 [ 8909.352330] RAX: 0000000000000004 RBX: ffff880703603d57 RCX: ffff880703603d98 [ 8909.361312] RDX: ffff880703603d78 RSI: 0000000000000000 RDI: ffff8803162f6000 [ 8909.370286] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [ 8909.379245] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [ 8909.388192] R13: ffffea00027e3c80 R14: 000000009f8f2001 R15: ffff880703603d08 [ 8909.397138] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 8909.407158] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8909.414560] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 8909.423523] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8909.432485] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 8909.441434] Stack: [ 8909.444648] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [ 8909.453938] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [ 8909.463221] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [ 8909.472514] Call Trace: [ 8909.476245] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 8909.485418] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 8909.494284] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 8909.502971] [] ? wake_up_state+0x20/0x20 [ 8909.510207] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 8909.519553] [] kthread+0xcf/0xe0 [ 8909.525991] [] ? kthread_create_on_node+0x140/0x140 [ 8909.534274] [] ret_from_fork+0x58/0x90 [ 8909.541294] [] ? kthread_create_on_node+0x140/0x140 [ 8909.549567] Code: 02 00 00 00 00 00 c7 05 89 86 02 00 01 00 00 00 e8 7c eb 97 ff e9 0f fc ff ff 0f 1f 80 00 00 00 00 45 89 e9 4c 8d 85 67 ff ff ff <48> 8b 4d a8 48 8d 55 88 48 8b b5 38 ff ff ff 48 8b bd 48 ff ff [ 8923.680346] INFO: rcu_sched self-detected stall on CPU { 24} (t=1500045 jiffies g=907112 c=907111 q=1415359) [ 8923.692648] Task dump for CPU 24: [ 8923.697398] OI_scrub R running task 0 5551 2 0x00000088 [ 8923.706319] ffff88083fde6dd0 00000000fb6db880 ffff88082da03db0 ffffffff810c47e8 [ 8923.715643] 0000000000000018 ffffffff81a1e7c0 ffff88082da03dc8 ffffffff810c8079 [ 8923.724960] 0000000000000009 ffff88082da03df8 ffffffff811378e0 ffff88082da101c0 [ 8923.734272] Call Trace: [ 8923.737975] [] sched_show_task+0xa8/0x110 [ 8923.745963] [] dump_cpu_task+0x39/0x70 [ 8923.752940] [] rcu_dump_cpu_stacks+0x90/0xd0 [ 8923.760477] [] rcu_check_callbacks+0x442/0x720 [ 8923.768185] [] ? tick_sched_handle.isra.13+0x60/0x60 [ 8923.776457] [] update_process_times+0x47/0x80 [ 8923.784027] [] tick_sched_handle.isra.13+0x25/0x60 [ 8923.792069] [] tick_sched_timer+0x41/0x70 [ 8923.799212] [] __hrtimer_run_queues+0xd2/0x260 [ 8923.806816] [] hrtimer_interrupt+0xb0/0x1e0 [ 8923.814118] [] ? call_softirq+0x1c/0x30 [ 8923.821000] [] local_apic_timer_interrupt+0x37/0x60 [ 8923.829040] [] smp_apic_timer_interrupt+0x3f/0x60 [ 8923.836871] [] apic_timer_interrupt+0x6d/0x80 [ 8923.844289] [] ? osd_inode_iteration+0x490/0xcc0 [osd_ldiskfs] [ 8923.854091] [] ? osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [ 8923.863197] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 8923.872109] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 8923.880732] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 8923.889144] [] ? wake_up_state+0x20/0x20 [ 8923.896102] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 8923.905182] [] kthread+0xcf/0xe0 [ 8923.911349] [] ? kthread_create_on_node+0x140/0x140 [ 8923.919370] [] ret_from_fork+0x58/0x90 [ 8923.926118] [] ? kthread_create_on_node+0x140/0x140 [ 8949.154233] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [ 8949.163628] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 8949.293143] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 8949.306317] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 8949.322378] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 8949.331682] RIP: 0010:[] [] osd_scrub_exec+0xe9/0x5b0 [osd_ldiskfs] [ 8949.343237] RSP: 0018:ffff880703603c78 EFLAGS: 00000282 [ 8949.350095] RAX: 0000000000280001 RBX: ffff880703603c37 RCX: 0000000000000000 [ 8949.359001] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 8949.367898] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [ 8949.376794] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [ 8949.385691] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [ 8949.394583] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 8949.404550] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8949.411894] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 8949.420801] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8949.429708] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 8949.438606] Stack: [ 8949.441774] ffff880703603d08 0000000000000001 ffff880703603df0 000000009f8f2001 [ 8949.451011] ffff880703603d57 000000009f8f2001 0000000000000004 0000000000000001 [ 8949.460258] 0000000000000004 ffff880703603d98 ffff880703603d78 0000000000000000 [ 8949.469507] Call Trace: [ 8949.473189] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [ 8949.482324] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 8949.491454] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 8949.500296] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 8949.508951] [] ? wake_up_state+0x20/0x20 [ 8949.516137] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 8949.525455] [] kthread+0xcf/0xe0 [ 8949.531851] [] ? kthread_create_on_node+0x140/0x140 [ 8949.540091] [] ret_from_fork+0x58/0x90 [ 8949.547072] [] ? kthread_create_on_node+0x140/0x140 [ 8949.555317] Code: 88 11 00 00 01 4c 89 ef e8 f5 9f e9 df 44 8b 4d 88 48 85 db 74 23 f6 83 18 0a 00 00 08 74 1a 4d 85 e4 74 15 41 8b 87 6c 14 00 00 <41> 39 84 24 0c 0a 00 00 0f 82 d9 00 00 00 31 c0 41 f6 87 74 14 [ 8977.155202] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [ 8977.164799] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 8977.296298] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 8977.309581] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 8977.325830] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 8977.335211] RIP: 0010:[] [] osd_inode_iteration+0x489/0xcc0 [osd_ldiskfs] [ 8977.347427] RSP: 0018:ffff880703603d18 EFLAGS: 00000293 [ 8977.354366] RAX: 0000000000000004 RBX: 000000009f8f2001 RCX: 0000000000000000 [ 8977.363341] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 8977.372312] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [ 8977.381263] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [ 8977.390207] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [ 8977.399153] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 8977.409170] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8977.416573] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 8977.425536] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8977.434487] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 8977.443424] Stack: [ 8977.446627] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [ 8977.455903] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [ 8977.465192] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [ 8977.474469] Call Trace: [ 8977.478181] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 8977.487345] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 8977.496223] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 8977.504891] [] ? wake_up_state+0x20/0x20 [ 8977.512098] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 8977.521442] [] kthread+0xcf/0xe0 [ 8977.527874] [] ? kthread_create_on_node+0x140/0x140 [ 8977.536151] [] ret_from_fork+0x58/0x90 [ 8977.543165] [] ? kthread_create_on_node+0x140/0x140 [ 8977.551438] Code: 00 e8 7c eb 97 ff e9 0f fc ff ff 0f 1f 80 00 00 00 00 45 89 e9 4c 8d 85 67 ff ff ff 48 8b 4d a8 48 8d 55 88 48 8b b5 38 ff ff ff <48> 8b bd 48 ff ff ff 48 8b 85 28 ff ff ff ff d0 85 c0 41 89 c5 [ 9005.156184] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [ 9005.165817] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 9005.297689] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 9005.310999] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 9005.327290] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 9005.336695] RIP: 0010:[] [] osd_inode_iteration+0x282/0xcc0 [osd_ldiskfs] [ 9005.348944] RSP: 0018:ffff880703603d18 EFLAGS: 00000287 [ 9005.355915] RAX: ffff88042bd78800 RBX: 000000009f8f2001 RCX: ffff88042bd78800 [ 9005.364912] RDX: 00000000000072cf RSI: ffff880419dc3a00 RDI: ffff88042bd79800 [ 9005.373892] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [ 9005.382860] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [ 9005.391815] R13: ffff880703603df0 R14: 000000009f8f2001 R15: ffff880703603d08 [ 9005.400761] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 9005.410799] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9005.418211] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 9005.427183] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 9005.436150] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 9005.445103] Stack: [ 9005.448322] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [ 9005.457616] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [ 9005.466906] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [ 9005.476214] Call Trace: [ 9005.479933] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 9005.489096] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 9005.497975] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 9005.506655] [] ? wake_up_state+0x20/0x20 [ 9005.513876] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 9005.523233] [] kthread+0xcf/0xe0 [ 9005.529676] [] ? kthread_create_on_node+0x140/0x140 [ 9005.537957] [] ret_from_fork+0x58/0x90 [ 9005.544974] [] ? kthread_create_on_node+0x140/0x140 [ 9005.553252] Code: 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 e8 77 5b f4 ff 42 8d 14 38 48 8b 45 88 48 8b 80 a8 03 00 00 48 3b 50 20 0f 83 57 01 00 00 <44> 0f b6 85 67 ff ff ff 48 8d 4d a8 48 8d 55 88 48 8b b5 38 ff [ 9033.157155] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [ 9033.166791] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 9033.298619] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 9033.311935] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 9033.328240] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 9033.337655] RIP: 0010:[] [] osd_inode_iteration+0x2ac/0xcc0 [osd_ldiskfs] [ 9033.349907] RSP: 0018:ffff880703603d18 EFLAGS: 00000202 [ 9033.356876] RAX: 0000000000000004 RBX: ffff880703603d98 RCX: ffff880703603d98 [ 9033.365873] RDX: ffff880703603d78 RSI: 0000000000000000 RDI: ffff8803162f6000 [ 9033.374863] RBP: ffff880703603df0 R08: 0000000000000001 R09: 0000000000000004 [ 9033.383834] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000001 [ 9033.392792] R13: 0000000000000004 R14: 000000009f8f2001 R15: 0000000000000018 [ 9033.401742] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 9033.411766] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9033.419169] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 9033.428130] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 9033.437085] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 9033.446026] Stack: [ 9033.449230] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [ 9033.458515] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [ 9033.467792] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [ 9033.477095] Call Trace: [ 9033.480809] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 9033.489966] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 9033.498821] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 9033.507486] [] ? wake_up_state+0x20/0x20 [ 9033.514694] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 9033.524036] [] kthread+0xcf/0xe0 [ 9033.530458] [] ? kthread_create_on_node+0x140/0x140 [ 9033.538727] [] ret_from_fork+0x58/0x90 [ 9033.545732] [] ? kthread_create_on_node+0x140/0x140 [ 9033.553995] Code: 00 44 0f b6 85 67 ff ff ff 48 8d 4d a8 48 8d 55 88 48 8b b5 38 ff ff ff 48 8b bd 48 ff ff ff 48 8b 85 30 ff ff ff ff d0 83 f8 03 <41> 89 c5 0f 84 4b 02 00 00 0f 8e 15 01 00 00 83 f8 05 66 90 0f [ 9061.158128] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [ 9061.167747] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 9061.299416] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 9061.312725] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 9061.329001] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 9061.338398] RIP: 0010:[] [] osd_scrub_exec+0x95/0x5b0 [osd_ldiskfs] [ 9061.350044] RSP: 0018:ffff880703603d08 EFLAGS: 00000292 [ 9061.356996] RAX: 0000000000000000 RBX: 000000009f8f2001 RCX: 0000000000000000 [ 9061.365981] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 9061.374952] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [ 9061.383907] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [ 9061.392848] R13: 0000000000000018 R14: ffff880703603c78 R15: 0000000000000286 [ 9061.401783] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 9061.411793] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9061.419184] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 9061.428134] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 9061.437085] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 9061.446026] Stack: [ 9061.449231] ffff880703603df0 ffffffffa121d1e9 ffffffffa121b990 ffffffffa1217a20 [ 9061.458512] ffff8800b2a36000 00000000810d354f ffff8803162f6000 ffff8800b2a37468 [ 9061.467778] 0000000020000000 010000000000000c 0000000000000000 0000000000000000 [ 9061.477063] Call Trace: [ 9061.480781] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [ 9061.489942] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 9061.499100] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 9061.507967] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 9061.516635] [] ? wake_up_state+0x20/0x20 [ 9061.523841] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 9061.533184] [] kthread+0xcf/0xe0 [ 9061.539612] [] ? kthread_create_on_node+0x140/0x140 [ 9061.547888] [] ret_from_fork+0x58/0x90 [ 9061.554897] [] ? kthread_create_on_node+0x140/0x140 [ 9061.563166] Code: 01 00 00 41 80 a7 74 14 00 00 fe 48 8b 4d d0 65 48 33 0c 25 28 00 00 00 0f 85 b1 03 00 00 48 83 c4 68 5b 41 5c 41 5d 41 5e 41 5f <5d> c3 66 0f 1f 84 00 00 00 00 00 4c 8d ae f0 10 00 00 44 89 4d [ 9089.159104] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [ 9089.168717] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 9089.300264] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 9089.313535] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 9089.329767] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 9089.339145] RIP: 0010:[] [] osd_scrub_exec+0xf7/0x5b0 [osd_ldiskfs] [ 9089.350765] RSP: 0018:ffff880703603c78 EFLAGS: 00000246 [ 9089.357695] RAX: 0000000000280001 RBX: ffff880703603c37 RCX: 0000000000000000 [ 9089.366664] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 9089.375626] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [ 9089.384568] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [ 9089.393509] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [ 9089.402449] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 9089.412467] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9089.419865] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 9089.428825] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 9089.437779] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 9089.446719] Stack: [ 9089.449925] ffff880703603d08 ffff880703603d57 ffffea00027e3c80 000000009f8f2001 [ 9089.459209] ffff880703603d57 ffff880703603d57 0000000000000000 ffff88042bd78800 [ 9089.468501] 00000000000072ce ffff880419dc3a00 ffff88042bd79800 ffffffffffffff10 [ 9089.477793] Call Trace: [ 9089.481506] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [ 9089.490668] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 9089.499829] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 9089.508684] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 9089.517343] [] ? wake_up_state+0x20/0x20 [ 9089.524544] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 9089.533889] [] kthread+0xcf/0xe0 [ 9089.540314] [] ? kthread_create_on_node+0x140/0x140 [ 9089.548585] [] ret_from_fork+0x58/0x90 [ 9089.555589] [] ? kthread_create_on_node+0x140/0x140 [ 9089.563852] Code: 8b 4d 88 48 85 db 74 23 f6 83 18 0a 00 00 08 74 1a 4d 85 e4 74 15 41 8b 87 6c 14 00 00 41 39 84 24 0c 0a 00 00 0f 82 d9 00 00 00 <31> c0 41 f6 87 74 14 00 00 04 0f 85 6e ff ff ff 41 83 f9 02 0f [ 9103.689617] INFO: rcu_sched self-detected stall on CPU { 24} (t=1680048 jiffies g=907112 c=907111 q=1422050) [ 9103.701962] Task dump for CPU 24: [ 9103.706694] OI_scrub R running task 0 5551 2 0x00000088 [ 9103.715606] ffff88083fde6dd0 00000000fb6db880 ffff88082da03db0 ffffffff810c47e8 [ 9103.724918] 0000000000000018 ffffffff81a1e7c0 ffff88082da03dc8 ffffffff810c8079 [ 9103.734203] 0000000000000009 ffff88082da03df8 ffffffff811378e0 ffff88082da101c0 [ 9103.743478] Call Trace: [ 9103.747162] [] sched_show_task+0xa8/0x110 [ 9103.755119] [] dump_cpu_task+0x39/0x70 [ 9103.762075] [] rcu_dump_cpu_stacks+0x90/0xd0 [ 9103.769590] [] rcu_check_callbacks+0x442/0x720 [ 9103.777270] [] ? tick_sched_handle.isra.13+0x60/0x60 [ 9103.785512] [] update_process_times+0x47/0x80 [ 9103.793058] [] tick_sched_handle.isra.13+0x25/0x60 [ 9103.801067] [] tick_sched_timer+0x41/0x70 [ 9103.808178] [] __hrtimer_run_queues+0xd2/0x260 [ 9103.815762] [] hrtimer_interrupt+0xb0/0x1e0 [ 9103.823028] [] ? call_softirq+0x1c/0x30 [ 9103.829885] [] local_apic_timer_interrupt+0x37/0x60 [ 9103.837904] [] smp_apic_timer_interrupt+0x3f/0x60 [ 9103.845718] [] apic_timer_interrupt+0x6d/0x80 [ 9103.853144] [] ? osd_inode_iteration+0x282/0xcc0 [osd_ldiskfs] [ 9103.862950] [] ? osd_inode_iteration+0x269/0xcc0 [osd_ldiskfs] [ 9103.872047] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 9103.880953] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 9103.889559] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 9103.897960] [] ? wake_up_state+0x20/0x20 [ 9103.904904] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 9103.913982] [] kthread+0xcf/0xe0 [ 9103.920155] [] ? kthread_create_on_node+0x140/0x140 [ 9103.928174] [] ret_from_fork+0x58/0x90 [ 9103.934933] [] ? kthread_create_on_node+0x140/0x140 [ 9129.160503] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [ 9129.169899] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 9129.299494] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 9129.312676] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 9129.328757] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 9129.338059] RIP: 0010:[] [] osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 9129.350006] RSP: 0018:ffff880703603d10 EFLAGS: 00000293 [ 9129.356868] RAX: ffffffffa121b990 RBX: 0000000000000000 RCX: 0000000000000000 [ 9129.365772] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 9129.374670] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [ 9129.383576] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffff880703603d57 [ 9129.392481] R13: 0000000000000004 R14: 000000009f8f2001 R15: ffffea00027e3c80 [ 9129.401375] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 9129.411331] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9129.418672] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 9129.427582] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 9129.436483] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 9129.445377] Stack: [ 9129.448541] ffffffffa121d1e9 ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 [ 9129.457783] 00000000810d354f ffff8803162f6000 ffff8800b2a37468 0000000020000000 [ 9129.467031] 010000000000000c 0000000000000000 0000000000000000 ffff8800b2a36000 [ 9129.476287] Call Trace: [ 9129.479978] [] ? osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [ 9129.489300] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 9129.498436] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 9129.507281] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 9129.515927] [] ? wake_up_state+0x20/0x20 [ 9129.523115] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 9129.532435] [] kthread+0xcf/0xe0 [ 9129.538834] [] ? kthread_create_on_node+0x140/0x140 [ 9129.547078] [] ret_from_fork+0x58/0x90 [ 9129.554067] [] ? kthread_create_on_node+0x140/0x140 [ 9129.562309] Code: b5 07 00 00 c7 05 85 a5 02 00 00 00 02 00 89 54 24 08 48 8b 40 40 ba 07 00 00 00 48 89 04 24 31 c0 e8 a5 03 98 ff e9 c7 fd ff ff <66> 66 66 66 90 55 48 89 e5 41 57 49 89 f7 41 56 41 55 41 54 45 [ 9157.161480] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [ 9157.171073] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 9157.302756] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 9157.316045] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 9157.332314] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 9157.341700] RIP: 0010:[] [] osd_scrub_exec+0xf7/0x5b0 [osd_ldiskfs] [ 9157.353344] RSP: 0018:ffff880703603c78 EFLAGS: 00000246 [ 9157.360286] RAX: 0000000000280001 RBX: ffff880703603c37 RCX: 0000000000000000 [ 9157.369265] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 9157.378230] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [ 9157.387190] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [ 9157.396140] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [ 9157.405090] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 9157.415120] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9157.422527] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 9157.431497] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 9157.440456] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 9157.449405] Stack: [ 9157.452620] ffffea00027e3c80 000000009f8f2001 ffff880703603df0 ffff880703603d57 [ 9157.461909] ffff880703603d57 000000009f8f2001 0000000000000004 ffff880703603d57 [ 9157.471190] ffffffffa121b990 0000000000000000 ffff880703603d78 ffff8800b2a36000 [ 9157.480472] Call Trace: [ 9157.484196] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 9157.493353] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [ 9157.502510] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 9157.511657] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 9157.520507] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 9157.529164] [] ? wake_up_state+0x20/0x20 [ 9157.536360] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 9157.545694] [] kthread+0xcf/0xe0 [ 9157.552118] [] ? kthread_create_on_node+0x140/0x140 [ 9157.560389] [] ret_from_fork+0x58/0x90 [ 9157.567387] [] ? kthread_create_on_node+0x140/0x140 [ 9157.575639] Code: 8b 4d 88 48 85 db 74 23 f6 83 18 0a 00 00 08 74 1a 4d 85 e4 74 15 41 8b 87 6c 14 00 00 41 39 84 24 0c 0a 00 00 0f 82 d9 00 00 00 <31> c0 41 f6 87 74 14 00 00 04 0f 85 6e ff ff ff 41 83 f9 02 0f [ 9185.162448] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [ 9185.172054] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 9185.303362] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 9185.316621] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 9185.332812] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 9185.342169] RIP: 0010:[] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [ 9185.354358] RSP: 0018:ffff880703603d18 EFLAGS: 00000292 [ 9185.361282] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000 [ 9185.370250] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 9185.379198] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [ 9185.388138] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffff880703603d57 [ 9185.397080] R13: 0000000000000004 R14: 000000009f8f2001 R15: 0000000000000018 [ 9185.406023] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 9185.416036] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9185.423428] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 9185.432373] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 9185.441311] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 9185.450236] Stack: [ 9185.453434] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [ 9185.462722] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [ 9185.471984] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [ 9185.481261] Call Trace: [ 9185.484968] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 9185.494126] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 9185.502983] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 9185.511635] [] ? wake_up_state+0x20/0x20 [ 9185.518839] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 9185.528182] [] kthread+0xcf/0xe0 [ 9185.534618] [] ? kthread_create_on_node+0x140/0x140 [ 9185.542892] [] ret_from_fork+0x58/0x90 [ 9185.549903] [] ? kthread_create_on_node+0x140/0x140 [ 9185.558180] Code: 00 00 45 89 e9 4c 8d 85 67 ff ff ff 48 8b 4d a8 48 8d 55 88 48 8b b5 38 ff ff ff 48 8b bd 48 ff ff ff 48 8b 85 28 ff ff ff ff d0 <85> c0 41 89 c5 75 1c 8b 85 58 ff ff ff 41 3b 04 24 72 10 48 8b [ 9213.163423] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [ 9213.173046] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 9213.304823] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 9213.318127] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 9213.334409] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 9213.343816] RIP: 0010:[] [] osd_inode_iteration+0x264/0xcc0 [osd_ldiskfs] [ 9213.356057] RSP: 0018:ffff880703603d18 EFLAGS: 00000286 [ 9213.363016] RAX: ffff8800b2a37468 RBX: 000000009f8f2001 RCX: 0000000000000000 [ 9213.372003] RDX: ffff880703603d78 RSI: ffff880419dc3a00 RDI: ffff88042bd79800 [ 9213.380981] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [ 9213.389943] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [ 9213.398894] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [ 9213.407842] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 9213.417863] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9213.425266] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 9213.434228] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 9213.443188] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 9213.452136] Stack: [ 9213.455346] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [ 9213.464628] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [ 9213.473908] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [ 9213.483205] Call Trace: [ 9213.486925] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 9213.496089] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 9213.504959] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 9213.513632] [] ? wake_up_state+0x20/0x20 [ 9213.520835] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 9213.530177] [] kthread+0xcf/0xe0 [ 9213.536606] [] ? kthread_create_on_node+0x140/0x140 [ 9213.544879] [] ret_from_fork+0x58/0x90 [ 9213.551896] [] ? kthread_create_on_node+0x140/0x140 [ 9213.560174] Code: 75 98 48 8b 7d 88 e8 dc 2d f5 ff 48 85 c0 48 89 45 90 0f 84 b3 08 00 00 44 8b 7d a0 48 8b 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 77 5b f4 ff 42 8d 14 38 48 8b 45 88 48 8b 80 a8 03 00 00 48 [ 9241.164399] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [ 9241.174039] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 9241.305878] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 9241.319183] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 9241.335470] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 9241.344882] RIP: 0010:[] [] osd_scrub_exec+0x1c/0x5b0 [osd_ldiskfs] [ 9241.356549] RSP: 0018:ffff880703603c78 EFLAGS: 00000296 [ 9241.363517] RAX: ffffffffa121b990 RBX: ffff880703603c37 RCX: 0000000000000000 [ 9241.372502] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 9241.381478] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [ 9241.390444] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [ 9241.399393] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [ 9241.408338] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 9241.418362] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9241.425772] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 9241.434736] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 9241.443696] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 9241.452646] Stack: [ 9241.455855] ffffea00027e3c80 000000009f8f2001 ffff880703603df0 0000000000000001 [ 9241.465134] ffff880703603d57 000000009f8f2001 0000000000000004 ffff880703603d57 [ 9241.474404] ffff8800b2a37468 0000000000000000 ffff880703603d78 ffff8800b2a36000 [ 9241.483700] Call Trace: [ 9241.487407] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [ 9241.496569] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 9241.505726] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 9241.514590] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 9241.523258] [] ? wake_up_state+0x20/0x20 [ 9241.530459] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 9241.539788] [] kthread+0xcf/0xe0 [ 9241.546217] [] ? kthread_create_on_node+0x140/0x140 [ 9241.554488] [] ret_from_fork+0x58/0x90 [ 9241.561505] [] ? kthread_create_on_node+0x140/0x140 [ 9241.569766] Code: 89 04 24 31 c0 e8 a5 03 98 ff e9 c7 fd ff ff 66 66 66 66 90 55 48 89 e5 41 57 49 89 f7 41 56 41 55 41 54 45 31 e4 53 48 83 ec 68 <48> 8b 9e 08 10 00 00 4c 89 45 90 65 48 8b 04 25 28 00 00 00 48 [ 9269.165380] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [ 9269.175004] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 9269.306578] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 9269.319860] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 9269.336107] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 9269.345492] RIP: 0010:[] [] osd_scrub_exec+0xf7/0x5b0 [osd_ldiskfs] [ 9269.357121] RSP: 0018:ffff880703603c78 EFLAGS: 00000246 [ 9269.364056] RAX: 0000000000280001 RBX: ffff880703603c37 RCX: 0000000000000000 [ 9269.373029] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 9269.381991] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [ 9269.390940] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [ 9269.399886] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [ 9269.408829] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 9269.418855] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9269.426258] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 9269.435212] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 9269.444164] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 9269.453102] Stack: [ 9269.456298] 0000000000000004 0000000000000001 ffff880703603df0 ffff880703603d98 [ 9269.465577] ffff880703603d57 000000009f8f2001 0000000000000004 ffff880703603d57 [ 9269.474862] 0000000020000000 0000000000000000 ffff880703603d78 ffff8800b2a36000 [ 9269.484141] Call Trace: [ 9269.487854] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [ 9269.497015] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 9269.506173] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 9269.515041] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 9269.523702] [] ? wake_up_state+0x20/0x20 [ 9269.530909] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 9269.540257] [] kthread+0xcf/0xe0 [ 9269.546683] [] ? kthread_create_on_node+0x140/0x140 [ 9269.554956] [] ret_from_fork+0x58/0x90 [ 9269.561971] [] ? kthread_create_on_node+0x140/0x140 [ 9269.570244] Code: 8b 4d 88 48 85 db 74 23 f6 83 18 0a 00 00 08 74 1a 4d 85 e4 74 15 41 8b 87 6c 14 00 00 41 39 84 24 0c 0a 00 00 0f 82 d9 00 00 00 <31> c0 41 f6 87 74 14 00 00 04 0f 85 6e ff ff ff 41 83 f9 02 0f [ 9283.698887] INFO: rcu_sched self-detected stall on CPU { 24} (t=1860051 jiffies g=907112 c=907111 q=1432823) [ 9283.711229] Task dump for CPU 24: [ 9283.715957] OI_scrub R running task 0 5551 2 0x00000088 [ 9283.724867] ffff88083fde6dd0 00000000fb6db880 ffff88082da03db0 ffffffff810c47e8 [ 9283.734190] 0000000000000018 ffffffff81a1e7c0 ffff88082da03dc8 ffffffff810c8079 [ 9283.743475] 0000000000000009 ffff88082da03df8 ffffffff811378e0 ffff88082da101c0 [ 9283.752736] Call Trace: [ 9283.756414] [] sched_show_task+0xa8/0x110 [ 9283.764381] [] dump_cpu_task+0x39/0x70 [ 9283.771332] [] rcu_dump_cpu_stacks+0x90/0xd0 [ 9283.778840] [] rcu_check_callbacks+0x442/0x720 [ 9283.786527] [] ? tick_sched_handle.isra.13+0x60/0x60 [ 9283.794772] [] update_process_times+0x47/0x80 [ 9283.802316] [] tick_sched_handle.isra.13+0x25/0x60 [ 9283.810325] [] tick_sched_timer+0x41/0x70 [ 9283.817438] [] __hrtimer_run_queues+0xd2/0x260 [ 9283.825023] [] hrtimer_interrupt+0xb0/0x1e0 [ 9283.832291] [] ? call_softirq+0x1c/0x30 [ 9283.839162] [] local_apic_timer_interrupt+0x37/0x60 [ 9283.847189] [] smp_apic_timer_interrupt+0x3f/0x60 [ 9283.855005] [] apic_timer_interrupt+0x6d/0x80 [ 9283.862441] [] ? osd_scrub_next+0x161/0x3c0 [osd_ldiskfs] [ 9283.871756] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [ 9283.880655] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 9283.889550] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 9283.898161] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 9283.906559] [] ? wake_up_state+0x20/0x20 [ 9283.913507] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 9283.922585] [] kthread+0xcf/0xe0 [ 9283.928750] [] ? kthread_create_on_node+0x140/0x140 [ 9283.936761] [] ret_from_fork+0x58/0x90 [ 9283.943514] [] ? kthread_create_on_node+0x140/0x140 [ 9309.166768] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [ 9309.176157] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 9309.305604] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 9309.318778] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 9309.334840] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 9309.344137] RIP: 0010:[] [] osd_inode_iteration+0x4a6/0xcc0 [osd_ldiskfs] [ 9309.356277] RSP: 0018:ffff880703603d18 EFLAGS: 00000246 [ 9309.363136] RAX: 0000000020000000 RBX: 000000009f8f2001 RCX: 0000000000000000 [ 9309.372040] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 9309.380937] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [ 9309.389833] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [ 9309.398728] R13: ffff880703603d08 R14: 000000009f8f2001 R15: ffff880703603d08 [ 9309.407623] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 9309.417598] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9309.424941] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 9309.433839] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 9309.442745] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 9309.451645] Stack: [ 9309.454813] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [ 9309.464061] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [ 9309.473317] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [ 9309.482573] Call Trace: [ 9309.486271] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 9309.495404] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 9309.504247] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 9309.512891] [] ? wake_up_state+0x20/0x20 [ 9309.520082] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 9309.529408] [] kthread+0xcf/0xe0 [ 9309.535808] [] ? kthread_create_on_node+0x140/0x140 [ 9309.544048] [] ret_from_fork+0x58/0x90 [ 9309.551034] [] ? kthread_create_on_node+0x140/0x140 [ 9309.559286] Code: 8b 4d a8 48 8d 55 88 48 8b b5 38 ff ff ff 48 8b bd 48 ff ff ff 48 8b 85 28 ff ff ff ff d0 85 c0 41 89 c5 75 1c 8b 85 58 ff ff ff <41> 3b 04 24 72 10 48 8b 85 50 ff ff ff 44 3b 30 0f 87 95 fd ff [ 9337.167749] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [ 9337.177347] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 9337.309158] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 9337.322466] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 9337.338766] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 9337.348171] RIP: 0010:[] [] ldiskfs_used_dirs_count+0x30/0x30 [ldiskfs] [ 9337.360244] RSP: 0018:ffff880703603d10 EFLAGS: 00000286 [ 9337.367215] RAX: ffff8800b2a37468 RBX: 0000000000000000 RCX: 0000000000000000 [ 9337.376209] RDX: ffff880703603d78 RSI: ffff880419dc3a00 RDI: ffff88042bd79800 [ 9337.385190] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [ 9337.394168] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffff880703603d57 [ 9337.403122] R13: 0000000000000004 R14: 000000009f8f2001 R15: ffffea00027e3c80 [ 9337.412070] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 9337.422093] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9337.429496] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 9337.438459] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 9337.447423] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 9337.456364] Stack: [ 9337.459570] ffffffffa121cfb9 ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 [ 9337.468851] 00000000810d354f ffff8803162f6000 ffff8800b2a37468 0000000020000000 [ 9337.478136] 010000000000000c 0000000000000000 0000000000000000 ffff8800b2a36000 [ 9337.487416] Call Trace: [ 9337.491143] [] ? osd_inode_iteration+0x269/0xcc0 [osd_ldiskfs] [ 9337.500500] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 9337.509664] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 9337.518536] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 9337.527200] [] ? wake_up_state+0x20/0x20 [ 9337.534398] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 9337.543743] [] kthread+0xcf/0xe0 [ 9337.550179] [] ? kthread_create_on_node+0x140/0x140 [ 9337.558454] [] ret_from_fork+0x58/0x90 [ 9337.565469] [] ? kthread_create_on_node+0x140/0x140 [ 9337.573742] Code: 55 48 8b 8f a8 03 00 00 31 c0 0f b7 56 10 48 89 e5 48 83 39 3f 76 07 0f b7 46 30 c1 e0 10 09 d0 5d c3 66 0f 1f 84 00 00 00 00 00 <66> 66 66 66 90 55 48 8b 8f a8 03 00 00 31 c0 0f b7 56 1c 48 89 [ 9365.168719] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [ 9365.178354] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 9365.310090] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 9365.323382] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 9365.339638] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 9365.349026] RIP: 0010:[] [] osd_inode_iteration+0x282/0xcc0 [osd_ldiskfs] [ 9365.361258] RSP: 0018:ffff880703603d18 EFLAGS: 00000287 [ 9365.368212] RAX: ffff88042bd78800 RBX: ffff880703603d98 RCX: ffff88042bd78800 [ 9365.377196] RDX: 00000000000072cf RSI: ffff880419dc3a00 RDI: ffff88042bd79800 [ 9365.386165] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [ 9365.395131] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000001 [ 9365.404091] R13: 0000000000000004 R14: 000000009f8f2001 R15: 0000000000000018 [ 9365.413040] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 9365.423064] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9365.430473] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 9365.439434] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 9365.448389] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 9365.457336] Stack: [ 9365.460543] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [ 9365.469835] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [ 9365.479124] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [ 9365.488406] Call Trace: [ 9365.492122] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 9365.501284] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 9365.510153] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 9365.518822] [] ? wake_up_state+0x20/0x20 [ 9365.526025] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 9365.535364] [] kthread+0xcf/0xe0 [ 9365.541795] [] ? kthread_create_on_node+0x140/0x140 [ 9365.550074] [] ret_from_fork+0x58/0x90 [ 9365.557087] [] ? kthread_create_on_node+0x140/0x140 [ 9365.565361] Code: 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 e8 77 5b f4 ff 42 8d 14 38 48 8b 45 88 48 8b 80 a8 03 00 00 48 3b 50 20 0f 83 57 01 00 00 <44> 0f b6 85 67 ff ff ff 48 8d 4d a8 48 8d 55 88 48 8b b5 38 ff [ 9393.169701] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [ 9393.179310] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 9393.311038] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 9393.324340] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 9393.340623] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 9393.350019] RIP: 0010:[] [] osd_inode_iteration+0x251/0xcc0 [osd_ldiskfs] [ 9393.362258] RSP: 0018:ffff880703603d18 EFLAGS: 00000286 [ 9393.369215] RAX: ffff8800b2a37468 RBX: 000000009f8f2001 RCX: 0000000000000000 [ 9393.378200] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 9393.387173] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [ 9393.396133] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [ 9393.405086] R13: ffff880703603df0 R14: 000000009f8f2001 R15: ffff880703603d08 [ 9393.414031] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 9393.424056] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9393.431459] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 9393.440422] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 9393.449377] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 9393.458329] Stack: [ 9393.461544] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [ 9393.470839] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [ 9393.480126] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [ 9393.489420] Call Trace: [ 9393.493146] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 9393.502317] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 9393.511195] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 9393.519874] [] ? wake_up_state+0x20/0x20 [ 9393.527093] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 9393.536441] [] kthread+0xcf/0xe0 [ 9393.542875] [] ? kthread_create_on_node+0x140/0x140 [ 9393.551145] [] ret_from_fork+0x58/0x90 [ 9393.558168] [] ? kthread_create_on_node+0x140/0x140 [ 9393.566448] Code: b0 00 00 00 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 00 8b 75 98 48 8b 7d 88 e8 dc 2d f5 ff 48 85 c0 48 89 45 90 0f 84 b3 08 00 00 <44> 8b 7d a0 48 8b 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 e8 77 [ 9421.170669] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [ 9421.180298] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 9421.312162] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 9421.325472] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 9421.341791] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 9421.351197] RIP: 0010:[] [] osd_scrub_exec+0x12/0x5b0 [osd_ldiskfs] [ 9421.362860] RSP: 0018:ffff880703603cf0 EFLAGS: 00000293 [ 9421.369822] RAX: ffffffffa121b990 RBX: 000000009f8f2001 RCX: 0000000000000000 [ 9421.378802] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 9421.387788] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [ 9421.396756] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffff880703603c70 [ 9421.405712] R13: 0000000000000202 R14: 0000000000000010 R15: ffffffffa1217b69 [ 9421.414652] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 9421.424673] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9421.432079] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 9421.441043] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 9421.450000] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 9421.458942] Stack: [ 9421.462149] 0000000000000004 00000000ffffffff 0000000000000001 ffff880703603df0 [ 9421.471430] ffffffffa121d1e9 ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 [ 9421.480730] 00000000810d354f ffff8803162f6000 ffff8800b2a37468 0000000020000000 [ 9421.490031] Call Trace: [ 9421.493759] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [ 9421.502940] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 9421.512100] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 9421.520966] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 9421.529627] [] ? wake_up_state+0x20/0x20 [ 9421.536838] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 9421.546186] [] kthread+0xcf/0xe0 [ 9421.552618] [] ? kthread_create_on_node+0x140/0x140 [ 9421.560880] [] ret_from_fork+0x58/0x90 [ 9421.567887] [] ? kthread_create_on_node+0x140/0x140 [ 9421.576156] Code: 48 8b 40 40 ba 07 00 00 00 48 89 04 24 31 c0 e8 a5 03 98 ff e9 c7 fd ff ff 66 66 66 66 90 55 48 89 e5 41 57 49 89 f7 41 56 41 55 <41> 54 45 31 e4 53 48 83 ec 68 48 8b 9e 08 10 00 00 4c 89 45 90 [ 9449.171645] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [ 9449.181279] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 9449.312901] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 9449.326186] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 9449.342430] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 9449.351819] RIP: 0010:[] [] osd_inode_iteration+0x489/0xcc0 [osd_ldiskfs] [ 9449.364051] RSP: 0018:ffff880703603d18 EFLAGS: 00000293 [ 9449.370990] RAX: 0000000000000004 RBX: ffff880703603d98 RCX: 0000000000000000 [ 9449.379966] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 9449.388933] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [ 9449.397884] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000001 [ 9449.406832] R13: 0000000000000004 R14: 000000009f8f2001 R15: ffffea00027e3c80 [ 9449.415780] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 9449.425810] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9449.433209] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 9449.442171] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 9449.451123] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 9449.460064] Stack: [ 9449.463270] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [ 9449.472566] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [ 9449.481857] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [ 9449.491143] Call Trace: [ 9449.494856] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 9449.504016] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 9449.512882] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 9449.521548] [] ? wake_up_state+0x20/0x20 [ 9449.528758] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 9449.538098] [] kthread+0xcf/0xe0 [ 9449.544532] [] ? kthread_create_on_node+0x140/0x140 [ 9449.552811] [] ret_from_fork+0x58/0x90 [ 9449.559820] [] ? kthread_create_on_node+0x140/0x140 [ 9449.568088] Code: 00 e8 7c eb 97 ff e9 0f fc ff ff 0f 1f 80 00 00 00 00 45 89 e9 4c 8d 85 67 ff ff ff 48 8b 4d a8 48 8d 55 88 48 8b b5 38 ff ff ff <48> 8b bd 48 ff ff ff 48 8b 85 28 ff ff ff ff d0 85 c0 41 89 c5 [ 9463.708158] INFO: rcu_sched self-detected stall on CPU { 24} (t=2040054 jiffies g=907112 c=907111 q=1439501) [ 9463.720487] Task dump for CPU 24: [ 9463.725229] OI_scrub R running task 0 5551 2 0x00000088 [ 9463.734143] ffff88083fde6dd0 00000000fb6db880 ffff88082da03db0 ffffffff810c47e8 [ 9463.743467] 0000000000000018 ffffffff81a1e7c0 ffff88082da03dc8 ffffffff810c8079 [ 9463.752764] 0000000000000009 ffff88082da03df8 ffffffff811378e0 ffff88082da101c0 [ 9463.762057] Call Trace: [ 9463.765753] [] sched_show_task+0xa8/0x110 [ 9463.773734] [] dump_cpu_task+0x39/0x70 [ 9463.780701] [] rcu_dump_cpu_stacks+0x90/0xd0 [ 9463.788230] [] rcu_check_callbacks+0x442/0x720 [ 9463.795930] [] ? tick_sched_handle.isra.13+0x60/0x60 [ 9463.804195] [] update_process_times+0x47/0x80 [ 9463.811765] [] tick_sched_handle.isra.13+0x25/0x60 [ 9463.819792] [] tick_sched_timer+0x41/0x70 [ 9463.826928] [] __hrtimer_run_queues+0xd2/0x260 [ 9463.834529] [] hrtimer_interrupt+0xb0/0x1e0 [ 9463.841813] [] ? call_softirq+0x1c/0x30 [ 9463.848687] [] local_apic_timer_interrupt+0x37/0x60 [ 9463.856718] [] smp_apic_timer_interrupt+0x3f/0x60 [ 9463.864541] [] apic_timer_interrupt+0x6d/0x80 [ 9463.871964] [] ? osd_scrub_next+0x167/0x3c0 [osd_ldiskfs] [ 9463.881276] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [ 9463.890171] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 9463.899063] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 9463.907665] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 9463.916073] [] ? wake_up_state+0x20/0x20 [ 9463.923023] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 9463.932105] [] kthread+0xcf/0xe0 [ 9463.938274] [] ? kthread_create_on_node+0x140/0x140 [ 9463.946283] [] ret_from_fork+0x58/0x90 [ 9463.953039] [] ? kthread_create_on_node+0x140/0x140 [ 9489.173038] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [ 9489.182437] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 9489.311963] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 9489.325131] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 9489.341186] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 9489.350478] RIP: 0010:[] [] osd_scrub_exec+0xe9/0x5b0 [osd_ldiskfs] [ 9489.362023] RSP: 0018:ffff880703603c78 EFLAGS: 00000282 [ 9489.368876] RAX: 0000000000280001 RBX: ffff880703603c37 RCX: 0000000000000000 [ 9489.377771] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 9489.386661] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [ 9489.395551] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [ 9489.404441] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [ 9489.413328] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 9489.423288] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9489.430624] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 9489.439523] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 9489.448421] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 9489.457312] Stack: [ 9489.460473] ffff880703603d08 000000009f8f2001 ffff880703603df0 000000009f8f2001 [ 9489.469714] ffff880703603d57 000000009f8f2001 0000000000000004 ffff880703603d57 [ 9489.478947] 0000000000000004 0000000000000000 ffff880703603d78 ffff8800b2a36000 [ 9489.488192] Call Trace: [ 9489.491874] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [ 9489.501001] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 9489.510124] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 9489.518957] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 9489.527598] [] ? wake_up_state+0x20/0x20 [ 9489.534770] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 9489.544089] [] kthread+0xcf/0xe0 [ 9489.550484] [] ? kthread_create_on_node+0x140/0x140 [ 9489.558730] [] ret_from_fork+0x58/0x90 [ 9489.565719] [] ? kthread_create_on_node+0x140/0x140 [ 9489.573952] Code: 88 11 00 00 01 4c 89 ef e8 f5 9f e9 df 44 8b 4d 88 48 85 db 74 23 f6 83 18 0a 00 00 08 74 1a 4d 85 e4 74 15 41 8b 87 6c 14 00 00 <41> 39 84 24 0c 0a 00 00 0f 82 d9 00 00 00 31 c0 41 f6 87 74 14 [ 9517.174013] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [ 9517.183614] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 9517.315222] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 9517.328511] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 9517.344762] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 9517.354144] RIP: 0010:[] [] osd_scrub_exec+0x30/0x5b0 [osd_ldiskfs] [ 9517.365788] RSP: 0018:ffff880703603c78 EFLAGS: 00000296 [ 9517.372722] RAX: 00000000fb6db880 RBX: ffff880703603c37 RCX: 0000000000000000 [ 9517.381694] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 9517.390653] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [ 9517.399602] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [ 9517.408545] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [ 9517.417490] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 9517.427514] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9517.434918] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 9517.443871] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 9517.452821] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 9517.461759] Stack: [ 9517.464953] ffff880703603d08 0000000000000001 ffffea00027e3c80 000000009f8f2001 [ 9517.474241] ffff880703603d57 0000000000000001 ffffffffa1217a20 ffff880703603d98 [ 9517.483533] ffff880703603d78 ffff8800b2a36000 ffff8803162f6000 ffffffffffffff10 [ 9517.492818] Call Trace: [ 9517.496526] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 9517.505392] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [ 9517.514549] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 9517.523687] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 9517.532546] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 9517.541208] [] ? wake_up_state+0x20/0x20 [ 9517.548415] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 9517.557753] [] kthread+0xcf/0xe0 [ 9517.564184] [] ? kthread_create_on_node+0x140/0x140 [ 9517.572459] [] ret_from_fork+0x58/0x90 [ 9517.579466] [] ? kthread_create_on_node+0x140/0x140 [ 9517.587736] Code: 55 48 89 e5 41 57 49 89 f7 41 56 41 55 41 54 45 31 e4 53 48 83 ec 68 48 8b 9e 08 10 00 00 4c 89 45 90 65 48 8b 04 25 28 00 00 00 <48> 89 45 d0 31 c0 48 8d 43 08 48 85 db 4c 0f 45 e0 41 83 f9 04 [ 9545.174996] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [ 9545.184603] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 9545.315987] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 9545.329249] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 9545.345457] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 9545.354820] RIP: 0010:[] [] osd_scrub_exec+0x30/0x5b0 [osd_ldiskfs] [ 9545.366439] RSP: 0018:ffff880703603c78 EFLAGS: 00000296 [ 9545.373365] RAX: 00000000fb6db880 RBX: ffff880703603c37 RCX: 0000000000000000 [ 9545.382323] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 9545.391274] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [ 9545.400224] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [ 9545.409168] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [ 9545.418122] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 9545.428150] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9545.435550] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 9545.444502] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 9545.453448] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 9545.462381] Stack: [ 9545.465585] 0000000000000004 0000000000000001 ffff880703603df0 ffff880703603d98 [ 9545.474866] ffff880703603d57 000000009f8f2001 0000000000000004 ffff880703603d57 [ 9545.484154] ffff8800b2a37468 0000000000000000 ffff880703603d78 ffff880419dc3a00 [ 9545.493428] Call Trace: [ 9545.497131] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [ 9545.506291] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 9545.515440] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 9545.524284] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 9545.532937] [] ? wake_up_state+0x20/0x20 [ 9545.540140] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 9545.549480] [] kthread+0xcf/0xe0 [ 9545.555908] [] ? kthread_create_on_node+0x140/0x140 [ 9545.564183] [] ret_from_fork+0x58/0x90 [ 9545.571187] [] ? kthread_create_on_node+0x140/0x140 [ 9545.579450] Code: 55 48 89 e5 41 57 49 89 f7 41 56 41 55 41 54 45 31 e4 53 48 83 ec 68 48 8b 9e 08 10 00 00 4c 89 45 90 65 48 8b 04 25 28 00 00 00 <48> 89 45 d0 31 c0 48 8d 43 08 48 85 db 4c 0f 45 e0 41 83 f9 04 [ 9573.175964] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [ 9573.185585] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 9573.317166] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 9573.330449] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 9573.346686] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 9573.356067] RIP: 0010:[] [] osd_inode_iteration+0x28a/0xcc0 [osd_ldiskfs] [ 9573.368292] RSP: 0018:ffff880703603d18 EFLAGS: 00000287 [ 9573.375230] RAX: ffff88042bd78800 RBX: ffff880703603d98 RCX: ffff88042bd78800 [ 9573.384206] RDX: 00000000000072cf RSI: ffff880419dc3a00 RDI: ffff88042bd79800 [ 9573.393165] RBP: ffff880703603df0 R08: 0000000000000001 R09: 0000000000000004 [ 9573.402117] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000001 [ 9573.411065] R13: 0000000000000004 R14: 000000009f8f2001 R15: ffffea00027e3c80 [ 9573.420012] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 9573.430025] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9573.437416] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 9573.446374] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 9573.455324] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 9573.464259] Stack: [ 9573.467462] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [ 9573.476746] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [ 9573.486030] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [ 9573.495318] Call Trace: [ 9573.499039] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 9573.508199] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 9573.517067] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 9573.525731] [] ? wake_up_state+0x20/0x20 [ 9573.532935] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 9573.542285] [] kthread+0xcf/0xe0 [ 9573.548706] [] ? kthread_create_on_node+0x140/0x140 [ 9573.556986] [] ret_from_fork+0x58/0x90 [ 9573.564003] [] ? kthread_create_on_node+0x140/0x140 [ 9573.572271] Code: a8 00 00 00 00 e8 77 5b f4 ff 42 8d 14 38 48 8b 45 88 48 8b 80 a8 03 00 00 48 3b 50 20 0f 83 57 01 00 00 44 0f b6 85 67 ff ff ff <48> 8d 4d a8 48 8d 55 88 48 8b b5 38 ff ff ff 48 8b bd 48 ff ff [ 9601.176946] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [ 9601.186575] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 9601.318391] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 9601.331703] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 9601.348002] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 9601.357405] RIP: 0010:[] [] osd_scrub_next+0x12/0x3c0 [osd_ldiskfs] [ 9601.369070] RSP: 0018:ffff880703603cf0 EFLAGS: 00000287 [ 9601.376027] RAX: ffffffffa1217a20 RBX: 000000009f8f2001 RCX: ffff880703603d98 [ 9601.385013] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 9601.393998] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [ 9601.402967] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffff880703603c70 [ 9601.411920] R13: 0000000000000282 R14: 0000000000000010 R15: ffffffffa1217a4b [ 9601.420869] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 9601.430893] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9601.438291] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 9601.447249] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 9601.456206] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 9601.465151] Stack: [ 9601.468361] 0000000000000000 00000000ffffffff 0000000000000001 ffff880703603df0 [ 9601.477648] ffffffffa121cff9 ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 [ 9601.486929] 00000000810d354f ffff8803162f6000 ffff8800b2a37468 0000000020000000 [ 9601.496227] Call Trace: [ 9601.499951] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [ 9601.509126] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 9601.518290] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 9601.527162] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 9601.535828] [] ? wake_up_state+0x20/0x20 [ 9601.543034] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 9601.552377] [] kthread+0xcf/0xe0 [ 9601.558812] [] ? kthread_create_on_node+0x140/0x140 [ 9601.567094] [] ret_from_fork+0x58/0x90 [ 9601.574102] [] ? kthread_create_on_node+0x140/0x140 [ 9601.582377] Code: 25 fc ff ff 48 83 c4 08 5b 41 5c 41 5d 41 5e 5d c3 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 89 e5 41 57 49 89 f7 41 56 41 55 <49> 89 fd 41 54 49 89 d4 53 48 89 cb 48 83 ec 70 65 48 8b 04 25 [ 9629.177917] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [ 9629.187543] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 9629.319182] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 9629.332470] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 9629.348727] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 9629.358119] RIP: 0010:[] [] osd_scrub_exec+0x79/0x5b0 [osd_ldiskfs] [ 9629.369762] RSP: 0018:ffff880703603c78 EFLAGS: 00000202 [ 9629.376701] RAX: 0000000000000000 RBX: ffff880703603c37 RCX: 00000000fb6db880 [ 9629.385677] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 9629.394643] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [ 9629.403594] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [ 9629.412544] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [ 9629.421496] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 9629.431525] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9629.438933] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 9629.447900] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 9629.456858] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 9629.465805] Stack: [ 9629.469017] ffff880703603d08 0000000000000001 ffffea00027e3c80 000000009f8f2001 [ 9629.478305] ffff880703603d57 ffff880703603d57 0000000000000000 ffff88042bd78800 [ 9629.487597] 00000000000072ce ffff880419dc3a00 ffff88042bd79800 ffffffffffffff10 [ 9629.496903] Call Trace: [ 9629.500624] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [ 9629.509792] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 9629.518953] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 9629.527818] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 9629.536485] [] ? wake_up_state+0x20/0x20 [ 9629.543699] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 9629.553059] [] kthread+0xcf/0xe0 [ 9629.559494] [] ? kthread_create_on_node+0x140/0x140 [ 9629.567778] [] ret_from_fork+0x58/0x90 [ 9629.574789] [] ? kthread_create_on_node+0x140/0x140 [ 9629.583065] Code: 07 74 4f 41 83 f9 02 74 78 48 89 ca 44 89 c9 e8 0e c6 ff ff 85 c0 41 89 c1 0f 84 a3 01 00 00 41 80 a7 74 14 00 00 fe 48 8b 4d d0 <65> 48 33 0c 25 28 00 00 00 0f 85 b1 03 00 00 48 83 c4 68 5b 41 [ 9643.717428] INFO: rcu_sched self-detected stall on CPU { 24} (t=2220057 jiffies g=907112 c=907111 q=1450326) [ 9643.729781] Task dump for CPU 24: [ 9643.734514] OI_scrub R running task 0 5551 2 0x00000088 [ 9643.743428] ffff88083fde6dd0 00000000fb6db880 ffff88082da03db0 ffffffff810c47e8 [ 9643.752747] 0000000000000018 ffffffff81a1e7c0 ffff88082da03dc8 ffffffff810c8079 [ 9643.762046] 0000000000000009 ffff88082da03df8 ffffffff811378e0 ffff88082da101c0 [ 9643.771332] Call Trace: [ 9643.775013] [] sched_show_task+0xa8/0x110 [ 9643.782976] [] dump_cpu_task+0x39/0x70 [ 9643.789931] [] rcu_dump_cpu_stacks+0x90/0xd0 [ 9643.797436] [] rcu_check_callbacks+0x442/0x720 [ 9643.805121] [] ? tick_sched_handle.isra.13+0x60/0x60 [ 9643.813388] [] update_process_times+0x47/0x80 [ 9643.820937] [] tick_sched_handle.isra.13+0x25/0x60 [ 9643.828953] [] tick_sched_timer+0x41/0x70 [ 9643.836078] [] __hrtimer_run_queues+0xd2/0x260 [ 9643.843668] [] hrtimer_interrupt+0xb0/0x1e0 [ 9643.850942] [] ? call_softirq+0x1c/0x30 [ 9643.857830] [] local_apic_timer_interrupt+0x37/0x60 [ 9643.865855] [] smp_apic_timer_interrupt+0x3f/0x60 [ 9643.873669] [] apic_timer_interrupt+0x6d/0x80 [ 9643.881101] [] ? osd_inode_iteration+0x49e/0xcc0 [osd_ldiskfs] [ 9643.890909] [] ? osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [ 9643.900018] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 9643.908927] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 9643.917555] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 9643.925974] [] ? wake_up_state+0x20/0x20 [ 9643.932932] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 9643.942013] [] kthread+0xcf/0xe0 [ 9643.948183] [] ? kthread_create_on_node+0x140/0x140 [ 9643.956211] [] ret_from_fork+0x58/0x90 [ 9643.962966] [] ? kthread_create_on_node+0x140/0x140 [ 9669.179310] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [ 9669.188695] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 9669.318246] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 9669.331422] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 9669.347486] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 9669.356784] RIP: 0010:[] [] osd_inode_iteration+0x2ac/0xcc0 [osd_ldiskfs] [ 9669.368926] RSP: 0018:ffff880703603d18 EFLAGS: 00000202 [ 9669.375779] RAX: 0000000000000004 RBX: 000000009f8f2001 RCX: ffff880703603d98 [ 9669.384687] RDX: ffff880703603d78 RSI: 0000000000000000 RDI: ffff8803162f6000 [ 9669.393587] RBP: ffff880703603df0 R08: 0000000000000001 R09: 0000000000000004 [ 9669.402482] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [ 9669.411385] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [ 9669.420284] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 9669.430251] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9669.437594] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 9669.446498] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 9669.455402] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 9669.464297] Stack: [ 9669.467464] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [ 9669.476713] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [ 9669.485962] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [ 9669.495219] Call Trace: [ 9669.498901] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 9669.508042] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 9669.516875] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 9669.525522] [] ? wake_up_state+0x20/0x20 [ 9669.532719] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 9669.542038] [] kthread+0xcf/0xe0 [ 9669.548435] [] ? kthread_create_on_node+0x140/0x140 [ 9669.556690] [] ret_from_fork+0x58/0x90 [ 9669.563660] [] ? kthread_create_on_node+0x140/0x140 [ 9669.571906] Code: 00 44 0f b6 85 67 ff ff ff 48 8d 4d a8 48 8d 55 88 48 8b b5 38 ff ff ff 48 8b bd 48 ff ff ff 48 8b 85 30 ff ff ff ff d0 83 f8 03 <41> 89 c5 0f 84 4b 02 00 00 0f 8e 15 01 00 00 83 f8 05 66 90 0f [ 9697.180284] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [ 9697.189868] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 9697.321625] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 9697.334929] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 9697.351224] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 9697.360625] RIP: 0010:[] [] osd_scrub_exec+0x30/0x5b0 [osd_ldiskfs] [ 9697.372284] RSP: 0018:ffff880703603c78 EFLAGS: 00000296 [ 9697.379243] RAX: 00000000fb6db880 RBX: ffff880703603c37 RCX: 0000000000000000 [ 9697.388227] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 9697.397202] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [ 9697.406162] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [ 9697.415113] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [ 9697.424059] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 9697.434085] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9697.441494] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 9697.450459] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 9697.459437] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 9697.468386] Stack: [ 9697.471600] ffffea00027e3c80 000000009f8f2001 ffff880703603df0 0000000000000001 [ 9697.480891] ffff880703603d57 000000009f8f2001 0000000000000004 ffff880703603d57 [ 9697.490184] ffff88042bd78800 ffff88042bd78800 00000000000072cf ffff880419dc3a00 [ 9697.499483] Call Trace: [ 9697.503211] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [ 9697.512383] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 9697.521544] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 9697.530414] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 9697.539082] [] ? wake_up_state+0x20/0x20 [ 9697.546292] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 9697.555638] [] kthread+0xcf/0xe0 [ 9697.562064] [] ? kthread_create_on_node+0x140/0x140 [ 9697.570334] [] ret_from_fork+0x58/0x90 [ 9697.577338] [] ? kthread_create_on_node+0x140/0x140 [ 9697.585609] Code: 55 48 89 e5 41 57 49 89 f7 41 56 41 55 41 54 45 31 e4 53 48 83 ec 68 48 8b 9e 08 10 00 00 4c 89 45 90 65 48 8b 04 25 28 00 00 00 <48> 89 45 d0 31 c0 48 8d 43 08 48 85 db 4c 0f 45 e0 41 83 f9 04 [ 9725.181259] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [ 9725.190894] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 9725.322493] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 9725.335777] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 9725.352022] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 9725.361402] RIP: 0010:[] [] ldiskfs_itable_unused_count+0x25/0x30 [ldiskfs] [ 9725.373830] RSP: 0018:ffff880703603d08 EFLAGS: 00000202 [ 9725.380768] RAX: 00000000000072ce RBX: 0000000000000001 RCX: ffff88042bd78800 [ 9725.389750] RDX: 00000000000072ce RSI: ffff880419dc3a00 RDI: ffff88042bd79800 [ 9725.398706] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [ 9725.407653] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [ 9725.416596] R13: 0000000000000018 R14: ffff880703603c78 R15: 0000000000000296 [ 9725.425536] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 9725.435550] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9725.442943] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 9725.451894] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 9725.460835] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 9725.469765] Stack: [ 9725.472958] ffff880703603df0 ffffffffa121cfb9 ffffffffa121b990 ffffffffa1217a20 [ 9725.482231] ffff8800b2a36000 00000000810d354f ffff8803162f6000 ffff8800b2a37468 [ 9725.491522] 0000000020000000 010000000000000c 0000000000000000 0000000000000000 [ 9725.500803] Call Trace: [ 9725.504519] [] osd_inode_iteration+0x269/0xcc0 [osd_ldiskfs] [ 9725.513664] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 9725.522815] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 9725.531670] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 9725.540330] [] ? wake_up_state+0x20/0x20 [ 9725.547541] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 9725.556875] [] kthread+0xcf/0xe0 [ 9725.563295] [] ? kthread_create_on_node+0x140/0x140 [ 9725.571566] [] ret_from_fork+0x58/0x90 [ 9725.578566] [] ? kthread_create_on_node+0x140/0x140 [ 9725.586824] Code: 84 00 00 00 00 00 66 66 66 66 90 55 48 8b 8f a8 03 00 00 31 c0 0f b7 56 1c 48 89 e5 48 83 39 3f 76 07 0f b7 46 32 c1 e0 10 09 d0 <5d> c3 66 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 89 16 48 8b [ 9753.182241] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [ 9753.191850] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 9753.323400] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 9753.336679] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 9753.352917] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 9753.362294] RIP: 0010:[] [] osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 9753.374026] RSP: 0018:ffff880703603d10 EFLAGS: 00000287 [ 9753.380959] RAX: ffffffffa1217a20 RBX: 000000009f8f2001 RCX: ffff880703603d98 [ 9753.389927] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 9753.398881] RBP: ffff880703603df0 R08: 0000000000000001 R09: 0000000000000004 [ 9753.407823] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [ 9753.416763] R13: ffff880703603df0 R14: 000000009f8f2001 R15: ffff880703603d08 [ 9753.425710] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 9753.435732] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9753.443125] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 9753.452089] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 9753.461037] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 9753.469981] Stack: [ 9753.473191] ffffffffa121cff9 ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 [ 9753.482503] 00000000810d354f ffff8803162f6000 ffff8800b2a37468 0000000020000000 [ 9753.491798] 010000000000000c 0000000000000000 0000000000000000 ffff8800b2a36000 [ 9753.501094] Call Trace: [ 9753.504814] [] ? osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [ 9753.514176] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 9753.523366] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 9753.532236] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 9753.540901] [] ? wake_up_state+0x20/0x20 [ 9753.548106] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 9753.557457] [] kthread+0xcf/0xe0 [ 9753.563901] [] ? kthread_create_on_node+0x140/0x140 [ 9753.572183] [] ret_from_fork+0x58/0x90 [ 9753.579206] [] ? kthread_create_on_node+0x140/0x140 [ 9753.587487] Code: f7 48 8d 04 80 48 c1 e0 03 48 8d 4c 02 10 48 01 c2 e8 25 fc ff ff 48 83 c4 08 5b 41 5c 41 5d 41 5e 5d c3 0f 1f 84 00 00 00 00 00 <66> 66 66 66 90 55 48 89 e5 41 57 49 89 f7 41 56 41 55 49 89 fd [ 9781.183216] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [ 9781.192849] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 9781.324586] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 9781.337881] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 9781.354151] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 9781.363542] RIP: 0010:[] [] osd_scrub_exec+0x96/0x5b0 [osd_ldiskfs] [ 9781.375200] RSP: 0018:ffff880703603d10 EFLAGS: 00000292 [ 9781.382146] RAX: 0000000000000000 RBX: 000000009f8f2001 RCX: 0000000000000000 [ 9781.391129] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 9781.400109] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [ 9781.409069] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [ 9781.418023] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [ 9781.426979] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 9781.437014] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9781.444426] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 9781.453394] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 9781.462356] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 9781.471302] Stack: [ 9781.474515] ffffffffa121d1e9 ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 [ 9781.483800] 00000000810d354f ffff8803162f6000 ffff8800b2a37468 0000000020000000 [ 9781.493102] 010000000000000c 0000000000000000 0000000000000000 ffff8800b2a36000 [ 9781.502395] Call Trace: [ 9781.506116] [] ? osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [ 9781.515479] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 9781.524648] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 9781.533518] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 9781.542188] [] ? wake_up_state+0x20/0x20 [ 9781.549399] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 9781.558747] [] kthread+0xcf/0xe0 [ 9781.565180] [] ? kthread_create_on_node+0x140/0x140 [ 9781.573460] [] ret_from_fork+0x58/0x90 [ 9781.580473] [] ? kthread_create_on_node+0x140/0x140 [ 9781.588751] Code: 00 00 41 80 a7 74 14 00 00 fe 48 8b 4d d0 65 48 33 0c 25 28 00 00 00 0f 85 b1 03 00 00 48 83 c4 68 5b 41 5c 41 5d 41 5e 41 5f 5d 66 0f 1f 84 00 00 00 00 00 4c 8d ae f0 10 00 00 44 89 4d 88 [ 9809.184192] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [ 9809.193813] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 9809.325456] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 9809.338745] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 9809.354999] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 9809.364387] RIP: 0010:[] [] osd_inode_iteration+0x269/0xcc0 [osd_ldiskfs] [ 9809.376612] RSP: 0018:ffff880703603d18 EFLAGS: 00000202 [ 9809.383556] RAX: 00000000000072ce RBX: ffff880703603d98 RCX: ffff88042bd78800 [ 9809.392539] RDX: 00000000000072ce RSI: ffff880419dc3a00 RDI: ffff88042bd79800 [ 9809.401503] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [ 9809.410460] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000001 [ 9809.419408] R13: 0000000000000004 R14: 000000009f8f2001 R15: ffffea00027e3c80 [ 9809.428358] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 9809.438382] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9809.445784] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 9809.454745] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 9809.463697] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 9809.472637] Stack: [ 9809.475840] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [ 9809.485129] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [ 9809.494428] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [ 9809.503712] Call Trace: [ 9809.507423] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 9809.516583] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 9809.525448] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 9809.534106] [] ? wake_up_state+0x20/0x20 [ 9809.541313] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 9809.550654] [] kthread+0xcf/0xe0 [ 9809.557076] [] ? kthread_create_on_node+0x140/0x140 [ 9809.565344] [] ret_from_fork+0x58/0x90 [ 9809.572353] [] ? kthread_create_on_node+0x140/0x140 [ 9809.580621] Code: 88 e8 dc 2d f5 ff 48 85 c0 48 89 45 90 0f 84 b3 08 00 00 44 8b 7d a0 48 8b 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 e8 77 5b f4 ff <42> 8d 14 38 48 8b 45 88 48 8b 80 a8 03 00 00 48 3b 50 20 0f 83 [ 9823.726699] INFO: rcu_sched self-detected stall on CPU { 24} (t=2400060 jiffies g=907112 c=907111 q=1457053) [ 9823.739027] Task dump for CPU 24: [ 9823.743757] OI_scrub R running task 0 5551 2 0x00000088 [ 9823.752674] ffff88083fde6dd0 00000000fb6db880 ffff88082da03db0 ffffffff810c47e8 [ 9823.761990] 0000000000000018 ffffffff81a1e7c0 ffff88082da03dc8 ffffffff810c8079 [ 9823.771306] 0000000000000009 ffff88082da03df8 ffffffff811378e0 ffff88082da101c0 [ 9823.780601] Call Trace: [ 9823.784282] [] sched_show_task+0xa8/0x110 [ 9823.792254] [] dump_cpu_task+0x39/0x70 [ 9823.799216] [] rcu_dump_cpu_stacks+0x90/0xd0 [ 9823.806752] [] rcu_check_callbacks+0x442/0x720 [ 9823.814439] [] ? tick_sched_handle.isra.13+0x60/0x60 [ 9823.822707] [] update_process_times+0x47/0x80 [ 9823.830272] [] tick_sched_handle.isra.13+0x25/0x60 [ 9823.838298] [] tick_sched_timer+0x41/0x70 [ 9823.845440] [] __hrtimer_run_queues+0xd2/0x260 [ 9823.853032] [] hrtimer_interrupt+0xb0/0x1e0 [ 9823.860305] [] ? call_softirq+0x1c/0x30 [ 9823.867186] [] local_apic_timer_interrupt+0x37/0x60 [ 9823.875219] [] smp_apic_timer_interrupt+0x3f/0x60 [ 9823.883040] [] apic_timer_interrupt+0x6d/0x80 [ 9823.890463] [] ? osd_inode_iteration+0x28a/0xcc0 [osd_ldiskfs] [ 9823.900255] [] ? osd_inode_iteration+0x269/0xcc0 [osd_ldiskfs] [ 9823.909350] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 9823.918249] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 9823.926866] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 9823.935278] [] ? wake_up_state+0x20/0x20 [ 9823.942238] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 9823.951329] [] kthread+0xcf/0xe0 [ 9823.957505] [] ? kthread_create_on_node+0x140/0x140 [ 9823.965520] [] ret_from_fork+0x58/0x90 [ 9823.972287] [] ? kthread_create_on_node+0x140/0x140 [ 9849.185579] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [ 9849.194976] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 9849.324562] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 9849.337728] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 9849.353804] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 9849.363106] RIP: 0010:[] [] osd_inode_iteration+0x282/0xcc0 [osd_ldiskfs] [ 9849.375249] RSP: 0018:ffff880703603d18 EFLAGS: 00000287 [ 9849.382103] RAX: ffff88042bd78800 RBX: 000000009f8f2001 RCX: ffff88042bd78800 [ 9849.391013] RDX: 00000000000072cf RSI: ffff880419dc3a00 RDI: ffff88042bd79800 [ 9849.399913] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [ 9849.408822] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [ 9849.417725] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [ 9849.426613] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 9849.436581] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9849.443928] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 9849.452838] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 9849.461747] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 9849.470649] Stack: [ 9849.473821] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [ 9849.483072] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [ 9849.492318] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [ 9849.501565] Call Trace: [ 9849.505250] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 9849.514390] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 9849.523238] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 9849.531887] [] ? wake_up_state+0x20/0x20 [ 9849.539081] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 9849.548415] [] kthread+0xcf/0xe0 [ 9849.554825] [] ? kthread_create_on_node+0x140/0x140 [ 9849.563080] [] ret_from_fork+0x58/0x90 [ 9849.570070] [] ? kthread_create_on_node+0x140/0x140 [ 9849.578316] Code: 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 e8 77 5b f4 ff 42 8d 14 38 48 8b 45 88 48 8b 80 a8 03 00 00 48 3b 50 20 0f 83 57 01 00 00 <44> 0f b6 85 67 ff ff ff 48 8d 4d a8 48 8d 55 88 48 8b b5 38 ff [ 9877.186554] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [ 9877.196155] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 9877.328006] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 9877.341327] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 9877.357624] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 9877.367039] RIP: 0010:[] [] osd_inode_iteration+0x251/0xcc0 [osd_ldiskfs] [ 9877.379281] RSP: 0018:ffff880703603d18 EFLAGS: 00000286 [ 9877.386247] RAX: ffff8800b2a37468 RBX: 000000009f8f2001 RCX: 0000000000000000 [ 9877.395239] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 9877.404223] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [ 9877.413192] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffff880703603d57 [ 9877.422151] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [ 9877.431107] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 9877.441139] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9877.448548] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 9877.457519] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 9877.466486] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 9877.475437] Stack: [ 9877.478655] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [ 9877.487951] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [ 9877.497234] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [ 9877.506545] Call Trace: [ 9877.510271] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 9877.519442] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 9877.528320] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 9877.537000] [] ? wake_up_state+0x20/0x20 [ 9877.544218] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 9877.553557] [] kthread+0xcf/0xe0 [ 9877.559991] [] ? kthread_create_on_node+0x140/0x140 [ 9877.568277] [] ret_from_fork+0x58/0x90 [ 9877.575300] [] ? kthread_create_on_node+0x140/0x140 [ 9877.583582] Code: b0 00 00 00 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 00 8b 75 98 48 8b 7d 88 e8 dc 2d f5 ff 48 85 c0 48 89 45 90 0f 84 b3 08 00 00 <44> 8b 7d a0 48 8b 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 e8 77 [ 9905.187530] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [ 9905.197161] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 9905.328966] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 9905.342276] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 9905.358580] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 9905.367987] RIP: 0010:[] [] osd_scrub_exec+0xf7/0x5b0 [osd_ldiskfs] [ 9905.379659] RSP: 0018:ffff880703603c78 EFLAGS: 00000246 [ 9905.386613] RAX: 0000000000280001 RBX: ffff880703603c37 RCX: 0000000000000000 [ 9905.395603] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 9905.404592] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [ 9905.413564] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [ 9905.422512] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [ 9905.431462] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 9905.441498] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9905.448907] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 9905.457882] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 9905.466845] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 9905.475783] Stack: [ 9905.478986] ffff880703603df0 000000009f8f2001 ffffea00027e3c80 000000009f8f2001 [ 9905.488266] ffff880703603d57 ffff880703603d57 ffff8800b2a37468 0000000000000000 [ 9905.497544] ffff880703603d78 ffff880419dc3a00 ffff88042bd79800 ffffffffffffff10 [ 9905.506850] Call Trace: [ 9905.510569] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [ 9905.519732] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 9905.528890] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 9905.537756] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 9905.546416] [] ? wake_up_state+0x20/0x20 [ 9905.553616] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 9905.562956] [] kthread+0xcf/0xe0 [ 9905.569378] [] ? kthread_create_on_node+0x140/0x140 [ 9905.577645] [] ret_from_fork+0x58/0x90 [ 9905.584662] [] ? kthread_create_on_node+0x140/0x140 [ 9905.592938] Code: 8b 4d 88 48 85 db 74 23 f6 83 18 0a 00 00 08 74 1a 4d 85 e4 74 15 41 8b 87 6c 14 00 00 41 39 84 24 0c 0a 00 00 0f 82 d9 00 00 00 <31> c0 41 f6 87 74 14 00 00 04 0f 85 6e ff ff ff 41 83 f9 02 0f [ 9933.188513] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [ 9933.198131] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 9933.329673] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 9933.342953] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 9933.359187] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 9933.368564] RIP: 0010:[] [] osd_inode_iteration+0x2ac/0xcc0 [osd_ldiskfs] [ 9933.380783] RSP: 0018:ffff880703603d18 EFLAGS: 00000202 [ 9933.387711] RAX: 0000000000000004 RBX: 0000000000000000 RCX: ffff880703603d98 [ 9933.396675] RDX: ffff880703603d78 RSI: 0000000000000000 RDI: ffff8803162f6000 [ 9933.405622] RBP: ffff880703603df0 R08: 0000000000000001 R09: 0000000000000004 [ 9933.414572] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffff880703603d57 [ 9933.423510] R13: 0000000000000004 R14: 000000009f8f2001 R15: 0000000000000018 [ 9933.432451] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 9933.442466] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9933.449857] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 9933.458810] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 9933.467766] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 9933.476695] Stack: [ 9933.479880] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [ 9933.489146] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [ 9933.498436] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [ 9933.507707] Call Trace: [ 9933.511399] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 9933.520543] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 9933.529407] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 9933.538065] [] ? wake_up_state+0x20/0x20 [ 9933.545262] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 9933.554598] [] kthread+0xcf/0xe0 [ 9933.561020] [] ? kthread_create_on_node+0x140/0x140 [ 9933.569289] [] ret_from_fork+0x58/0x90 [ 9933.576294] [] ? kthread_create_on_node+0x140/0x140 [ 9933.584559] Code: 00 44 0f b6 85 67 ff ff ff 48 8d 4d a8 48 8d 55 88 48 8b b5 38 ff ff ff 48 8b bd 48 ff ff ff 48 8b 85 30 ff ff ff ff d0 83 f8 03 <41> 89 c5 0f 84 4b 02 00 00 0f 8e 15 01 00 00 83 f8 05 66 90 0f [ 9961.189480] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [ 9961.199108] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 9961.330870] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 9961.344170] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 9961.360436] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 9961.369833] RIP: 0010:[] [] osd_scrub_next+0x149/0x3c0 [osd_ldiskfs] [ 9961.381597] RSP: 0018:ffff880703603c70 EFLAGS: 00000202 [ 9961.388550] RAX: 0000000000000004 RBX: ffff880703603c37 RCX: ffff880703603d98 [ 9961.397531] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 9961.406502] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [ 9961.415458] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [ 9961.424412] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [ 9961.433357] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 9961.443381] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9961.450789] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 9961.459752] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 9961.468712] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 9961.477660] Stack: [ 9961.480874] 000000009f8f2001 ffff880703603d08 ffff880703603d57 ffffea00027e3c80 [ 9961.490166] 000000009f8f2001 ffff880703603d57 ffff880703603d57 0000000000000000 [ 9961.499462] ffff88042bd78800 00000000000072ce ffff880419dc3a00 ffff88042bd79800 [ 9961.508757] Call Trace: [ 9961.512481] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [ 9961.521644] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 9961.530803] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 9961.539670] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 9961.548322] [] ? wake_up_state+0x20/0x20 [ 9961.555527] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 9961.564864] [] kthread+0xcf/0xe0 [ 9961.571292] [] ? kthread_create_on_node+0x140/0x140 [ 9961.579562] [] ret_from_fork+0x58/0x90 [ 9961.586572] [] ? kthread_create_on_node+0x140/0x140 [ 9961.594839] Code: 45 90 74 49 49 8d 9f 10 11 00 00 48 89 df e8 df 6a 47 e0 41 c7 87 58 10 00 00 02 00 00 00 48 89 df e8 fc 69 47 e0 b8 05 00 00 00 <48> 8b 75 d0 65 48 33 34 25 28 00 00 00 0f 85 4c 02 00 00 48 83 [ 9989.190462] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [ 9989.200087] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 9989.331717] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 9989.345006] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 9989.361257] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 9989.370630] RIP: 0010:[] [] osd_inode_iteration+0x4a6/0xcc0 [osd_ldiskfs] [ 9989.382859] RSP: 0018:ffff880703603d18 EFLAGS: 00000246 [ 9989.389797] RAX: 0000000020000000 RBX: 000000009f8f2001 RCX: 0000000000000000 [ 9989.398772] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 9989.407736] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [ 9989.416694] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffff880703603d57 [ 9989.425642] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [ 9989.434605] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 9989.444627] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9989.452024] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 9989.460989] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 9989.469943] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 9989.478885] Stack: [ 9989.482094] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [ 9989.491389] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [ 9989.500686] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [ 9989.509981] Call Trace: [ 9989.513695] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 9989.522865] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 9989.531741] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 9989.540412] [] ? wake_up_state+0x20/0x20 [ 9989.547624] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 9989.556972] [] kthread+0xcf/0xe0 [ 9989.563410] [] ? kthread_create_on_node+0x140/0x140 [ 9989.571697] [] ret_from_fork+0x58/0x90 [ 9989.578723] [] ? kthread_create_on_node+0x140/0x140 [ 9989.587003] Code: 8b 4d a8 48 8d 55 88 48 8b b5 38 ff ff ff 48 8b bd 48 ff ff ff 48 8b 85 28 ff ff ff ff d0 85 c0 41 89 c5 75 1c 8b 85 58 ff ff ff <41> 3b 04 24 72 10 48 8b 85 50 ff ff ff 44 3b 30 0f 87 95 fd ff [10003.735970] INFO: rcu_sched self-detected stall on CPU { 24} (t=2580063 jiffies g=907112 c=907111 q=1468877) [10003.748283] Task dump for CPU 24: [10003.753037] OI_scrub R running task 0 5551 2 0x00000088 [10003.761982] ffff88083fde6dd0 00000000fb6db880 ffff88082da03db0 ffffffff810c47e8 [10003.771305] 0000000000000018 ffffffff81a1e7c0 ffff88082da03dc8 ffffffff810c8079 [10003.780622] 0000000000000009 ffff88082da03df8 ffffffff811378e0 ffff88082da101c0 [10003.789913] Call Trace: [10003.793615] [] sched_show_task+0xa8/0x110 [10003.801615] [] dump_cpu_task+0x39/0x70 [10003.808591] [] rcu_dump_cpu_stacks+0x90/0xd0 [10003.816135] [] rcu_check_callbacks+0x442/0x720 [10003.823844] [] ? tick_sched_handle.isra.13+0x60/0x60 [10003.832114] [] update_process_times+0x47/0x80 [10003.839685] [] tick_sched_handle.isra.13+0x25/0x60 [10003.847727] [] tick_sched_timer+0x41/0x70 [10003.854867] [] __hrtimer_run_queues+0xd2/0x260 [10003.862469] [] hrtimer_interrupt+0xb0/0x1e0 [10003.869766] [] ? call_softirq+0x1c/0x30 [10003.876656] [] local_apic_timer_interrupt+0x37/0x60 [10003.884695] [] smp_apic_timer_interrupt+0x3f/0x60 [10003.892525] [] apic_timer_interrupt+0x6d/0x80 [10003.899952] [] ? osd_scrub_exec+0x3d/0x5b0 [osd_ldiskfs] [10003.909194] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [10003.918095] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [10003.927010] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [10003.935613] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [10003.944024] [] ? wake_up_state+0x20/0x20 [10003.950982] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [10003.960065] [] kthread+0xcf/0xe0 [10003.966222] [] ? kthread_create_on_node+0x140/0x140 [10003.974229] [] ret_from_fork+0x58/0x90 [10003.980985] [] ? kthread_create_on_node+0x140/0x140 [10029.191857] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [10029.201226] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [10029.331105] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [10029.344274] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [10029.360332] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [10029.369630] RIP: 0010:[] [] osd_inode_iteration+0x251/0xcc0 [osd_ldiskfs] [10029.381768] RSP: 0018:ffff880703603d18 EFLAGS: 00000286 [10029.388629] RAX: ffff8800b2a37468 RBX: 000000009f8f2001 RCX: 0000000000000000 [10029.397527] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [10029.406434] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [10029.415339] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [10029.424237] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [10029.433137] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [10029.443101] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10029.450443] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [10029.459340] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [10029.468256] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [10029.477160] Stack: [10029.480327] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [10029.489577] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [10029.498829] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [10029.508078] Call Trace: [10029.511767] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [10029.520894] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [10029.529737] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [10029.538390] [] ? wake_up_state+0x20/0x20 [10029.545582] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [10029.554912] [] kthread+0xcf/0xe0 [10029.561371] [] ? kthread_create_on_node+0x140/0x140 [10029.569624] [] ret_from_fork+0x58/0x90 [10029.576605] [] ? kthread_create_on_node+0x140/0x140 [10029.584845] Code: b0 00 00 00 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 00 8b 75 98 48 8b 7d 88 e8 dc 2d f5 ff 48 85 c0 48 89 45 90 0f 84 b3 08 00 00 <44> 8b 7d a0 48 8b 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 e8 77 [10057.192832] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [10057.202451] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [10057.334725] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [10057.348033] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [10057.364338] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [10057.373740] RIP: 0010:[] [] osd_inode_iteration+0x264/0xcc0 [osd_ldiskfs] [10057.386015] RSP: 0018:ffff880703603d18 EFLAGS: 00000286 [10057.392987] RAX: ffff8800b2a37468 RBX: ffff880703603d98 RCX: 0000000000000000 [10057.401984] RDX: ffff880703603d78 RSI: ffff880419dc3a00 RDI: ffff88042bd79800 [10057.410980] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [10057.419952] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000001 [10057.428908] R13: 0000000000000004 R14: 000000009f8f2001 R15: ffffea00027e3c80 [10057.437865] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [10057.447890] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10057.455297] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [10057.464262] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [10057.473232] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [10057.482217] Stack: [10057.485435] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [10057.494720] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [10057.504014] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [10057.513292] Call Trace: [10057.517016] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [10057.526184] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [10057.535059] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [10057.543751] [] ? wake_up_state+0x20/0x20 [10057.550970] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [10057.560322] [] kthread+0xcf/0xe0 [10057.566754] [] ? kthread_create_on_node+0x140/0x140 [10057.575044] [] ret_from_fork+0x58/0x90 [10057.582059] [] ? kthread_create_on_node+0x140/0x140 [10057.590339] Code: 75 98 48 8b 7d 88 e8 dc 2d f5 ff 48 85 c0 48 89 45 90 0f 84 b3 08 00 00 44 8b 7d a0 48 8b 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 77 5b f4 ff 42 8d 14 38 48 8b 45 88 48 8b 80 a8 03 00 00 48 [10085.193806] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [10085.203473] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [10085.335776] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [10085.349077] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [10085.365372] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [10085.374777] RIP: 0010:[] [] osd_scrub_next+0x2b/0x3c0 [osd_ldiskfs] [10085.386448] RSP: 0018:ffff880703603c70 EFLAGS: 00000282 [10085.393402] RAX: 00000000fb6db880 RBX: ffff880703603c37 RCX: ffff880703603d98 [10085.402390] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [10085.411376] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [10085.420352] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [10085.429304] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [10085.438254] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [10085.448280] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10085.455689] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [10085.464665] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [10085.473631] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [10085.482585] Stack: [10085.485810] 000000009f8f2001 ffff880703603d08 0000000000000001 ffff880703603df0 [10085.495116] 000000009f8f2001 ffff880703603d57 000000009f8f2001 0000000000000004 [10085.504409] ffff880703603d57 ffff88042bd78800 ffff88042bd78800 00000000000072cf [10085.513712] Call Trace: [10085.517452] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [10085.526619] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [10085.535784] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [10085.544657] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [10085.553330] [] ? wake_up_state+0x20/0x20 [10085.560537] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [10085.569900] [] kthread+0xcf/0xe0 [10085.576322] [] ? kthread_create_on_node+0x140/0x140 [10085.584604] [] ret_from_fork+0x58/0x90 [10085.591614] [] ? kthread_create_on_node+0x140/0x140 [10085.599880] Code: 66 66 66 66 90 55 48 89 e5 41 57 49 89 f7 41 56 41 55 49 89 fd 41 54 49 89 d4 53 48 89 cb 48 83 ec 70 65 48 8b 04 25 28 00 00 00 <48> 89 45 d0 31 c0 48 8b 05 f8 59 9c ff 48 85 c0 75 7b 41 f6 87 [10113.194783] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [10113.204427] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [10113.336591] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [10113.349872] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [10113.366120] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [10113.375505] RIP: 0010:[] [] osd_inode_iteration+0x264/0xcc0 [osd_ldiskfs] [10113.387743] RSP: 0018:ffff880703603d18 EFLAGS: 00000286 [10113.394686] RAX: ffff8800b2a37468 RBX: 000000009f8f2001 RCX: 0000000000000000 [10113.403667] RDX: ffff880703603d78 RSI: ffff880419dc3a00 RDI: ffff88042bd79800 [10113.412629] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [10113.421596] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [10113.430543] R13: ffff880703603d08 R14: 000000009f8f2001 R15: ffff880703603d08 [10113.439498] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [10113.449531] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10113.456941] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [10113.465912] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [10113.474877] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [10113.483824] Stack: [10113.487043] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [10113.496338] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [10113.505642] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [10113.514941] Call Trace: [10113.518666] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [10113.527840] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [10113.536723] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [10113.545401] [] ? wake_up_state+0x20/0x20 [10113.552616] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [10113.561991] [] kthread+0xcf/0xe0 [10113.568434] [] ? kthread_create_on_node+0x140/0x140 [10113.576734] [] ret_from_fork+0x58/0x90 [10113.583757] [] ? kthread_create_on_node+0x140/0x140 [10113.592048] Code: 75 98 48 8b 7d 88 e8 dc 2d f5 ff 48 85 c0 48 89 45 90 0f 84 b3 08 00 00 44 8b 7d a0 48 8b 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 77 5b f4 ff 42 8d 14 38 48 8b 45 88 48 8b 80 a8 03 00 00 48 [10141.195759] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [10141.205390] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [10141.337809] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [10141.351142] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [10141.367453] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [10141.376868] RIP: 0010:[] [] osd_scrub_next+0x167/0x3c0 [osd_ldiskfs] [10141.388651] RSP: 0018:ffff880703603d00 EFLAGS: 00000282 [10141.395630] RAX: 0000000000000004 RBX: 0000000000000001 RCX: ffff880703603d98 [10141.404626] RDX: ffff880703603d78 RSI: 0000000000000000 RDI: ffff8803162f6000 [10141.413611] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [10141.422580] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [10141.431535] R13: 0000000000000018 R14: ffff880703603c78 R15: 0000000000000286 [10141.440485] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [10141.450506] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10141.457908] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [10141.466870] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [10141.475833] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [10141.484767] Stack: [10141.487974] 0000000000000001 ffff880703603df0 ffffffffa121cff9 ffffffffa121b990 [10141.497272] ffffffffa1217a20 ffff8800b2a36000 00000000810d354f ffff8803162f6000 [10141.506555] ffff8800b2a37468 0000000020000000 010000000000000c 0000000000000000 [10141.515848] Call Trace: [10141.519581] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [10141.528742] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [10141.537899] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [10141.546764] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [10141.555433] [] ? wake_up_state+0x20/0x20 [10141.562635] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [10141.571982] [] kthread+0xcf/0xe0 [10141.578404] [] ? kthread_create_on_node+0x140/0x140 [10141.586684] [] ret_from_fork+0x58/0x90 [10141.593686] [] ? kthread_create_on_node+0x140/0x140 [10141.601953] Code: 48 89 df e8 fc 69 47 e0 b8 05 00 00 00 48 8b 75 d0 65 48 33 34 25 28 00 00 00 0f 85 4c 02 00 00 48 83 c4 70 5b 41 5c 41 5d 41 5e <41> 5f 5d c3 48 8b 05 be 58 9c ff 48 85 c0 0f 84 c2 fe ff ff 0f [10169.196727] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [10169.206353] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [10169.338361] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [10169.351646] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [10169.367892] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [10169.377273] RIP: 0010:[] [] osd_scrub_next+0x12/0x3c0 [osd_ldiskfs] [10169.388944] RSP: 0018:ffff880703603cf0 EFLAGS: 00000287 [10169.395892] RAX: ffffffffa1217a20 RBX: 000000009f8f2001 RCX: ffff880703603d98 [10169.404871] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [10169.413840] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [10169.422789] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffff880703603c78 [10169.431735] R13: 0000000000000286 R14: 0000000000000010 R15: ffffffffa121ba64 [10169.440685] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [10169.450711] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10169.458116] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [10169.467084] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [10169.476051] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [10169.484998] Stack: [10169.488213] 0000000000000000 00000000ffffffff 0000000000000001 ffff880703603df0 [10169.497503] ffffffffa121cff9 ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 [10169.506805] 00000000810d354f ffff8803162f6000 ffff8800b2a37468 0000000020000000 [10169.516107] Call Trace: [10169.519820] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [10169.528995] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [10169.538154] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [10169.547024] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [10169.555700] [] ? wake_up_state+0x20/0x20 [10169.562915] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [10169.572281] [] kthread+0xcf/0xe0 [10169.578720] [] ? kthread_create_on_node+0x140/0x140 [10169.587025] [] ret_from_fork+0x58/0x90 [10169.594042] [] ? kthread_create_on_node+0x140/0x140 [10169.602318] Code: 25 fc ff ff 48 83 c4 08 5b 41 5c 41 5d 41 5e 5d c3 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 89 e5 41 57 49 89 f7 41 56 41 55 <49> 89 fd 41 54 49 89 d4 53 48 89 cb 48 83 ec 70 65 48 8b 04 25 [10183.745240] INFO: rcu_sched self-detected stall on CPU { 24} (t=2760066 jiffies g=907112 c=907111 q=1553403) [10183.757576] Task dump for CPU 24: [10183.762313] OI_scrub R running task 0 5551 2 0x00000088 [10183.771234] ffff88083fde6dd0 00000000fb6db880 ffff88082da03db0 ffffffff810c47e8 [10183.780564] 0000000000000018 ffffffff81a1e7c0 ffff88082da03dc8 ffffffff810c8079 [10183.789870] 0000000000000009 ffff88082da03df8 ffffffff811378e0 ffff88082da101c0 [10183.799149] Call Trace: [10183.802831] [] sched_show_task+0xa8/0x110 [10183.810815] [] dump_cpu_task+0x39/0x70 [10183.817776] [] rcu_dump_cpu_stacks+0x90/0xd0 [10183.825317] [] rcu_check_callbacks+0x442/0x720 [10183.833021] [] ? tick_sched_handle.isra.13+0x60/0x60 [10183.841290] [] update_process_times+0x47/0x80 [10183.848843] [] tick_sched_handle.isra.13+0x25/0x60 [10183.856868] [] tick_sched_timer+0x41/0x70 [10183.864012] [] __hrtimer_run_queues+0xd2/0x260 [10183.871616] [] hrtimer_interrupt+0xb0/0x1e0 [10183.878918] [] ? call_softirq+0x1c/0x30 [10183.885806] [] local_apic_timer_interrupt+0x37/0x60 [10183.893840] [] smp_apic_timer_interrupt+0x3f/0x60 [10183.901660] [] apic_timer_interrupt+0x6d/0x80 [10183.909097] [] ? osd_scrub_next+0x350/0x3c0 [osd_ldiskfs] [10183.918432] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [10183.927337] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [10183.936251] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [10183.945152] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [10183.953757] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [10183.962159] [] ? wake_up_state+0x20/0x20 [10183.969109] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [10183.978206] [] kthread+0xcf/0xe0 [10183.984384] [] ? kthread_create_on_node+0x140/0x140 [10183.992401] [] ret_from_fork+0x58/0x90 [10183.999147] [] ? kthread_create_on_node+0x140/0x140 [10202.504635] Lustre: soaked-MDT0000: Client e211cd09-6b0c-e43b-059c-f657101042a0 (at 192.168.1.117@o2ib100) reconnecting [10202.517650] Lustre: soaked-MDT0000: Connection restored to e211cd09-6b0c-e43b-059c-f657101042a0 (at 192.168.1.117@o2ib100) [10209.198127] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [10209.207553] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [10209.337628] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [10209.350826] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [10209.366902] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [10209.376200] RIP: 0010:[] [] osd_inode_iteration+0x4a6/0xcc0 [osd_ldiskfs] [10209.388366] RSP: 0018:ffff880703603d18 EFLAGS: 00000246 [10209.395260] RAX: 0000000020000000 RBX: 0000000000000001 RCX: 0000000000000000 [10209.404200] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [10209.413133] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [10209.422063] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [10209.430997] R13: ffffea00027e3c80 R14: 000000009f8f2001 R15: 0000000000000018 [10209.439922] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [10209.449927] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10209.457304] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [10209.466250] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [10209.475206] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [10209.484146] Stack: [10209.487363] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [10209.496644] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [10209.505931] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [10209.515222] Call Trace: [10209.518942] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [10209.528106] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [10209.536975] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [10209.545639] [] ? wake_up_state+0x20/0x20 [10209.552855] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [10209.562223] [] kthread+0xcf/0xe0 [10209.568664] [] ? kthread_create_on_node+0x140/0x140 [10209.576950] [] ret_from_fork+0x58/0x90 [10209.583979] [] ? kthread_create_on_node+0x140/0x140 [10209.592271] Code: 8b 4d a8 48 8d 55 88 48 8b b5 38 ff ff ff 48 8b bd 48 ff ff ff 48 8b 85 28 ff ff ff ff d0 85 c0 41 89 c5 75 1c 8b 85 58 ff ff ff <41> 3b 04 24 72 10 48 8b 85 50 ff ff ff 44 3b 30 0f 87 95 fd ff [10237.199102] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [10237.208785] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [10237.340666] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [10237.353970] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [10237.370258] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [10237.379649] RIP: 0010:[] [] osd_scrub_exec+0xf7/0x5b0 [osd_ldiskfs] [10237.391333] RSP: 0018:ffff880703603c78 EFLAGS: 00000246 [10237.398293] RAX: 0000000000280001 RBX: ffff880703603c37 RCX: 0000000000000000 [10237.407274] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [10237.416256] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [10237.425215] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [10237.434169] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [10237.443126] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [10237.453155] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10237.460567] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [10237.469532] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [10237.478489] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [10237.487434] Stack: [10237.490651] ffff880703603d08 ffff880703603d57 ffff880703603df0 000000009f8f2001 [10237.499944] ffff880703603d57 000000009f8f2001 0000000000000004 ffff880703603d57 [10237.509234] ffff8800b2a37468 0000000000000000 ffff880703603d78 ffff8800b2a36000 [10237.518525] Call Trace: [10237.522253] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [10237.531424] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [10237.540584] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [10237.549459] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [10237.558115] [] ? wake_up_state+0x20/0x20 [10237.565318] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [10237.574662] [] kthread+0xcf/0xe0 [10237.581086] [] ? kthread_create_on_node+0x140/0x140 [10237.589365] [] ret_from_fork+0x58/0x90 [10237.596378] [] ? kthread_create_on_node+0x140/0x140 [10237.604657] Code: 8b 4d 88 48 85 db 74 23 f6 83 18 0a 00 00 08 74 1a 4d 85 e4 74 15 41 8b 87 6c 14 00 00 41 39 84 24 0c 0a 00 00 0f 82 d9 00 00 00 <31> c0 41 f6 87 74 14 00 00 04 0f 85 6e ff ff ff 41 83 f9 02 0f [10265.200078] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [10265.209776] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [10265.341785] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [10265.355076] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [10265.371313] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [10265.380696] RIP: 0010:[] [] osd_scrub_exec+0x12/0x5b0 [osd_ldiskfs] [10265.392344] RSP: 0018:ffff880703603cf0 EFLAGS: 00000293 [10265.399282] RAX: ffffffffa121b990 RBX: 000000009f8f2001 RCX: 0000000000000000 [10265.408258] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [10265.417222] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [10265.426171] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffff880703603c78 [10265.435120] R13: ffff880703603d08 R14: 0000000000000010 R15: ffffffffa121ba79 [10265.444067] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [10265.454095] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10265.461501] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [10265.470467] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [10265.479429] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [10265.488374] Stack: [10265.491595] 0000000000000004 00000000ffffffff 0000000000000001 ffff880703603df0 [10265.500887] ffffffffa121d1e9 ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 [10265.510192] 00000000810d354f ffff8803162f6000 ffff8800b2a37468 0000000020000000 [10265.519482] Call Trace: [10265.523210] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [10265.532378] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [10265.541541] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [10265.550415] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [10265.559093] [] ? wake_up_state+0x20/0x20 [10265.566308] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [10265.575660] [] kthread+0xcf/0xe0 [10265.582093] [] ? kthread_create_on_node+0x140/0x140 [10265.590375] [] ret_from_fork+0x58/0x90 [10265.597389] [] ? kthread_create_on_node+0x140/0x140 [10265.605661] Code: 48 8b 40 40 ba 07 00 00 00 48 89 04 24 31 c0 e8 a5 03 98 ff e9 c7 fd ff ff 66 66 66 66 90 55 48 89 e5 41 57 49 89 f7 41 56 41 55 <41> 54 45 31 e4 53 48 83 ec 68 48 8b 9e 08 10 00 00 4c 89 45 90 [10293.201053] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [10293.210698] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [10293.342808] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [10293.356110] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [10293.372398] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [10293.381793] RIP: 0010:[] [] osd_scrub_next+0x9/0x3c0 [osd_ldiskfs] [10293.393366] RSP: 0018:ffff880703603d08 EFLAGS: 00000287 [10293.400315] RAX: ffffffffa1217a20 RBX: 000000009f8f2001 RCX: ffff880703603d98 [10293.409296] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [10293.418258] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [10293.427229] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [10293.436192] R13: ffff880703603d08 R14: ffff880703603c78 R15: 0000000000000246 [10293.445152] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [10293.455196] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10293.462610] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [10293.471583] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [10293.480553] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [10293.489502] Stack: [10293.492725] ffff880703603df0 ffffffffa121cff9 ffffffffa121b990 ffffffffa1217a20 [10293.502029] ffff8800b2a36000 00000000810d354f ffff8803162f6000 ffff8800b2a37468 [10293.511333] 0000000020000000 010000000000000c 0000000000000000 0000000000000000 [10293.520633] Call Trace: [10293.524368] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [10293.533544] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [10293.542698] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [10293.551557] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [10293.560236] [] ? wake_up_state+0x20/0x20 [10293.567441] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [10293.576792] [] kthread+0xcf/0xe0 [10293.583226] [] ? kthread_create_on_node+0x140/0x140 [10293.591500] [] ret_from_fork+0x58/0x90 [10293.598514] [] ? kthread_create_on_node+0x140/0x140 [10293.606793] Code: 48 8d 4c 02 10 48 01 c2 e8 25 fc ff ff 48 83 c4 08 5b 41 5c 41 5d 41 5e 5d c3 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 89 e5 <41> 57 49 89 f7 41 56 41 55 49 89 fd 41 54 49 89 d4 53 48 89 cb [10316.493459] Lustre: soaked-MDT0000: Client e211cd09-6b0c-e43b-059c-f657101042a0 (at 192.168.1.117@o2ib100) reconnecting [10316.508215] Lustre: soaked-MDT0000: Connection restored to e211cd09-6b0c-e43b-059c-f657101042a0 (at 192.168.1.117@o2ib100) [10321.202024] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [10321.211646] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [10321.343645] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [10321.356930] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [10321.373175] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [10321.382558] RIP: 0010:[] [] ldiskfs_itable_unused_count+0x16/0x30 [ldiskfs] [10321.395018] RSP: 0018:ffff880703603d08 EFLAGS: 00000246 [10321.401980] RAX: 0000000000000000 RBX: 0000000000000001 RCX: ffff88042bd78800 [10321.410973] RDX: 00000000000072ce RSI: ffff880419dc3a00 RDI: ffff88042bd79800 [10321.419957] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [10321.428944] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [10321.437936] R13: 0000000000000018 R14: ffff880703603c78 R15: 0000000000000296 [10321.446927] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [10321.456991] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10321.464427] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [10321.473412] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [10321.482389] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [10321.491362] Stack: [10321.494620] ffff880703603df0 ffffffffa121cfb9 ffffffffa121b990 ffffffffa1217a20 [10321.503939] ffff8800b2a36000 00000000810d354f ffff8803162f6000 ffff8800b2a37468 [10321.513260] 0000000020000000 010000000000000c 0000000000000000 0000000000000000 [10321.522581] Call Trace: [10321.526355] [] osd_inode_iteration+0x269/0xcc0 [osd_ldiskfs] [10321.535556] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [10321.544725] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [10321.553628] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [10321.562332] [] ? wake_up_state+0x20/0x20 [10321.569578] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [10321.578955] [] kthread+0xcf/0xe0 [10321.585469] [] ? kthread_create_on_node+0x140/0x140 [10321.593789] [] ret_from_fork+0x58/0x90 [10321.600835] [] ? kthread_create_on_node+0x140/0x140 [10321.609143] Code: 07 0f b7 46 30 c1 e0 10 09 d0 5d c3 66 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 8b 8f a8 03 00 00 31 c0 0f b7 56 1c 48 89 e5 <48> 83 39 3f 76 07 0f b7 46 32 c1 e0 10 09 d0 5d c3 66 0f 1f 84 [10349.202997] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [10349.212708] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [10349.344462] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [10349.357748] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [10349.373986] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [10349.383366] RIP: 0010:[] [] osd_scrub_exec+0xe9/0x5b0 [osd_ldiskfs] [10349.395016] RSP: 0018:ffff880703603c78 EFLAGS: 00000282 [10349.401962] RAX: 0000000000280001 RBX: ffff880703603c37 RCX: 0000000000000000 [10349.410935] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [10349.419890] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [10349.428839] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [10349.437787] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [10349.446739] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [10349.456768] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10349.464175] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [10349.473159] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [10349.482122] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [10349.491079] Stack: [10349.494296] 0000000000000004 ffff880703603d57 ffff880703603df0 0000000000000000 [10349.503599] ffff880703603d57 000000009f8f2001 0000000000000004 ffff880703603d57 [10349.512896] ffff8800b2a37468 0000000000000000 ffff880703603d78 ffff880419dc3a00 [10349.522200] Call Trace: [10349.525921] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [10349.535109] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [10349.544281] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [10349.553152] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [10349.561838] [] ? wake_up_state+0x20/0x20 [10349.569054] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [10349.578416] [] kthread+0xcf/0xe0 [10349.584854] [] ? kthread_create_on_node+0x140/0x140 [10349.593155] [] ret_from_fork+0x58/0x90 [10349.600166] [] ? kthread_create_on_node+0x140/0x140 [10349.608443] Code: 88 11 00 00 01 4c 89 ef e8 f5 9f e9 df 44 8b 4d 88 48 85 db 74 23 f6 83 18 0a 00 00 08 74 1a 4d 85 e4 74 15 41 8b 87 6c 14 00 00 <41> 39 84 24 0c 0a 00 00 0f 82 d9 00 00 00 31 c0 41 f6 87 74 14 [10363.754505] INFO: rcu_sched self-detected stall on CPU { 24} (t=2940069 jiffies g=907112 c=907111 q=2019436) [10363.766914] Task dump for CPU 24: [10363.771645] OI_scrub R running task 0 5551 2 0x00000088 [10363.780554] ffff88083fde6dd0 00000000fb6db880 ffff88082da03db0 ffffffff810c47e8 [10363.789878] 0000000000000018 ffffffff81a1e7c0 ffff88082da03dc8 ffffffff810c8079 [10363.799189] 0000000000000009 ffff88082da03df8 ffffffff811378e0 ffff88082da101c0 [10363.808479] Call Trace: [10363.812161] [] sched_show_task+0xa8/0x110 [10363.820135] [] dump_cpu_task+0x39/0x70 [10363.827101] [] rcu_dump_cpu_stacks+0x90/0xd0 [10363.834630] [] rcu_check_callbacks+0x442/0x720 [10363.842328] [] ? tick_sched_handle.isra.13+0x60/0x60 [10363.850582] [] update_process_times+0x47/0x80 [10363.858130] [] tick_sched_handle.isra.13+0x25/0x60 [10363.866145] [] tick_sched_timer+0x41/0x70 [10363.873255] [] __hrtimer_run_queues+0xd2/0x260 [10363.880840] [] hrtimer_interrupt+0xb0/0x1e0 [10363.888114] [] ? call_softirq+0x1c/0x30 [10363.894987] [] local_apic_timer_interrupt+0x37/0x60 [10363.903021] [] smp_apic_timer_interrupt+0x3f/0x60 [10363.910845] [] apic_timer_interrupt+0x6d/0x80 [10363.918272] [] ? osd_inode_iteration+0x473/0xcc0 [osd_ldiskfs] [10363.928104] [] ? osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [10363.937207] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [10363.946116] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [10363.954738] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [10363.963148] [] ? wake_up_state+0x20/0x20 [10363.970098] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [10363.979191] [] kthread+0xcf/0xe0 [10363.985362] [] ? kthread_create_on_node+0x140/0x140 [10363.993385] [] ret_from_fork+0x58/0x90 [10364.000141] [] ? kthread_create_on_node+0x140/0x140 [10389.204398] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [10389.213823] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [10389.344008] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [10389.357209] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [10389.373294] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [10389.382606] RIP: 0010:[] [] osd_scrub_exec+0xf9/0x5b0 [osd_ldiskfs] [10389.394168] RSP: 0018:ffff880703603c78 EFLAGS: 00000246 [10389.401034] RAX: 0000000000000000 RBX: ffff880703603c37 RCX: 0000000000000000 [10389.409943] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [10389.418849] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [10389.427752] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [10389.436662] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [10389.445565] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [10389.455541] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10389.462893] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [10389.471798] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [10389.480710] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [10389.489616] Stack: [10389.492799] ffff880703603d08 ffff880703603d57 ffff880703603df0 000000009f8f2001 [10389.502049] ffff880703603d57 000000009f8f2001 0000000000000004 ffff880703603d57 [10389.511320] 0000000000000000 0000000000000000 ffff880703603d78 ffff8800b2a36000 [10389.520587] Call Trace: [10389.524291] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [10389.533436] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [10389.542575] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [10389.551421] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [10389.560079] [] ? wake_up_state+0x20/0x20 [10389.567271] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [10389.576596] [] kthread+0xcf/0xe0 [10389.582996] [] ? kthread_create_on_node+0x140/0x140 [10389.591258] [] ret_from_fork+0x58/0x90 [10389.598242] [] ? kthread_create_on_node+0x140/0x140 [10389.606490] Code: 88 48 85 db 74 23 f6 83 18 0a 00 00 08 74 1a 4d 85 e4 74 15 41 8b 87 6c 14 00 00 41 39 84 24 0c 0a 00 00 0f 82 d9 00 00 00 31 c0 <41> f6 87 74 14 00 00 04 0f 85 6e ff ff ff 41 83 f9 02 0f 84 64 [10417.205373] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [10417.215003] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [10417.347255] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [10417.360550] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [10417.376843] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [10417.386237] RIP: 0010:[] [] osd_scrub_exec+0xe9/0x5b0 [osd_ldiskfs] [10417.397910] RSP: 0018:ffff880703603c78 EFLAGS: 00000282 [10417.404869] RAX: 0000000000280001 RBX: ffff880703603c37 RCX: 0000000000000000 [10417.413853] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [10417.422831] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [10417.431804] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [10417.440761] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [10417.449719] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [10417.459754] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10417.467168] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [10417.476146] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [10417.485108] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [10417.494063] Stack: [10417.497276] ffff880703603d08 000000009f8f2001 ffff880703603df0 000000009f8f2001 [10417.506550] ffff880703603d57 000000009f8f2001 0000000000000004 ffff880703603d57 [10417.515837] 0000000000000000 0000000000000000 ffff880703603d78 ffff8800b2a36000 [10417.525133] Call Trace: [10417.528862] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [10417.538032] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [10417.547187] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [10417.556051] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [10417.564707] [] ? wake_up_state+0x20/0x20 [10417.571918] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [10417.581290] [] kthread+0xcf/0xe0 [10417.587722] [] ? kthread_create_on_node+0x140/0x140 [10417.596007] [] ret_from_fork+0x58/0x90 [10417.603018] [] ? kthread_create_on_node+0x140/0x140 [10417.611294] Code: 88 11 00 00 01 4c 89 ef e8 f5 9f e9 df 44 8b 4d 88 48 85 db 74 23 f6 83 18 0a 00 00 08 74 1a 4d 85 e4 74 15 41 8b 87 6c 14 00 00 <41> 39 84 24 0c 0a 00 00 0f 82 d9 00 00 00 31 c0 41 f6 87 74 14 [10445.206349] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [10445.216014] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [10445.348133] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [10445.361426] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [10445.377682] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [10445.387069] RIP: 0010:[] [] osd_inode_iteration+0x264/0xcc0 [osd_ldiskfs] [10445.399318] RSP: 0018:ffff880703603d18 EFLAGS: 00000286 [10445.406263] RAX: ffff8800b2a37468 RBX: 000000009f8f2001 RCX: 0000000000000000 [10445.415246] RDX: ffff880703603d78 RSI: ffff880419dc3a00 RDI: ffff88042bd79800 [10445.424217] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [10445.433178] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [10445.442133] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [10445.451087] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [10445.461121] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10445.468532] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [10445.477508] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [10445.486467] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [10445.495415] Stack: [10445.498642] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [10445.507945] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [10445.517231] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [10445.526515] Call Trace: [10445.530244] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [10445.539418] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [10445.548299] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [10445.556974] [] ? wake_up_state+0x20/0x20 [10445.564208] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [10445.573579] [] kthread+0xcf/0xe0 [10445.580018] [] ? kthread_create_on_node+0x140/0x140 [10445.588318] [] ret_from_fork+0x58/0x90 [10445.595339] [] ? kthread_create_on_node+0x140/0x140 [10445.603624] Code: 75 98 48 8b 7d 88 e8 dc 2d f5 ff 48 85 c0 48 89 45 90 0f 84 b3 08 00 00 44 8b 7d a0 48 8b 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 77 5b f4 ff 42 8d 14 38 48 8b 45 88 48 8b 80 a8 03 00 00 48 [10473.207317] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [10473.216972] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [10473.349154] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [10473.362468] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [10473.378772] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [10473.388185] RIP: 0010:[] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [10473.400446] RSP: 0018:ffff880703603d18 EFLAGS: 00000292 [10473.407417] RAX: 0000000000000000 RBX: 000000009f8f2001 RCX: 0000000000000000 [10473.416401] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [10473.425387] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [10473.434355] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffff880703603d57 [10473.443312] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [10473.452270] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [10473.462308] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10473.469720] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [10473.478691] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [10473.487667] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [10473.496622] Stack: [10473.499853] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [10473.509154] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [10473.518451] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [10473.527749] Call Trace: [10473.531479] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [10473.540649] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [10473.549533] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [10473.558216] [] ? wake_up_state+0x20/0x20 [10473.565434] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [10473.574789] [] kthread+0xcf/0xe0 [10473.581223] [] ? kthread_create_on_node+0x140/0x140 [10473.589510] [] ret_from_fork+0x58/0x90 [10473.596526] [] ? kthread_create_on_node+0x140/0x140 [10473.604793] Code: 00 00 45 89 e9 4c 8d 85 67 ff ff ff 48 8b 4d a8 48 8d 55 88 48 8b b5 38 ff ff ff 48 8b bd 48 ff ff ff 48 8b 85 28 ff ff ff ff d0 <85> c0 41 89 c5 75 1c 8b 85 58 ff ff ff 41 3b 04 24 72 10 48 8b [10501.208292] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [10501.217946] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [10501.350183] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [10501.363509] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [10501.379846] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [10501.389266] RIP: 0010:[] [] osd_inode_iteration+0x282/0xcc0 [osd_ldiskfs] [10501.401519] RSP: 0018:ffff880703603d18 EFLAGS: 00000287 [10501.408487] RAX: ffff88042bd78800 RBX: 000000009f8f2001 RCX: ffff88042bd78800 [10501.417480] RDX: 00000000000072cf RSI: ffff880419dc3a00 RDI: ffff88042bd79800 [10501.426466] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [10501.435453] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000001 [10501.444419] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [10501.453372] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [10501.463412] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10501.470827] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [10501.479812] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [10501.488781] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [10501.497730] Stack: [10501.500953] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [10501.510232] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [10501.519537] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [10501.528836] Call Trace: [10501.532567] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [10501.541743] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [10501.550616] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [10501.559308] [] ? wake_up_state+0x20/0x20 [10501.566520] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [10501.575875] [] kthread+0xcf/0xe0 [10501.582303] [] ? kthread_create_on_node+0x140/0x140 [10501.590584] [] ret_from_fork+0x58/0x90 [10501.597601] [] ? kthread_create_on_node+0x140/0x140 [10501.605879] Code: 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 e8 77 5b f4 ff 42 8d 14 38 48 8b 45 88 48 8b 80 a8 03 00 00 48 3b 50 20 0f 83 57 01 00 00 <44> 0f b6 85 67 ff ff ff 48 8d 4d a8 48 8d 55 88 48 8b b5 38 ff [10529.209275] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [10529.218933] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [10529.351295] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [10529.364613] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [10529.380931] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [10529.390341] RIP: 0010:[] [] osd_scrub_exec+0xf7/0x5b0 [osd_ldiskfs] [10529.402024] RSP: 0018:ffff880703603c78 EFLAGS: 00000246 [10529.408991] RAX: 0000000000280001 RBX: ffff880703603c37 RCX: 0000000000000000 [10529.417992] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [10529.426979] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [10529.435950] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [10529.444909] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [10529.453871] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [10529.463905] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10529.471318] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [10529.480292] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [10529.489263] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [10529.498232] Stack: [10529.501476] 0000000000000004 0000000000000001 ffff880703603df0 ffff880703603d98 [10529.510775] ffff880703603d57 000000009f8f2001 0000000000000004 0000000000000001 [10529.520068] 0000000000000004 ffff880703603d98 ffff880703603d78 0000000000000000 [10529.529373] Call Trace: [10529.533130] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [10529.542301] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [10529.551473] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [10529.560350] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [10529.569031] [] ? wake_up_state+0x20/0x20 [10529.576246] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [10529.585622] [] kthread+0xcf/0xe0 [10529.592050] [] ? kthread_create_on_node+0x140/0x140 [10529.600349] [] ret_from_fork+0x58/0x90 [10529.607374] [] ? kthread_create_on_node+0x140/0x140 [10529.615657] Code: 8b 4d 88 48 85 db 74 23 f6 83 18 0a 00 00 08 74 1a 4d 85 e4 74 15 41 8b 87 6c 14 00 00 41 39 84 24 0c 0a 00 00 0f 82 d9 00 00 00 <31> c0 41 f6 87 74 14 00 00 04 0f 85 6e ff ff ff 41 83 f9 02 0f [10543.763775] INFO: rcu_sched self-detected stall on CPU { 24} (t=3120072 jiffies g=907112 c=907111 q=2916916) [10543.776187] Task dump for CPU 24: [10543.780929] OI_scrub R running task 0 5551 2 0x00000088 [10543.789862] ffff88083fde6dd0 00000000fb6db880 ffff88082da03db0 ffffffff810c47e8 [10543.799254] 0000000000000018 ffffffff81a1e7c0 ffff88082da03dc8 ffffffff810c8079 [10543.808560] 0000000000000009 ffff88082da03df8 ffffffff811378e0 ffff88082da101c0 [10543.817835] Call Trace: [10543.821516] [] sched_show_task+0xa8/0x110 [10543.829490] [] dump_cpu_task+0x39/0x70 [10543.836455] [] rcu_dump_cpu_stacks+0x90/0xd0 [10543.843986] [] rcu_check_callbacks+0x442/0x720 [10543.851675] [] ? tick_sched_handle.isra.13+0x60/0x60 [10543.859926] [] update_process_times+0x47/0x80 [10543.867474] [] tick_sched_handle.isra.13+0x25/0x60 [10543.875496] [] tick_sched_timer+0x41/0x70 [10543.882624] [] __hrtimer_run_queues+0xd2/0x260 [10543.890215] [] hrtimer_interrupt+0xb0/0x1e0 [10543.897494] [] ? call_softirq+0x1c/0x30 [10543.904379] [] local_apic_timer_interrupt+0x37/0x60 [10543.912405] [] smp_apic_timer_interrupt+0x3f/0x60 [10543.920235] [] apic_timer_interrupt+0x6d/0x80 [10543.927658] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [10543.937285] [] ? osd_scrub_exec+0x14/0x5b0 [osd_ldiskfs] [10543.945803] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [10543.954709] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [10543.963621] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [10543.972227] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [10543.980641] [] ? wake_up_state+0x20/0x20 [10543.987588] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [10543.996678] [] kthread+0xcf/0xe0 [10544.002856] [] ? kthread_create_on_node+0x140/0x140 [10544.010873] [] ret_from_fork+0x58/0x90 [10544.017619] [] ? kthread_create_on_node+0x140/0x140 [10569.210661] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [10569.220064] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [10569.350004] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [10569.363194] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [10569.379239] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [10569.388536] RIP: 0010:[] [] osd_scrub_exec+0x5/0x5b0 [osd_ldiskfs] [10569.399998] RSP: 0018:ffff880703603d10 EFLAGS: 00000293 [10569.406866] RAX: ffffffffa121b990 RBX: 000000009f8f2001 RCX: 0000000000000000 [10569.415762] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [10569.424657] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [10569.433552] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000001 [10569.442440] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [10569.451331] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [10569.461294] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10569.468638] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [10569.477556] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [10569.486461] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [10569.495366] Stack: [10569.498542] ffffffffa121d1e9 ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 [10569.507795] 00000000810d354f ffff8803162f6000 ffff8800b2a37468 0000000020000000 [10569.517048] 010000000000000c 0000000000000000 0000000000000000 ffff8800b2a36000 [10569.526293] Call Trace: [10569.529991] [] ? osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [10569.539325] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [10569.548465] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [10569.557314] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [10569.565967] [] ? wake_up_state+0x20/0x20 [10569.573148] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [10569.582473] [] kthread+0xcf/0xe0 [10569.588880] [] ? kthread_create_on_node+0x140/0x140 [10569.597130] [] ret_from_fork+0x58/0x90 [10569.604115] [] ? kthread_create_on_node+0x140/0x140 [10569.612383] Code: 05 85 a5 02 00 00 00 02 00 89 54 24 08 48 8b 40 40 ba 07 00 00 00 48 89 04 24 31 c0 e8 a5 03 98 ff e9 c7 fd ff ff 66 66 66 66 90 <55> 48 89 e5 41 57 49 89 f7 41 56 41 55 41 54 45 31 e4 53 48 83 [10597.211642] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [10597.221250] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [10597.353468] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [10597.366768] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [10597.383023] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [10597.392411] RIP: 0010:[] [] osd_scrub_exec+0x96/0x5b0 [osd_ldiskfs] [10597.404060] RSP: 0018:ffff880703603d10 EFLAGS: 00000292 [10597.411007] RAX: 0000000000000000 RBX: ffff880703603d57 RCX: 0000000000000000 [10597.419993] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [10597.428968] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [10597.437936] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [10597.446892] R13: ffffea00027e3c80 R14: 000000009f8f2001 R15: ffff880703603d08 [10597.455850] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [10597.465885] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10597.473301] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [10597.482292] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [10597.491259] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [10597.500220] Stack: [10597.503441] ffffffffa121d1e9 ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 [10597.512747] 00000000810d354f ffff8803162f6000 ffff8800b2a37468 0000000020000000 [10597.522051] 010000000000000c 0000000000000000 0000000000000000 ffff8800b2a36000 [10597.531358] Call Trace: [10597.535106] [] ? osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [10597.544472] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [10597.553648] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [10597.562522] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [10597.571207] [] ? wake_up_state+0x20/0x20 [10597.578426] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [10597.587794] [] kthread+0xcf/0xe0 [10597.594242] [] ? kthread_create_on_node+0x140/0x140 [10597.602554] [] ret_from_fork+0x58/0x90 [10597.609575] [] ? kthread_create_on_node+0x140/0x140 [10597.617857] Code: 00 00 41 80 a7 74 14 00 00 fe 48 8b 4d d0 65 48 33 0c 25 28 00 00 00 0f 85 b1 03 00 00 48 83 c4 68 5b 41 5c 41 5d 41 5e 41 5f 5d 66 0f 1f 84 00 00 00 00 00 4c 8d ae f0 10 00 00 44 89 4d 88 [10625.212618] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [10625.222284] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [10625.354758] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [10625.368065] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [10625.384337] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [10625.393743] RIP: 0010:[] [] osd_scrub_exec+0x23/0x5b0 [osd_ldiskfs] [10625.405403] RSP: 0018:ffff880703603c78 EFLAGS: 00000296 [10625.412355] RAX: ffffffffa121b990 RBX: ffff880703603c37 RCX: 0000000000000000 [10625.421344] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [10625.430321] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [10625.439285] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [10625.448237] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [10625.457198] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [10625.467238] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10625.474659] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [10625.483641] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [10625.492613] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [10625.501568] Stack: [10625.504780] ffff880703603d08 0000000000000001 ffffea00027e3c80 000000009f8f2001 [10625.514068] ffff880703603d57 ffff880703603d57 0000000000000000 0000000000000000 [10625.523373] ffff880703603d78 ffff8800b2a36000 ffff8803162f6000 ffffffffffffff10 [10625.532666] Call Trace: [10625.536400] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [10625.545564] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [10625.554722] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [10625.563590] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [10625.572258] [] ? wake_up_state+0x20/0x20 [10625.579468] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [10625.588830] [] kthread+0xcf/0xe0 [10625.595262] [] ? kthread_create_on_node+0x140/0x140 [10625.603547] [] ret_from_fork+0x58/0x90 [10625.610550] [] ? kthread_create_on_node+0x140/0x140 [10625.618827] Code: 03 98 ff e9 c7 fd ff ff 66 66 66 66 90 55 48 89 e5 41 57 49 89 f7 41 56 41 55 41 54 45 31 e4 53 48 83 ec 68 48 8b 9e 08 10 00 00 <4c> 89 45 90 65 48 8b 04 25 28 00 00 00 48 89 45 d0 31 c0 48 8d [10653.213594] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [10653.223244] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [10653.355075] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [10653.368364] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [10653.384630] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [10653.394008] RIP: 0010:[] [] osd_inode_iteration+0x490/0xcc0 [osd_ldiskfs] [10653.406239] RSP: 0018:ffff880703603d18 EFLAGS: 00000293 [10653.413185] RAX: 0000000000000004 RBX: 000000009f8f2001 RCX: 0000000000000000 [10653.422159] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [10653.431131] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [10653.440082] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [10653.449028] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [10653.457968] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [10653.467999] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10653.475410] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [10653.484380] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [10653.493338] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [10653.502289] Stack: [10653.505500] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [10653.514795] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [10653.524072] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [10653.533369] Call Trace: [10653.537080] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [10653.546244] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [10653.555114] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [10653.563780] [] ? wake_up_state+0x20/0x20 [10653.570986] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [10653.580341] [] kthread+0xcf/0xe0 [10653.586778] [] ? kthread_create_on_node+0x140/0x140 [10653.595070] [] ret_from_fork+0x58/0x90 [10653.602088] [] ? kthread_create_on_node+0x140/0x140 [10653.610374] Code: 0f fc ff ff 0f 1f 80 00 00 00 00 45 89 e9 4c 8d 85 67 ff ff ff 48 8b 4d a8 48 8d 55 88 48 8b b5 38 ff ff ff 48 8b bd 48 ff ff ff <48> 8b 85 28 ff ff ff ff d0 85 c0 41 89 c5 75 1c 8b 85 58 ff ff [10681.214565] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [10681.224245] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [10681.356243] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [10681.369552] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [10681.385849] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [10681.395254] RIP: 0010:[] [] osd_scrub_exec+0x1c/0x5b0 [osd_ldiskfs] [10681.406922] RSP: 0018:ffff880703603c78 EFLAGS: 00000296 [10681.413882] RAX: ffffffffa121b990 RBX: ffff880703603c37 RCX: 0000000000000000 [10681.422882] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [10681.431867] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [10681.440835] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [10681.449801] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [10681.458756] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [10681.468792] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10681.476203] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [10681.485182] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [10681.494150] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [10681.503108] Stack: [10681.506321] 0000000000000004 0000000000000001 ffff880703603df0 ffff880703603d98 [10681.515603] ffff880703603d57 000000009f8f2001 0000000000000004 ffff880703603d57 [10681.524890] ffff8800b2a37468 0000000000000000 ffff880703603d78 ffff8800b2a36000 [10681.534184] Call Trace: [10681.537921] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [10681.547087] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [10681.556242] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [10681.565115] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [10681.573795] [] ? wake_up_state+0x20/0x20 [10681.581003] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [10681.590354] [] kthread+0xcf/0xe0 [10681.596783] [] ? kthread_create_on_node+0x140/0x140 [10681.605084] [] ret_from_fork+0x58/0x90 [10681.612096] [] ? kthread_create_on_node+0x140/0x140 [10681.620361] Code: 89 04 24 31 c0 e8 a5 03 98 ff e9 c7 fd ff ff 66 66 66 66 90 55 48 89 e5 41 57 49 89 f7 41 56 41 55 41 54 45 31 e4 53 48 83 ec 68 <48> 8b 9e 08 10 00 00 4c 89 45 90 65 48 8b 04 25 28 00 00 00 48 [10709.215546] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [10709.225199] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [10709.357261] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [10709.370544] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [10709.386795] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [10709.396167] RIP: 0010:[] [] osd_scrub_exec+0xf7/0x5b0 [osd_ldiskfs] [10709.407812] RSP: 0018:ffff880703603c78 EFLAGS: 00000246 [10709.414747] RAX: 0000000000280001 RBX: ffff880703603c37 RCX: 0000000000000000 [10709.423727] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [10709.432695] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [10709.441654] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [10709.450613] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [10709.459577] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [10709.469616] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10709.477030] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [10709.486001] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [10709.494953] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [10709.503894] Stack: [10709.507098] 0000000000000004 0000000000000001 ffff880703603df0 ffff880703603d98 [10709.516388] ffff880703603d57 000000009f8f2001 0000000000000004 0000000000000001 [10709.525695] 0000000000000004 ffff880703603d98 ffff880703603d78 0000000000000000 [10709.534987] Call Trace: [10709.538705] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [10709.547878] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [10709.557040] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [10709.565911] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [10709.574597] [] ? wake_up_state+0x20/0x20 [10709.581815] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [10709.591169] [] kthread+0xcf/0xe0 [10709.597601] [] ? kthread_create_on_node+0x140/0x140 [10709.605882] [] ret_from_fork+0x58/0x90 [10709.612893] [] ? kthread_create_on_node+0x140/0x140 [10709.621171] Code: 8b 4d 88 48 85 db 74 23 f6 83 18 0a 00 00 08 74 1a 4d 85 e4 74 15 41 8b 87 6c 14 00 00 41 39 84 24 0c 0a 00 00 0f 82 d9 00 00 00 <31> c0 41 f6 87 74 14 00 00 04 0f 85 6e ff ff ff 41 83 f9 02 0f [10723.773051] INFO: rcu_sched self-detected stall on CPU { 24} (t=3300075 jiffies g=907112 c=907111 q=3476409) [10723.785411] Task dump for CPU 24: [10723.790137] OI_scrub R running task 0 5551 2 0x00000088 [10723.799055] ffff88083fde6dd0 00000000fb6db880 ffff88082da03db0 ffffffff810c47e8 [10723.808390] 0000000000000018 ffffffff81a1e7c0 ffff88082da03dc8 ffffffff810c8079 [10723.817691] 0000000000000009 ffff88082da03df8 ffffffff811378e0 ffff88082da101c0 [10723.826979] Call Trace: [10723.830668] [] sched_show_task+0xa8/0x110 [10723.838663] [] dump_cpu_task+0x39/0x70 [10723.845632] [] rcu_dump_cpu_stacks+0x90/0xd0 [10723.853157] [] rcu_check_callbacks+0x442/0x720 [10723.860860] [] ? tick_sched_handle.isra.13+0x60/0x60 [10723.869128] [] update_process_times+0x47/0x80 [10723.876685] [] tick_sched_handle.isra.13+0x25/0x60 [10723.884705] [] tick_sched_timer+0x41/0x70 [10723.891831] [] __hrtimer_run_queues+0xd2/0x260 [10723.899422] [] hrtimer_interrupt+0xb0/0x1e0 [10723.906695] [] ? call_softirq+0x1c/0x30 [10723.913575] [] local_apic_timer_interrupt+0x37/0x60 [10723.921600] [] smp_apic_timer_interrupt+0x3f/0x60 [10723.929411] [] apic_timer_interrupt+0x6d/0x80 [10723.936848] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [10723.946481] [] ? osd_scrub_exec+0x12/0x5b0 [osd_ldiskfs] [10723.954998] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [10723.963914] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [10723.972814] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [10723.981423] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [10723.989829] [] ? wake_up_state+0x20/0x20 [10723.996776] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [10724.005878] [] kthread+0xcf/0xe0 [10724.012054] [] ? kthread_create_on_node+0x140/0x140 [10724.020071] [] ret_from_fork+0x58/0x90 [10724.026822] [] ? kthread_create_on_node+0x140/0x140 [10749.216940] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [10749.226394] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [10749.356196] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [10749.369377] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [10749.385434] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [10749.394712] RIP: 0010:[] [] osd_scrub_exec+0x95/0x5b0 [osd_ldiskfs] [10749.406284] RSP: 0018:ffff880703603d08 EFLAGS: 00000292 [10749.413149] RAX: 0000000000000000 RBX: 000000009f8f2001 RCX: 0000000000000000 [10749.422045] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [10749.430956] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [10749.439859] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [10749.448748] R13: 0000000000000018 R14: ffff880703603c78 R15: 0000000000000296 [10749.457639] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [10749.467593] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10749.474927] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [10749.483825] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [10749.492727] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [10749.501621] Stack: [10749.504787] ffff880703603df0 ffffffffa121d1e9 ffffffffa121b990 ffffffffa1217a20 [10749.514020] ffff8800b2a36000 00000000810d354f ffff8803162f6000 ffff8800b2a37468 [10749.523264] 0000000020000000 010000000000000c 0000000000000000 0000000000000000 [10749.532513] Call Trace: [10749.536192] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [10749.545322] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [10749.554446] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [10749.563282] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [10749.571918] [] ? wake_up_state+0x20/0x20 [10749.579086] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [10749.588392] [] kthread+0xcf/0xe0 [10749.594795] [] ? kthread_create_on_node+0x140/0x140 [10749.603048] [] ret_from_fork+0x58/0x90 [10749.610027] [] ? kthread_create_on_node+0x140/0x140 [10749.618262] Code: 01 00 00 41 80 a7 74 14 00 00 fe 48 8b 4d d0 65 48 33 0c 25 28 00 00 00 0f 85 b1 03 00 00 48 83 c4 68 5b 41 5c 41 5d 41 5e 41 5f <5d> c3 66 0f 1f 84 00 00 00 00 00 4c 8d ae f0 10 00 00 44 89 4d [10777.217907] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [10777.227572] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [10777.359410] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [10777.372704] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [10777.388945] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [10777.398329] RIP: 0010:[] [] ldiskfs_itable_unused_count+0xd/0x30 [ldiskfs] [10777.410668] RSP: 0018:ffff880703603d08 EFLAGS: 00000286 [10777.417635] RAX: ffff8800b2a37468 RBX: ffff880703603d57 RCX: ffff88042bd78800 [10777.426607] RDX: ffff880703603d78 RSI: ffff880419dc3a00 RDI: ffff88042bd79800 [10777.435574] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [10777.444522] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [10777.453471] R13: ffffea00027e3c80 R14: ffff880703603c70 R15: ffff880703603d08 [10777.462417] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [10777.472428] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10777.479845] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [10777.488811] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [10777.497764] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [10777.506711] Stack: [10777.509920] ffff880703603df0 ffffffffa121cfb9 ffffffffa121b990 ffffffffa1217a20 [10777.519205] ffff8800b2a36000 00000000810d354f ffff8803162f6000 ffff8800b2a37468 [10777.528504] 0000000020000000 010000000000000c 0000000000000000 0000000000000000 [10777.537804] Call Trace: [10777.541540] [] ? osd_inode_iteration+0x269/0xcc0 [osd_ldiskfs] [10777.550906] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [10777.560072] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [10777.568935] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [10777.577598] [] ? wake_up_state+0x20/0x20 [10777.584811] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [10777.594175] [] kthread+0xcf/0xe0 [10777.600614] [] ? kthread_create_on_node+0x140/0x140 [10777.608893] [] ret_from_fork+0x58/0x90 [10777.615903] [] ? kthread_create_on_node+0x140/0x140 [10777.624174] Code: 10 48 89 e5 48 83 39 3f 76 07 0f b7 46 30 c1 e0 10 09 d0 5d c3 66 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 8b 8f a8 03 00 00 <31> c0 0f b7 56 1c 48 89 e5 48 83 39 3f 76 07 0f b7 46 32 c1 e0 [10805.218891] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [10805.228550] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [10805.360338] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [10805.373668] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [10805.389927] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [10805.399323] RIP: 0010:[] [] osd_inode_iteration+0x255/0xcc0 [osd_ldiskfs] [10805.411565] RSP: 0018:ffff880703603d18 EFLAGS: 00000286 [10805.418512] RAX: ffff8800b2a37468 RBX: 000000009f8f2001 RCX: 0000000000000000 [10805.427501] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [10805.436477] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [10805.445438] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [10805.454389] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [10805.463345] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [10805.473379] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10805.480799] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [10805.489771] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [10805.498722] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [10805.507669] Stack: [10805.510881] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [10805.520175] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [10805.529483] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [10805.538793] Call Trace: [10805.542523] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [10805.551691] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [10805.560567] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [10805.569240] [] ? wake_up_state+0x20/0x20 [10805.576449] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [10805.585795] [] kthread+0xcf/0xe0 [10805.592226] [] ? kthread_create_on_node+0x140/0x140 [10805.600512] [] ret_from_fork+0x58/0x90 [10805.607528] [] ? kthread_create_on_node+0x140/0x140 [10805.615802] Code: 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 00 8b 75 98 48 8b 7d 88 e8 dc 2d f5 ff 48 85 c0 48 89 45 90 0f 84 b3 08 00 00 44 8b 7d a0 <48> 8b 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 e8 77 5b f4 ff 42 [10825.856229] perf: interrupt took too long (4030 > 4028), lowering kernel.perf_event_max_sample_rate to 49000 [10853.220561] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [10853.230232] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [10853.362727] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [10853.376057] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [10853.392348] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [10853.401756] RIP: 0010:[] [] osd_scrub_exec+0xcf/0x5b0 [osd_ldiskfs] [10853.413442] RSP: 0018:ffff880703603c78 EFLAGS: 00000246 [10853.420415] RAX: ffff8803162f1008 RBX: ffff880703603c37 RCX: 0000000000000000 [10853.429419] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [10853.438410] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [10853.447385] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [10853.456352] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [10853.465325] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [10853.475369] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10853.482802] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [10853.491800] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [10853.500781] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [10853.509741] Stack: [10853.512982] ffffea00027e3c80 000000009f8f2001 ffff880703603df0 ffff880703603d57 [10853.522307] ffff880703603d57 000000009f8f2001 0000000000000004 ffff880703603d57 [10853.531631] ffff8800b2a37468 0000000000000000 ffff880703603d78 ffff8800b2a36000 [10853.540920] Call Trace: [10853.544665] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [10853.553856] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [10853.563041] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [10853.571925] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [10853.580631] [] ? wake_up_state+0x20/0x20 [10853.587856] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [10853.597234] [] kthread+0xcf/0xe0 [10853.603680] [] ? kthread_create_on_node+0x140/0x140 [10853.611992] [] ret_from_fork+0x58/0x90 [10853.619018] [] ? kthread_create_on_node+0x140/0x140 [10853.627313] Code: 10 00 00 44 89 4d 88 4c 89 ef e8 1d fb 46 e0 41 83 87 68 14 00 00 01 49 83 87 88 11 00 00 01 4c 89 ef e8 f5 9f e9 df 44 8b 4d 88 <48> 85 db 74 23 f6 83 18 0a 00 00 08 74 1a 4d 85 e4 74 15 41 8b [10881.221544] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [10881.231218] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [10881.363455] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [10881.376735] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [10881.392986] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [10881.402370] RIP: 0010:[] [] osd_inode_iteration+0x251/0xcc0 [osd_ldiskfs] [10881.414611] RSP: 0018:ffff880703603d18 EFLAGS: 00000286 [10881.421558] RAX: ffff8800b2a37468 RBX: ffff880703603d57 RCX: 0000000000000000 [10881.430540] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [10881.439504] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [10881.448455] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [10881.457403] R13: ffffea00027e3c80 R14: 000000009f8f2001 R15: 0000000000000018 [10881.466353] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [10881.476381] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10881.483785] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [10881.492755] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [10881.501710] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [10881.510660] Stack: [10881.513866] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [10881.523160] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [10881.532450] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [10881.541731] Call Trace: [10881.545460] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [10881.554624] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [10881.563492] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [10881.572172] [] ? wake_up_state+0x20/0x20 [10881.579383] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [10881.588735] [] kthread+0xcf/0xe0 [10881.595164] [] ? kthread_create_on_node+0x140/0x140 [10881.603455] [] ret_from_fork+0x58/0x90 [10881.610472] [] ? kthread_create_on_node+0x140/0x140 [10881.618744] Code: b0 00 00 00 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 00 8b 75 98 48 8b 7d 88 e8 dc 2d f5 ff 48 85 c0 48 89 45 90 0f 84 b3 08 00 00 <44> 8b 7d a0 48 8b 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 e8 77 [10903.782330] INFO: rcu_sched self-detected stall on CPU { 24} (t=3480078 jiffies g=907112 c=907111 q=4879055) [10903.794692] Task dump for CPU 24: [10903.799431] OI_scrub R running task 0 5551 2 0x00000088 [10903.808393] ffff88083fde6dd0 00000000fb6db880 ffff88082da03db0 ffffffff810c47e8 [10903.817711] 0000000000000018 ffffffff81a1e7c0 ffff88082da03dc8 ffffffff810c8079 [10903.827020] 0000000000000009 ffff88082da03df8 ffffffff811378e0 ffff88082da101c0 [10903.836305] Call Trace: [10903.840010] [] sched_show_task+0xa8/0x110 [10903.848010] [] dump_cpu_task+0x39/0x70 [10903.854999] [] rcu_dump_cpu_stacks+0x90/0xd0 [10903.862531] [] rcu_check_callbacks+0x442/0x720 [10903.870234] [] ? tick_sched_handle.isra.13+0x60/0x60 [10903.878500] [] update_process_times+0x47/0x80 [10903.886064] [] tick_sched_handle.isra.13+0x25/0x60 [10903.894086] [] tick_sched_timer+0x41/0x70 [10903.901218] [] __hrtimer_run_queues+0xd2/0x260 [10903.908818] [] hrtimer_interrupt+0xb0/0x1e0 [10903.916107] [] ? call_softirq+0x1c/0x30 [10903.922986] [] local_apic_timer_interrupt+0x37/0x60 [10903.931021] [] smp_apic_timer_interrupt+0x3f/0x60 [10903.938847] [] apic_timer_interrupt+0x6d/0x80 [10903.946269] [] ? osd_scrub_next+0x14d/0x3c0 [osd_ldiskfs] [10903.955599] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [10903.964498] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [10903.973395] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [10903.981991] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [10903.990398] [] ? wake_up_state+0x20/0x20 [10903.997344] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [10904.006425] [] kthread+0xcf/0xe0 [10904.012595] [] ? kthread_create_on_node+0x140/0x140 [10904.020605] [] ret_from_fork+0x58/0x90 [10904.027358] [] ? kthread_create_on_node+0x140/0x140 [10929.223216] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [10929.232609] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [10929.362863] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [10929.376050] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [10929.392125] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [10929.401422] RIP: 0010:[] [] osd_inode_iteration+0x255/0xcc0 [osd_ldiskfs] [10929.413570] RSP: 0018:ffff880703603d18 EFLAGS: 00000286 [10929.420434] RAX: ffff8800b2a37468 RBX: 000000009f8f2001 RCX: 0000000000000000 [10929.429342] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [10929.438237] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [10929.447136] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffff880703603d57 [10929.456040] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [10929.464941] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [10929.474920] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10929.482269] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [10929.491200] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [10929.500105] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [10929.509008] Stack: [10929.512173] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [10929.521409] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [10929.530643] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [10929.539892] Call Trace: [10929.543590] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [10929.552731] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [10929.561581] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [10929.570226] [] ? wake_up_state+0x20/0x20 [10929.577418] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [10929.586740] [] kthread+0xcf/0xe0 [10929.593140] [] ? kthread_create_on_node+0x140/0x140 [10929.601391] [] ret_from_fork+0x58/0x90 [10929.608377] [] ? kthread_create_on_node+0x140/0x140 [10929.616632] Code: 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 00 8b 75 98 48 8b 7d 88 e8 dc 2d f5 ff 48 85 c0 48 89 45 90 0f 84 b3 08 00 00 44 8b 7d a0 <48> 8b 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 e8 77 5b f4 ff 42 [10934.532800] Lustre: soaked-MDT0000: Client 91eacee7-b83c-19ec-9326-47c5ebda930d (at 192.168.1.116@o2ib100) reconnecting [10934.546060] Lustre: soaked-MDT0000: Connection restored to 91eacee7-b83c-19ec-9326-47c5ebda930d (at 192.168.1.116@o2ib100) [10957.224185] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [10957.233862] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [10957.365840] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [10957.379151] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [10957.395463] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [10957.404874] RIP: 0010:[] [] osd_scrub_next+0x2b/0x3c0 [osd_ldiskfs] [10957.416552] RSP: 0018:ffff880703603c70 EFLAGS: 00000282 [10957.423523] RAX: 00000000fb6db880 RBX: ffff880703603c37 RCX: ffff880703603d98 [10957.432530] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [10957.441524] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [10957.450512] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [10957.459502] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [10957.468494] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [10957.478569] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10957.486013] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [10957.495018] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [10957.504009] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [10957.512989] Stack: [10957.516248] 000000009f8f2001 ffff880703603d08 000000009f8f2001 ffffea00027e3c80 [10957.525586] 000000009f8f2001 ffff880703603d57 0000000000000001 ffffffffa1217a20 [10957.534903] ffff880703603d98 ffff880703603d78 ffff8800b2a36000 ffff8803162f6000 [10957.544224] Call Trace: [10957.547975] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [10957.556881] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [10957.566071] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [10957.575261] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [10957.584149] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [10957.592843] [] ? wake_up_state+0x20/0x20 [10957.600079] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [10957.609444] [] kthread+0xcf/0xe0 [10957.615903] [] ? kthread_create_on_node+0x140/0x140 [10957.624208] [] ret_from_fork+0x58/0x90 [10957.631249] [] ? kthread_create_on_node+0x140/0x140 [10957.639548] Code: 66 66 66 66 90 55 48 89 e5 41 57 49 89 f7 41 56 41 55 49 89 fd 41 54 49 89 d4 53 48 89 cb 48 83 ec 70 65 48 8b 04 25 28 00 00 00 <48> 89 45 d0 31 c0 48 8b 05 f8 59 9c ff 48 85 c0 75 7b 41 f6 87 [10985.225153] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [10985.234781] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [10985.366298] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [10985.379566] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [10985.395770] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [10985.405131] RIP: 0010:[] [] osd_scrub_exec+0xe9/0x5b0 [osd_ldiskfs] [10985.416754] RSP: 0018:ffff880703603c78 EFLAGS: 00000282 [10985.423678] RAX: 0000000000280001 RBX: ffff880703603c37 RCX: 0000000000000000 [10985.432634] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [10985.441581] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [10985.450522] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [10985.459464] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [10985.468410] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [10985.478442] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10985.485853] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [10985.494815] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [10985.503851] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [10985.512814] Stack: [10985.516083] 0000000000000004 0000000000000001 ffff880703603df0 ffff880703603d98 [10985.525432] ffff880703603d57 000000009f8f2001 0000000000000004 ffff880703603d57 [10985.534871] ffff8800b2a37468 0000000000000000 ffff880703603d78 ffff880419dc3a00 [10985.544158] Call Trace: [10985.548004] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [10985.557178] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [10985.566344] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [10985.575206] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [10985.583990] [] ? wake_up_state+0x20/0x20 [10985.591337] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [10985.600697] [] kthread+0xcf/0xe0 [10985.607437] [] ? kthread_create_on_node+0x140/0x140 [10985.615723] [] ret_from_fork+0x58/0x90 [10985.622741] [] ? kthread_create_on_node+0x140/0x140 [10985.631028] Code: 88 11 00 00 01 4c 89 ef e8 f5 9f e9 df 44 8b 4d 88 48 85 db 74 23 f6 83 18 0a 00 00 08 74 1a 4d 85 e4 74 15 41 8b 87 6c 14 00 00 <41> 39 84 24 0c 0a 00 00 0f 82 d9 00 00 00 31 c0 41 f6 87 74 14 [11013.226129] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [11013.235811] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [11013.367626] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [11013.380910] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [11013.397154] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [11013.406543] RIP: 0010:[] [] osd_scrub_exec+0xf7/0x5b0 [osd_ldiskfs] [11013.418203] RSP: 0018:ffff880703603c78 EFLAGS: 00000246 [11013.425139] RAX: 0000000000280001 RBX: ffff880703603c37 RCX: 0000000000000000 [11013.434104] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [11013.443065] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [11013.452021] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [11013.460968] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [11013.469916] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [11013.479941] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11013.487342] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [11013.496312] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [11013.505264] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [11013.514207] Stack: [11013.517411] ffff880703603d08 000000009f8f2001 ffff880703603df0 000000009f8f2001 [11013.526688] ffff880703603d57 000000009f8f2001 0000000000000004 ffff880703603d57 [11013.535988] ffff88042bd78800 ffff88042bd78800 00000000000072cf ffff880419dc3a00 [11013.545266] Call Trace: [11013.548974] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [11013.558133] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [11013.567286] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [11013.576139] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [11013.584801] [] ? wake_up_state+0x20/0x20 [11013.592008] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [11013.601349] [] kthread+0xcf/0xe0 [11013.607771] [] ? kthread_create_on_node+0x140/0x140 [11013.616041] [] ret_from_fork+0x58/0x90 [11013.623052] [] ? kthread_create_on_node+0x140/0x140 [11013.631315] Code: 8b 4d 88 48 85 db 74 23 f6 83 18 0a 00 00 08 74 1a 4d 85 e4 74 15 41 8b 87 6c 14 00 00 41 39 84 24 0c 0a 00 00 0f 82 d9 00 00 00 <31> c0 41 f6 87 74 14 00 00 04 0f 85 6e ff ff ff 41 83 f9 02 0f [11041.227110] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [11041.236799] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [11041.368518] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [11041.381807] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [11041.398071] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [11041.407788] RIP: 0010:[] [] osd_inode_iteration+0x49e/0xcc0 [osd_ldiskfs] [11041.420373] RSP: 0018:ffff880703603d18 EFLAGS: 00000246 [11041.427512] RAX: 0000000000000000 RBX: 000000009f8f2001 RCX: 0000000000000000 [11041.436592] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [11041.445756] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [11041.454914] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffff880703603d57 [11041.464162] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [11041.473191] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [11041.483471] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11041.490902] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [11041.499873] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [11041.508967] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [11041.518095] Stack: [11041.521368] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [11041.530688] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [11041.539976] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [11041.549268] Call Trace: [11041.552999] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [11041.562181] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [11041.571057] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [11041.579730] [] ? wake_up_state+0x20/0x20 [11041.586933] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [11041.596300] [] kthread+0xcf/0xe0 [11041.602744] [] ? kthread_create_on_node+0x140/0x140 [11041.611029] [] ret_from_fork+0x58/0x90 [11041.618047] [] ? kthread_create_on_node+0x140/0x140 [11041.626332] Code: 4c 8d 85 67 ff ff ff 48 8b 4d a8 48 8d 55 88 48 8b b5 38 ff ff ff 48 8b bd 48 ff ff ff 48 8b 85 28 ff ff ff ff d0 85 c0 41 89 c5 <75> 1c 8b 85 58 ff ff ff 41 3b 04 24 72 10 48 8b 85 50 ff ff ff [11069.228080] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [11069.237693] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [11069.369611] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [11069.382923] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [11069.399242] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [11069.408648] RIP: 0010:[] [] osd_inode_iteration+0x251/0xcc0 [osd_ldiskfs] [11069.420917] RSP: 0018:ffff880703603d18 EFLAGS: 00000286 [11069.427891] RAX: ffff8800b2a37468 RBX: 000000009f8f2001 RCX: 0000000000000000 [11069.436884] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [11069.445867] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [11069.454837] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [11069.463792] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [11069.472743] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [11069.482769] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11069.490187] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [11069.499157] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [11069.508128] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [11069.517086] Stack: [11069.520302] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [11069.529600] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [11069.538886] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [11069.548188] Call Trace: [11069.551908] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [11069.561078] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [11069.569957] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [11069.578642] [] ? wake_up_state+0x20/0x20 [11069.585854] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [11069.595208] [] kthread+0xcf/0xe0 [11069.601638] [] ? kthread_create_on_node+0x140/0x140 [11069.609915] [] ret_from_fork+0x58/0x90 [11069.616926] [] ? kthread_create_on_node+0x140/0x140 [11069.625192] Code: b0 00 00 00 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 00 8b 75 98 48 8b 7d 88 e8 dc 2d f5 ff 48 85 c0 48 89 45 90 0f 84 b3 08 00 00 <44> 8b 7d a0 48 8b 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 e8 77 [11083.791587] INFO: rcu_sched self-detected stall on CPU { 24} (t=3660081 jiffies g=907112 c=907111 q=5905472) [11083.803940] Task dump for CPU 24: [11083.808688] OI_scrub R running task 0 5551 2 0x00000088 [11083.817614] ffff88083fde6dd0 00000000fb6db880 ffff88082da03db0 ffffffff810c47e8 [11083.826933] 0000000000000018 ffffffff81a1e7c0 ffff88082da03dc8 ffffffff810c8079 [11083.836248] 0000000000000009 ffff88082da03df8 ffffffff811378e0 ffff88082da101c0 [11083.845559] Call Trace: [11083.849274] [] sched_show_task+0xa8/0x110 [11083.857279] [] dump_cpu_task+0x39/0x70 [11083.864271] [] rcu_dump_cpu_stacks+0x90/0xd0 [11083.871807] [] rcu_check_callbacks+0x442/0x720 [11083.879523] [] ? tick_sched_handle.isra.13+0x60/0x60 [11083.887800] [] update_process_times+0x47/0x80 [11083.895370] [] tick_sched_handle.isra.13+0x25/0x60 [11083.903400] [] tick_sched_timer+0x41/0x70 [11083.910541] [] __hrtimer_run_queues+0xd2/0x260 [11083.918142] [] hrtimer_interrupt+0xb0/0x1e0 [11083.925425] [] ? call_softirq+0x1c/0x30 [11083.932322] [] local_apic_timer_interrupt+0x37/0x60 [11083.940362] [] smp_apic_timer_interrupt+0x3f/0x60 [11083.948205] [] apic_timer_interrupt+0x6d/0x80 [11083.955629] [] ? osd_scrub_next+0x169/0x3c0 [osd_ldiskfs] [11083.964961] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [11083.973862] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [11083.982759] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [11083.991362] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [11083.999767] [] ? wake_up_state+0x20/0x20 [11084.006714] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [11084.015800] [] kthread+0xcf/0xe0 [11084.021960] [] ? kthread_create_on_node+0x140/0x140 [11084.029976] [] ret_from_fork+0x58/0x90 [11084.036734] [] ? kthread_create_on_node+0x140/0x140 [11109.229472] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [11109.238868] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [11109.368571] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [11109.381759] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [11109.397834] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [11109.407133] RIP: 0010:[] [] osd_inode_iteration+0x255/0xcc0 [osd_ldiskfs] [11109.419303] RSP: 0018:ffff880703603d18 EFLAGS: 00000286 [11109.426166] RAX: ffff8800b2a37468 RBX: ffff880703603d98 RCX: 0000000000000000 [11109.435079] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [11109.443987] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [11109.452895] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000001 [11109.461801] R13: 0000000000000004 R14: 000000009f8f2001 R15: 0000000000000018 [11109.470704] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [11109.480670] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11109.488008] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [11109.496935] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [11109.505838] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [11109.514725] Stack: [11109.517886] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [11109.527135] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [11109.536396] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [11109.545669] Call Trace: [11109.549351] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [11109.558476] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [11109.567332] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [11109.576096] [] ? wake_up_state+0x20/0x20 [11109.583306] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [11109.592618] [] kthread+0xcf/0xe0 [11109.599009] [] ? kthread_create_on_node+0x140/0x140 [11109.607248] [] ret_from_fork+0x58/0x90 [11109.614231] [] ? kthread_create_on_node+0x140/0x140 [11109.622468] Code: 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 00 8b 75 98 48 8b 7d 88 e8 dc 2d f5 ff 48 85 c0 48 89 45 90 0f 84 b3 08 00 00 44 8b 7d a0 <48> 8b 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 e8 77 5b f4 ff 42 [11129.530033] Lustre: soaked-MDT0000: Connection restored to 7ea3a484-c027-1559-8163-dc85944c882c (at 192.168.1.143@o2ib100) [11137.230449] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [11137.240115] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [11137.372046] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [11137.385354] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [11137.401670] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [11137.411073] RIP: 0010:[] [] osd_scrub_next+0x2f/0x3c0 [osd_ldiskfs] [11137.422755] RSP: 0018:ffff880703603c70 EFLAGS: 00000282 [11137.429712] RAX: 00000000fb6db880 RBX: ffff880703603c37 RCX: ffff880703603d98 [11137.438708] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [11137.447691] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [11137.456658] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [11137.465624] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [11137.474591] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [11137.484641] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11137.492075] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [11137.501073] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [11137.510061] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [11137.519039] Stack: [11137.522278] 000000009f8f2001 ffff880703603d08 ffff880703603d57 ffff880703603df0 [11137.531610] 000000009f8f2001 ffff880703603d57 000000009f8f2001 0000000000000004 [11137.540935] ffff880703603d57 0000000020000000 0000000000000000 ffff880703603d78 [11137.550268] Call Trace: [11137.554016] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [11137.563209] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [11137.572399] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [11137.581289] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [11137.589989] [] ? wake_up_state+0x20/0x20 [11137.597220] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [11137.606604] [] kthread+0xcf/0xe0 [11137.613058] [] ? kthread_create_on_node+0x140/0x140 [11137.621370] [] ret_from_fork+0x58/0x90 [11137.628408] [] ? kthread_create_on_node+0x140/0x140 [11137.636704] Code: 90 55 48 89 e5 41 57 49 89 f7 41 56 41 55 49 89 fd 41 54 49 89 d4 53 48 89 cb 48 83 ec 70 65 48 8b 04 25 28 00 00 00 48 89 45 d0 <31> c0 48 8b 05 f8 59 9c ff 48 85 c0 75 7b 41 f6 87 58 10 00 00 [11165.231430] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [11165.241114] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [11165.372816] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [11165.386202] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [11165.402458] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [11165.411842] RIP: 0010:[] [] osd_scrub_exec+0xf7/0x5b0 [osd_ldiskfs] [11165.423503] RSP: 0018:ffff880703603c78 EFLAGS: 00000246 [11165.430444] RAX: 0000000000280001 RBX: ffff880703603c37 RCX: 0000000000000000 [11165.439423] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [11165.448387] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [11165.457336] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [11165.466289] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [11165.475241] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [11165.485269] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11165.492678] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [11165.501641] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [11165.510594] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [11165.519535] Stack: [11165.522751] ffffea00027e3c80 000000009f8f2001 0000000000000004 0000000000000001 [11165.532045] ffff880703603d57 ffff880703603d98 ffff880703603d78 ffff8800b2a36000 [11165.541378] ffff8803162f6000 ffffffffffffff10 ffffffffa1217a32 0000000000000010 [11165.550678] Call Trace: [11165.554399] [] ? osd_scrub_next+0x12/0x3c0 [osd_ldiskfs] [11165.563182] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [11165.572349] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [11165.581508] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [11165.590369] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [11165.599050] [] ? wake_up_state+0x20/0x20 [11165.606259] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [11165.615617] [] kthread+0xcf/0xe0 [11165.622049] [] ? kthread_create_on_node+0x140/0x140 [11165.630335] [] ret_from_fork+0x58/0x90 [11165.637346] [] ? kthread_create_on_node+0x140/0x140 [11165.645620] Code: 8b 4d 88 48 85 db 74 23 f6 83 18 0a 00 00 08 74 1a 4d 85 e4 74 15 41 8b 87 6c 14 00 00 41 39 84 24 0c 0a 00 00 0f 82 d9 00 00 00 <31> c0 41 f6 87 74 14 00 00 04 0f 85 6e ff ff ff 41 83 f9 02 0f [11193.232400] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [11193.242056] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [11193.373579] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [11193.386865] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [11193.403092] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [11193.412466] RIP: 0010:[] [] osd_scrub_next+0x12/0x3c0 [osd_ldiskfs] [11193.424098] RSP: 0018:ffff880703603cf0 EFLAGS: 00000287 [11193.431040] RAX: ffffffffa1217a20 RBX: 000000009f8f2001 RCX: ffff880703603d98 [11193.440009] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [11193.448972] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [11193.457916] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffff880703603c70 [11193.466876] R13: ffff880703603d08 R14: 0000000000000010 R15: ffffffffa1217a4f [11193.475840] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [11193.485863] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11193.493274] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [11193.502236] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [11193.511190] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [11193.520134] Stack: [11193.523349] 0000000000000000 00000000ffffffff 0000000000000001 ffff880703603df0 [11193.532659] ffffffffa121cff9 ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 [11193.541970] 00000000810d354f ffff8803162f6000 ffff8800b2a37468 0000000020000000 [11193.551249] Call Trace: [11193.554971] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [11193.564144] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [11193.573320] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [11193.582190] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [11193.590865] [] ? wake_up_state+0x20/0x20 [11193.598076] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [11193.607434] [] kthread+0xcf/0xe0 [11193.613874] [] ? kthread_create_on_node+0x140/0x140 [11193.622159] [] ret_from_fork+0x58/0x90 [11193.629172] [] ? kthread_create_on_node+0x140/0x140 [11193.637455] Code: 25 fc ff ff 48 83 c4 08 5b 41 5c 41 5d 41 5e 5d c3 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 89 e5 41 57 49 89 f7 41 56 41 55 <49> 89 fd 41 54 49 89 d4 53 48 89 cb 48 83 ec 70 65 48 8b 04 25 [11221.233380] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [11221.243073] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [11221.375053] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [11221.388344] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [11221.404595] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [11221.413983] RIP: 0010:[] [] osd_inode_iteration+0x28a/0xcc0 [osd_ldiskfs] [11221.426245] RSP: 0018:ffff880703603d18 EFLAGS: 00000287 [11221.433199] RAX: ffff88042bd78800 RBX: 0000000000000000 RCX: ffff88042bd78800 [11221.442182] RDX: 00000000000072cf RSI: ffff880419dc3a00 RDI: ffff88042bd79800 [11221.451161] RBP: ffff880703603df0 R08: 0000000000000001 R09: 0000000000000004 [11221.460132] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffff880703603d57 [11221.469101] R13: 0000000000000004 R14: 000000009f8f2001 R15: ffffea00027e3c80 [11221.478061] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [11221.488103] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11221.495515] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [11221.504485] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [11221.513446] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [11221.522395] Stack: [11221.525609] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [11221.534909] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [11221.544213] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [11221.553503] Call Trace: [11221.557230] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [11221.566396] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [11221.575267] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [11221.583937] [] ? wake_up_state+0x20/0x20 [11221.591145] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [11221.600502] [] kthread+0xcf/0xe0 [11221.606934] [] ? kthread_create_on_node+0x140/0x140 [11221.615222] [] ret_from_fork+0x58/0x90 [11221.622237] [] ? kthread_create_on_node+0x140/0x140 [11221.630512] Code: a8 00 00 00 00 e8 77 5b f4 ff 42 8d 14 38 48 8b 45 88 48 8b 80 a8 03 00 00 48 3b 50 20 0f 83 57 01 00 00 44 0f b6 85 67 ff ff ff <48> 8d 4d a8 48 8d 55 88 48 8b b5 38 ff ff ff 48 8b bd 48 ff ff [11249.234349] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [11249.244012] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [11249.375975] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [11249.389300] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [11249.405600] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [11249.415012] RIP: 0010:[] [] osd_scrub_next+0x12/0x3c0 [osd_ldiskfs] [11249.426698] RSP: 0018:ffff880703603cf0 EFLAGS: 00000287 [11249.433659] RAX: ffffffffa1217a20 RBX: 000000009f8f2001 RCX: ffff880703603d98 [11249.442648] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [11249.451626] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [11249.460592] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffff880703603c78 [11249.469545] R13: 0000000000000246 R14: 0000000000000010 R15: ffffffffa121ba12 [11249.478502] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [11249.488534] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11249.495944] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [11249.504904] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [11249.513865] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [11249.522818] Stack: [11249.526027] 0000000000000000 00000000ffffffff 0000000000000001 ffff880703603df0 [11249.535324] ffffffffa121cff9 ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 [11249.544624] 00000000810d354f ffff8803162f6000 ffff8800b2a37468 0000000020000000 [11249.553934] Call Trace: [11249.557660] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [11249.566834] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [11249.575995] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [11249.584867] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [11249.593538] [] ? wake_up_state+0x20/0x20 [11249.600753] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [11249.610106] [] kthread+0xcf/0xe0 [11249.616536] [] ? kthread_create_on_node+0x140/0x140 [11249.624820] [] ret_from_fork+0x58/0x90 [11249.631837] [] ? kthread_create_on_node+0x140/0x140 [11249.640104] Code: 25 fc ff ff 48 83 c4 08 5b 41 5c 41 5d 41 5e 5d c3 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 89 e5 41 57 49 89 f7 41 56 41 55 <49> 89 fd 41 54 49 89 d4 53 48 89 cb 48 83 ec 70 65 48 8b 04 25 [11263.800871] INFO: rcu_sched self-detected stall on CPU { 24} (t=3840084 jiffies g=907112 c=907111 q=6877352) [11263.813258] Task dump for CPU 24: [11263.817996] OI_scrub R running task 0 5551 2 0x00000088 [11263.826928] ffff88083fde6dd0 00000000fb6db880 ffff88082da03db0 ffffffff810c47e8 [11263.836243] 0000000000000018 ffffffff81a1e7c0 ffff88082da03dc8 ffffffff810c8079 [11263.845556] 0000000000000009 ffff88082da03df8 ffffffff811378e0 ffff88082da101c0 [11263.854834] Call Trace: [11263.858516] [] sched_show_task+0xa8/0x110 [11263.866478] [] dump_cpu_task+0x39/0x70 [11263.873446] [] rcu_dump_cpu_stacks+0x90/0xd0 [11263.880972] [] rcu_check_callbacks+0x442/0x720 [11263.888671] [] ? tick_sched_handle.isra.13+0x60/0x60 [11263.896921] [] update_process_times+0x47/0x80 [11263.904467] [] tick_sched_handle.isra.13+0x25/0x60 [11263.912479] [] tick_sched_timer+0x41/0x70 [11263.919595] [] __hrtimer_run_queues+0xd2/0x260 [11263.927176] [] hrtimer_interrupt+0xb0/0x1e0 [11263.934464] [] ? call_softirq+0x1c/0x30 [11263.941346] [] local_apic_timer_interrupt+0x37/0x60 [11263.949379] [] smp_apic_timer_interrupt+0x3f/0x60 [11263.957205] [] apic_timer_interrupt+0x6d/0x80 [11263.964626] [] ? ldiskfs_itable_unused_count+0x23/0x30 [ldiskfs] [11263.974656] [] osd_inode_iteration+0x269/0xcc0 [osd_ldiskfs] [11263.983563] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [11263.992465] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [11264.001077] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [11264.009484] [] ? wake_up_state+0x20/0x20 [11264.016434] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [11264.025520] [] kthread+0xcf/0xe0 [11264.031697] [] ? kthread_create_on_node+0x140/0x140 [11264.039726] [] ret_from_fork+0x58/0x90 [11264.046483] [] ? kthread_create_on_node+0x140/0x140 [11289.235743] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [11289.245166] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [11289.374875] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [11289.388049] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [11289.404115] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [11289.413415] RIP: 0010:[] [] osd_scrub_next+0x9/0x3c0 [osd_ldiskfs] [11289.424885] RSP: 0018:ffff880703603d08 EFLAGS: 00000287 [11289.431739] RAX: ffffffffa1217a20 RBX: 000000009f8f2001 RCX: ffff880703603d98 [11289.440645] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [11289.449537] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [11289.458438] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [11289.467339] R13: 0000000000000018 R14: ffff880703603c70 R15: 0000000000000282 [11289.476224] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [11289.486179] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11289.493519] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [11289.502418] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [11289.511319] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [11289.520205] Stack: [11289.523361] ffff880703603df0 ffffffffa121cff9 ffffffffa121b990 ffffffffa1217a20 [11289.532571] ffff8800b2a36000 00000000810d354f ffff8803162f6000 ffff8800b2a37468 [11289.541814] 0000000020000000 010000000000000c 0000000000000000 0000000000000000 [11289.551072] Call Trace: [11289.554757] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [11289.563883] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [11289.573005] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [11289.581837] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [11289.590472] [] ? wake_up_state+0x20/0x20 [11289.597644] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [11289.606960] [] kthread+0xcf/0xe0 [11289.613348] [] ? kthread_create_on_node+0x140/0x140 [11289.621586] [] ret_from_fork+0x58/0x90 [11289.628564] [] ? kthread_create_on_node+0x140/0x140 [11289.636796] Code: 48 8d 4c 02 10 48 01 c2 e8 25 fc ff ff 48 83 c4 08 5b 41 5c 41 5d 41 5e 5d c3 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 89 e5 <41> 57 49 89 f7 41 56 41 55 49 89 fd 41 54 49 89 d4 53 48 89 cb [11317.236725] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [11317.246345] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [11317.378088] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [11317.391396] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [11317.407642] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [11317.417018] RIP: 0010:[] [] osd_inode_iteration+0x473/0xcc0 [osd_ldiskfs] [11317.429271] RSP: 0018:ffff880703603d18 EFLAGS: 00000293 [11317.436206] RAX: 0000000000000004 RBX: ffff880703603d98 RCX: ffff880703603d98 [11317.445187] RDX: ffff880703603d78 RSI: 0000000000000000 RDI: ffff8803162f6000 [11317.454153] RBP: ffff880703603df0 R08: 0000000000000001 R09: 0000000000000004 [11317.463108] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000001 [11317.472055] R13: 0000000000000004 R14: 000000009f8f2001 R15: 0000000000000018 [11317.480995] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [11317.491014] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11317.498406] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [11317.507368] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [11317.516321] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [11317.525259] Stack: [11317.528464] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [11317.537744] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [11317.547032] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [11317.556299] Call Trace: [11317.560014] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [11317.569176] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [11317.578044] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [11317.586712] [] ? wake_up_state+0x20/0x20 [11317.593908] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [11317.603253] [] kthread+0xcf/0xe0 [11317.609691] [] ? kthread_create_on_node+0x140/0x140 [11317.617973] [] ret_from_fork+0x58/0x90 [11317.624976] [] ? kthread_create_on_node+0x140/0x140 [11317.633240] Code: 00 00 48 c7 05 9b 86 02 00 00 00 00 00 c7 05 89 86 02 00 01 00 00 00 e8 7c eb 97 ff e9 0f fc ff ff 0f 1f 80 00 00 00 00 45 89 e9 <4c> 8d 85 67 ff ff ff 48 8b 4d a8 48 8d 55 88 48 8b b5 38 ff ff [11332.823838] perf: interrupt took too long (5039 > 5037), lowering kernel.perf_event_max_sample_rate to 39000 [11357.238111] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [11357.247813] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [11357.379703] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [11357.393006] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [11357.409288] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [11357.418687] RIP: 0010:[] [] osd_scrub_exec+0x30/0x5b0 [osd_ldiskfs] [11357.430343] RSP: 0018:ffff880703603c78 EFLAGS: 00000296 [11357.437288] RAX: 00000000fb6db880 RBX: ffff880703603c37 RCX: 0000000000000000 [11357.446279] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [11357.455256] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [11357.464219] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [11357.473186] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [11357.482146] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [11357.492188] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11357.499614] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [11357.508605] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [11357.517582] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [11357.526554] Stack: [11357.529785] 0000000000000004 0000000000000001 ffff880703603df0 ffff880703603d98 [11357.539108] ffff880703603d57 000000009f8f2001 0000000000000004 ffff880703603d57 [11357.548426] ffff88042bd78800 ffff88042bd78800 00000000000072cf ffff880419dc3a00 [11357.557744] Call Trace: [11357.561479] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [11357.570670] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [11357.579862] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [11357.588743] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [11357.597424] [] ? wake_up_state+0x20/0x20 [11357.604655] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [11357.614019] [] kthread+0xcf/0xe0 [11357.620465] [] ? kthread_create_on_node+0x140/0x140 [11357.628762] [] ret_from_fork+0x58/0x90 [11357.635792] [] ? kthread_create_on_node+0x140/0x140 [11357.644092] Code: 55 48 89 e5 41 57 49 89 f7 41 56 41 55 41 54 45 31 e4 53 48 83 ec 68 48 8b 9e 08 10 00 00 4c 89 45 90 65 48 8b 04 25 28 00 00 00 <48> 89 45 d0 31 c0 48 8d 43 08 48 85 db 4c 0f 45 e0 41 83 f9 04 [11385.239093] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [11385.248775] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [11385.380726] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [11385.394016] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [11385.410258] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [11385.419643] RIP: 0010:[] [] osd_inode_iteration+0x264/0xcc0 [osd_ldiskfs] [11385.431887] RSP: 0018:ffff880703603d18 EFLAGS: 00000286 [11385.438830] RAX: ffff8800b2a37468 RBX: 000000009f8f2001 RCX: 0000000000000000 [11385.447809] RDX: ffff880703603d78 RSI: ffff880419dc3a00 RDI: ffff88042bd79800 [11385.456779] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [11385.465737] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [11385.474682] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [11385.483628] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [11385.493658] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11385.501067] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [11385.510034] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [11385.518985] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [11385.527923] Stack: [11385.531137] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [11385.540432] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [11385.549724] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [11385.559020] Call Trace: [11385.562738] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [11385.571898] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [11385.580776] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [11385.589443] [] ? wake_up_state+0x20/0x20 [11385.596655] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [11385.606008] [] kthread+0xcf/0xe0 [11385.612438] [] ? kthread_create_on_node+0x140/0x140 [11385.620730] [] ret_from_fork+0x58/0x90 [11385.627749] [] ? kthread_create_on_node+0x140/0x140 [11385.636027] Code: 75 98 48 8b 7d 88 e8 dc 2d f5 ff 48 85 c0 48 89 45 90 0f 84 b3 08 00 00 44 8b 7d a0 48 8b 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 77 5b f4 ff 42 8d 14 38 48 8b 45 88 48 8b 80 a8 03 00 00 48 [11412.545528] Lustre: soaked-MDT0000: Client 91eacee7-b83c-19ec-9326-47c5ebda930d (at 192.168.1.116@o2ib100) reconnecting [11412.558814] Lustre: Skipped 1 previous similar message [11412.565640] Lustre: soaked-MDT0000: Connection restored to 91eacee7-b83c-19ec-9326-47c5ebda930d (at 192.168.1.116@o2ib100) [11413.240063] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [11413.249725] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [11413.381777] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [11413.395083] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [11413.411375] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [11413.420779] RIP: 0010:[] [] osd_inode_iteration+0x489/0xcc0 [osd_ldiskfs] [11413.433038] RSP: 0018:ffff880703603d18 EFLAGS: 00000293 [11413.440021] RAX: 0000000000000004 RBX: 000000009f8f2001 RCX: 0000000000000000 [11413.449040] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [11413.458052] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [11413.467053] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [11413.476056] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [11413.485061] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [11413.495150] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11413.502607] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [11413.511623] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [11413.520626] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [11413.529628] Stack: [11413.532900] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [11413.542247] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [11413.551589] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [11413.560931] Call Trace: [11413.564699] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [11413.573912] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [11413.582835] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [11413.591556] [] ? wake_up_state+0x20/0x20 [11413.598818] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [11413.608224] [] kthread+0xcf/0xe0 [11413.614712] [] ? kthread_create_on_node+0x140/0x140 [11413.623037] [] ret_from_fork+0x58/0x90 [11413.630102] [] ? kthread_create_on_node+0x140/0x140 [11413.638444] Code: 00 e8 7c eb 97 ff e9 0f fc ff ff 0f 1f 80 00 00 00 00 45 89 e9 4c 8d 85 67 ff ff ff 48 8b 4d a8 48 8d 55 88 48 8b b5 38 ff ff ff <48> 8b bd 48 ff ff ff 48 8b 85 28 ff ff ff ff d0 85 c0 41 89 c5 [11441.241045] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [11441.250738] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [11441.382767] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [11441.396077] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [11441.412387] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [11441.421789] RIP: 0010:[] [] osd_scrub_next+0x2b/0x3c0 [osd_ldiskfs] [11441.433457] RSP: 0018:ffff880703603c70 EFLAGS: 00000282 [11441.440425] RAX: 00000000fb6db880 RBX: ffff880703603c37 RCX: ffff880703603d98 [11441.449417] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [11441.458394] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [11441.467362] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [11441.476323] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [11441.485278] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [11441.495294] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11441.502700] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [11441.511666] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [11441.520631] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [11441.529586] Stack: [11441.532791] 000000009f8f2001 ffff880703603d08 0000000000000001 ffff880703603df0 [11441.542082] 000000009f8f2001 ffff880703603d57 000000009f8f2001 0000000000000004 [11441.551374] ffff880703603d57 ffff8800b2a37468 0000000000000000 ffff880703603d78 [11441.560663] Call Trace: [11441.564387] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [11441.573556] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [11441.582716] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [11441.591583] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [11441.600247] [] ? wake_up_state+0x20/0x20 [11441.607458] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [11441.616799] [] kthread+0xcf/0xe0 [11441.623232] [] ? kthread_create_on_node+0x140/0x140 [11441.631503] [] ret_from_fork+0x58/0x90 [11441.638507] [] ? kthread_create_on_node+0x140/0x140 [11441.646780] Code: 66 66 66 66 90 55 48 89 e5 41 57 49 89 f7 41 56 41 55 49 89 fd 41 54 49 89 d4 53 48 89 cb 48 83 ec 70 65 48 8b 04 25 28 00 00 00 <48> 89 45 d0 31 c0 48 8b 05 f8 59 9c ff 48 85 c0 75 7b 41 f6 87 [11443.810127] INFO: rcu_sched self-detected stall on CPU { 24} (t=4020087 jiffies g=907112 c=907111 q=7512209) [11443.822505] Task dump for CPU 24: [11443.827223] OI_scrub R running task 0 5551 2 0x00000088 [11443.836145] ffff88083fde6dd0 00000000fb6db880 ffff88082da03db0 ffffffff810c47e8 [11443.845475] 0000000000000018 ffffffff81a1e7c0 ffff88082da03dc8 ffffffff810c8079 [11443.854763] 0000000000000009 ffff88082da03df8 ffffffff811378e0 ffff88082da101c0 [11443.864038] Call Trace: [11443.867713] [] sched_show_task+0xa8/0x110 [11443.875685] [] dump_cpu_task+0x39/0x70 [11443.882643] [] rcu_dump_cpu_stacks+0x90/0xd0 [11443.890157] [] rcu_check_callbacks+0x442/0x720 [11443.897838] [] ? tick_sched_handle.isra.13+0x60/0x60 [11443.906084] [] update_process_times+0x47/0x80 [11443.913626] [] tick_sched_handle.isra.13+0x25/0x60 [11443.921641] [] tick_sched_timer+0x41/0x70 [11443.928766] [] __hrtimer_run_queues+0xd2/0x260 [11443.936345] [] hrtimer_interrupt+0xb0/0x1e0 [11443.943613] [] ? call_softirq+0x1c/0x30 [11443.950482] [] local_apic_timer_interrupt+0x37/0x60 [11443.958503] [] smp_apic_timer_interrupt+0x3f/0x60 [11443.966325] [] apic_timer_interrupt+0x6d/0x80 [11443.973744] [] ? osd_inode_iteration+0x271/0xcc0 [osd_ldiskfs] [11443.983561] [] ? osd_inode_iteration+0x269/0xcc0 [osd_ldiskfs] [11443.992658] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [11444.001558] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [11444.010166] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [11444.018568] [] ? wake_up_state+0x20/0x20 [11444.025510] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [11444.034601] [] kthread+0xcf/0xe0 [11444.040765] [] ? kthread_create_on_node+0x140/0x140 [11444.048797] [] ret_from_fork+0x58/0x90 [11444.055544] [] ? kthread_create_on_node+0x140/0x140 [11469.242020] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [11469.251457] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [11469.381350] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [11469.394537] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [11469.410632] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [11469.419936] RIP: 0010:[] [] osd_scrub_next+0xb/0x3c0 [osd_ldiskfs] [11469.431398] RSP: 0018:ffff880703603d00 EFLAGS: 00000287 [11469.438257] RAX: ffffffffa1217a20 RBX: 000000009f8f2001 RCX: ffff880703603d98 [11469.447162] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [11469.456076] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [11469.464972] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [11469.473872] R13: ffff880703603d08 R14: ffff880703603c70 R15: 0000000000000282 [11469.482761] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [11469.492727] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11469.500074] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [11469.508985] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [11469.517879] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [11469.526781] Stack: [11469.529960] 0000000000000001 ffff880703603df0 ffffffffa121cff9 ffffffffa121b990 [11469.539228] ffffffffa1217a20 ffff8800b2a36000 00000000810d354f ffff8803162f6000 [11469.548478] ffff8800b2a37468 0000000020000000 010000000000000c 0000000000000000 [11469.557742] Call Trace: [11469.561434] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [11469.570571] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [11469.579705] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [11469.588545] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [11469.597198] [] ? wake_up_state+0x20/0x20 [11469.604385] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [11469.613705] [] kthread+0xcf/0xe0 [11469.620099] [] ? kthread_create_on_node+0x140/0x140 [11469.628350] [] ret_from_fork+0x58/0x90 [11469.635325] [] ? kthread_create_on_node+0x140/0x140 [11469.643563] Code: 4c 02 10 48 01 c2 e8 25 fc ff ff 48 83 c4 08 5b 41 5c 41 5d 41 5e 5d c3 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 89 e5 41 57 <49> 89 f7 41 56 41 55 49 89 fd 41 54 49 89 d4 53 48 89 cb 48 83 [11497.242988] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [11497.252635] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [11497.384482] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [11497.397767] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [11497.414014] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [11497.423395] RIP: 0010:[] [] osd_scrub_exec+0xf7/0x5b0 [osd_ldiskfs] [11497.435049] RSP: 0018:ffff880703603c78 EFLAGS: 00000246 [11497.441984] RAX: 0000000000280001 RBX: ffff880703603c37 RCX: 0000000000000000 [11497.450958] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [11497.459919] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [11497.468867] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [11497.477814] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [11497.486762] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [11497.496788] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11497.504198] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [11497.513164] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [11497.522119] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [11497.531057] Stack: [11497.534266] 0000000000000004 0000000000000001 ffff880703603df0 ffff880703603d98 [11497.543569] ffff880703603d57 000000009f8f2001 0000000000000004 0000000000000001 [11497.552859] ffff88042bd78800 ffff88042bd78800 00000000000072cf ffff880419dc3a00 [11497.562149] Call Trace: [11497.565869] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [11497.575043] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [11497.584217] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [11497.593069] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [11497.601746] [] ? wake_up_state+0x20/0x20 [11497.608962] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [11497.618314] [] kthread+0xcf/0xe0 [11497.624757] [] ? kthread_create_on_node+0x140/0x140 [11497.633029] [] ret_from_fork+0x58/0x90 [11497.640045] [] ? kthread_create_on_node+0x140/0x140 [11497.648334] Code: 8b 4d 88 48 85 db 74 23 f6 83 18 0a 00 00 08 74 1a 4d 85 e4 74 15 41 8b 87 6c 14 00 00 41 39 84 24 0c 0a 00 00 0f 82 d9 00 00 00 <31> c0 41 f6 87 74 14 00 00 04 0f 85 6e ff ff ff 41 83 f9 02 0f [11525.243964] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [11525.253644] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [11525.385519] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [11525.398806] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [11525.415079] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [11525.424468] RIP: 0010:[] [] osd_inode_iteration+0x26d/0xcc0 [osd_ldiskfs] [11525.436708] RSP: 0018:ffff880703603d18 EFLAGS: 00000202 [11525.443654] RAX: 00000000000072ce RBX: 000000009f8f2001 RCX: ffff88042bd78800 [11525.452633] RDX: 00000000000072cf RSI: ffff880419dc3a00 RDI: ffff88042bd79800 [11525.461601] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [11525.470561] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000001 [11525.479509] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [11525.488455] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [11525.498485] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11525.505891] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [11525.514865] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [11525.523814] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [11525.532750] Stack: [11525.535957] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [11525.545248] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [11525.554543] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [11525.563842] Call Trace: [11525.567565] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [11525.576721] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [11525.585593] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [11525.594255] [] ? wake_up_state+0x20/0x20 [11525.601463] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [11525.610811] [] kthread+0xcf/0xe0 [11525.617246] [] ? kthread_create_on_node+0x140/0x140 [11525.625516] [] ret_from_fork+0x58/0x90 [11525.632522] [] ? kthread_create_on_node+0x140/0x140 [11525.640795] Code: f5 ff 48 85 c0 48 89 45 90 0f 84 b3 08 00 00 44 8b 7d a0 48 8b 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 e8 77 5b f4 ff 42 8d 14 38 <48> 8b 45 88 48 8b 80 a8 03 00 00 48 3b 50 20 0f 83 57 01 00 00 [11553.244946] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [11553.254612] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [11553.386621] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [11553.399926] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [11553.416220] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [11553.425624] RIP: 0010:[] [] osd_inode_iteration+0x264/0xcc0 [osd_ldiskfs] [11553.437863] RSP: 0018:ffff880703603d18 EFLAGS: 00000286 [11553.444826] RAX: ffff8800b2a37468 RBX: 000000009f8f2001 RCX: 0000000000000000 [11553.453811] RDX: ffff880703603d78 RSI: ffff880419dc3a00 RDI: ffff88042bd79800 [11553.462788] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [11553.471747] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffff880703603d57 [11553.480704] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [11553.489659] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [11553.499693] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11553.507106] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [11553.516075] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [11553.525051] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [11553.534010] Stack: [11553.537229] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [11553.546523] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [11553.555813] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [11553.565116] Call Trace: [11553.568860] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [11553.578038] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [11553.586914] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [11553.595608] [] ? wake_up_state+0x20/0x20 [11553.602834] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [11553.612185] [] kthread+0xcf/0xe0 [11553.618626] [] ? kthread_create_on_node+0x140/0x140 [11553.626917] [] ret_from_fork+0x58/0x90 [11553.633941] [] ? kthread_create_on_node+0x140/0x140 [11553.642216] Code: 75 98 48 8b 7d 88 e8 dc 2d f5 ff 48 85 c0 48 89 45 90 0f 84 b3 08 00 00 44 8b 7d a0 48 8b 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 77 5b f4 ff 42 8d 14 38 48 8b 45 88 48 8b 80 a8 03 00 00 48 [11581.245926] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [11581.255597] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [11581.388183] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [11581.401508] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [11581.417832] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [11581.427254] RIP: 0010:[] [] osd_inode_iteration+0x2be/0xcc0 [osd_ldiskfs] [11581.439521] RSP: 0018:ffff880703603d18 EFLAGS: 00000297 [11581.446496] RAX: 0000000000000004 RBX: ffff880703603d98 RCX: ffff880703603d98 [11581.455496] RDX: ffff880703603d78 RSI: 0000000000000000 RDI: ffff8803162f6000 [11581.464495] RBP: ffff880703603df0 R08: 0000000000000001 R09: 0000000000000004 [11581.473469] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000001 [11581.482432] R13: 0000000000000004 R14: 000000009f8f2001 R15: ffffea00027e3c80 [11581.491388] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [11581.501411] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11581.508822] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [11581.517798] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [11581.526762] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [11581.535719] Stack: [11581.538934] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [11581.548237] ffff8803162f6000 ffff8800b2a37468[11581.551605] Lustre: soaked-MDT0000: Connection restored to 91eacee7-b83c-19ec-9326-47c5ebda930d (at 192.168.1.116@o2ib100) 0000000020000000 010000000000000c [11581.571917] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [11581.581232] Call Trace: [11581.584988] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [11581.594185] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [11581.603084] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [11581.611784] [] ? wake_up_state+0x20/0x20 [11581.619025] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [11581.628403] [] kthread+0xcf/0xe0 [11581.634866] [] ? kthread_create_on_node+0x140/0x140 [11581.643171] [] ret_from_fork+0x58/0x90 [11581.650210] [] ? kthread_create_on_node+0x140/0x140 [11581.658511] Code: 8b b5 38 ff ff ff 48 8b bd 48 ff ff ff 48 8b 85 30 ff ff ff ff d0 83 f8 03 41 89 c5 0f 84 4b 02 00 00 0f 8e 15 01 00 00 83 f8 05 <66> 90 0f 84 8a 02 00 00 83 f8 06 0f 85 a1 01 00 00 48 8b 7d 90 [11609.246890] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [11609.256565] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [11609.388541] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [11609.401846] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [11609.418139] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [11609.427539] RIP: 0010:[] [] osd_scrub_next+0x14d/0x3c0 [osd_ldiskfs] [11609.439303] RSP: 0018:ffff880703603c70 EFLAGS: 00000202 [11609.446267] RAX: 0000000000000004 RBX: ffff880703603c37 RCX: ffff880703603d98 [11609.455263] RDX: ffff880703603d78 RSI: 00000000fb6db880 RDI: ffff8803162f6000 [11609.464250] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [11609.473243] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [11609.482223] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [11609.491212] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [11609.501275] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11609.508717] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [11609.517729] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [11609.526721] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [11609.535699] Stack: [11609.538949] 000000009f8f2001 ffff880703603d08 0000000000000001 ffff880703603df0 [11609.548297] 000000009f8f2001 ffff880703603d57 000000009f8f2001 0000000000000004 [11609.557629] ffff880703603d57 ffff88042bd78800 ffff88042bd78800 00000000000072cf [11609.566939] Call Trace: [11609.570685] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [11609.579853] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [11609.589016] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [11609.597879] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [11609.606544] [] ? wake_up_state+0x20/0x20 [11609.613745] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [11609.623086] [] kthread+0xcf/0xe0 [11609.629515] [] ? kthread_create_on_node+0x140/0x140 [11609.637791] [] ret_from_fork+0x58/0x90 [11609.644802] [] ? kthread_create_on_node+0x140/0x140 [11609.653063] Code: 49 8d 9f 10 11 00 00 48 89 df e8 df 6a 47 e0 41 c7 87 58 10 00 00 02 00 00 00 48 89 df e8 fc 69 47 e0 b8 05 00 00 00 48 8b 75 d0 <65> 48 33 34 25 28 00 00 00 0f 85 4c 02 00 00 48 83 c4 70 5b 41 [11623.819412] INFO: rcu_sched self-detected stall on CPU { 24} (t=4200090 jiffies g=907112 c=907111 q=8013065) [11623.831774] Task dump for CPU 24: [11623.836517] OI_scrub R running task 0 5551 2 0x00000088 [11623.845430] ffff88083fde6dd0 00000000fb6db880 ffff88082da03db0 ffffffff810c47e8 [11623.854745] 0000000000000018 ffffffff81a1e7c0 ffff88082da03dc8 ffffffff810c8079 [11623.864041] 0000000000000009 ffff88082da03df8 ffffffff811378e0 ffff88082da101c0 [11623.873308] Call Trace: [11623.876982] [] sched_show_task+0xa8/0x110 [11623.884951] [] dump_cpu_task+0x39/0x70 [11623.891914] [] rcu_dump_cpu_stacks+0x90/0xd0 [11623.899437] [] rcu_check_callbacks+0x442/0x720 [11623.907131] [] ? tick_sched_handle.isra.13+0x60/0x60 [11623.915390] [] update_process_times+0x47/0x80 [11623.922940] [] tick_sched_handle.isra.13+0x25/0x60 [11623.930953] [] tick_sched_timer+0x41/0x70 [11623.938091] [] __hrtimer_run_queues+0xd2/0x260 [11623.945675] [] hrtimer_interrupt+0xb0/0x1e0 [11623.952957] [] ? call_softirq+0x1c/0x30 [11623.959832] [] local_apic_timer_interrupt+0x37/0x60 [11623.967857] [] smp_apic_timer_interrupt+0x3f/0x60 [11623.975670] [] apic_timer_interrupt+0x6d/0x80 [11623.983087] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [11623.992705] [] ? osd_scrub_exec+0x14/0x5b0 [osd_ldiskfs] [11624.001223] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [11624.010119] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [11624.019023] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [11624.027627] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [11624.036028] [] ? wake_up_state+0x20/0x20 [11624.042968] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [11624.052074] [] kthread+0xcf/0xe0 [11624.058252] [] ? kthread_create_on_node+0x140/0x140 [11624.066278] [] ret_from_fork+0x58/0x90 [11624.073032] [] ? kthread_create_on_node+0x140/0x140 [11649.248288] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [11649.257684] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [11649.387709] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [11649.400884] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [11649.416927] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [11649.426214] RIP: 0010:[] [] osd_scrub_next+0x1b/0x3c0 [osd_ldiskfs] [11649.437776] RSP: 0018:ffff880703603ce0 EFLAGS: 00000287 [11649.444639] RAX: ffffffffa1217a20 RBX: ffff880703603c70 RCX: ffff880703603d98 [11649.453542] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [11649.462445] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [11649.471338] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000010 [11649.480237] R13: ffffffffa1217b6d R14: ffffffffffffff10 R15: ffff8803162f6000 [11649.489129] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [11649.499096] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11649.506445] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [11649.515371] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [11649.524283] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [11649.533175] Stack: [11649.536354] ffff880419dc3a00 ffff8800b2a3746c 0000000000000000 00000000ffffffff [11649.545619] 0000000000000001 ffff880703603df0 ffffffffa121cff9 ffffffffa121b990 [11649.554883] ffffffffa1217a20 ffff8800b2a36000 00000000810d354f ffff8803162f6000 [11649.564119] Call Trace: [11649.567819] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [11649.576965] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [11649.586100] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [11649.594948] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [11649.603594] [] ? wake_up_state+0x20/0x20 [11649.610769] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [11649.620093] [] kthread+0xcf/0xe0 [11649.626498] [] ? kthread_create_on_node+0x140/0x140 [11649.634757] [] ret_from_fork+0x58/0x90 [11649.641744] [] ? kthread_create_on_node+0x140/0x140 [11649.649994] Code: 41 5c 41 5d 41 5e 5d c3 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 89 e5 41 57 49 89 f7 41 56 41 55 49 89 fd 41 54 49 89 d4 53 <48> 89 cb 48 83 ec 70 65 48 8b 04 25 28 00 00 00 48 89 45 d0 31 [11677.249258] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [11677.258874] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [11677.390612] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [11677.403910] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [11677.420178] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [11677.429562] RIP: 0010:[] [] osd_inode_iteration+0x489/0xcc0 [osd_ldiskfs] [11677.441813] RSP: 0018:ffff880703603d18 EFLAGS: 00000293 [11677.448762] RAX: 0000000000000004 RBX: 000000009f8f2001 RCX: 0000000000000000 [11677.457747] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [11677.466722] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [11677.475685] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [11677.484643] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [11677.493603] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [11677.503647] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11677.511063] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [11677.520037] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [11677.529004] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [11677.537957] Stack: [11677.541167] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [11677.550460] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [11677.559762] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [11677.569039] Call Trace: [11677.572761] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [11677.581931] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [11677.590807] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [11677.599483] [] ? wake_up_state+0x20/0x20 [11677.606694] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [11677.616060] [] kthread+0xcf/0xe0 [11677.622497] [] ? kthread_create_on_node+0x140/0x140 [11677.630794] [] ret_from_fork+0x58/0x90 [11677.637812] [] ? kthread_create_on_node+0x140/0x140 [11677.646095] Code: 00 e8 7c eb 97 ff e9 0f fc ff ff 0f 1f 80 00 00 00 00 45 89 e9 4c 8d 85 67 ff ff ff 48 8b 4d a8 48 8d 55 88 48 8b b5 38 ff ff ff <48> 8b bd 48 ff ff ff 48 8b 85 28 ff ff ff ff d0 85 c0 41 89 c5 [11705.250235] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [11705.259925] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [11705.391818] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [11705.405125] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [11705.421421] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [11705.430816] RIP: 0010:[] [] osd_inode_iteration+0x2ac/0xcc0 [osd_ldiskfs] [11705.443069] RSP: 0018:ffff880703603d18 EFLAGS: 00000202 [11705.450031] RAX: 0000000000000004 RBX: 000000009f8f2001 RCX: ffff880703603d98 [11705.459011] RDX: ffff880703603d78 RSI: 0000000000000000 RDI: ffff8803162f6000 [11705.467991] RBP: ffff880703603df0 R08: 0000000000000001 R09: 0000000000000004 [11705.476960] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000001 [11705.485914] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [11705.494857] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [11705.504876] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11705.512279] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [11705.521237] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [11705.530194] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [11705.539143] Stack: [11705.542351] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [11705.551644] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [11705.560929] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [11705.570232] Call Trace: [11705.573949] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [11705.583110] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [11705.591976] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [11705.600647] [] ? wake_up_state+0x20/0x20 [11705.607855] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [11705.617221] [] kthread+0xcf/0xe0 [11705.623644] [] ? kthread_create_on_node+0x140/0x140 [11705.631923] [] ret_from_fork+0x58/0x90 [11705.638938] [] ? kthread_create_on_node+0x140/0x140 [11705.647212] Code: 00 44 0f b6 85 67 ff ff ff 48 8d 4d a8 48 8d 55 88 48 8b b5 38 ff ff ff 48 8b bd 48 ff ff ff 48 8b 85 30 ff ff ff ff d0 83 f8 03 <41> 89 c5 0f 84 4b 02 00 00 0f 8e 15 01 00 00 83 f8 05 66 90 0f [11733.251217] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [11733.260874] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [11733.392776] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [11733.406078] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [11733.422367] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [11733.431773] RIP: 0010:[] [] osd_inode_iteration+0x28a/0xcc0 [osd_ldiskfs] [11733.444028] RSP: 0018:ffff880703603d18 EFLAGS: 00000287 [11733.450986] RAX: ffff88042bd78800 RBX: 000000009f8f2001 RCX: ffff88042bd78800 [11733.459970] RDX: 00000000000072cf RSI: ffff880419dc3a00 RDI: ffff88042bd79800 [11733.468946] RBP: ffff880703603df0 R08: 0000000000000001 R09: 0000000000000004 [11733.477916] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [11733.486868] R13: ffff880703603df0 R14: 000000009f8f2001 R15: 0000000000000018 [11733.495827] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [11733.505855] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11733.513283] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [11733.522257] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [11733.531230] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [11733.540186] Stack: [11733.543407] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [11733.552703] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [11733.562008] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [11733.571322] Call Trace: [11733.575061] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [11733.584235] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [11733.593116] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [11733.601803] [] ? wake_up_state+0x20/0x20 [11733.609030] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [11733.618393] [] kthread+0xcf/0xe0 [11733.624830] [] ? kthread_create_on_node+0x140/0x140 [11733.633130] [] ret_from_fork+0x58/0x90 [11733.640140] [] ? kthread_create_on_node+0x140/0x140 [11733.648417] Code: a8 00 00 00 00 e8 77 5b f4 ff 42 8d 14 38 48 8b 45 88 48 8b 80 a8 03 00 00 48 3b 50 20 0f 83 57 01 00 00 44 0f b6 85 67 ff ff ff <48> 8d 4d a8 48 8d 55 88 48 8b b5 38 ff ff ff 48 8b bd 48 ff ff [11761.252185] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [11761.261851] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [11761.393804] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [11761.407118] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [11761.423439] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [11761.432847] RIP: 0010:[] [] osd_scrub_exec+0xd4/0x5b0 [osd_ldiskfs] [11761.444517] RSP: 0018:ffff880703603c78 EFLAGS: 00000286 [11761.451489] RAX: ffff8803162f1008 RBX: ffff880703603c37 RCX: 0000000000000000 [11761.460481] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [11761.469464] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [11761.478430] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [11761.487386] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [11761.496337] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [11761.506374] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11761.513779] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [11761.522754] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [11761.531717] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [11761.540663] Stack: [11761.543883] 0000000000000004 0000000000000001 0000000000000004 ffff880703603d98 [11761.553170] ffff880703603d57 0000000000000000 ffff8803162f6000 ffffffffffffff10 [11761.562455] ffffffffa1217b80 0000000000000010 0000000000000282 ffff880703603ce0 [11761.571750] Call Trace: [11761.575477] [] ? osd_scrub_next+0x160/0x3c0 [osd_ldiskfs] [11761.584344] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [11761.593508] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [11761.602667] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [11761.611529] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [11761.620192] [] ? wake_up_state+0x20/0x20 [11761.627395] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [11761.636738] [] kthread+0xcf/0xe0 [11761.643171] [] ? kthread_create_on_node+0x140/0x140 [11761.651450] [] ret_from_fork+0x58/0x90 [11761.658456] [] ? kthread_create_on_node+0x140/0x140 [11761.666721] Code: 4d 88 4c 89 ef e8 1d fb 46 e0 41 83 87 68 14 00 00 01 49 83 87 88 11 00 00 01 4c 89 ef e8 f5 9f e9 df 44 8b 4d 88 48 85 db 74 23 83 18 0a 00 00 08 74 1a 4d 85 e4 74 15 41 8b 87 6c 14 00 00 [11789.253161] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [11789.262830] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [11789.394327] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [11789.407600] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [11789.423807] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [11789.433180] RIP: 0010:[] [] ldiskfs_itable_unused_count+0x23/0x30 [ldiskfs] [11789.445602] RSP: 0018:ffff880703603d08 EFLAGS: 00000297 [11789.452535] RAX: 0000000000000000 RBX: 000000009f8f2001 RCX: ffff88042bd78800 [11789.461501] RDX: 00000000000072ce RSI: ffff880419dc3a00 RDI: ffff88042bd79800 [11789.470453] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [11789.479403] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [11789.488352] R13: ffff880703603d08 R14: ffff880703603c70 R15: 0000000000000282 [11789.497304] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [11789.507333] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11789.514745] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [11789.523700] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [11789.532649] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [11789.541586] Stack: [11789.544788] ffff880703603df0 ffffffffa121cfb9 ffffffffa121b990 ffffffffa1217a20 [11789.554083] ffff8800b2a36000 00000000810d354f ffff8803162f6000 ffff8800b2a37468 [11789.563376] 0000000020000000 010000000000000c 0000000000000000 0000000000000000 [11789.572667] Call Trace: [11789.576394] [] osd_inode_iteration+0x269/0xcc0 [osd_ldiskfs] [11789.585550] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [11789.594704] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [11789.603562] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [11789.612221] [] ? wake_up_state+0x20/0x20 [11789.619429] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [11789.628775] [] kthread+0xcf/0xe0 [11789.635219] [] ? kthread_create_on_node+0x140/0x140 [11789.643500] [] ret_from_fork+0x58/0x90 [11789.650509] [] ? kthread_create_on_node+0x140/0x140 [11789.658783] Code: 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 8b 8f a8 03 00 00 31 c0 0f b7 56 1c 48 89 e5 48 83 39 3f 76 07 0f b7 46 32 c1 e0 10 <09> d0 5d c3 66 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 89 16 [11803.828669] INFO: rcu_sched self-detected stall on CPU { 24} (t=4380093 jiffies g=907112 c=907111 q=8604937) [11803.841047] Task dump for CPU 24: [11803.845787] OI_scrub R running task 0 5551 2 0x00000088 [11803.854725] ffff88083fde6dd0 00000000fb6db880 ffff88082da03db0 ffffffff810c47e8 [11803.864043] 0000000000000018 ffffffff81a1e7c0 ffff88082da03dc8 ffffffff810c8079 [11803.873358] 0000000000000009 ffff88082da03df8 ffffffff811378e0 ffff88082da101c0 [11803.882648] Call Trace: [11803.886331] [] sched_show_task+0xa8/0x110 [11803.894318] [] dump_cpu_task+0x39/0x70 [11803.901280] [] rcu_dump_cpu_stacks+0x90/0xd0 [11803.908793] [] rcu_check_callbacks+0x442/0x720 [11803.916479] [] ? tick_sched_handle.isra.13+0x60/0x60 [11803.924731] [] update_process_times+0x47/0x80 [11803.932270] [] tick_sched_handle.isra.13+0x25/0x60 [11803.940290] [] tick_sched_timer+0x41/0x70 [11803.947407] [] __hrtimer_run_queues+0xd2/0x260 [11803.954993] [] hrtimer_interrupt+0xb0/0x1e0 [11803.962266] [] ? call_softirq+0x1c/0x30 [11803.969139] [] local_apic_timer_interrupt+0x37/0x60 [11803.977162] [] smp_apic_timer_interrupt+0x3f/0x60 [11803.984992] [] apic_timer_interrupt+0x6d/0x80 [11803.992415] [] ? osd_inode_iteration+0x2be/0xcc0 [osd_ldiskfs] [11804.002231] [] ? osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [11804.011338] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [11804.020243] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [11804.028857] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [11804.037264] [] ? wake_up_state+0x20/0x20 [11804.044214] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [11804.053300] [] kthread+0xcf/0xe0 [11804.059469] [] ? kthread_create_on_node+0x140/0x140 [11804.067490] [] ret_from_fork+0x58/0x90 [11804.074244] [] ? kthread_create_on_node+0x140/0x140 [11829.254554] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [11829.263999] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [11829.393719] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [11829.406897] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [11829.422975] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [11829.432271] RIP: 0010:[] [] osd_scrub_next+0x14d/0x3c0 [osd_ldiskfs] [11829.443935] RSP: 0018:ffff880703603c70 EFLAGS: 00000202 [11829.450794] RAX: 0000000000000004 RBX: ffff880703603c37 RCX: ffff880703603d98 [11829.459690] RDX: ffff880703603d78 RSI: 00000000fb6db880 RDI: ffff8803162f6000 [11829.468590] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [11829.477487] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [11829.486386] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [11829.495281] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [11829.505240] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11829.512586] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [11829.521493] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [11829.530402] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [11829.539301] Stack: [11829.542472] 000000009f8f2001 ffff880703603d08 0000000000000001 ffff880703603df0 [11829.551729] 000000009f8f2001 ffff880703603d57 000000009f8f2001 0000000000000004 [11829.560971] ffff880703603d57 0000000000000000 0000000000000000 ffff880703603d78 [11829.570219] Call Trace: [11829.573910] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [11829.583051] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [11829.592188] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [11829.601026] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [11829.609671] [] ? wake_up_state+0x20/0x20 [11829.616866] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [11829.626179] [] kthread+0xcf/0xe0 [11829.632575] [] ? kthread_create_on_node+0x140/0x140 [11829.640818] [] ret_from_fork+0x58/0x90 [11829.647794] [] ? kthread_create_on_node+0x140/0x140 [11829.656032] Code: 49 8d 9f 10 11 00 00 48 89 df e8 df 6a 47 e0 41 c7 87 58 10 00 00 02 00 00 00 48 89 df e8 fc 69 47 e0 b8 05 00 00 00 48 8b 75 d0 <65> 48 33 34 25 28 00 00 00 0f 85 4c 02 00 00 48 83 c4 70 5b 41 [11857.255529] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [11857.265210] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [11857.396889] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [11857.410172] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [11857.426417] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [11857.435796] RIP: 0010:[] [] osd_inode_iteration+0x28a/0xcc0 [osd_ldiskfs] [11857.448035] RSP: 0018:ffff880703603d18 EFLAGS: 00000287 [11857.454984] RAX: ffff88042bd78800 RBX: 0000000000000001 RCX: ffff88042bd78800 [11857.463968] RDX: 00000000000072cf RSI: ffff880419dc3a00 RDI: ffff88042bd79800 [11857.472931] RBP: ffff880703603df0 R08: 0000000000000001 R09: 0000000000000004 [11857.481886] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [11857.490837] R13: ffffea00027e3c80 R14: 000000009f8f2001 R15: ffff880703603d08 [11857.499795] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [11857.509827] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11857.517238] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [11857.526207] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [11857.535166] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [11857.544112] Stack: [11857.547315] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [11857.556612] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [11857.565914] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [11857.575203] Call Trace: [11857.578923] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [11857.588103] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [11857.596985] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [11857.605668] [] ? wake_up_state+0x20/0x20 [11857.612883] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [11857.622240] [] kthread+0xcf/0xe0 [11857.628678] [] ? kthread_create_on_node+0x140/0x140 [11857.636970] [] ret_from_fork+0x58/0x90 [11857.643990] [] ? kthread_create_on_node+0x140/0x140 [11857.652268] Code: a8 00 00 00 00 e8 77 5b f4 ff 42 8d 14 38 48 8b 45 88 48 8b 80 a8 03 00 00 48 3b 50 20 0f 83 57 01 00 00 44 0f b6 85 67 ff ff ff <48> 8d 4d a8 48 8d 55 88 48 8b b5 38 ff ff ff 48 8b bd 48 ff ff [11885.256505] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [11885.266132] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [11885.398122] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [11885.411444] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [11885.427748] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [11885.437159] RIP: 0010:[] [] osd_scrub_next+0x149/0x3c0 [osd_ldiskfs] [11885.448931] RSP: 0018:ffff880703603c70 EFLAGS: 00000202 [11885.455898] RAX: 0000000000000004 RBX: ffff880703603c37 RCX: ffff880703603d98 [11885.464897] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [11885.473887] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [11885.482855] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [11885.491824] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [11885.500780] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [11885.510815] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11885.518227] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [11885.527196] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [11885.536161] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [11885.545110] Stack: [11885.548325] 000000009f8f2001 0000000000000004 0000000000000001 0000000000000004 [11885.557607] ffff880703603d98 ffff880703603d57 0000000000000000 ffff8803162f6000 [11885.566902] ffffffffffffff10 ffffffffa1217b80 0000000000000010 0000000000000282 [11885.576198] Call Trace: [11885.579930] [] ? osd_scrub_next+0x160/0x3c0 [osd_ldiskfs] [11885.588811] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [11885.597978] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [11885.607142] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [11885.616016] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [11885.624681] [] ? wake_up_state+0x20/0x20 [11885.631889] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [11885.641245] [] kthread+0xcf/0xe0 [11885.647674] [] ? kthread_create_on_node+0x140/0x140 [11885.655956] [] ret_from_fork+0x58/0x90 [11885.662963] [] ? kthread_create_on_node+0x140/0x140 [11885.671223] Code: 45 90 74 49 49 8d 9f 10 11 00 00 48 89 df e8 df 6a 47 e0 41 c7 87 58 10 00 00 02 00 00 00 48 89 df e8 fc 69 47 e0 b8 05 00 00 00 <48> 8b 75 d0 65 48 33 34 25 28 00 00 00 0f 85 4c 02 00 00 48 83 [11911.485625] Lustre: soaked-MDT0000: Connection restored to 91eacee7-b83c-19ec-9326-47c5ebda930d (at 192.168.1.116@o2ib100) [11913.257481] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [11913.267187] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [11913.398846] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [11913.412131] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [11913.428365] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [11913.437745] RIP: 0010:[] [] ldiskfs_itable_unused_count+0x6/0x30 [ldiskfs] [11913.450082] RSP: 0018:ffff880703603d08 EFLAGS: 00000286 [11913.457032] RAX: ffff8800b2a37468 RBX: 000000009f8f2001 RCX: 0000000000000000 [11913.466018] RDX: ffff880703603d78 RSI: ffff880419dc3a00 RDI: ffff88042bd79800 [11913.475000] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [11913.483970] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [11913.492951] R13: 0000000000000018 R14: ffff880703603c78 R15: 0000000000000246 [11913.501931] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [11913.511971] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11913.519381] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [11913.528353] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [11913.537321] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [11913.546285] Stack: [11913.549537] ffff880703603df0 ffffffffa121cfb9 ffffffffa121b990 ffffffffa1217a20 [11913.558831] ffff8800b2a36000 00000000810d354f ffff8803162f6000 ffff8800b2a37468 [11913.568130] 0000000020000000 010000000000000c 0000000000000000 0000000000000000 [11913.577427] Call Trace: [11913.581179] [] ? osd_inode_iteration+0x269/0xcc0 [osd_ldiskfs] [11913.590554] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [11913.599731] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [11913.608621] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [11913.617315] [] ? wake_up_state+0x20/0x20 [11913.624559] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [11913.633932] [] kthread+0xcf/0xe0 [11913.640386] [] ? kthread_create_on_node+0x140/0x140 [11913.648696] [] ret_from_fork+0x58/0x90 [11913.655727] [] ? kthread_create_on_node+0x140/0x140 [11913.664022] Code: 00 00 31 c0 0f b7 56 10 48 89 e5 48 83 39 3f 76 07 0f b7 46 30 c1 e0 10 09 d0 5d c3 66 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 <48> 8b 8f a8 03 00 00 31 c0 0f b7 56 1c 48 89 e5 48 83 39 3f 76 [11941.258456] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [11941.268116] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [11941.399876] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [11941.413170] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [11941.429433] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [11941.438816] RIP: 0010:[] [] osd_inode_iteration+0x473/0xcc0 [osd_ldiskfs] [11941.451057] RSP: 0018:ffff880703603d18 EFLAGS: 00000293 [11941.458005] RAX: 0000000000000004 RBX: 000000009f8f2001 RCX: ffff880703603d98 [11941.466986] RDX: ffff880703603d78 RSI: 0000000000000000 RDI: ffff8803162f6000 [11941.475948] RBP: ffff880703603df0 R08: 0000000000000001 R09: 0000000000000004 [11941.484895] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000001 [11941.493843] R13: ffff880703603df0 R14: 000000009f8f2001 R15: ffffea00027e3c80 [11941.502793] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [11941.512824] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11941.520219] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [11941.529192] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [11941.538151] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [11941.547114] Stack: [11941.550317] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [11941.559612] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [11941.568917] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [11941.578217] Call Trace: [11941.581928] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [11941.591095] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [11941.599968] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [11941.608646] [] ? wake_up_state+0x20/0x20 [11941.615860] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [11941.625212] [] kthread+0xcf/0xe0 [11941.631642] [] ? kthread_create_on_node+0x140/0x140 [11941.639920] [] ret_from_fork+0x58/0x90 [11941.646931] [] ? kthread_create_on_node+0x140/0x140 [11941.655208] Code: 00 00 48 c7 05 9b 86 02 00 00 00 00 00 c7 05 89 86 02 00 01 00 00 00 e8 7c eb 97 ff e9 0f fc ff ff 0f 1f 80 00 00 00 00 45 89 e9 <4c> 8d 85 67 ff ff ff 48 8b 4d a8 48 8d 55 88 48 8b b5 38 ff ff [11969.259438] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [11969.269078] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [11969.401075] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [11969.414380] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [11969.430682] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [11969.440091] RIP: 0010:[] [] osd_inode_iteration+0x473/0xcc0 [osd_ldiskfs] [11969.452346] RSP: 0018:ffff880703603d18 EFLAGS: 00000293 [11969.459314] RAX: 0000000000000004 RBX: 000000009f8f2001 RCX: ffff880703603d98 [11969.468307] RDX: ffff880703603d78 RSI: 0000000000000000 RDI: ffff8803162f6000 [11969.477291] RBP: ffff880703603df0 R08: 0000000000000001 R09: 0000000000000004 [11969.486249] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffff880703603d57 [11969.495200] R13: ffff880703603df0 R14: 000000009f8f2001 R15: 0000000000000018 [11969.504149] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [11969.514174] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11969.521590] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [11969.530567] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [11969.539534] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [11969.548483] Stack: [11969.551701] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [11969.560994] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [11969.570278] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [11969.579595] Call Trace: [11969.583326] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [11969.592500] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [11969.601373] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [11969.610054] [] ? wake_up_state+0x20/0x20 [11969.617281] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [11969.626639] [] kthread+0xcf/0xe0 [11969.633070] [] ? kthread_create_on_node+0x140/0x140 [11969.641359] [] ret_from_fork+0x58/0x90 [11969.648379] [] ? kthread_create_on_node+0x140/0x140 [11969.656654] Code: 00 00 48 c7 05 9b 86 02 00 00 00 00 00 c7 05 89 86 02 00 01 00 00 00 e8 7c eb 97 ff e9 0f fc ff ff 0f 1f 80 00 00 00 00 45 89 e9 <4c> 8d 85 67 ff ff ff 48 8b 4d a8 48 8d 55 88 48 8b b5 38 ff ff [11983.837939] INFO: rcu_sched self-detected stall on CPU { 24} (t=4560096 jiffies g=907112 c=907111 q=9683906) [11983.850317] Task dump for CPU 24: [11983.855078] OI_scrub R running task 0 5551 2 0x00000088 [11983.864020] ffff88083fde6dd0 00000000fb6db880 ffff88082da03db0 ffffffff810c47e8 [11983.873340] 0000000000000018 ffffffff81a1e7c0 ffff88082da03dc8 ffffffff810c8079 [11983.882678] 0000000000000009 ffff88082da03df8 ffffffff811378e0 ffff88082da101c0 [11983.891984] Call Trace: [11983.895688] [] sched_show_task+0xa8/0x110 [11983.903681] [] dump_cpu_task+0x39/0x70 [11983.910657] [] rcu_dump_cpu_stacks+0x90/0xd0 [11983.918202] [] rcu_check_callbacks+0x442/0x720 [11983.925908] [] ? tick_sched_handle.isra.13+0x60/0x60 [11983.934185] [] update_process_times+0x47/0x80 [11983.941761] [] tick_sched_handle.isra.13+0x25/0x60 [11983.949792] [] tick_sched_timer+0x41/0x70 [11983.956927] [] __hrtimer_run_queues+0xd2/0x260 [11983.964536] [] hrtimer_interrupt+0xb0/0x1e0 [11983.971830] [] ? call_softirq+0x1c/0x30 [11983.978726] [] local_apic_timer_interrupt+0x37/0x60 [11983.986780] [] smp_apic_timer_interrupt+0x3f/0x60 [11983.994629] [] apic_timer_interrupt+0x6d/0x80 [11984.002065] [] ? osd_inode_iteration+0x26d/0xcc0 [osd_ldiskfs] [11984.011886] [] ? osd_inode_iteration+0x269/0xcc0 [osd_ldiskfs] [11984.020984] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [11984.029893] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [11984.038509] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [11984.046923] [] ? wake_up_state+0x20/0x20 [11984.053883] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [11984.062973] [] kthread+0xcf/0xe0 [11984.069149] [] ? kthread_create_on_node+0x140/0x140 [11984.077176] [] ret_from_fork+0x58/0x90 [11984.083943] [] ? kthread_create_on_node+0x140/0x140 [12009.260833] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [12009.270264] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [12009.400034] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [12009.413214] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [12009.429289] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [12009.438585] RIP: 0010:[] [] osd_inode_iteration+0x255/0xcc0 [osd_ldiskfs] [12009.450733] RSP: 0018:ffff880703603d18 EFLAGS: 00000286 [12009.457590] RAX: ffff8800b2a37468 RBX: 000000009f8f2001 RCX: 0000000000000000 [12009.466498] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [12009.475408] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [12009.484306] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000001 [12009.493206] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [12009.502103] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [12009.512067] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12009.519409] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [12009.528311] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12009.537215] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [12009.546110] Stack: [12009.549274] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [12009.558498] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [12009.567749] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [12009.577023] Call Trace: [12009.580718] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [12009.589863] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [12009.598709] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [12009.607365] [] ? wake_up_state+0x20/0x20 [12009.614557] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [12009.623894] [] kthread+0xcf/0xe0 [12009.630290] [] ? kthread_create_on_node+0x140/0x140 [12009.638534] [] ret_from_fork+0x58/0x90 [12009.645517] [] ? kthread_create_on_node+0x140/0x140 [12009.653764] Code: 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 00 8b 75 98 48 8b 7d 88 e8 dc 2d f5 ff 48 85 c0 48 89 45 90 0f 84 b3 08 00 00 44 8b 7d a0 <48> 8b 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 e8 77 5b f4 ff 42 [12037.261800] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [12037.271477] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [12037.403502] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [12037.416808] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [12037.433108] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [12037.442509] RIP: 0010:[] [] osd_scrub_next+0x3d/0x3c0 [osd_ldiskfs] [12037.454189] RSP: 0018:ffff880703603c70 EFLAGS: 00000246 [12037.461141] RAX: 0000000000000000 RBX: ffff880703603c37 RCX: ffff880703603d98 [12037.470132] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [12037.479113] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [12037.488064] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [12037.497015] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [12037.505961] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [12037.515985] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12037.523390] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [12037.532356] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12037.541321] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [12037.550271] Stack: [12037.553486] 000000009f8f2001 ffff880703603d08 ffff880703603d57 ffff880703603df0 [12037.562769] 000000009f8f2001 ffff880703603d57 000000009f8f2001 0000000000000004 [12037.572069] ffff880703603d57 ffff88042bd78800 ffff88042bd78800 00000000000072cf [12037.581379] Call Trace: [12037.585103] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [12037.594271] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [12037.603437] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [12037.612306] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [12037.620970] [] ? wake_up_state+0x20/0x20 [12037.628174] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [12037.637518] [] kthread+0xcf/0xe0 [12037.643946] [] ? kthread_create_on_node+0x140/0x140 [12037.652227] [] ret_from_fork+0x58/0x90 [12037.659232] [] ? kthread_create_on_node+0x140/0x140 [12037.667496] Code: 49 89 fd 41 54 49 89 d4 53 48 89 cb 48 83 ec 70 65 48 8b 04 25 28 00 00 00 48 89 45 d0 31 c0 48 8b 05 f8 59 9c ff 48 85 c0 75 7b <41> f6 87 58 10 00 00 08 0f 84 62 03 00 00 4d 8d b7 e0 10 00 00 [12065.262776] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [12065.272470] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [12065.404143] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [12065.417427] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [12065.433670] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [12065.443051] RIP: 0010:[] [] osd_inode_iteration+0x4ac/0xcc0 [osd_ldiskfs] [12065.455282] RSP: 0018:ffff880703603d18 EFLAGS: 00000216 [12065.462227] RAX: 0000000020000000 RBX: 0000000000000001 RCX: 0000000000000000 [12065.471201] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [12065.480165] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [12065.489122] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [12065.498071] R13: ffffea00027e3c80 R14: 000000009f8f2001 R15: 0000000000000018 [12065.507020] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [12065.517046] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12065.524451] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [12065.533414] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12065.542358] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [12065.551296] Stack: [12065.554501] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [12065.563784] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [12065.573083] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [12065.582355] Call Trace: [12065.586069] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [12065.595233] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [12065.604104] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [12065.612774] [] ? wake_up_state+0x20/0x20 [12065.619981] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [12065.629346] [] kthread+0xcf/0xe0 [12065.635776] [] ? kthread_create_on_node+0x140/0x140 [12065.644055] [] ret_from_fork+0x58/0x90 [12065.651070] [] ? kthread_create_on_node+0x140/0x140 [12065.659351] Code: 88 48 8b b5 38 ff ff ff 48 8b bd 48 ff ff ff 48 8b 85 28 ff ff ff ff d0 85 c0 41 89 c5 75 1c 8b 85 58 ff ff ff 41 3b 04 24 72 10 <48> 8b 85 50 ff ff ff 44 3b 30 0f 87 95 fd ff ff 45 31 ff e9 a2 [12078.568324] Lustre: soaked-MDT0000: Client af5c0dbc-bc84-dcde-3672-645b97702b86 (at 192.168.1.122@o2ib100) reconnecting [12078.583209] Lustre: Skipped 2 previous similar messages [12093.263752] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [12093.273463] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [12093.405225] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [12093.418520] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [12093.434800] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [12093.444195] RIP: 0010:[] [] osd_inode_iteration+0x251/0xcc0 [osd_ldiskfs] [12093.456444] RSP: 0018:ffff880703603d18 EFLAGS: 00000286 [12093.463407] RAX: ffff8800b2a37468 RBX: ffff880703603d98 RCX: 0000000000000000 [12093.472402] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [12093.481391] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [12093.490374] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000001 [12093.499355] R13: 0000000000000004 R14: 000000009f8f2001 R15: 0000000000000018 [12093.508335] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [12093.518379] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12093.525807] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [12093.534783] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12093.543764] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [12093.552727] Stack: [12093.555960] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [12093.565270] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [12093.574579] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [12093.583887] Call Trace: [12093.587626] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [12093.596822] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [12093.605706] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [12093.614387] [] ? wake_up_state+0x20/0x20 [12093.621614] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [12093.630969] [] kthread+0xcf/0xe0 [12093.637420] [] ? kthread_create_on_node+0x140/0x140 [12093.645713] [] ret_from_fork+0x58/0x90 [12093.652738] [] ? kthread_create_on_node+0x140/0x140 [12093.661019] Code: b0 00 00 00 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 00 8b 75 98 48 8b 7d 88 e8 dc 2d f5 ff 48 85 c0 48 89 45 90 0f 84 b3 08 00 00 <44> 8b 7d a0 48 8b 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 e8 77 [12121.264726] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [12121.274436] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [12121.406348] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [12121.419646] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [12121.435923] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [12121.445317] RIP: 0010:[] [] osd_inode_iteration+0x473/0xcc0 [osd_ldiskfs] [12121.457556] RSP: 0018:ffff880703603d18 EFLAGS: 00000293 [12121.464516] RAX: 0000000000000004 RBX: 000000009f8f2001 RCX: ffff880703603d98 [12121.473496] RDX: ffff880703603d78 RSI: 0000000000000000 RDI: ffff8803162f6000 [12121.482465] RBP: ffff880703603df0 R08: 0000000000000001 R09: 0000000000000004 [12121.491416] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffff880703603d57 [12121.500355] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [12121.509294] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [12121.519313] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12121.526714] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [12121.535673] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12121.544634] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [12121.553583] Stack: [12121.556798] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [12121.566094] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [12121.575384] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [12121.584697] Call Trace: [12121.588422] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [12121.597590] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [12121.606470] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [12121.615159] [] ? wake_up_state+0x20/0x20 [12121.622374] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [12121.631722] [] kthread+0xcf/0xe0 [12121.638156] [] ? kthread_create_on_node+0x140/0x140 [12121.646434] [] ret_from_fork+0x58/0x90 [12121.653446] [] ? kthread_create_on_node+0x140/0x140 [12121.661726] Code: 00 00 48 c7 05 9b 86 02 00 00 00 00 00 c7 05 89 86 02 00 01 00 00 00 e8 7c eb 97 ff e9 0f fc ff ff 0f 1f 80 00 00 00 00 45 89 e9 <4c> 8d 85 67 ff ff ff 48 8b 4d a8 48 8d 55 88 48 8b b5 38 ff ff [12149.265708] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [12149.275384] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [12149.407428] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [12149.420739] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [12149.437033] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [12149.446437] RIP: 0010:[] [] osd_inode_iteration+0x264/0xcc0 [osd_ldiskfs] [12149.458695] RSP: 0018:ffff880703603d18 EFLAGS: 00000286 [12149.465666] RAX: ffff8800b2a37468 RBX: 000000009f8f2001 RCX: 0000000000000000 [12149.474656] RDX: ffff880703603d78 RSI: ffff880419dc3a00 RDI: ffff88042bd79800 [12149.483632] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [12149.492605] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [12149.501569] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [12149.510513] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [12149.520544] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12149.527948] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [12149.536910] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12149.545875] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [12149.554830] Stack: [12149.558051] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [12149.567344] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [12149.576644] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [12149.585973] Call Trace: [12149.589701] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [12149.598872] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [12149.607747] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [12149.616427] [] ? wake_up_state+0x20/0x20 [12149.623643] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [12149.632992] [] kthread+0xcf/0xe0 [12149.639431] [] ? kthread_create_on_node+0x140/0x140 [12149.647709] [] ret_from_fork+0x58/0x90 [12149.654730] [] ? kthread_create_on_node+0x140/0x140 [12149.663006] Code: 75 98 48 8b 7d 88 e8 dc 2d f5 ff 48 85 c0 48 89 45 90 0f 84 b3 08 00 00 44 8b 7d a0 48 8b 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 77 5b f4 ff 42 8d 14 38 48 8b 45 88 48 8b 80 a8 03 00 00 48 [12163.847209] INFO: rcu_sched self-detected stall on CPU { 24} (t=4740099 jiffies g=907112 c=907111 q=10773653) [12163.859658] Task dump for CPU 24: [12163.864408] OI_scrub R running task 0 5551 2 0x00000088 [12163.873335] ffff88083fde6dd0 00000000fb6db880 ffff88082da03db0 ffffffff810c47e8 [12163.882668] 0000000000000018 ffffffff81a1e7c0 ffff88082da03dc8 ffffffff810c8079 [12163.891978] 0000000000000009 ffff88082da03df8 ffffffff811378e0 ffff88082da101c0 [12163.901285] Call Trace: [12163.904985] [] sched_show_task+0xa8/0x110 [12163.912979] [] dump_cpu_task+0x39/0x70 [12163.919954] [] rcu_dump_cpu_stacks+0x90/0xd0 [12163.927496] [] rcu_check_callbacks+0x442/0x720 [12163.935200] [] ? tick_sched_handle.isra.13+0x60/0x60 [12163.943468] [] update_process_times+0x47/0x80 [12163.951037] [] tick_sched_handle.isra.13+0x25/0x60 [12163.959069] [] tick_sched_timer+0x41/0x70 [12163.966195] [] __hrtimer_run_queues+0xd2/0x260 [12163.973801] [] hrtimer_interrupt+0xb0/0x1e0 [12163.981098] [] ? call_softirq+0x1c/0x30 [12163.987982] [] local_apic_timer_interrupt+0x37/0x60 [12163.996024] [] smp_apic_timer_interrupt+0x3f/0x60 [12164.003852] [] apic_timer_interrupt+0x6d/0x80 [12164.011284] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [12164.020901] [] ? osd_scrub_exec+0x9/0x5b0 [osd_ldiskfs] [12164.029318] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [12164.038222] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [12164.047123] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [12164.055742] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [12164.064144] [] ? wake_up_state+0x20/0x20 [12164.071087] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [12164.080167] [] kthread+0xcf/0xe0 [12164.086334] [] ? kthread_create_on_node+0x140/0x140 [12164.094355] [] ret_from_fork+0x58/0x90 [12164.101106] [] ? kthread_create_on_node+0x140/0x140 [12189.267096] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [12189.276529] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [12189.406129] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [12189.419305] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [12189.435345] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [12189.444628] RIP: 0010:[] [] ldiskfs_itable_unused_count+0x16/0x30 [ldiskfs] [12189.456954] RSP: 0018:ffff880703603d08 EFLAGS: 00000246 [12189.463814] RAX: 0000000000000000 RBX: 0000000000000001 RCX: ffff88042bd78800 [12189.472711] RDX: 00000000000072ce RSI: ffff880419dc3a00 RDI: ffff88042bd79800 [12189.481683] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [12189.490582] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [12189.499473] R13: 0000000000000018 R14: ffff880703603c78 R15: 0000000000000246 [12189.508358] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [12189.518305] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12189.525643] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [12189.534552] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12189.543449] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [12189.552337] Stack: [12189.555497] ffff880703603df0 ffffffffa121cfb9 ffffffffa121b990 ffffffffa1217a20 [12189.564733] ffff8800b2a36000 00000000810d354f ffff8803162f6000 ffff8800b2a37468 [12189.573981] 0000000020000000 010000000000000c 0000000000000000 0000000000000000 [12189.583238] Call Trace: [12189.586933] [] osd_inode_iteration+0x269/0xcc0 [osd_ldiskfs] [12189.596069] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [12189.605196] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [12189.614033] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [12189.622665] [] ? wake_up_state+0x20/0x20 [12189.629831] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [12189.639135] [] kthread+0xcf/0xe0 [12189.645525] [] ? kthread_create_on_node+0x140/0x140 [12189.653757] [] ret_from_fork+0x58/0x90 [12189.660731] [] ? kthread_create_on_node+0x140/0x140 [12189.668965] Code: 07 0f b7 46 30 c1 e0 10 09 d0 5d c3 66 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 8b 8f a8 03 00 00 31 c0 0f b7 56 1c 48 89 e5 <48> 83 39 3f 76 07 0f b7 46 32 c1 e0 10 09 d0 5d c3 66 0f 1f 84 [12217.268072] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [12217.277695] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [12217.409406] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [12217.422696] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [12217.438951] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [12217.448329] RIP: 0010:[] [] osd_inode_iteration+0x2be/0xcc0 [osd_ldiskfs] [12217.460563] RSP: 0018:ffff880703603d18 EFLAGS: 00000297 [12217.467500] RAX: 0000000000000004 RBX: 000000009f8f2001 RCX: ffff880703603d98 [12217.476481] RDX: ffff880703603d78 RSI: 0000000000000000 RDI: ffff8803162f6000 [12217.485454] RBP: ffff880703603df0 R08: 0000000000000001 R09: 0000000000000004 [12217.494409] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffff880703603d57 [12217.503341] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [12217.512287] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [12217.522307] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12217.529715] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [12217.538676] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12217.547633] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [12217.556575] Stack: [12217.559783] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [12217.569064] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [12217.578378] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [12217.587670] Call Trace: [12217.591393] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [12217.600555] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [12217.609427] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [12217.618107] [] ? wake_up_state+0x20/0x20 [12217.625319] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [12217.634667] [] kthread+0xcf/0xe0 [12217.641098] [] ? kthread_create_on_node+0x140/0x140 [12217.649382] [] ret_from_fork+0x58/0x90 [12217.656385] [] ? kthread_create_on_node+0x140/0x140 [12217.664657] Code: 8b b5 38 ff ff ff 48 8b bd 48 ff ff ff 48 8b 85 30 ff ff ff ff d0 83 f8 03 41 89 c5 0f 84 4b 02 00 00 0f 8e 15 01 00 00 83 f8 05 <66> 90 0f 84 8a 02 00 00 83 f8 06 0f 85 a1 01 00 00 48 8b 7d 90 [12245.269046] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [12245.278709] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [12245.410609] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [12245.423919] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [12245.440210] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [12245.449618] RIP: 0010:[] [] osd_inode_iteration+0x282/0xcc0 [osd_ldiskfs] [12245.461873] RSP: 0018:ffff880703603d18 EFLAGS: 00000287 [12245.468836] RAX: ffff88042bd78800 RBX: ffff880703603d98 RCX: ffff88042bd78800 [12245.477826] RDX: 00000000000072cf RSI: ffff880419dc3a00 RDI: ffff88042bd79800 [12245.486806] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [12245.495788] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000001 [12245.504740] R13: 0000000000000004 R14: 000000009f8f2001 R15: 0000000000000018 [12245.513687] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [12245.523707] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12245.531110] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [12245.540069] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12245.549024] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [12245.557964] Stack: [12245.561168] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [12245.570441] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [12245.579727] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [12245.589019] Call Trace: [12245.592742] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [12245.601901] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [12245.610767] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [12245.619436] [] ? wake_up_state+0x20/0x20 [12245.626643] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [12245.635982] [] kthread+0xcf/0xe0 [12245.642404] [] ? kthread_create_on_node+0x140/0x140 [12245.650675] [] ret_from_fork+0x58/0x90 [12245.657680] [] ? kthread_create_on_node+0x140/0x140 [12245.665945] Code: 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 e8 77 5b f4 ff 42 8d 14 38 48 8b 45 88 48 8b 80 a8 03 00 00 48 3b 50 20 0f 83 57 01 00 00 <44> 0f b6 85 67 ff ff ff 48 8d 4d a8 48 8d 55 88 48 8b b5 38 ff [12273.270021] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [12273.279687] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [12273.411523] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [12273.424827] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [12273.441103] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [12273.450509] RIP: 0010:[] [] osd_inode_iteration+0x282/0xcc0 [osd_ldiskfs] [12273.462751] RSP: 0018:ffff880703603d18 EFLAGS: 00000287 [12273.469708] RAX: ffff88042bd78800 RBX: 000000009f8f2001 RCX: ffff88042bd78800 [12273.478690] RDX: 00000000000072cf RSI: ffff880419dc3a00 RDI: ffff88042bd79800 [12273.487671] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [12273.496631] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [12273.505579] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [12273.514526] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [12273.524540] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12273.531943] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [12273.540898] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12273.549861] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [12273.558813] Stack: [12273.562035] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [12273.571327] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [12273.580616] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [12273.589918] Call Trace: [12273.593645] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [12273.602813] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [12273.611690] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [12273.620378] [] ? wake_up_state+0x20/0x20 [12273.627596] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [12273.636938] [] kthread+0xcf/0xe0 [12273.643374] [] ? kthread_create_on_node+0x140/0x140 [12273.651657] [] ret_from_fork+0x58/0x90 [12273.658672] [] ? kthread_create_on_node+0x140/0x140 [12273.666947] Code: 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 e8 77 5b f4 ff 42 8d 14 38 48 8b 45 88 48 8b 80 a8 03 00 00 48 3b 50 20 0f 83 57 01 00 00 <44> 0f b6 85 67 ff ff ff 48 8d 4d a8 48 8d 55 88 48 8b b5 38 ff [12301.270998] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [12301.280618] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [12301.412444] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [12301.425753] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [12301.442050] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [12301.451458] RIP: 0010:[] [] osd_inode_iteration+0x251/0xcc0 [osd_ldiskfs] [12301.463718] RSP: 0018:ffff880703603d18 EFLAGS: 00000286 [12301.470685] RAX: ffff8800b2a37468 RBX: 0000000000000001 RCX: 0000000000000000 [12301.479676] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [12301.488658] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [12301.497625] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [12301.506578] R13: ffffea00027e3c80 R14: 000000009f8f2001 R15: ffff880703603d08 [12301.515532] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [12301.525562] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12301.532972] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [12301.541938] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12301.550902] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [12301.559851] Stack: [12301.563072] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [12301.572358] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [12301.581633] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [12301.590935] Call Trace: [12301.594661] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [12301.603830] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [12301.612714] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [12301.621387] [] ? wake_up_state+0x20/0x20 [12301.628596] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [12301.637954] [] kthread+0xcf/0xe0 [12301.644387] [] ? kthread_create_on_node+0x140/0x140 [12301.652668] [] ret_from_fork+0x58/0x90 [12301.659682] [] ? kthread_create_on_node+0x140/0x140 [12301.667955] Code: b0 00 00 00 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 00 8b 75 98 48 8b 7d 88 e8 dc 2d f5 ff 48 85 c0 48 89 45 90 0f 84 b3 08 00 00 <44> 8b 7d a0 48 8b 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 e8 77 [12329.271972] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [12329.281647] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [12329.413521] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [12329.426829] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [12329.443128] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [12329.452534] RIP: 0010:[] [] osd_inode_iteration+0x251/0xcc0 [osd_ldiskfs] [12329.464794] RSP: 0018:ffff880703603d18 EFLAGS: 00000286 [12329.471758] RAX: ffff8800b2a37468 RBX: 000000009f8f2001 RCX: 0000000000000000 [12329.480750] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [12329.489741] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [12329.498706] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [12329.507666] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [12329.516618] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [12329.526646] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12329.534054] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [12329.543032] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12329.551996] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [12329.560945] Stack: [12329.564158] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [12329.573439] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [12329.582734] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [12329.592026] Call Trace: [12329.595758] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [12329.604922] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [12329.613796] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [12329.622472] [] ? wake_up_state+0x20/0x20 [12329.629688] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [12329.639044] [] kthread+0xcf/0xe0 [12329.645480] [] ? kthread_create_on_node+0x140/0x140 [12329.653759] [] ret_from_fork+0x58/0x90 [12329.660771] [] ? kthread_create_on_node+0x140/0x140 [12329.669042] Code: b0 00 00 00 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 00 8b 75 98 48 8b 7d 88 e8 dc 2d f5 ff 48 85 c0 48 89 45 90 0f 84 b3 08 00 00 <44> 8b 7d a0 48 8b 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 e8 77 [12343.856480] INFO: rcu_sched self-detected stall on CPU { 24} (t=4920102 jiffies g=907112 c=907111 q=11879033) [12343.869035] Task dump for CPU 24: [12343.873790] OI_scrub R running task 0 5551 2 0x00000088 [12343.882719] ffff88083fde6dd0 00000000fb6db880 ffff88082da03db0 ffffffff810c47e8 [12343.892056] 0000000000000018 ffffffff81a1e7c0 ffff88082da03dc8 ffffffff810c8079 [12343.901382] 0000000000000009 ffff88082da03df8 ffffffff811378e0 ffff88082da101c0 [12343.910682] Call Trace: [12343.914382] [] sched_show_task+0xa8/0x110 [12343.922377] [] dump_cpu_task+0x39/0x70 [12343.929357] [] rcu_dump_cpu_stacks+0x90/0xd0 [12343.936892] [] rcu_check_callbacks+0x442/0x720 [12343.944606] [] ? tick_sched_handle.isra.13+0x60/0x60 [12343.952883] [] update_process_times+0x47/0x80 [12343.960453] [] tick_sched_handle.isra.13+0x25/0x60 [12343.968484] [] tick_sched_timer+0x41/0x70 [12343.975611] [] __hrtimer_run_queues+0xd2/0x260 [12343.983210] [] hrtimer_interrupt+0xb0/0x1e0 [12343.990513] [] ? call_softirq+0x1c/0x30 [12343.997392] [] local_apic_timer_interrupt+0x37/0x60 [12344.005425] [] smp_apic_timer_interrupt+0x3f/0x60 [12344.013261] [] apic_timer_interrupt+0x6d/0x80 [12344.020685] [] ? osd_inode_iteration+0x2a0/0xcc0 [osd_ldiskfs] [12344.030498] [] ? osd_inode_iteration+0x269/0xcc0 [osd_ldiskfs] [12344.039595] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [12344.048499] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [12344.057110] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [12344.065524] [] ? wake_up_state+0x20/0x20 [12344.072474] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [12344.081562] [] kthread+0xcf/0xe0 [12344.087730] [] ? kthread_create_on_node+0x140/0x140 [12344.095747] [] ret_from_fork+0x58/0x90 [12344.102497] [] ? kthread_create_on_node+0x140/0x140 [12369.273366] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [12369.282758] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [12369.412406] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [12369.425591] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [12369.441674] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [12369.450969] RIP: 0010:[] [] osd_scrub_exec+0xdb/0x5b0 [osd_ldiskfs] [12369.462530] RSP: 0018:ffff880703603c78 EFLAGS: 00000202 [12369.469387] RAX: ffff8803162f1008 RBX: ffff880703603c37 RCX: 0000000000000000 [12369.478291] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [12369.487181] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [12369.496077] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [12369.504974] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [12369.513868] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [12369.523836] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12369.531179] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [12369.540086] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12369.548994] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [12369.557900] Stack: [12369.561067] ffff880703603d08 0000000000000001 ffff880703603df0 000000009f8f2001 [12369.570305] ffff880703603d57 000000009f8f2001 0000000000000004 ffff880703603d57 [12369.579569] 0000000020000000 0000000000000000 ffff880703603d78 ffff8800b2a36000 [12369.588833] Call Trace: [12369.592510] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [12369.601648] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [12369.610781] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [12369.619631] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [12369.628279] [] ? wake_up_state+0x20/0x20 [12369.635460] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [12369.644772] [] kthread+0xcf/0xe0 [12369.651169] [] ? kthread_create_on_node+0x140/0x140 [12369.659420] [] ret_from_fork+0x58/0x90 [12369.666398] [] ? kthread_create_on_node+0x140/0x140 [12369.674638] Code: fb 46 e0 41 83 87 68 14 00 00 01 49 83 87 88 11 00 00 01 4c 89 ef e8 f5 9f e9 df 44 8b 4d 88 48 85 db 74 23 f6 83 18 0a 00 00 08 <74> 1a 4d 85 e4 74 15 41 8b 87 6c 14 00 00 41 39 84 24 0c 0a 00 [12397.274341] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [12397.283932] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [12397.415627] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [12397.428917] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [12397.445160] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [12397.454541] RIP: 0010:[] [] osd_scrub_next+0x2b/0x3c0 [osd_ldiskfs] [12397.466192] RSP: 0018:ffff880703603c70 EFLAGS: 00000282 [12397.473131] RAX: 00000000fb6db880 RBX: ffff880703603c37 RCX: ffff880703603d98 [12397.482111] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [12397.491073] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [12397.500022] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [12397.508971] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [12397.517920] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [12397.527950] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12397.535357] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [12397.544327] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12397.553279] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [12397.562226] Stack: [12397.565437] 000000009f8f2001 ffff880703603d08 000000009f8f2001 ffff880703603df0 [12397.574726] 000000009f8f2001 ffff880703603d57 000000009f8f2001 0000000000000004 [12397.584016] 0000000000000001 ffff88042bd78800 ffff880703603d98 ffff880703603d78 [12397.593307] Call Trace: [12397.597023] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [12397.606180] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [12397.615334] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [12397.624190] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [12397.632843] [] ? wake_up_state+0x20/0x20 [12397.640049] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [12397.649397] [] kthread+0xcf/0xe0 [12397.655820] [] ? kthread_create_on_node+0x140/0x140 [12397.664088] [] ret_from_fork+0x58/0x90 [12397.671095] [] ? kthread_create_on_node+0x140/0x140 [12397.679353] Code: 66 66 66 66 90 55 48 89 e5 41 57 49 89 f7 41 56 41 55 49 89 fd 41 54 49 89 d4 53 48 89 cb 48 83 ec 70 65 48 8b 04 25 28 00 00 00 <48> 89 45 d0 31 c0 48 8b 05 f8 59 9c ff 48 85 c0 75 7b 41 f6 87 [12425.275316] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [12425.284956] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [12425.416590] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [12425.429868] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [12425.446102] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [12425.455478] RIP: 0010:[] [] osd_inode_iteration+0x473/0xcc0 [osd_ldiskfs] [12425.467709] RSP: 0018:ffff880703603d18 EFLAGS: 00000293 [12425.474646] RAX: 0000000000000004 RBX: ffff880703603d98 RCX: ffff880703603d98 [12425.483619] RDX: ffff880703603d78 RSI: 0000000000000000 RDI: ffff8803162f6000 [12425.492571] RBP: ffff880703603df0 R08: 0000000000000001 R09: 0000000000000004 [12425.501516] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000001 [12425.510462] R13: 0000000000000004 R14: 000000009f8f2001 R15: ffffea00027e3c80 [12425.519394] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [12425.529419] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12425.536811] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [12425.545769] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12425.554719] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [12425.563654] Stack: [12425.566850] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [12425.576120] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [12425.585419] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [12425.594709] Call Trace: [12425.598420] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [12425.607577] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [12425.616458] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [12425.625123] [] ? wake_up_state+0x20/0x20 [12425.632323] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [12425.641670] [] kthread+0xcf/0xe0 [12425.648095] [] ? kthread_create_on_node+0x140/0x140 [12425.656368] [] ret_from_fork+0x58/0x90 [12425.663374] [] ? kthread_create_on_node+0x140/0x140 [12425.671648] Code: 00 00 48 c7 05 9b 86 02 00 00 00 00 00 c7 05 89 86 02 00 01 00 00 00 e8 7c eb 97 ff e9 0f fc ff ff 0f 1f 80 00 00 00 00 45 89 e9 <4c> 8d 85 67 ff ff ff 48 8b 4d a8 48 8d 55 88 48 8b b5 38 ff ff [12453.276293] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [12453.285944] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [12453.417832] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [12453.431135] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [12453.447424] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [12453.456833] RIP: 0010:[] [] osd_scrub_exec+0xf9/0x5b0 [osd_ldiskfs] [12453.468503] RSP: 0018:ffff880703603c78 EFLAGS: 00000246 [12453.475457] RAX: 0000000000000000 RBX: ffff880703603c37 RCX: 0000000000000000 [12453.484446] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [12453.493427] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [12453.502376] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [12453.511326] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [12453.520271] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [12453.530290] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12453.537698] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [12453.546662] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12453.555622] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [12453.564569] Stack: [12453.567783] ffff880703603d08 ffff880703603d57 ffff880703603df0 000000009f8f2001 [12453.577072] ffff880703603d57 000000009f8f2001 0000000000000004 ffff880703603d57 [12453.586364] 0000000000000000 0000000000000000 ffff880703603d78 ffff8800b2a36000 [12453.595661] Call Trace: [12453.599388] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [12453.608556] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [12453.617720] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [12453.626594] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [12453.635283] [] ? wake_up_state+0x20/0x20 [12453.642492] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [12453.651839] [] kthread+0xcf/0xe0 [12453.658267] [] ? kthread_create_on_node+0x140/0x140 [12453.666548] [] ret_from_fork+0x58/0x90 [12453.673559] [] ? kthread_create_on_node+0x140/0x140 [12453.681828] Code: 88 48 85 db 74 23 f6 83 18 0a 00 00 08 74 1a 4d 85 e4 74 15 41 8b 87 6c 14 00 00 41 39 84 24 0c 0a 00 00 0f 82 d9 00 00 00 31 c0 <41> f6 87 74 14 00 00 04 0f 85 6e ff ff ff 41 83 f9 02 0f 84 64 [12481.277268] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [12481.286943] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [12481.418594] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [12481.431881] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [12481.448133] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [12481.457518] RIP: 0010:[] [] osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [12481.469260] RSP: 0018:ffff880703603d10 EFLAGS: 00000287 [12481.476201] RAX: ffffffffa1217a20 RBX: 000000009f8f2001 RCX: ffff880703603d98 [12481.485179] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [12481.494154] RBP: ffff880703603df0 R08: 0000000000000001 R09: 0000000000000004 [12481.503111] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000001 [12481.512062] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [12481.521016] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [12481.531043] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12481.538453] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [12481.547418] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12481.556378] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [12481.565317] Stack: [12481.568530] ffffffffa121cff9 ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 [12481.577816] 00000000810d354f ffff8803162f6000 ffff8800b2a37468 0000000020000000 [12481.587104] 010000000000000c 0000000000000000 0000000000000000 ffff8800b2a36000 [12481.596393] Call Trace: [12481.600111] [] ? osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [12481.609471] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [12481.618626] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [12481.627492] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [12481.636155] [] ? wake_up_state+0x20/0x20 [12481.643372] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [12481.652735] [] kthread+0xcf/0xe0 [12481.659166] [] ? kthread_create_on_node+0x140/0x140 [12481.667445] [] ret_from_fork+0x58/0x90 [12481.674457] [] ? kthread_create_on_node+0x140/0x140 [12481.682727] Code: f7 48 8d 04 80 48 c1 e0 03 48 8d 4c 02 10 48 01 c2 e8 25 fc ff ff 48 83 c4 08 5b 41 5c 41 5d 41 5e 5d c3 0f 1f 84 00 00 00 00 00 <66> 66 66 66 90 55 48 89 e5 41 57 49 89 f7 41 56 41 55 49 89 fd [12509.278242] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [12509.287999] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [12509.419781] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [12509.433072] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [12509.449331] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [12509.458717] RIP: 0010:[] [] osd_inode_iteration+0x269/0xcc0 [osd_ldiskfs] [12509.470976] RSP: 0018:ffff880703603d18 EFLAGS: 00000202 [12509.477927] RAX: 00000000000072ce RBX: 000000009f8f2001 RCX: ffff88042bd78800 [12509.486910] RDX: 00000000000072ce RSI: ffff880419dc3a00 RDI: ffff88042bd79800 [12509.495870] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [12509.504835] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffff880703603d57 [12509.513787] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [12509.522749] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [12509.532781] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12509.540192] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [12509.549164] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12509.558126] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [12509.567076] Stack: [12509.570292] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [12509.579599] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [12509.588895] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [12509.598204] Call Trace: [12509.601929] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [12509.611098] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [12509.619976] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [12509.628649] [] ? wake_up_state+0x20/0x20 [12509.635859] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [12509.645198] [] kthread+0xcf/0xe0 [12509.651630] [] ? kthread_create_on_node+0x140/0x140 [12509.659915] [] ret_from_fork+0x58/0x90 [12509.666931] [] ? kthread_create_on_node+0x140/0x140 [12509.675212] Code: 88 e8 dc 2d f5 ff 48 85 c0 48 89 45 90 0f 84 b3 08 00 00 44 8b 7d a0 48 8b 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 e8 77 5b f4 ff <42> 8d 14 38 48 8b 45 88 48 8b 80 a8 03 00 00 48 3b 50 20 0f 83 [12523.865751] INFO: rcu_sched self-detected stall on CPU { 24} (t=5100105 jiffies g=907112 c=907111 q=13351514) [12523.878243] Task dump for CPU 24: [12523.882999] OI_scrub R running task 0 5551 2 0x00000088 [12523.891935] ffff88083fde6dd0 00000000fb6db880 ffff88082da03db0 ffffffff810c47e8 [12523.901272] 0000000000000018 ffffffff81a1e7c0 ffff88082da03dc8 ffffffff810c8079 [12523.910592] 0000000000000009 ffff88082da03df8 ffffffff811378e0 ffff88082da101c0 [12523.919898] Call Trace: [12523.923604] [] sched_show_task+0xa8/0x110 [12523.931603] [] dump_cpu_task+0x39/0x70 [12523.938580] [] rcu_dump_cpu_stacks+0x90/0xd0 [12523.946114] [] rcu_check_callbacks+0x442/0x720 [12523.953822] [] ? tick_sched_handle.isra.13+0x60/0x60 [12523.962101] [] update_process_times+0x47/0x80 [12523.969667] [] tick_sched_handle.isra.13+0x25/0x60 [12523.977699] [] tick_sched_timer+0x41/0x70 [12523.984846] [] __hrtimer_run_queues+0xd2/0x260 [12523.992449] [] hrtimer_interrupt+0xb0/0x1e0 [12523.999747] [] ? call_softirq+0x1c/0x30 [12524.006630] [] local_apic_timer_interrupt+0x37/0x60 [12524.014667] [] smp_apic_timer_interrupt+0x3f/0x60 [12524.022496] [] apic_timer_interrupt+0x6d/0x80 [12524.029923] [] ? osd_inode_iteration+0x4a0/0xcc0 [osd_ldiskfs] [12524.039746] [] ? osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [12524.048850] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [12524.057758] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [12524.066383] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [12524.074797] [] ? wake_up_state+0x20/0x20 [12524.081760] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [12524.090848] [] kthread+0xcf/0xe0 [12524.097022] [] ? kthread_create_on_node+0x140/0x140 [12524.105032] [] ret_from_fork+0x58/0x90 [12524.111793] [] ? kthread_create_on_node+0x140/0x140 [12549.279635] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [12549.289106] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [12549.418758] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [12549.431934] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [12549.448014] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [12549.457313] RIP: 0010:[] [] osd_scrub_exec+0x34/0x5b0 [osd_ldiskfs] [12549.468877] RSP: 0018:ffff880703603c78 EFLAGS: 00000296 [12549.475738] RAX: 00000000fb6db880 RBX: ffff880703603c37 RCX: 0000000000000000 [12549.484649] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [12549.493549] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [12549.502448] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [12549.511348] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [12549.520242] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [12549.530209] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12549.537560] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [12549.546466] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12549.555373] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [12549.564280] Stack: [12549.567450] 0000000000000004 ffff880703603d57 ffff880703603df0 0000000000000000 [12549.576720] ffff880703603d57 000000009f8f2001 0000000000000004 ffff880703603d57 [12549.585977] ffff8800b2a37468 0000000000000000 ffff880703603d78 ffff880419dc3a00 [12549.595242] Call Trace: [12549.598946] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [12549.608087] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [12549.617232] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [12549.626075] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [12549.634746] [] ? wake_up_state+0x20/0x20 [12549.641933] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [12549.651250] [] kthread+0xcf/0xe0 [12549.657645] [] ? kthread_create_on_node+0x140/0x140 [12549.665903] [] ret_from_fork+0x58/0x90 [12549.672891] [] ? kthread_create_on_node+0x140/0x140 [12549.681135] Code: 41 57 49 89 f7 41 56 41 55 41 54 45 31 e4 53 48 83 ec 68 48 8b 9e 08 10 00 00 4c 89 45 90 65 48 8b 04 25 28 00 00 00 48 89 45 d0 <31> c0 48 8d 43 08 48 85 db 4c 0f 45 e0 41 83 f9 04 0f 84 84 00 [12577.280611] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [12577.290217] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [12577.421899] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [12577.435188] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [12577.451440] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [12577.460825] RIP: 0010:[] [] osd_inode_iteration+0x26d/0xcc0 [osd_ldiskfs] [12577.473065] RSP: 0018:ffff880703603d18 EFLAGS: 00000202 [12577.480009] RAX: 00000000000072ce RBX: 000000009f8f2001 RCX: ffff88042bd78800 [12577.488997] RDX: 00000000000072cf RSI: ffff880419dc3a00 RDI: ffff88042bd79800 [12577.497963] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [12577.506921] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [12577.515876] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [12577.524824] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [12577.534868] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12577.542295] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [12577.551267] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12577.560229] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [12577.569172] Stack: [12577.572389] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [12577.581679] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [12577.590974] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [12577.600267] Call Trace: [12577.603985] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [12577.613147] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [12577.622023] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [12577.630693] [] ? wake_up_state+0x20/0x20 [12577.637897] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [12577.647249] [] kthread+0xcf/0xe0 [12577.653668] [] ? kthread_create_on_node+0x140/0x140 [12577.661952] [] ret_from_fork+0x58/0x90 [12577.668963] [] ? kthread_create_on_node+0x140/0x140 [12577.677233] Code: f5 ff 48 85 c0 48 89 45 90 0f 84 b3 08 00 00 44 8b 7d a0 48 8b 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 e8 77 5b f4 ff 42 8d 14 38 <48> 8b 45 88 48 8b 80 a8 03 00 00 48 3b 50 20 0f 83 57 01 00 00 [12605.281586] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [12605.291246] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [12605.423158] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [12605.436467] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [12605.452748] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [12605.462155] RIP: 0010:[] [] osd_scrub_exec+0xf7/0x5b0 [osd_ldiskfs] [12605.473822] RSP: 0018:ffff880703603c78 EFLAGS: 00000246 [12605.480785] RAX: 0000000000280001 RBX: ffff880703603c37 RCX: 0000000000000000 [12605.489762] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [12605.498735] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [12605.507692] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [12605.516649] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [12605.525611] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [12605.535643] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12605.543057] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [12605.552030] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12605.560998] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [12605.569947] Stack: [12605.573159] ffffea00027e3c80 000000009f8f2001 ffff880703603df0 ffff880703603d57 [12605.582446] ffff880703603d57 000000009f8f2001 0000000000000004 ffff880703603d57 [12605.591716] ffff88042bd78800 ffff88042bd78800 00000000000072cf ffff880419dc3a00 [12605.600992] Call Trace: [12605.604718] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [12605.613876] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [12605.623028] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [12605.631892] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [12605.640563] [] ? wake_up_state+0x20/0x20 [12605.647762] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [12605.657099] [] kthread+0xcf/0xe0 [12605.663523] [] ? kthread_create_on_node+0x140/0x140 [12605.671792] [] ret_from_fork+0x58/0x90 [12605.678792] [] ? kthread_create_on_node+0x140/0x140 [12605.687051] Code: 8b 4d 88 48 85 db 74 23 f6 83 18 0a 00 00 08 74 1a 4d 85 e4 74 15 41 8b 87 6c 14 00 00 41 39 84 24 0c 0a 00 00 0f 82 d9 00 00 00 <31> c0 41 f6 87 74 14 00 00 04 0f 85 6e ff ff ff 41 83 f9 02 0f [12633.282562] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [12633.292237] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [12633.423764] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [12633.437043] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [12633.453282] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [12633.462658] RIP: 0010:[] [] osd_inode_iteration+0x4a6/0xcc0 [osd_ldiskfs] [12633.474903] RSP: 0018:ffff880703603d18 EFLAGS: 00000246 [12633.481828] RAX: 0000000020000000 RBX: ffff880703603d98 RCX: 0000000000000000 [12633.490801] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [12633.499753] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [12633.508700] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000001 [12633.517632] R13: 0000000000000004 R14: 000000009f8f2001 R15: ffffea00027e3c80 [12633.526578] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [12633.536594] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12633.543989] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [12633.552935] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12633.561877] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [12633.570807] Stack: [12633.574011] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [12633.583290] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [12633.592580] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [12633.601851] Call Trace: [12633.605552] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [12633.614701] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [12633.623566] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [12633.632236] [] ? wake_up_state+0x20/0x20 [12633.639443] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [12633.648784] [] kthread+0xcf/0xe0 [12633.655212] [] ? kthread_create_on_node+0x140/0x140 [12633.663498] [] ret_from_fork+0x58/0x90 [12633.670505] [] ? kthread_create_on_node+0x140/0x140 [12633.678779] Code: 8b 4d a8 48 8d 55 88 48 8b b5 38 ff ff ff 48 8b bd 48 ff ff ff 48 8b 85 28 ff ff ff ff d0 85 c0 41 89 c5 75 1c 8b 85 58 ff ff ff <41> 3b 04 24 72 10 48 8b 85 50 ff ff ff 44 3b 30 0f 87 95 fd ff [12661.283537] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [12661.293223] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [12661.425131] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [12661.438431] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [12661.454723] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [12661.464126] RIP: 0010:[] [] osd_inode_iteration+0x251/0xcc0 [osd_ldiskfs] [12661.476353] RSP: 0018:ffff880703603d18 EFLAGS: 00000286 [12661.483309] RAX: ffff8800b2a37468 RBX: ffff880703603d57 RCX: 0000000000000000 [12661.492303] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [12661.501285] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [12661.510233] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [12661.519179] R13: ffffea00027e3c80 R14: 000000009f8f2001 R15: ffff880703603d08 [12661.528120] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [12661.538139] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12661.545537] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [12661.554511] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12661.563553] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [12661.572499] Stack: [12661.575708] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [12661.585004] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [12661.594286] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [12661.603575] Call Trace: [12661.607308] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [12661.616472] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [12661.625344] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [12661.634020] [] ? wake_up_state+0x20/0x20 [12661.641240] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [12661.650600] [] kthread+0xcf/0xe0 [12661.657035] [] ? kthread_create_on_node+0x140/0x140 [12661.665326] [] ret_from_fork+0x58/0x90 [12661.672336] [] ? kthread_create_on_node+0x140/0x140 [12661.680605] Code: b0 00 00 00 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 00 8b 75 98 48 8b 7d 88 e8 dc 2d f5 ff 48 85 c0 48 89 45 90 0f 84 b3 08 00 00 <44> 8b 7d a0 48 8b 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 e8 77 [12689.284512] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [12689.294162] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [12689.426076] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [12689.439391] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [12689.455681] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [12689.465097] RIP: 0010:[] [] osd_inode_iteration+0x255/0xcc0 [osd_ldiskfs] [12689.477362] RSP: 0018:ffff880703603d18 EFLAGS: 00000286 [12689.484327] RAX: ffff8800b2a37468 RBX: ffff880703603d98 RCX: 0000000000000000 [12689.493318] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [12689.502297] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [12689.511269] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000001 [12689.520212] R13: 0000000000000004 R14: 000000009f8f2001 R15: 0000000000000018 [12689.529156] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [12689.539174] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12689.546580] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [12689.555546] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12689.564499] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [12689.573444] Stack: [12689.576647] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [12689.585918] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [12689.595201] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [12689.604497] Call Trace: [12689.608212] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [12689.617367] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [12689.626230] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [12689.634905] [] ? wake_up_state+0x20/0x20 [12689.642111] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [12689.651455] [] kthread+0xcf/0xe0 [12689.657877] [] ? kthread_create_on_node+0x140/0x140 [12689.666146] [] ret_from_fork+0x58/0x90 [12689.673148] [] ? kthread_create_on_node+0x140/0x140 [12689.681410] Code: 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 00 8b 75 98 48 8b 7d 88 e8 dc 2d f5 ff 48 85 c0 48 89 45 90 0f 84 b3 08 00 00 44 8b 7d a0 <48> 8b 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 e8 77 5b f4 ff 42 [12703.875030] INFO: rcu_sched self-detected stall on CPU { 24} (t=5280108 jiffies g=907112 c=907111 q=14494531) [12703.887510] Task dump for CPU 24: [12703.892253] OI_scrub R running task 0 5551 2 0x00000088 [12703.901175] ffff88083fde6dd0 00000000fb6db880 ffff88082da03db0 ffffffff810c47e8 [12703.910512] 0000000000000018 ffffffff81a1e7c0 ffff88082da03dc8 ffffffff810c8079 [12703.919832] 0000000000000009 ffff88082da03df8 ffffffff811378e0 ffff88082da101c0 [12703.929123] Call Trace: [12703.932815] [] sched_show_task+0xa8/0x110 [12703.940816] [] dump_cpu_task+0x39/0x70 [12703.947781] [] rcu_dump_cpu_stacks+0x90/0xd0 [12703.955307] [] rcu_check_callbacks+0x442/0x720 [12703.963013] [] ? tick_sched_handle.isra.13+0x60/0x60 [12703.971274] [] update_process_times+0x47/0x80 [12703.978833] [] tick_sched_handle.isra.13+0x25/0x60 [12703.986856] [] tick_sched_timer+0x41/0x70 [12703.993982] [] __hrtimer_run_queues+0xd2/0x260 [12704.001575] [] hrtimer_interrupt+0xb0/0x1e0 [12704.008861] [] ? call_softirq+0x1c/0x30 [12704.015736] [] local_apic_timer_interrupt+0x37/0x60 [12704.023779] [] smp_apic_timer_interrupt+0x3f/0x60 [12704.031600] [] apic_timer_interrupt+0x6d/0x80 [12704.039015] [] ? osd_scrub_next+0x350/0x3c0 [osd_ldiskfs] [12704.048335] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [12704.057228] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [12704.066119] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [12704.074720] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [12704.083135] [] ? wake_up_state+0x20/0x20 [12704.090088] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [12704.099171] [] kthread+0xcf/0xe0 [12704.105343] [] ? kthread_create_on_node+0x140/0x140 [12704.113363] [] ret_from_fork+0x58/0x90 [12704.120119] [] ? kthread_create_on_node+0x140/0x140 [12719.514071] Lustre: soaked-MDT0000: Client e211cd09-6b0c-e43b-059c-f657101042a0 (at 192.168.1.117@o2ib100) reconnecting [12719.528038] Lustre: Skipped 1 previous similar message [12719.535516] Lustre: soaked-MDT0000: Connection restored to e211cd09-6b0c-e43b-059c-f657101042a0 (at 192.168.1.117@o2ib100) [12719.549633] Lustre: Skipped 2 previous similar messages [12729.285906] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [12729.295372] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [12729.425421] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [12729.438612] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [12729.454695] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [12729.464008] RIP: 0010:[] [] osd_scrub_exec+0xf7/0x5b0 [osd_ldiskfs] [12729.475608] RSP: 0018:ffff880703603c78 EFLAGS: 00000246 [12729.482513] RAX: 0000000000280001 RBX: ffff880703603c37 RCX: 0000000000000000 [12729.491460] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [12729.500403] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [12729.509342] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [12729.518288] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [12729.527243] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [12729.537268] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12729.544670] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [12729.553640] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12729.562613] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [12729.571570] Stack: [12729.574799] 0000000000000004 0000000000000001 ffff880703603df0 ffff880703603d98 [12729.584113] ffff880703603d57 000000009f8f2001 0000000000000004 ffff880703603d57 [12729.593437] 0000000020000000 0000000000000000 ffff880703603d78 ffff8800b2a36000 [12729.602730] Call Trace: [12729.606484] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [12729.615682] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [12729.624855] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [12729.633748] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [12729.642437] [] ? wake_up_state+0x20/0x20 [12729.649675] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [12729.659057] [] kthread+0xcf/0xe0 [12729.665516] [] ? kthread_create_on_node+0x140/0x140 [12729.673814] [] ret_from_fork+0x58/0x90 [12729.680845] [] ? kthread_create_on_node+0x140/0x140 [12729.689137] Code: 8b 4d 88 48 85 db 74 23 f6 83 18 0a 00 00 08 74 1a 4d 85 e4 74 15 41 8b 87 6c 14 00 00 41 39 84 24 0c 0a 00 00 0f 82 d9 00 00 00 <31> c0 41 f6 87 74 14 00 00 04 0f 85 6e ff ff ff 41 83 f9 02 0f [12757.286890] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [12757.296557] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [12757.428393] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [12757.441679] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [12757.457938] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [12757.467325] RIP: 0010:[] [] osd_inode_iteration+0x28a/0xcc0 [osd_ldiskfs] [12757.479563] RSP: 0018:ffff880703603d18 EFLAGS: 00000287 [12757.486502] RAX: ffff88042bd78800 RBX: 000000009f8f2001 RCX: ffff88042bd78800 [12757.495480] RDX: 00000000000072cf RSI: ffff880419dc3a00 RDI: ffff88042bd79800 [12757.504445] RBP: ffff880703603df0 R08: 0000000000000001 R09: 0000000000000004 [12757.513417] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [12757.522378] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [12757.531338] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [12757.541375] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12757.548792] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [12757.557774] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12757.566739] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [12757.575691] Stack: [12757.578901] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [12757.588212] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [12757.597512] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [12757.606825] Call Trace: [12757.610558] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [12757.619733] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [12757.628605] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [12757.637285] [] ? wake_up_state+0x20/0x20 [12757.644500] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [12757.653860] [] kthread+0xcf/0xe0 [12757.660300] [] ? kthread_create_on_node+0x140/0x140 [12757.668588] [] ret_from_fork+0x58/0x90 [12757.675610] [] ? kthread_create_on_node+0x140/0x140 [12757.683891] Code: a8 00 00 00 00 e8 77 5b f4 ff 42 8d 14 38 48 8b 45 88 48 8b 80 a8 03 00 00 48 3b 50 20 0f 83 57 01 00 00 44 0f b6 85 67 ff ff ff <48> 8d 4d a8 48 8d 55 88 48 8b b5 38 ff ff ff 48 8b bd 48 ff ff [12785.287856] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [12785.297507] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [12785.429694] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [12785.443006] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [12785.459313] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [12785.468721] RIP: 0010:[] [] osd_scrub_next+0x149/0x3c0 [osd_ldiskfs] [12785.480494] RSP: 0018:ffff880703603c70 EFLAGS: 00000202 [12785.487458] RAX: 0000000000000004 RBX: ffff880703603c37 RCX: ffff880703603d98 [12785.496448] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [12785.505444] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [12785.514424] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [12785.523380] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [12785.532334] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [12785.542373] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12785.549771] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [12785.558737] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12785.567705] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [12785.576655] Stack: [12785.579869] 000000009f8f2001 ffff880703603d08 000000009f8f2001 ffff880703603df0 [12785.589157] 000000009f8f2001 ffff880703603d57 000000009f8f2001 0000000000000004 [12785.598432] ffff880703603d57 ffff88042bd78800 ffff88042bd78800 00000000000072cf [12785.607731] Call Trace: [12785.611459] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [12785.620619] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [12785.629785] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [12785.638657] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [12785.647320] [] ? wake_up_state+0x20/0x20 [12785.654520] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [12785.663863] [] kthread+0xcf/0xe0 [12785.670285] [] ? kthread_create_on_node+0x140/0x140 [12785.678560] [] ret_from_fork+0x58/0x90 [12785.685563] [] ? kthread_create_on_node+0x140/0x140 [12785.693825] Code: 45 90 74 49 49 8d 9f 10 11 00 00 48 89 df e8 df 6a 47 e0 41 c7 87 58 10 00 00 02 00 00 00 48 89 df e8 fc 69 47 e0 b8 05 00 00 00 <48> 8b 75 d0 65 48 33 34 25 28 00 00 00 0f 85 4c 02 00 00 48 83 [12813.288831] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [12813.298478] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [12813.430275] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [12813.443548] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [12813.459775] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [12813.469162] RIP: 0010:[] [] osd_inode_iteration+0x4ac/0xcc0 [osd_ldiskfs] [12813.481399] RSP: 0018:ffff880703603d18 EFLAGS: 00000216 [12813.488338] RAX: 0000000020000000 RBX: 000000009f8f2001 RCX: 0000000000000000 [12813.497322] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [12813.506290] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [12813.515239] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [12813.524182] R13: ffff880703603d08 R14: 000000009f8f2001 R15: ffff880703603d08 [12813.533134] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [12813.543159] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12813.550560] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [12813.559527] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12813.568488] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [12813.577440] Stack: [12813.580658] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [12813.589956] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [12813.599245] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [12813.608541] Call Trace: [12813.612259] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [12813.621426] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [12813.630305] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [12813.638977] [] ? wake_up_state+0x20/0x20 [12813.646185] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [12813.655544] [] kthread+0xcf/0xe0 [12813.661980] [] ? kthread_create_on_node+0x140/0x140 [12813.670270] [] ret_from_fork+0x58/0x90 [12813.677287] [] ? kthread_create_on_node+0x140/0x140 [12813.685571] Code: 88 48 8b b5 38 ff ff ff 48 8b bd 48 ff ff ff 48 8b 85 28 ff ff ff ff d0 85 c0 41 89 c5 75 1c 8b 85 58 ff ff ff 41 3b 04 24 72 10 <48> 8b 85 50 ff ff ff 44 3b 30 0f 87 95 fd ff ff 45 31 ff e9 a2 [12841.289821] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [12841.299509] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [12841.431746] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [12841.445057] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [12841.461354] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [12841.470758] RIP: 0010:[] [] osd_scrub_exec+0x3d/0x5b0 [osd_ldiskfs] [12841.482429] RSP: 0018:ffff880703603c78 EFLAGS: 00000286 [12841.489397] RAX: ffff8803162f1008 RBX: ffff880703603c37 RCX: 0000000000000000 [12841.498393] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [12841.507381] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [12841.516348] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [12841.525307] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [12841.534257] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [12841.544282] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12841.551687] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [12841.560652] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12841.569617] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [12841.578567] Stack: [12841.581796] 0000000000000004 0000000000000001 ffffffffa1217a20 ffff880703603d98 [12841.591080] ffff880703603d57 ffff8800b2a36000 ffff8803162f6000 ffffffffffffff10 [12841.600387] ffffffffa1217a3b 0000000000000010 0000000000000287 ffff880703603ce0 [12841.609676] Call Trace: [12841.613407] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [12841.622280] [] ? osd_scrub_next+0x1b/0x3c0 [osd_ldiskfs] [12841.631058] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [12841.640222] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [12841.649385] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [12841.658248] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [12841.666917] [] ? wake_up_state+0x20/0x20 [12841.674120] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [12841.683472] [] kthread+0xcf/0xe0 [12841.689903] [] ? kthread_create_on_node+0x140/0x140 [12841.698173] [] ret_from_fork+0x58/0x90 [12841.705185] [] ? kthread_create_on_node+0x140/0x140 [12841.713459] Code: 41 54 45 31 e4 53 48 83 ec 68 48 8b 9e 08 10 00 00 4c 89 45 90 65 48 8b 04 25 28 00 00 00 48 89 45 d0 31 c0 48 8d 43 08 48 85 db <4c> 0f 45 e0 41 83 f9 04 0f 84 84 00 00 00 41 83 f9 07 74 4f 41 [12869.290789] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [12869.300417] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [12869.432078] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [12869.445342] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [12869.461545] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [12869.470902] RIP: 0010:[] [] osd_scrub_exec+0x95/0x5b0 [osd_ldiskfs] [12869.482531] RSP: 0018:ffff880703603d08 EFLAGS: 00000292 [12869.489450] RAX: 0000000000000000 RBX: ffff880703603d57 RCX: 0000000000000000 [12869.498414] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [12869.507365] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [12869.516324] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [12869.525282] R13: ffffea00027e3c80 R14: ffff880703603c70 R15: ffff880703603d08 [12869.534242] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [12869.544278] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12869.551685] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [12869.560647] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12869.569596] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [12869.578546] Stack: [12869.581779] ffff880703603df0 ffffffffa121d1e9 ffffffffa121b990 ffffffffa1217a20 [12869.591092] ffff8800b2a36000 00000000810d354f ffff8803162f6000 ffff8800b2a37468 [12869.600379] 0000000020000000 010000000000000c 0000000000000000 0000000000000000 [12869.609656] Call Trace: [12869.613376] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [12869.622540] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [12869.631688] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [12869.640543] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [12869.649222] [] ? wake_up_state+0x20/0x20 [12869.656418] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [12869.665769] [] kthread+0xcf/0xe0 [12869.672207] [] ? kthread_create_on_node+0x140/0x140 [12869.680491] [] ret_from_fork+0x58/0x90 [12869.687506] [] ? kthread_create_on_node+0x140/0x140 [12869.695792] Code: 01 00 00 41 80 a7 74 14 00 00 fe 48 8b 4d d0 65 48 33 0c 25 28 00 00 00 0f 85 b1 03 00 00 48 83 c4 68 5b 41 5c 41 5d 41 5e 41 5f <5d> c3 66 0f 1f 84 00 00 00 00 00 4c 8d ae f0 10 00 00 44 89 4d [12883.884298] INFO: rcu_sched self-detected stall on CPU { 24} (t=5460111 jiffies g=907112 c=907111 q=15277119) [12883.896715] Task dump for CPU 24: [12883.901434] OI_scrub R running task 0 5551 2 0x00000088 [12883.910341] ffff88083fde6dd0 00000000fb6db880 ffff88082da03db0 ffffffff810c47e8 [12883.919661] 0000000000000018 ffffffff81a1e7c0 ffff88082da03dc8 ffffffff810c8079 [12883.928958] 0000000000000009 ffff88082da03df8 ffffffff811378e0 ffff88082da101c0 [12883.938250] Call Trace: [12883.941932] [] sched_show_task+0xa8/0x110 [12883.949919] [] dump_cpu_task+0x39/0x70 [12883.956871] [] rcu_dump_cpu_stacks+0x90/0xd0 [12883.964384] [] rcu_check_callbacks+0x442/0x720 [12883.972074] [] ? tick_sched_handle.isra.13+0x60/0x60 [12883.980327] [] update_process_times+0x47/0x80 [12883.987880] [] tick_sched_handle.isra.13+0x25/0x60 [12883.995890] [] tick_sched_timer+0x41/0x70 [12884.003012] [] __hrtimer_run_queues+0xd2/0x260 [12884.010600] [] hrtimer_interrupt+0xb0/0x1e0 [12884.017871] [] ? call_softirq+0x1c/0x30 [12884.024745] [] local_apic_timer_interrupt+0x37/0x60 [12884.032775] [] smp_apic_timer_interrupt+0x3f/0x60 [12884.040591] [] apic_timer_interrupt+0x6d/0x80 [12884.048021] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [12884.057658] [] ? osd_scrub_exec+0x12/0x5b0 [osd_ldiskfs] [12884.066172] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [12884.075083] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [12884.083985] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [12884.092602] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [12884.101007] [] ? wake_up_state+0x20/0x20 [12884.107956] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [12884.117038] [] kthread+0xcf/0xe0 [12884.123218] [] ? kthread_create_on_node+0x140/0x140 [12884.131232] [] ret_from_fork+0x58/0x90 [12884.137993] [] ? kthread_create_on_node+0x140/0x140 [12909.292184] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [12909.301577] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [12909.431348] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [12909.444530] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [12909.460567] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [12909.469846] RIP: 0010:[] [] osd_inode_iteration+0x2be/0xcc0 [osd_ldiskfs] [12909.481987] RSP: 0018:ffff880703603d18 EFLAGS: 00000297 [12909.488853] RAX: 0000000000000004 RBX: 000000009f8f2001 RCX: ffff880703603d98 [12909.497749] RDX: ffff880703603d78 RSI: 0000000000000000 RDI: ffff8803162f6000 [12909.506648] RBP: ffff880703603df0 R08: 0000000000000001 R09: 0000000000000004 [12909.515550] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000001 [12909.524439] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [12909.533323] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [12909.543287] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12909.550631] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [12909.559537] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12909.568436] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [12909.577333] Stack: [12909.580509] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [12909.589757] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [12909.598993] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [12909.608258] Call Trace: [12909.611943] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [12909.621071] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [12909.629908] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [12909.638550] [] ? wake_up_state+0x20/0x20 [12909.645735] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [12909.655059] [] kthread+0xcf/0xe0 [12909.661451] [] ? kthread_create_on_node+0x140/0x140 [12909.669714] [] ret_from_fork+0x58/0x90 [12909.676697] [] ? kthread_create_on_node+0x140/0x140 [12909.684943] Code: 8b b5 38 ff ff ff 48 8b bd 48 ff ff ff 48 8b 85 30 ff ff ff ff d0 83 f8 03 41 89 c5 0f 84 4b 02 00 00 0f 8e 15 01 00 00 83 f8 05 <66> 90 0f 84 8a 02 00 00 83 f8 06 0f 85 a1 01 00 00 48 8b 7d 90 [12937.293158] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [12937.302797] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [12937.434998] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [12937.448317] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [12937.464610] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [12937.474024] RIP: 0010:[] [] osd_inode_iteration+0x4a6/0xcc0 [osd_ldiskfs] [12937.486273] RSP: 0018:ffff880703603d18 EFLAGS: 00000246 [12937.493230] RAX: 0000000020000000 RBX: 0000000000000000 RCX: 0000000000000000 [12937.502232] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [12937.511218] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [12937.520177] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffff880703603d57 [12937.529132] R13: 0000000000000004 R14: 000000009f8f2001 R15: ffffea00027e3c80 [12937.538085] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [12937.548112] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12937.555515] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [12937.564487] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12937.573455] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [12937.582402] Stack: [12937.585621] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [12937.594901] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [12937.604180] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [12937.613473] Call Trace: [12937.617217] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [12937.626381] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [12937.635261] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [12937.643953] [] ? wake_up_state+0x20/0x20 [12937.651166] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [12937.660534] [] kthread+0xcf/0xe0 [12937.666971] [] ? kthread_create_on_node+0x140/0x140 [12937.675259] [] ret_from_fork+0x58/0x90 [12937.682278] [] ? kthread_create_on_node+0x140/0x140 [12937.690549] Code: 8b 4d a8 48 8d 55 88 48 8b b5 38 ff ff ff 48 8b bd 48 ff ff ff 48 8b 85 28 ff ff ff ff d0 85 c0 41 89 c5 75 1c 8b 85 58 ff ff ff <41> 3b 04 24 72 10 48 8b 85 50 ff ff ff 44 3b 30 0f 87 95 fd ff [12965.294134] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [12965.303818] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [12965.435850] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [12965.449156] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [12965.465443] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [12965.474835] RIP: 0010:[] [] osd_scrub_exec+0xf7/0x5b0 [osd_ldiskfs] [12965.486497] RSP: 0018:ffff880703603c78 EFLAGS: 00000246 [12965.493454] RAX: 0000000000280001 RBX: ffff880703603c37 RCX: 0000000000000000 [12965.502434] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [12965.511408] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [12965.520365] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [12965.529312] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [12965.538267] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [12965.548293] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12965.555695] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [12965.564671] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12965.573636] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [12965.582589] Stack: [12965.585807] ffff880703603df0 0000000000000001 ffff880703603df0 000000009f8f2001 [12965.595108] ffff880703603d57 000000009f8f2001 0000000000000004 ffff880703603d57 [12965.604400] ffff88042bd78800 ffff88042bd78800 00000000000072cf ffff880419dc3a00 [12965.613699] Call Trace: [12965.617428] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [12965.626583] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [12965.635739] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [12965.644615] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [12965.653290] [] ? wake_up_state+0x20/0x20 [12965.660498] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [12965.669856] [] kthread+0xcf/0xe0 [12965.676286] [] ? kthread_create_on_node+0x140/0x140 [12965.684567] [] ret_from_fork+0x58/0x90 [12965.691578] [] ? kthread_create_on_node+0x140/0x140 [12965.699834] Code: 8b 4d 88 48 85 db 74 23 f6 83 18 0a 00 00 08 74 1a 4d 85 e4 74 15 41 8b 87 6c 14 00 00 41 39 84 24 0c 0a 00 00 0f 82 d9 00 00 00 <31> c0 41 f6 87 74 14 00 00 04 0f 85 6e ff ff ff 41 83 f9 02 0f [12993.295109] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [12993.304759] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [12993.436954] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [12993.450243] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [12993.466504] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [12993.475891] RIP: 0010:[] [] osd_scrub_exec+0x95/0x5b0 [osd_ldiskfs] [12993.487546] RSP: 0018:ffff880703603d08 EFLAGS: 00000292 [12993.494488] RAX: 0000000000000000 RBX: 0000000000000001 RCX: 0000000000000000 [12993.503465] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [12993.512438] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [12993.521399] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [12993.530350] R13: ffffea00027e3c80 R14: ffff880703603c78 R15: ffff880703603d08 [12993.539299] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [12993.549327] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12993.556735] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [12993.565709] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12993.574664] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [12993.583607] Stack: [12993.586823] ffff880703603df0 ffffffffa121d1e9 ffffffffa121b990 ffffffffa1217a20 [12993.596094] ffff8800b2a36000 00000000810d354f ffff8803162f6000 ffff8800b2a37468 [12993.605381] 0000000020000000 010000000000000c 0000000000000000 0000000000000000 [12993.614662] Call Trace: [12993.618389] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [12993.627553] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [12993.636718] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [12993.645588] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [12993.654264] [] ? wake_up_state+0x20/0x20 [12993.661475] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [12993.670831] [] kthread+0xcf/0xe0 [12993.677270] [] ? kthread_create_on_node+0x140/0x140 [12993.685552] [] ret_from_fork+0x58/0x90 [12993.692566] [] ? kthread_create_on_node+0x140/0x140 [12993.700840] Code: 01 00 00 41 80 a7 74 14 00 00 fe 48 8b 4d d0 65 48 33 0c 25 28 00 00 00 0f 85 b1 03 00 00 48 83 c4 68 5b 41 5c 41 5d 41 5e 41 5f <5d> c3 66 0f 1f 84 00 00 00 00 00 4c 8d ae f0 10 00 00 44 89 4d [13021.296077] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [13021.305734] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [13021.437866] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [13021.451159] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [13021.467436] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [13021.476833] RIP: 0010:[] [] osd_scrub_next+0x161/0x3c0 [osd_ldiskfs] [13021.488603] RSP: 0018:ffff880703603ce8 EFLAGS: 00000282 [13021.495547] RAX: 0000000000000004 RBX: ffff880703603c78 RCX: ffff880703603d98 [13021.504535] RDX: ffff880703603d78 RSI: 0000000000000000 RDI: ffff8803162f6000 [13021.513498] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [13021.522455] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000010 [13021.531406] R13: ffffffffa121b9c4 R14: ffffffffffffff10 R15: ffff8803162f6000 [13021.540358] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [13021.550390] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13021.557795] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [13021.566757] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13021.575717] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [13021.584662] Stack: [13021.587875] ffff8800b2a3746c 0000000000000000 00000000ffffffff 0000000000000001 [13021.597176] ffff880703603df0 ffffffffa121cff9 ffffffffa121b990 ffffffffa1217a20 [13021.606474] ffff8800b2a36000 00000000810d354f ffff8803162f6000 ffff8800b2a37468 [13021.615770] Call Trace: [13021.619499] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [13021.628668] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [13021.637832] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [13021.646700] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [13021.655367] [] ? wake_up_state+0x20/0x20 [13021.662580] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [13021.671937] [] kthread+0xcf/0xe0 [13021.678381] [] ? kthread_create_on_node+0x140/0x140 [13021.686671] [] ret_from_fork+0x58/0x90 [13021.693687] [] ? kthread_create_on_node+0x140/0x140 [13021.701959] Code: 00 00 02 00 00 00 48 89 df e8 fc 69 47 e0 b8 05 00 00 00 48 8b 75 d0 65 48 33 34 25 28 00 00 00 0f 85 4c 02 00 00 48 83 c4 70 5b <41> 5c 41 5d 41 5e 41 5f 5d c3 48 8b 05 be 58 9c ff 48 85 c0 0f [13049.297053] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [13049.306735] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [13049.438732] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [13049.452020] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [13049.468293] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [13049.477697] RIP: 0010:[] [] osd_scrub_exec+0x23/0x5b0 [osd_ldiskfs] [13049.489356] RSP: 0018:ffff880703603c78 EFLAGS: 00000296 [13049.496302] RAX: ffffffffa121b990 RBX: ffff880703603c37 RCX: 0000000000000000 [13049.505292] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [13049.514252] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [13049.523208] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [13049.532158] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [13049.541111] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [13049.551150] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13049.558567] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [13049.567542] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13049.576513] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [13049.585465] Stack: [13049.588682] ffff880703603d08 000000009f8f2001 ffff880703603df0 000000009f8f2001 [13049.597977] ffff880703603d57 000000009f8f2001 0000000000000004 ffff880703603d57 [13049.607267] ffff8800b2a37468 0000000000000000 ffff880703603d78 ffff8800b2a36000 [13049.616565] Call Trace: [13049.620287] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [13049.629454] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [13049.638615] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [13049.647478] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [13049.656140] [] ? wake_up_state+0x20/0x20 [13049.663349] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [13049.672707] [] kthread+0xcf/0xe0 [13049.679137] [] ? kthread_create_on_node+0x140/0x140 [13049.687414] [] ret_from_fork+0x58/0x90 [13049.694426] [] ? kthread_create_on_node+0x140/0x140 [13049.702694] Code: 03 98 ff e9 c7 fd ff ff 66 66 66 66 90 55 48 89 e5 41 57 49 89 f7 41 56 41 55 41 54 45 31 e4 53 48 83 ec 68 48 8b 9e 08 10 00 00 <4c> 89 45 90 65 48 8b 04 25 28 00 00 00 48 89 45 d0 31 c0 48 8d [13063.893562] INFO: rcu_sched self-detected stall on CPU { 24} (t=5640114 jiffies g=907112 c=907111 q=15703893) [13063.906080] Task dump for CPU 24: [13063.910821] OI_scrub R running task 0 5551 2 0x00000088 [13063.919745] ffff88083fde6dd0 00000000fb6db880 ffff88082da03db0 ffffffff810c47e8 [13063.929067] 0000000000000018 ffffffff81a1e7c0 ffff88082da03dc8 ffffffff810c8079 [13063.938370] 0000000000000009 ffff88082da03df8 ffffffff811378e0 ffff88082da101c0 [13063.947636] Call Trace: [13063.951316] [] sched_show_task+0xa8/0x110 [13063.959297] [] dump_cpu_task+0x39/0x70 [13063.966254] [] rcu_dump_cpu_stacks+0x90/0xd0 [13063.973779] [] rcu_check_callbacks+0x442/0x720 [13063.981468] [] ? tick_sched_handle.isra.13+0x60/0x60 [13063.989729] [] update_process_times+0x47/0x80 [13063.997282] [] tick_sched_handle.isra.13+0x25/0x60 [13064.005307] [] tick_sched_timer+0x41/0x70 [13064.012431] [] __hrtimer_run_queues+0xd2/0x260 [13064.020014] [] hrtimer_interrupt+0xb0/0x1e0 [13064.027281] [] ? call_softirq+0x1c/0x30 [13064.034140] [] local_apic_timer_interrupt+0x37/0x60 [13064.042160] [] smp_apic_timer_interrupt+0x3f/0x60 [13064.049989] [] apic_timer_interrupt+0x6d/0x80 [13064.057418] [] ? osd_scrub_exec+0xf7/0x5b0 [osd_ldiskfs] [13064.066652] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [13064.075549] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [13064.084440] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [13064.093049] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [13064.101450] [] ? wake_up_state+0x20/0x20 [13064.108401] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [13064.117487] [] kthread+0xcf/0xe0 [13064.123647] [] ? kthread_create_on_node+0x140/0x140 [13064.131658] [] ret_from_fork+0x58/0x90 [13064.138398] [] ? kthread_create_on_node+0x140/0x140 [13089.298448] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [13089.307873] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [13089.437773] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [13089.450950] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [13089.467018] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [13089.476307] RIP: 0010:[] [] osd_scrub_next+0x167/0x3c0 [osd_ldiskfs] [13089.487974] RSP: 0018:ffff880703603d00 EFLAGS: 00000282 [13089.494827] RAX: 0000000000000004 RBX: 0000000000000001 RCX: ffff880703603d98 [13089.503738] RDX: ffff880703603d78 RSI: 0000000000000000 RDI: ffff8803162f6000 [13089.512627] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [13089.521520] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [13089.530410] R13: 0000000000000018 R14: ffff880703603c78 R15: 0000000000000296 [13089.539296] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [13089.549249] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13089.556574] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [13089.565472] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13089.574369] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [13089.583248] Stack: [13089.586409] 0000000000000001 ffff880703603df0 ffffffffa121cff9 ffffffffa121b990 [13089.595641] ffffffffa1217a20 ffff8800b2a36000 00000000810d354f ffff8803162f6000 [13089.604867] ffff8800b2a37468 0000000020000000 010000000000000c 0000000000000000 [13089.614094] Call Trace: [13089.617753] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [13089.626872] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [13089.635987] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [13089.644811] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [13089.653455] [] ? wake_up_state+0x20/0x20 [13089.660627] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [13089.669931] [] kthread+0xcf/0xe0 [13089.676326] [] ? kthread_create_on_node+0x140/0x140 [13089.684576] [] ret_from_fork+0x58/0x90 [13089.691550] [] ? kthread_create_on_node+0x140/0x140 [13089.699791] Code: 48 89 df e8 fc 69 47 e0 b8 05 00 00 00 48 8b 75 d0 65 48 33 34 25 28 00 00 00 0f 85 4c 02 00 00 48 83 c4 70 5b 41 5c 41 5d 41 5e <41> 5f 5d c3 48 8b 05 be 58 9c ff 48 85 c0 0f 84 c2 fe ff ff 0f [13117.299422] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [13117.309070] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [13117.440718] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [13117.453993] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [13117.470224] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [13117.479602] RIP: 0010:[] [] osd_scrub_next+0x160/0x3c0 [osd_ldiskfs] [13117.491347] RSP: 0018:ffff880703603ce0 EFLAGS: 00000282 [13117.498275] RAX: 0000000000000004 RBX: ffff880703603c78 RCX: ffff880703603d98 [13117.507240] RDX: ffff880703603d78 RSI: 0000000000000000 RDI: ffff8803162f6000 [13117.516201] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [13117.525138] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000010 [13117.534073] R13: ffffffffa121ba87 R14: ffffffffffffff10 R15: ffff8803162f6000 [13117.543009] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [13117.553027] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13117.560434] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [13117.569393] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13117.578351] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [13117.587305] Stack: [13117.590518] ffff880419dc3a00 ffff8800b2a3746c 0000000000000000 00000000ffffffff [13117.599816] 0000000000000001 ffff880703603df0 ffffffffa121cff9 ffffffffa121b990 [13117.609117] ffffffffa1217a20 ffff8800b2a36000 00000000810d354f ffff8803162f6000 [13117.618412] Call Trace: [13117.622132] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [13117.631297] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [13117.640458] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [13117.649321] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [13117.657983] [] ? wake_up_state+0x20/0x20 [13117.665198] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [13117.674550] [] kthread+0xcf/0xe0 [13117.680977] [] ? kthread_create_on_node+0x140/0x140 [13117.689283] [] ret_from_fork+0x58/0x90 [13117.696299] [] ? kthread_create_on_node+0x140/0x140 [13117.704574] Code: 10 00 00 02 00 00 00 48 89 df e8 fc 69 47 e0 b8 05 00 00 00 48 8b 75 d0 65 48 33 34 25 28 00 00 00 0f 85 4c 02 00 00 48 83 c4 70 <5b> 41 5c 41 5d 41 5e 41 5f 5d c3 48 8b 05 be 58 9c ff 48 85 c0 [13145.300397] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [13145.310074] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [13145.441818] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [13145.455109] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [13145.471374] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [13145.480764] RIP: 0010:[] [] ldiskfs_itable_unused_count+0x6/0x30 [ldiskfs] [13145.493112] RSP: 0018:ffff880703603d08 EFLAGS: 00000286 [13145.500064] RAX: ffff8800b2a37468 RBX: 0000000000000001 RCX: 0000000000000000 [13145.509044] RDX: ffff880703603d78 RSI: ffff880419dc3a00 RDI: ffff88042bd79800 [13145.518030] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [13145.526985] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [13145.535927] R13: 0000000000000018 R14: ffff880703603c78 R15: 0000000000000282 [13145.544873] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [13145.554899] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13145.562310] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [13145.571278] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13145.580239] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [13145.589188] Stack: [13145.592407] ffff880703603df0 ffffffffa121cfb9 ffffffffa121b990 ffffffffa1217a20 [13145.601701] ffff8800b2a36000 00000000810d354f ffff8803162f6000 ffff8800b2a37468 [13145.610999] 0000000020000000 010000000000000c 0000000000000000 0000000000000000 [13145.620295] Call Trace: [13145.624031] [] ? osd_inode_iteration+0x269/0xcc0 [osd_ldiskfs] [13145.633389] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [13145.642548] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [13145.651418] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [13145.660079] [] ? wake_up_state+0x20/0x20 [13145.667287] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [13145.676640] [] kthread+0xcf/0xe0 [13145.683076] [] ? kthread_create_on_node+0x140/0x140 [13145.691359] [] ret_from_fork+0x58/0x90 [13145.698368] [] ? kthread_create_on_node+0x140/0x140 [13145.706642] Code: 00 00 31 c0 0f b7 56 10 48 89 e5 48 83 39 3f 76 07 0f b7 46 30 c1 e0 10 09 d0 5d c3 66 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 <48> 8b 8f a8 03 00 00 31 c0 0f b7 56 1c 48 89 e5 48 83 39 3f 76 [13173.301372] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [13173.311009] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [13173.442884] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [13173.456182] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [13173.472446] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [13173.481838] RIP: 0010:[] [] osd_scrub_exec+0xd4/0x5b0 [osd_ldiskfs] [13173.493500] RSP: 0018:ffff880703603c78 EFLAGS: 00000286 [13173.500439] RAX: ffff8803162f1008 RBX: ffff880703603c37 RCX: 0000000000000000 [13173.509423] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [13173.518386] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [13173.527335] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [13173.536281] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [13173.545232] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [13173.555265] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13173.562670] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [13173.571638] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13173.580597] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [13173.589546] Stack: [13173.592760] ffff880703603df0 ffff880703603d57 ffff880703603df0 000000009f8f2001 [13173.602054] ffff880703603d57 000000009f8f2001 0000000000000004 0000000000000001 [13173.611362] 0000000000000004 ffff880703603d98 ffff880703603d78 0000000000000000 [13173.620650] Call Trace: [13173.624373] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [13173.633555] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [13173.642717] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [13173.651582] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [13173.660244] [] ? wake_up_state+0x20/0x20 [13173.667452] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [13173.676804] [] kthread+0xcf/0xe0 [13173.683232] [] ? kthread_create_on_node+0x140/0x140 [13173.691523] [] ret_from_fork+0x58/0x90 [13173.698532] [] ? kthread_create_on_node+0x140/0x140 [13173.706798] Code: 4d 88 4c 89 ef e8 1d fb 46 e0 41 83 87 68 14 00 00 01 49 83 87 88 11 00 00 01 4c 89 ef e8 f5 9f e9 df 44 8b 4d 88 48 85 db 74 23 83 18 0a 00 00 08 74 1a 4d 85 e4 74 15 41 8b 87 6c 14 00 00 [13201.302348] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [13201.312026] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [13201.443757] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [13201.457044] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [13201.473293] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [13201.482676] RIP: 0010:[] [] osd_scrub_exec+0x3d/0x5b0 [osd_ldiskfs] [13201.494328] RSP: 0018:ffff880703603c78 EFLAGS: 00000286 [13201.501266] RAX: ffff8803162f1008 RBX: ffff880703603c37 RCX: 0000000000000000 [13201.510251] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [13201.519220] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [13201.528168] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [13201.537121] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [13201.546076] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [13201.556103] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13201.563508] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [13201.572475] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13201.581441] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [13201.590382] Stack: [13201.593589] ffff880703603df0 000000009f8f2001 ffff880703603df0 000000009f8f2001 [13201.602887] ffff880703603d57 000000009f8f2001 0000000000000004 ffff880703603d57 [13201.612170] ffff88042bd78800 ffff88042bd78800 00000000000072cf ffff880419dc3a00 [13201.621464] Call Trace: [13201.625198] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [13201.634358] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [13201.643514] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [13201.652379] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [13201.661040] [] ? wake_up_state+0x20/0x20 [13201.668247] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [13201.677587] [] kthread+0xcf/0xe0 [13201.684017] [] ? kthread_create_on_node+0x140/0x140 [13201.692293] [] ret_from_fork+0x58/0x90 [13201.699304] [] ? kthread_create_on_node+0x140/0x140 [13201.707564] Code: 41 54 45 31 e4 53 48 83 ec 68 48 8b 9e 08 10 00 00 4c 89 45 90 65 48 8b 04 25 28 00 00 00 48 89 45 d0 31 c0 48 8d 43 08 48 85 db <4c> 0f 45 e0 41 83 f9 04 0f 84 84 00 00 00 41 83 f9 07 74 4f 41 [13229.303323] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [13229.312997] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [13229.444770] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [13229.458062] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [13229.474300] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [13229.483691] RIP: 0010:[] [] osd_scrub_exec+0x1c/0x5b0 [osd_ldiskfs] [13229.495340] RSP: 0018:ffff880703603c78 EFLAGS: 00000296 [13229.502273] RAX: ffffffffa121b990 RBX: ffff880703603c37 RCX: 0000000000000000 [13229.511244] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [13229.520201] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [13229.529152] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [13229.538094] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [13229.547037] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [13229.557060] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13229.564460] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [13229.573431] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13229.582378] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [13229.591321] Stack: [13229.594525] ffff880703603d08 0000000000000001 ffffea00027e3c80 000000009f8f2001 [13229.603808] ffff880703603d57 ffff880703603d57 00000000000072ce ffff88042bd78800 [13229.613103] 00000000000072ce ffff880419dc3a00 ffff88042bd79800 ffffffffffffff10 [13229.622405] Call Trace: [13229.626125] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [13229.635300] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [13229.644459] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [13229.653319] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [13229.661987] [] ? wake_up_state+0x20/0x20 [13229.669194] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [13229.678541] [] kthread+0xcf/0xe0 [13229.684977] [] ? kthread_create_on_node+0x140/0x140 [13229.693274] [] ret_from_fork+0x58/0x90 [13229.700282] [] ? kthread_create_on_node+0x140/0x140 [13229.708554] Code: 89 04 24 31 c0 e8 a5 03 98 ff e9 c7 fd ff ff 66 66 66 66 90 55 48 89 e5 41 57 49 89 f7 41 56 41 55 41 54 45 31 e4 53 48 83 ec 68 <48> 8b 9e 08 10 00 00 4c 89 45 90 65 48 8b 04 25 28 00 00 00 48 [13243.902832] INFO: rcu_sched self-detected stall on CPU { 24} (t=5820117 jiffies g=907112 c=907111 q=16299401) [13243.915297] Task dump for CPU 24: [13243.920023] OI_scrub R running task 0 5551 2 0x00000088 [13243.928960] ffff88083fde6dd0 00000000fb6db880 ffff88082da03db0 ffffffff810c47e8 [13243.938277] 0000000000000018 ffffffff81a1e7c0 ffff88082da03dc8 ffffffff810c8079 [13243.947556] 0000000000000009 ffff88082da03df8 ffffffff811378e0 ffff88082da101c0 [13243.956839] Call Trace: [13243.960524] [] sched_show_task+0xa8/0x110 [13243.968518] [] dump_cpu_task+0x39/0x70 [13243.975483] [] rcu_dump_cpu_stacks+0x90/0xd0 [13243.983007] [] rcu_check_callbacks+0x442/0x720 [13243.990704] [] ? tick_sched_handle.isra.13+0x60/0x60 [13243.998972] [] update_process_times+0x47/0x80 [13244.006527] [] tick_sched_handle.isra.13+0x25/0x60 [13244.014540] [] tick_sched_timer+0x41/0x70 [13244.021669] [] __hrtimer_run_queues+0xd2/0x260 [13244.029257] [] hrtimer_interrupt+0xb0/0x1e0 [13244.036545] [] ? call_softirq+0x1c/0x30 [13244.043435] [] local_apic_timer_interrupt+0x37/0x60 [13244.051469] [] smp_apic_timer_interrupt+0x3f/0x60 [13244.059287] [] apic_timer_interrupt+0x6d/0x80 [13244.066710] [] ? osd_scrub_next+0x2b/0x3c0 [osd_ldiskfs] [13244.075956] [] ? osd_scrub_exec+0x8d/0x5b0 [osd_ldiskfs] [13244.084471] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [13244.093375] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [13244.102278] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [13244.110881] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [13244.119293] [] ? wake_up_state+0x20/0x20 [13244.126241] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [13244.135341] [] kthread+0xcf/0xe0 [13244.141521] [] ? kthread_create_on_node+0x140/0x140 [13244.149554] [] ret_from_fork+0x58/0x90 [13244.156305] [] ? kthread_create_on_node+0x140/0x140 [13269.304723] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [13269.314164] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [13269.443867] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [13269.457038] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [13269.473064] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [13269.482341] RIP: 0010:[] [] osd_scrub_exec+0x9/0x5b0 [osd_ldiskfs] [13269.493802] RSP: 0018:ffff880703603d08 EFLAGS: 00000293 [13269.500654] RAX: ffffffffa121b990 RBX: 000000009f8f2001 RCX: 0000000000000000 [13269.509547] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [13269.518448] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [13269.527349] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [13269.536242] R13: 0000000000000018 R14: ffff880703603c78 R15: 0000000000000282 [13269.545122] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [13269.555073] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13269.562404] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [13269.571300] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13269.580193] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [13269.589076] Stack: [13269.592229] ffff880703603df0 ffffffffa121d1e9 ffffffffa121b990 ffffffffa1217a20 [13269.601479] ffff8800b2a36000 00000000810d354f ffff8803162f6000 ffff8800b2a37468 [13269.610720] 0000000020000000 010000000000000c 0000000000000000 0000000000000000 [13269.619973] Call Trace: [13269.623651] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [13269.632780] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [13269.641907] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [13269.650739] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [13269.659379] [] ? wake_up_state+0x20/0x20 [13269.666544] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [13269.675845] [] kthread+0xcf/0xe0 [13269.682240] [] ? kthread_create_on_node+0x140/0x140 [13269.690484] [] ret_from_fork+0x58/0x90 [13269.697462] [] ? kthread_create_on_node+0x140/0x140 [13269.705687] Code: 00 00 00 02 00 89 54 24 08 48 8b 40 40 ba 07 00 00 00 48 89 04 24 31 c0 e8 a5 03 98 ff e9 c7 fd ff ff 66 66 66 66 90 55 48 89 e5 <41> 57 49 89 f7 41 56 41 55 41 54 45 31 e4 53 48 83 ec 68 48 8b [13297.305699] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [13297.315338] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [13297.447098] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [13297.460372] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [13297.476608] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [13297.485976] RIP: 0010:[] [] osd_scrub_next+0x350/0x3c0 [osd_ldiskfs] [13297.497714] RSP: 0018:ffff880703603c70 EFLAGS: 00000246 [13297.504641] RAX: 0000000000000000 RBX: ffff880703603c37 RCX: ffff880703603d98 [13297.513610] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [13297.522569] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [13297.531516] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [13297.540461] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [13297.549403] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [13297.559433] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13297.566837] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [13297.575801] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13297.584761] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [13297.593705] Stack: [13297.596924] 000000009f8f2001 ffff880703603d08 000000009f8f2001 ffff880703603df0 [13297.606222] 000000009f8f2001 ffff880703603d57 000000009f8f2001 0000000000000004 [13297.615527] ffff880703603d57 0000000000000000 0000000000000000 ffff880703603d78 [13297.624827] Call Trace: [13297.628542] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [13297.637702] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [13297.646857] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [13297.655714] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [13297.664369] [] ? wake_up_state+0x20/0x20 [13297.671572] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [13297.680913] [] kthread+0xcf/0xe0 [13297.687336] [] ? kthread_create_on_node+0x140/0x140 [13297.695614] [] ret_from_fork+0x58/0x90 [13297.702618] [] ? kthread_create_on_node+0x140/0x140 [13297.710882] Code: 97 ff 48 8b 55 90 48 8b 4d 88 44 8b 85 74 ff ff ff e9 1d ff ff ff 48 89 d7 44 89 45 90 e8 f9 67 47 e0 44 8b 45 90 0f 1f 44 00 00 <45> 84 c0 b8 04 00 00 00 0f 85 eb fd ff ff 49 8d b7 6c 14 00 00 [13325.306667] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [13325.316359] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [13325.448145] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [13325.461437] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [13325.477678] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [13325.487057] RIP: 0010:[] [] osd_scrub_exec+0x82/0x5b0 [osd_ldiskfs] [13325.498697] RSP: 0018:ffff880703603c78 EFLAGS: 00000246 [13325.505633] RAX: 0000000000000000 RBX: ffff880703603c37 RCX: 0000000000000000 [13325.514613] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [13325.523580] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [13325.532525] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [13325.541467] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [13325.550410] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [13325.560431] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13325.567831] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [13325.576789] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13325.585735] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [13325.594670] Stack: [13325.597870] ffff880703603d08 000000009f8f2001 ffff880703603df0 000000009f8f2001 [13325.607138] ffff880703603d57 000000009f8f2001 0000000000000004 0000000000000001 [13325.616433] 0000000000000004 ffff880703603d98 ffff880703603d78 0000000000000000 [13325.625721] Call Trace: [13325.629427] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [13325.638598] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [13325.647751] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [13325.656605] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [13325.665259] [] ? wake_up_state+0x20/0x20 [13325.672460] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [13325.681808] [] kthread+0xcf/0xe0 [13325.688231] [] ? kthread_create_on_node+0x140/0x140 [13325.696501] [] ret_from_fork+0x58/0x90 [13325.703504] [] ? kthread_create_on_node+0x140/0x140 [13325.711769] Code: 48 89 ca 44 89 c9 e8 0e c6 ff ff 85 c0 41 89 c1 0f 84 a3 01 00 00 41 80 a7 74 14 00 00 fe 48 8b 4d d0 65 48 33 0c 25 28 00 00 00 <0f> 85 b1 03 00 00 48 83 c4 68 5b 41 5c 41 5d 41 5e 41 5f 5d c3 [13353.307643] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [13353.317359] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [13353.449125] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [13353.462409] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [13353.478654] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [13353.488023] RIP: 0010:[] [] osd_inode_iteration+0x269/0xcc0 [osd_ldiskfs] [13353.500248] RSP: 0018:ffff880703603d18 EFLAGS: 00000202 [13353.507185] RAX: 00000000000072ce RBX: 000000009f8f2001 RCX: ffff88042bd78800 [13353.516165] RDX: 00000000000072ce RSI: ffff880419dc3a00 RDI: ffff88042bd79800 [13353.525121] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [13353.534068] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [13353.543012] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [13353.551961] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [13353.561982] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13353.569387] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [13353.578345] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13353.587290] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [13353.596229] Stack: [13353.599444] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [13353.608725] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [13353.618025] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [13353.627317] Call Trace: [13353.631039] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [13353.640209] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [13353.649077] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [13353.657751] [] ? wake_up_state+0x20/0x20 [13353.664955] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [13353.674309] [] kthread+0xcf/0xe0 [13353.680740] [] ? kthread_create_on_node+0x140/0x140 [13353.689024] [] ret_from_fork+0x58/0x90 [13353.696036] [] ? kthread_create_on_node+0x140/0x140 [13353.704314] Code: 88 e8 dc 2d f5 ff 48 85 c0 48 89 45 90 0f 84 b3 08 00 00 44 8b 7d a0 48 8b 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 e8 77 5b f4 ff <42> 8d 14 38 48 8b 45 88 48 8b 80 a8 03 00 00 48 3b 50 20 0f 83 [13373.553202] Lustre: soaked-MDT0000: Client e211cd09-6b0c-e43b-059c-f657101042a0 (at 192.168.1.117@o2ib100) reconnecting [13373.566529] Lustre: Skipped 1 previous similar message [13373.573360] Lustre: soaked-MDT0000: Connection restored to e211cd09-6b0c-e43b-059c-f657101042a0 (at 192.168.1.117@o2ib100) [13373.586773] Lustre: Skipped 1 previous similar message [13381.308618] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [13381.318309] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [13381.450048] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [13381.463337] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [13381.479591] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [13381.488987] RIP: 0010:[] [] osd_scrub_next+0x1b/0x3c0 [osd_ldiskfs] [13381.500653] RSP: 0018:ffff880703603ce0 EFLAGS: 00000287 [13381.507616] RAX: ffffffffa1217a20 RBX: ffff880703603c78 RCX: ffff880703603d98 [13381.516619] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [13381.525627] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [13381.534637] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000010 [13381.543644] R13: ffffffffa121ba64 R14: ffffffffffffff10 R15: ffff8803162f6000 [13381.552650] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [13381.562723] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13381.570173] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [13381.579182] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13381.588185] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [13381.597192] Stack: [13381.600451] ffff880419dc3a00 ffff8800b2a3746c 0000000000000000 00000000ffffffff [13381.609794] 0000000000000001 ffff880703603df0 ffffffffa121cff9 ffffffffa121b990 [13381.619138] ffffffffa1217a20 ffff8800b2a36000 00000000810d354f ffff8803162f6000 [13381.628497] Call Trace: [13381.632283] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [13381.641496] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [13381.650715] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [13381.659633] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [13381.668363] [] ? wake_up_state+0x20/0x20 [13381.675625] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [13381.685014] [] kthread+0xcf/0xe0 [13381.691495] [] ? kthread_create_on_node+0x140/0x140 [13381.699827] [] ret_from_fork+0x58/0x90 [13381.706892] [] ? kthread_create_on_node+0x140/0x140 [13381.715210] Code: 41 5c 41 5d 41 5e 5d c3 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 89 e5 41 57 49 89 f7 41 56 41 55 49 89 fd 41 54 49 89 d4 53 <48> 89 cb 48 83 ec 70 65 48 8b 04 25 28 00 00 00 48 89 45 d0 31 [13409.309601] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [13409.319277] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [13409.451128] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [13409.464420] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [13409.480677] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [13409.490066] RIP: 0010:[] [] osd_scrub_next+0x1e/0x3c0 [osd_ldiskfs] [13409.501719] RSP: 0018:ffff880703603ce0 EFLAGS: 00000287 [13409.508667] RAX: ffffffffa1217a20 RBX: ffff880703603c70 RCX: ffff880703603d98 [13409.517649] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [13409.526611] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [13409.535568] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000010 [13409.544512] R13: ffffffffa1217a4b R14: ffffffffffffff10 R15: ffff8803162f6000 [13409.553468] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [13409.563502] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13409.570913] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [13409.579877] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13409.588823] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [13409.597757] Stack: [13409.600968] ffff880419dc3a00 ffff8800b2a3746c 0000000000000000 00000000ffffffff [13409.610251] 0000000000000001 ffff880703603df0 ffffffffa121cff9 ffffffffa121b990 [13409.619561] ffffffffa1217a20 ffff8800b2a36000 00000000810d354f ffff8803162f6000 [13409.628878] Call Trace: [13409.632610] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [13409.641781] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [13409.650947] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [13409.659800] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [13409.668472] [] ? wake_up_state+0x20/0x20 [13409.675679] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [13409.685033] [] kthread+0xcf/0xe0 [13409.691470] [] ? kthread_create_on_node+0x140/0x140 [13409.699748] [] ret_from_fork+0x58/0x90 [13409.706765] [] ? kthread_create_on_node+0x140/0x140 [13409.715034] Code: 5d 41 5e 5d c3 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 89 e5 41 57 49 89 f7 41 56 41 55 49 89 fd 41 54 49 89 d4 53 48 89 cb <48> 83 ec 70 65 48 8b 04 25 28 00 00 00 48 89 45 d0 31 c0 48 8b [13423.912103] INFO: rcu_sched self-detected stall on CPU { 24} (t=6000120 jiffies g=907112 c=907111 q=17487651) [13423.924595] Task dump for CPU 24: [13423.929321] OI_scrub R running task 0 5551 2 0x00000088 [13423.938250] ffff88083fde6dd0 00000000fb6db880 ffff88082da03db0 ffffffff810c47e8 [13423.947574] 0000000000000018 ffffffff81a1e7c0 ffff88082da03dc8 ffffffff810c8079 [13423.956874] 0000000000000009 ffff88082da03df8 ffffffff811378e0 ffff88082da101c0 [13423.966156] Call Trace: [13423.969836] [] sched_show_task+0xa8/0x110 [13423.977809] [] dump_cpu_task+0x39/0x70 [13423.984763] [] rcu_dump_cpu_stacks+0x90/0xd0 [13423.992281] [] rcu_check_callbacks+0x442/0x720 [13423.999970] [] ? tick_sched_handle.isra.13+0x60/0x60 [13424.008224] [] update_process_times+0x47/0x80 [13424.015781] [] tick_sched_handle.isra.13+0x25/0x60 [13424.023795] [] tick_sched_timer+0x41/0x70 [13424.030918] [] __hrtimer_run_queues+0xd2/0x260 [13424.038508] [] hrtimer_interrupt+0xb0/0x1e0 [13424.045790] [] ? call_softirq+0x1c/0x30 [13424.052672] [] local_apic_timer_interrupt+0x37/0x60 [13424.060697] [] smp_apic_timer_interrupt+0x3f/0x60 [13424.068510] [] apic_timer_interrupt+0x6d/0x80 [13424.075940] [] ? osd_scrub_exec+0xd4/0x5b0 [osd_ldiskfs] [13424.085186] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [13424.094091] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [13424.102998] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [13424.111613] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [13424.120015] [] ? wake_up_state+0x20/0x20 [13424.126959] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [13424.136044] [] kthread+0xcf/0xe0 [13424.142211] [] ? kthread_create_on_node+0x140/0x140 [13424.150232] [] ret_from_fork+0x58/0x90 [13424.156982] [] ? kthread_create_on_node+0x140/0x140 [13449.310988] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [13449.320402] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [13449.450001] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [13449.463183] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [13449.479246] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [13449.488543] RIP: 0010:[] [] osd_inode_iteration+0x473/0xcc0 [osd_ldiskfs] [13449.500690] RSP: 0018:ffff880703603d18 EFLAGS: 00000293 [13449.507546] RAX: 0000000000000004 RBX: 000000009f8f2001 RCX: ffff880703603d98 [13449.516456] RDX: ffff880703603d78 RSI: 0000000000000000 RDI: ffff8803162f6000 [13449.525347] RBP: ffff880703603df0 R08: 0000000000000001 R09: 0000000000000004 [13449.534235] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000001 [13449.543133] R13: ffff880703603d08 R14: 000000009f8f2001 R15: ffffea00027e3c80 [13449.552018] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [13449.561984] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13449.569325] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [13449.578222] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13449.587118] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [13449.596012] Stack: [13449.599171] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [13449.608409] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [13449.617663] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [13449.626919] Call Trace: [13449.630607] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [13449.639726] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [13449.648567] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [13449.657220] [] ? wake_up_state+0x20/0x20 [13449.664406] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [13449.673727] [] kthread+0xcf/0xe0 [13449.680122] [] ? kthread_create_on_node+0x140/0x140 [13449.688375] [] ret_from_fork+0x58/0x90 [13449.695355] [] ? kthread_create_on_node+0x140/0x140 [13449.703596] Code: 00 00 48 c7 05 9b 86 02 00 00 00 00 00 c7 05 89 86 02 00 01 00 00 00 e8 7c eb 97 ff e9 0f fc ff ff 0f 1f 80 00 00 00 00 45 89 e9 <4c> 8d 85 67 ff ff ff 48 8b 4d a8 48 8d 55 88 48 8b b5 38 ff ff [13459.556421] Lustre: 3694:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1494366804/real 1494366804] req@ffff88028d932d00 x1566943502434336/t0(0) o104->soaked-MDT0000@192.168.1.118@o2ib100:15/16 lens 296/224 e 0 to 1 dl 1494366811 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [13477.311965] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [13477.321663] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [13477.453686] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [13477.466998] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [13477.483303] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [13477.492717] RIP: 0010:[] [] osd_inode_iteration+0x47a/0xcc0 [osd_ldiskfs] [13477.504990] RSP: 0018:ffff880703603d18 EFLAGS: 00000293 [13477.511972] RAX: 0000000000000004 RBX: 000000009f8f2001 RCX: ffff880703603d98 [13477.520990] RDX: ffff880703603d78 RSI: 0000000000000000 RDI: ffff8803162f6000 [13477.530001] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [13477.539010] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [13477.548030] R13: ffff880703603d08 R14: 000000009f8f2001 R15: ffff880703603d08 [13477.557045] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [13477.567136] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13477.574603] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [13477.583621] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13477.592631] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [13477.601636] Stack: [13477.604924] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [13477.614278] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [13477.623625] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [13477.632970] Call Trace: [13477.636758] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [13477.645984] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [13477.654905] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [13477.663634] [] ? wake_up_state+0x20/0x20 [13477.670901] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [13477.680312] [] kthread+0xcf/0xe0 [13477.686802] [] ? kthread_create_on_node+0x140/0x140 [13477.695136] [] ret_from_fork+0x58/0x90 [13477.702213] [] ? kthread_create_on_node+0x140/0x140 [13477.710536] Code: 02 00 00 00 00 00 c7 05 89 86 02 00 01 00 00 00 e8 7c eb 97 ff e9 0f fc ff ff 0f 1f 80 00 00 00 00 45 89 e9 4c 8d 85 67 ff ff ff <48> 8b 4d a8 48 8d 55 88 48 8b b5 38 ff ff ff 48 8b bd 48 ff ff [13505.312939] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [13505.322653] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [13505.454675] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [13505.467986] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [13505.484289] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [13505.493698] RIP: 0010:[] [] osd_scrub_next+0x3d/0x3c0 [osd_ldiskfs] [13505.505375] RSP: 0018:ffff880703603c70 EFLAGS: 00000246 [13505.512337] RAX: 0000000000000000 RBX: ffff880703603c37 RCX: ffff880703603d98 [13505.521326] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [13505.530305] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [13505.539278] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [13505.548234] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [13505.557194] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [13505.567223] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13505.574632] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [13505.583602] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13505.592567] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [13505.601516] Stack: [13505.604735] 000000009f8f2001 ffff880703603d08 0000000000000001 ffff880703603df0 [13505.614007] 000000009f8f2001 ffff880703603d57 000000009f8f2001 0000000000000004 [13505.623299] ffff880703603d57 ffff88042bd78800 ffff88042bd78800 00000000000072cf [13505.632599] Call Trace: [13505.636320] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [13505.645482] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [13505.654646] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [13505.663511] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [13505.672174] [] ? wake_up_state+0x20/0x20 [13505.679375] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [13505.688714] [] kthread+0xcf/0xe0 [13505.695138] [] ? kthread_create_on_node+0x140/0x140 [13505.703418] [] ret_from_fork+0x58/0x90 [13505.710421] [] ? kthread_create_on_node+0x140/0x140 [13505.718688] Code: 49 89 fd 41 54 49 89 d4 53 48 89 cb 48 83 ec 70 65 48 8b 04 25 28 00 00 00 48 89 45 d0 31 c0 48 8b 05 f8 59 9c ff 48 85 c0 75 7b <41> f6 87 58 10 00 00 08 0f 84 62 03 00 00 4d 8d b7 e0 10 00 00 [13533.313914] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [13533.323577] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [13533.455299] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [13533.468581] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [13533.484813] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [13533.494192] RIP: 0010:[] [] osd_inode_iteration+0x47a/0xcc0 [osd_ldiskfs] [13533.506417] RSP: 0018:ffff880703603d18 EFLAGS: 00000293 [13533.513354] RAX: 0000000000000004 RBX: 000000009f8f2001 RCX: ffff880703603d98 [13533.522329] RDX: ffff880703603d78 RSI: 0000000000000000 RDI: ffff8803162f6000 [13533.531291] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [13533.540240] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [13533.549185] R13: ffff880703603d08 R14: 000000009f8f2001 R15: ffff880703603d08 [13533.558132] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [13533.568159] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13533.575563] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [13533.584529] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13533.593473] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [13533.602413] Stack: [13533.605617] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [13533.614903] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [13533.624223] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [13533.633503] Call Trace: [13533.637217] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [13533.646390] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [13533.655264] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [13533.663936] [] ? wake_up_state+0x20/0x20 [13533.671137] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [13533.680483] [] kthread+0xcf/0xe0 [13533.686917] [] ? kthread_create_on_node+0x140/0x140 [13533.695198] [] ret_from_fork+0x58/0x90 [13533.702223] [] ? kthread_create_on_node+0x140/0x140 [13533.710489] Code: 02 00 00 00 00 00 c7 05 89 86 02 00 01 00 00 00 e8 7c eb 97 ff e9 0f fc ff ff 0f 1f 80 00 00 00 00 45 89 e9 4c 8d 85 67 ff ff ff <48> 8b 4d a8 48 8d 55 88 48 8b b5 38 ff ff ff 48 8b bd 48 ff ff [13561.314889] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [13561.324557] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [13561.456559] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [13561.469867] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [13561.486154] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [13561.495562] RIP: 0010:[] [] ldiskfs_used_dirs_count+0x30/0x30 [ldiskfs] [13561.507627] RSP: 0018:ffff880703603d10 EFLAGS: 00000286 [13561.514592] RAX: ffff8800b2a37468 RBX: ffff880703603d57 RCX: 0000000000000000 [13561.523583] RDX: ffff880703603d78 RSI: ffff880419dc3a00 RDI: ffff88042bd79800 [13561.532565] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [13561.541533] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [13561.550490] R13: ffffea00027e3c80 R14: 000000009f8f2001 R15: 0000000000000018 [13561.559440] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [13561.569476] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13561.576882] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [13561.585849] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13561.594809] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [13561.603755] Stack: [13561.606963] ffffffffa121cfb9 ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 [13561.616261] 00000000810d354f ffff8803162f6000 ffff8800b2a37468 0000000020000000 [13561.625556] 010000000000000c 0000000000000000 0000000000000000 ffff8800b2a36000 [13561.634858] Call Trace: [13561.638603] [] ? osd_inode_iteration+0x269/0xcc0 [osd_ldiskfs] [13561.647967] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [13561.657133] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [13561.666009] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [13561.674685] [] ? wake_up_state+0x20/0x20 [13561.681894] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [13561.691236] [] kthread+0xcf/0xe0 [13561.697677] [] ? kthread_create_on_node+0x140/0x140 [13561.705958] [] ret_from_fork+0x58/0x90 [13561.712970] [] ? kthread_create_on_node+0x140/0x140 [13561.721246] Code: 55 48 8b 8f a8 03 00 00 31 c0 0f b7 56 10 48 89 e5 48 83 39 3f 76 07 0f b7 46 30 c1 e0 10 09 d0 5d c3 66 0f 1f 84 00 00 00 00 00 <66> 66 66 66 90 55 48 8b 8f a8 03 00 00 31 c0 0f b7 56 1c 48 89 [13589.315865] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [13589.325550] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [13589.457328] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [13589.470619] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [13589.486883] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [13589.496272] RIP: 0010:[] [] osd_scrub_next+0x3d/0x3c0 [osd_ldiskfs] [13589.507922] RSP: 0018:ffff880703603c70 EFLAGS: 00000246 [13589.514865] RAX: 0000000000000000 RBX: ffff880703603c37 RCX: ffff880703603d98 [13589.523847] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [13589.532819] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [13589.541774] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [13589.550720] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [13589.559665] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [13589.569700] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13589.577122] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [13589.586098] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13589.595060] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [13589.604010] Stack: [13589.607227] 000000009f8f2001 ffff880703603d08 ffff880703603d57 ffffea00027e3c80 [13589.616522] 000000009f8f2001 ffff880703603d57 ffff880703603d57 ffffffffa121b990 [13589.625834] 0000000000000000 ffff880703603d78 ffff8800b2a36000 ffff8803162f6000 [13589.635126] Call Trace: [13589.638846] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [13589.648005] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [13589.657164] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [13589.666308] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [13589.675152] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [13589.683802] [] ? wake_up_state+0x20/0x20 [13589.690996] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [13589.700327] [] kthread+0xcf/0xe0 [13589.706741] [] ? kthread_create_on_node+0x140/0x140 [13589.715014] [] ret_from_fork+0x58/0x90 [13589.722005] [] ? kthread_create_on_node+0x140/0x140 [13589.730259] Code: 49 89 fd 41 54 49 89 d4 53 48 89 cb 48 83 ec 70 65 48 8b 04 25 28 00 00 00 48 89 45 d0 31 c0 48 8b 05 f8 59 9c ff 48 85 c0 75 7b <41> f6 87 58 10 00 00 08 0f 84 62 03 00 00 4d 8d b7 e0 10 00 00 [13603.921386] INFO: rcu_sched self-detected stall on CPU { 24} (t=6180123 jiffies g=907112 c=907111 q=19283214) [13603.933835] Task dump for CPU 24: [13603.938544] OI_scrub R running task 0 5551 2 0x00000088 [13603.947461] ffff88083fde6dd0 00000000fb6db880 ffff88082da03db0 ffffffff810c47e8 [13603.956762] 0000000000000018 ffffffff81a1e7c0 ffff88082da03dc8 ffffffff810c8079 [13603.966040] 0000000000000009 ffff88082da03df8 ffffffff811378e0 ffff88082da101c0 [13603.975285] Call Trace: [13603.978940] [] sched_show_task+0xa8/0x110 [13603.986898] [] dump_cpu_task+0x39/0x70 [13603.993828] [] rcu_dump_cpu_stacks+0x90/0xd0 [13604.001319] [] rcu_check_callbacks+0x442/0x720 [13604.008982] [] ? tick_sched_handle.isra.13+0x60/0x60 [13604.017206] [] update_process_times+0x47/0x80 [13604.024726] [] tick_sched_handle.isra.13+0x25/0x60 [13604.032713] [] tick_sched_timer+0x41/0x70 [13604.039806] [] __hrtimer_run_queues+0xd2/0x260 [13604.047363] [] hrtimer_interrupt+0xb0/0x1e0 [13604.054626] [] ? call_softirq+0x1c/0x30 [13604.061486] [] local_apic_timer_interrupt+0x37/0x60 [13604.069508] [] smp_apic_timer_interrupt+0x3f/0x60 [13604.077329] [] apic_timer_interrupt+0x6d/0x80 [13604.084745] [] ? osd_inode_iteration+0x489/0xcc0 [osd_ldiskfs] [13604.094561] [] ? osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [13604.103652] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [13604.112549] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [13604.121144] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [13604.129537] [] ? wake_up_state+0x20/0x20 [13604.136479] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [13604.145568] [] kthread+0xcf/0xe0 [13604.151736] [] ? kthread_create_on_node+0x140/0x140 [13604.159754] [] ret_from_fork+0x58/0x90 [13604.166495] [] ? kthread_create_on_node+0x140/0x140 [13629.317264] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [13629.326675] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [13629.456991] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [13629.470172] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [13629.486251] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [13629.495559] RIP: 0010:[] [] osd_inode_iteration+0x4ac/0xcc0 [osd_ldiskfs] [13629.507710] RSP: 0018:ffff880703603d18 EFLAGS: 00000216 [13629.514579] RAX: 0000000020000000 RBX: 000000009f8f2001 RCX: 0000000000000000 [13629.523495] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [13629.532404] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [13629.541317] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffff880703603d57 [13629.550228] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [13629.559136] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [13629.569123] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13629.576481] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [13629.585403] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13629.594332] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [13629.603249] Stack: [13629.606431] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [13629.615694] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [13629.624949] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [13629.634214] Call Trace: [13629.637905] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [13629.647054] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [13629.655903] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [13629.664572] [] ? wake_up_state+0x20/0x20 [13629.671779] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [13629.681120] [] kthread+0xcf/0xe0 [13629.687533] [] ? kthread_create_on_node+0x140/0x140 [13629.695799] [] ret_from_fork+0x58/0x90 [13629.702792] [] ? kthread_create_on_node+0x140/0x140 [13629.711048] Code: 88 48 8b b5 38 ff ff ff 48 8b bd 48 ff ff ff 48 8b 85 28 ff ff ff ff d0 85 c0 41 89 c5 75 1c 8b 85 58 ff ff ff 41 3b 04 24 72 10 <48> 8b 85 50 ff ff ff 44 3b 30 0f 87 95 fd ff ff 45 31 ff e9 a2 [13657.318239] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [13657.327886] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [13657.459789] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [13657.473106] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [13657.489431] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [13657.498837] RIP: 0010:[] [] osd_inode_iteration+0x4a0/0xcc0 [osd_ldiskfs] [13657.511108] RSP: 0018:ffff880703603d18 EFLAGS: 00000246 [13657.518081] RAX: 0000000000000000 RBX: 000000009f8f2001 RCX: 0000000000000000 [13657.527080] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [13657.536063] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [13657.545038] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [13657.554001] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [13657.562961] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [13657.572996] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13657.580411] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [13657.589391] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13657.598363] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [13657.607322] Stack: [13657.610542] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [13657.619836] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [13657.629124] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [13657.638432] Call Trace: [13657.642158] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [13657.651336] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [13657.660214] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [13657.668892] [] ? wake_up_state+0x20/0x20 [13657.676112] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [13657.685462] [] kthread+0xcf/0xe0 [13657.691896] [] ? kthread_create_on_node+0x140/0x140 [13657.700180] [] ret_from_fork+0x58/0x90 [13657.707203] [] ? kthread_create_on_node+0x140/0x140 [13657.715486] Code: 85 67 ff ff ff 48 8b 4d a8 48 8d 55 88 48 8b b5 38 ff ff ff 48 8b bd 48 ff ff ff 48 8b 85 28 ff ff ff ff d0 85 c0 41 89 c5 75 1c <8b> 85 58 ff ff ff 41 3b 04 24 72 10 48 8b 85 50 ff ff ff 44 3b [13685.319213] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [13685.328834] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [13685.460808] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [13685.474115] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [13685.490398] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [13685.499813] RIP: 0010:[] [] osd_scrub_exec+0x95/0x5b0 [osd_ldiskfs] [13685.511494] RSP: 0018:ffff880703603d08 EFLAGS: 00000292 [13685.518460] RAX: 0000000000000000 RBX: 0000000000000001 RCX: 0000000000000000 [13685.527448] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [13685.536430] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [13685.545389] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [13685.554348] R13: 0000000000000018 R14: ffff880703603c78 R15: 0000000000000246 [13685.563294] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [13685.573306] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13685.580707] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [13685.589667] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13685.598625] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [13685.607567] Stack: [13685.610774] ffff880703603df0 ffffffffa121d1e9 ffffffffa121b990 ffffffffa1217a20 [13685.620056] ffff8800b2a36000 00000000810d354f ffff8803162f6000 ffff8800b2a37468 [13685.629345] 0000000020000000 010000000000000c 0000000000000000 0000000000000000 [13685.638639] Call Trace: [13685.642364] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [13685.651520] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [13685.660678] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [13685.669546] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [13685.678209] [] ? wake_up_state+0x20/0x20 [13685.685410] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [13685.694758] [] kthread+0xcf/0xe0 [13685.701182] [] ? kthread_create_on_node+0x140/0x140 [13685.709467] [] ret_from_fork+0x58/0x90 [13685.716474] [] ? kthread_create_on_node+0x140/0x140 [13685.724736] Code: 01 00 00 41 80 a7 74 14 00 00 fe 48 8b 4d d0 65 48 33 0c 25 28 00 00 00 0f 85 b1 03 00 00 48 83 c4 68 5b 41 5c 41 5d 41 5e 41 5f <5d> c3 66 0f 1f 84 00 00 00 00 00 4c 8d ae f0 10 00 00 44 89 4d [13713.320184] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [13713.329789] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [13713.461453] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [13713.474730] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [13713.490960] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [13713.500334] RIP: 0010:[] [] osd_scrub_exec+0x96/0x5b0 [osd_ldiskfs] [13713.511973] RSP: 0018:ffff880703603d10 EFLAGS: 00000292 [13713.518905] RAX: 0000000000000000 RBX: 000000009f8f2001 RCX: 0000000000000000 [13713.527872] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [13713.536827] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [13713.545771] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [13713.554718] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [13713.563659] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [13713.573683] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13713.581079] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [13713.590040] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13713.599002] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [13713.607943] Stack: [13713.611150] ffffffffa121d1e9 ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 [13713.620441] 00000000810d354f ffff8803162f6000 ffff8800b2a37468 0000000020000000 [13713.629747] 010000000000000c 0000000000000000 0000000000000000 ffff8800b2a36000 [13713.639037] Call Trace: [13713.642754] [] ? osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [13713.652113] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [13713.661278] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [13713.670142] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [13713.678813] [] ? wake_up_state+0x20/0x20 [13713.686023] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [13713.695365] [] kthread+0xcf/0xe0 [13713.701798] [] ? kthread_create_on_node+0x140/0x140 [13713.710077] [] ret_from_fork+0x58/0x90 [13713.717090] [] ? kthread_create_on_node+0x140/0x140 [13713.725371] Code: 00 00 41 80 a7 74 14 00 00 fe 48 8b 4d d0 65 48 33 0c 25 28 00 00 00 0f 85 b1 03 00 00 48 83 c4 68 5b 41 5c 41 5d 41 5e 41 5f 5d 66 0f 1f 84 00 00 00 00 00 4c 8d ae f0 10 00 00 44 89 4d 88 [13741.321167] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [13741.330815] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [13741.462640] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [13741.475932] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [13741.492197] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [13741.501584] RIP: 0010:[] [] osd_scrub_exec+0x1c/0x5b0 [osd_ldiskfs] [13741.513250] RSP: 0018:ffff880703603c78 EFLAGS: 00000296 [13741.520197] RAX: ffffffffa121b990 RBX: ffff880703603c37 RCX: 0000000000000000 [13741.529179] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [13741.538153] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [13741.547115] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [13741.556068] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [13741.565019] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [13741.575057] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13741.582466] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [13741.591444] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13741.600412] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [13741.609361] Stack: [13741.612582] ffffea00027e3c80 000000009f8f2001 0000000000000004 0000000000000001 [13741.621881] ffff880703603d57 ffff880703603d98 ffff880703603d78 ffff8800b2a36000 [13741.631189] ffff8803162f6000 ffffffffffffff10 ffffffffa1217a32 0000000000000010 [13741.640496] Call Trace: [13741.644220] [] ? osd_scrub_next+0x12/0x3c0 [osd_ldiskfs] [13741.653013] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [13741.662185] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [13741.671345] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [13741.680208] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [13741.688888] [] ? wake_up_state+0x20/0x20 [13741.696098] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [13741.705461] [] kthread+0xcf/0xe0 [13741.711900] [] ? kthread_create_on_node+0x140/0x140 [13741.720178] [] ret_from_fork+0x58/0x90 [13741.727195] [] ? kthread_create_on_node+0x140/0x140 [13741.735474] Code: 89 04 24 31 c0 e8 a5 03 98 ff e9 c7 fd ff ff 66 66 66 66 90 55 48 89 e5 41 57 49 89 f7 41 56 41 55 41 54 45 31 e4 53 48 83 ec 68 <48> 8b 9e 08 10 00 00 4c 89 45 90 65 48 8b 04 25 28 00 00 00 48 [13769.322143] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [13769.331792] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [13769.463570] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [13769.476842] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [13769.493081] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [13769.502452] RIP: 0010:[] [] osd_scrub_exec+0x8d/0x5b0 [osd_ldiskfs] [13769.514095] RSP: 0018:ffff880703603ce8 EFLAGS: 00000292 [13769.521031] RAX: 0000000000000000 RBX: ffff880703603c78 RCX: 0000000000000000 [13769.530004] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [13769.538968] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [13769.547921] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000010 [13769.556870] R13: ffffffffa121ba87 R14: ffffffffffffff10 R15: ffff8803162f6000 [13769.565829] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [13769.575856] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13769.583270] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [13769.592233] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13769.601192] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [13769.610131] Stack: [13769.613358] ffff8800b2a3746c 0000000000000004 00000000ffffffff 0000000000000001 [13769.622659] ffff880703603df0 ffffffffa121d1e9 ffffffffa121b990 ffffffffa1217a20 [13769.631961] ffff8800b2a36000 00000000810d354f ffff8803162f6000 ffff8800b2a37468 [13769.641248] Call Trace: [13769.644974] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [13769.654148] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [13769.663303] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [13769.672161] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [13769.680830] [] ? wake_up_state+0x20/0x20 [13769.688046] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [13769.697392] [] kthread+0xcf/0xe0 [13769.703822] [] ? kthread_create_on_node+0x140/0x140 [13769.712111] [] ret_from_fork+0x58/0x90 [13769.719128] [] ? kthread_create_on_node+0x140/0x140 [13769.727397] Code: 85 c0 41 89 c1 0f 84 a3 01 00 00 41 80 a7 74 14 00 00 fe 48 8b 4d d0 65 48 33 0c 25 28 00 00 00 0f 85 b1 03 00 00 48 83 c4 68 5b <41> 5c 41 5d 41 5e 41 5f 5d c3 66 0f 1f 84 00 00 00 00 00 4c 8d [13783.930657] INFO: rcu_sched self-detected stall on CPU { 24} (t=6360126 jiffies g=907112 c=907111 q=19635298) [13783.943127] Task dump for CPU 24: [13783.947863] OI_scrub R running task 0 5551 2 0x00000088 [13783.956793] ffff88083fde6dd0 00000000fb6db880 ffff88082da03db0 ffffffff810c47e8 [13783.966125] 0000000000000018 ffffffff81a1e7c0 ffff88082da03dc8 ffffffff810c8079 [13783.975421] 0000000000000009 ffff88082da03df8 ffffffff811378e0 ffff88082da101c0 [13783.984707] Call Trace: [13783.988403] [] sched_show_task+0xa8/0x110 [13783.996378] [] dump_cpu_task+0x39/0x70 [13784.003348] [] rcu_dump_cpu_stacks+0x90/0xd0 [13784.010870] [] rcu_check_callbacks+0x442/0x720 [13784.018568] [] ? tick_sched_handle.isra.13+0x60/0x60 [13784.026818] [] update_process_times+0x47/0x80 [13784.034371] [] tick_sched_handle.isra.13+0x25/0x60 [13784.042383] [] tick_sched_timer+0x41/0x70 [13784.049518] [] __hrtimer_run_queues+0xd2/0x260 [13784.057114] [] hrtimer_interrupt+0xb0/0x1e0 [13784.064394] [] ? call_softirq+0x1c/0x30 [13784.071276] [] local_apic_timer_interrupt+0x37/0x60 [13784.079306] [] smp_apic_timer_interrupt+0x3f/0x60 [13784.087124] [] apic_timer_interrupt+0x6d/0x80 [13784.094550] [] ? osd_inode_iteration+0x2be/0xcc0 [osd_ldiskfs] [13784.104370] [] ? osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [13784.113476] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [13784.122383] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [13784.130998] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [13784.139415] [] ? wake_up_state+0x20/0x20 [13784.146367] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [13784.155463] [] kthread+0xcf/0xe0 [13784.161639] [] ? kthread_create_on_node+0x140/0x140 [13784.169664] [] ret_from_fork+0x58/0x90 [13784.176425] [] ? kthread_create_on_node+0x140/0x140 [13809.323529] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [13809.332951] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [13809.462700] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [13809.475886] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [13809.491957] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [13809.501261] RIP: 0010:[] [] osd_inode_iteration+0x2be/0xcc0 [osd_ldiskfs] [13809.513404] RSP: 0018:ffff880703603d18 EFLAGS: 00000297 [13809.520265] RAX: 0000000000000004 RBX: 000000009f8f2001 RCX: ffff880703603d98 [13809.529172] RDX: ffff880703603d78 RSI: 0000000000000000 RDI: ffff8803162f6000 [13809.538087] RBP: ffff880703603df0 R08: 0000000000000001 R09: 0000000000000004 [13809.546990] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [13809.555881] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [13809.564775] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [13809.574740] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13809.582084] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [13809.590988] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13809.599893] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [13809.608788] Stack: [13809.611961] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [13809.621200] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [13809.630455] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [13809.639708] Call Trace: [13809.643382] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [13809.652515] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [13809.661365] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [13809.670020] [] ? wake_up_state+0x20/0x20 [13809.677211] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [13809.686540] [] kthread+0xcf/0xe0 [13809.692946] [] ? kthread_create_on_node+0x140/0x140 [13809.701196] [] ret_from_fork+0x58/0x90 [13809.708185] [] ? kthread_create_on_node+0x140/0x140 [13809.716432] Code: 8b b5 38 ff ff ff 48 8b bd 48 ff ff ff 48 8b 85 30 ff ff ff ff d0 83 f8 03 41 89 c5 0f 84 4b 02 00 00 0f 8e 15 01 00 00 83 f8 05 <66> 90 0f 84 8a 02 00 00 83 f8 06 0f 85 a1 01 00 00 48 8b 7d 90 [13837.324505] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [13837.334146] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [13837.466086] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [13837.479383] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [13837.495670] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [13837.505073] RIP: 0010:[] [] osd_scrub_next+0x3d/0x3c0 [osd_ldiskfs] [13837.516743] RSP: 0018:ffff880703603c70 EFLAGS: 00000246 [13837.523701] RAX: 0000000000000000 RBX: ffff880703603c37 RCX: ffff880703603d98 [13837.532685] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [13837.541660] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [13837.550626] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [13837.559578] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [13837.568534] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [13837.578561] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13837.585975] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [13837.594943] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13837.603913] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [13837.612870] Stack: [13837.616084] 000000009f8f2001 ffff880703603d08 0000000000000001 ffff880703603df0 [13837.625376] 000000009f8f2001 ffff880703603d57 000000009f8f2001 0000000000000004 [13837.634659] ffff880703603d57 0000000000000004 0000000000000000 ffff880703603d78 [13837.643972] Call Trace: [13837.647702] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [13837.656865] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [13837.666024] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [13837.674891] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [13837.683565] [] ? wake_up_state+0x20/0x20 [13837.690768] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [13837.700112] [] kthread+0xcf/0xe0 [13837.706544] [] ? kthread_create_on_node+0x140/0x140 [13837.714817] [] ret_from_fork+0x58/0x90 [13837.721822] [] ? kthread_create_on_node+0x140/0x140 [13837.730092] Code: 49 89 fd 41 54 49 89 d4 53 48 89 cb 48 83 ec 70 65 48 8b 04 25 28 00 00 00 48 89 45 d0 31 c0 48 8b 05 f8 59 9c ff 48 85 c0 75 7b <41> f6 87 58 10 00 00 08 0f 84 62 03 00 00 4d 8d b7 e0 10 00 00 [13865.325479] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [13865.335106] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [13865.466818] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [13865.480106] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [13865.496347] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [13865.505727] RIP: 0010:[] [] osd_scrub_exec+0xd4/0x5b0 [osd_ldiskfs] [13865.517374] RSP: 0018:ffff880703603c78 EFLAGS: 00000286 [13865.524313] RAX: ffff8803162f1008 RBX: ffff880703603c37 RCX: 0000000000000000 [13865.533286] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [13865.542254] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [13865.551200] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [13865.560138] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [13865.569075] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [13865.579100] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13865.586501] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [13865.595464] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13865.604415] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [13865.613354] Stack: [13865.616568] ffff880703603d08 0000000000000001 ffff880703603df0 000000009f8f2001 [13865.625855] ffff880703603d57 000000009f8f2001 0000000000000004 ffff880703603d57 [13865.635140] ffff8800b2a37468 0000000000000000 ffff880703603d78 ffff8800b2a36000 [13865.644420] Call Trace: [13865.648130] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [13865.657297] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [13865.666452] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [13865.675310] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [13865.683963] [] ? wake_up_state+0x20/0x20 [13865.691168] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [13865.700517] [] kthread+0xcf/0xe0 [13865.706943] [] ? kthread_create_on_node+0x140/0x140 [13865.715215] [] ret_from_fork+0x58/0x90 [13865.722219] [] ? kthread_create_on_node+0x140/0x140 [13865.730481] Code: 4d 88 4c 89 ef e8 1d fb 46 e0 41 83 87 68 14 00 00 01 49 83 87 88 11 00 00 01 4c 89 ef e8 f5 9f e9 df 44 8b 4d 88 48 85 db 74 23 83 18 0a 00 00 08 74 1a 4d 85 e4 74 15 41 8b 87 6c 14 00 00 [13893.326455] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [13893.336147] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [13893.467888] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [13893.481171] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [13893.497417] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [13893.506807] RIP: 0010:[] [] osd_inode_iteration+0x255/0xcc0 [osd_ldiskfs] [13893.519035] RSP: 0018:ffff880703603d18 EFLAGS: 00000286 [13893.525983] RAX: ffff8800b2a37468 RBX: 000000009f8f2001 RCX: 0000000000000000 [13893.534961] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [13893.543931] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [13893.552888] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [13893.561834] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [13893.570789] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [13893.580822] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13893.588226] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [13893.597198] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13893.606151] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [13893.615091] Stack: [13893.618297] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [13893.627598] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [13893.636877] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [13893.646157] Call Trace: [13893.649870] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [13893.659039] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [13893.667917] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [13893.676585] [] ? wake_up_state+0x20/0x20 [13893.683790] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [13893.693139] [] kthread+0xcf/0xe0 [13893.699579] [] ? kthread_create_on_node+0x140/0x140 [13893.707858] [] ret_from_fork+0x58/0x90 [13893.714873] [] ? kthread_create_on_node+0x140/0x140 [13893.723158] Code: 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 00 8b 75 98 48 8b 7d 88 e8 dc 2d f5 ff 48 85 c0 48 89 45 90 0f 84 b3 08 00 00 44 8b 7d a0 <48> 8b 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 e8 77 5b f4 ff 42 [13921.327432] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [13921.337135] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [13921.468993] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [13921.482310] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [13921.498591] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [13921.508007] RIP: 0010:[] [] osd_scrub_next+0x160/0x3c0 [osd_ldiskfs] [13921.519769] RSP: 0018:ffff880703603ce0 EFLAGS: 00000282 [13921.526729] RAX: 0000000000000004 RBX: ffff880703603c78 RCX: ffff880703603d98 [13921.535715] RDX: ffff880703603d78 RSI: 0000000000000000 RDI: ffff8803162f6000 [13921.544696] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [13921.553669] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000010 [13921.562619] R13: ffffffffa121ba89 R14: ffffffffffffff10 R15: ffff8803162f6000 [13921.571571] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [13921.581597] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13921.589007] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [13921.597967] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13921.606929] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [13921.615873] Stack: [13921.619081] ffff880419dc3a00 ffff8800b2a3746c 0000000000000000 00000000ffffffff [13921.628354] 0000000000000001 ffff880703603df0 ffffffffa121cff9 ffffffffa121b990 [13921.637643] ffffffffa1217a20 ffff8800b2a36000 00000000810d354f ffff8803162f6000 [13921.646948] Call Trace: [13921.650678] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [13921.659852] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [13921.669006] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [13921.677880] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [13921.686557] [] ? wake_up_state+0x20/0x20 [13921.693764] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [13921.703108] [] kthread+0xcf/0xe0 [13921.709543] [] ? kthread_create_on_node+0x140/0x140 [13921.717821] [] ret_from_fork+0x58/0x90 [13921.724833] [] ? kthread_create_on_node+0x140/0x140 [13921.733101] Code: 10 00 00 02 00 00 00 48 89 df e8 fc 69 47 e0 b8 05 00 00 00 48 8b 75 d0 65 48 33 34 25 28 00 00 00 0f 85 4c 02 00 00 48 83 c4 70 <5b> 41 5c 41 5d 41 5e 41 5f 5d c3 48 8b 05 be 58 9c ff 48 85 c0 [13949.328405] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [13949.338087] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [13949.469845] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [13949.483133] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [13949.499379] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [13949.508763] RIP: 0010:[] [] osd_inode_iteration+0x255/0xcc0 [osd_ldiskfs] [13949.521002] RSP: 0018:ffff880703603d18 EFLAGS: 00000286 [13949.527945] RAX: ffff8800b2a37468 RBX: 000000009f8f2001 RCX: 0000000000000000 [13949.536924] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [13949.545900] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [13949.554854] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000001 [13949.563804] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [13949.572752] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [13949.582786] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13949.590197] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [13949.599162] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13949.608122] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [13949.617064] Stack: [13949.620275] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [13949.629576] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [13949.638878] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [13949.648190] Call Trace: [13949.651907] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [13949.661071] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [13949.669940] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [13949.678605] [] ? wake_up_state+0x20/0x20 [13949.685807] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [13949.695154] [] kthread+0xcf/0xe0 [13949.701589] [] ? kthread_create_on_node+0x140/0x140 [13949.709875] [] ret_from_fork+0x58/0x90 [13949.716884] [] ? kthread_create_on_node+0x140/0x140 [13949.725162] Code: 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 00 8b 75 98 48 8b 7d 88 e8 dc 2d f5 ff 48 85 c0 48 89 45 90 0f 84 b3 08 00 00 44 8b 7d a0 <48> 8b 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 e8 77 5b f4 ff 42 [13963.939914] INFO: rcu_sched self-detected stall on CPU { 24} (t=6540129 jiffies g=907112 c=907111 q=21002172) [13963.952393] Task dump for CPU 24: [13963.957150] OI_scrub R running task 0 5551 2 0x00000088 [13963.966092] ffff88083fde6dd0 00000000fb6db880 ffff88082da03db0 ffffffff810c47e8 [13963.975419] 0000000000000018 ffffffff81a1e7c0 ffff88082da03dc8 ffffffff810c8079 [13963.984733] 0000000000000009 ffff88082da03df8 ffffffff811378e0 ffff88082da101c0 [13963.994016] Call Trace: [13963.997724] [] sched_show_task+0xa8/0x110 [13964.005727] [] dump_cpu_task+0x39/0x70 [13964.012711] [] rcu_dump_cpu_stacks+0x90/0xd0 [13964.020247] [] rcu_check_callbacks+0x442/0x720 [13964.027947] [] ? tick_sched_handle.isra.13+0x60/0x60 [13964.036218] [] update_process_times+0x47/0x80 [13964.043781] [] tick_sched_handle.isra.13+0x25/0x60 [13964.051815] [] tick_sched_timer+0x41/0x70 [13964.058950] [] __hrtimer_run_queues+0xd2/0x260 [13964.066548] [] hrtimer_interrupt+0xb0/0x1e0 [13964.073837] [] ? call_softirq+0x1c/0x30 [13964.080717] [] local_apic_timer_interrupt+0x37/0x60 [13964.088752] [] smp_apic_timer_interrupt+0x3f/0x60 [13964.096580] [] apic_timer_interrupt+0x6d/0x80 [13964.104002] [] ? ldiskfs_itable_unused_count+0x23/0x30 [ldiskfs] [13964.114031] [] osd_inode_iteration+0x269/0xcc0 [osd_ldiskfs] [13964.122940] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [13964.131839] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [13964.140449] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [13964.148858] [] ? wake_up_state+0x20/0x20 [13964.155808] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [13964.164886] [] kthread+0xcf/0xe0 [13964.171051] [] ? kthread_create_on_node+0x140/0x140 [13964.179067] [] ret_from_fork+0x58/0x90 [13964.185823] [] ? kthread_create_on_node+0x140/0x140 [13989.329799] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [13989.339217] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [13989.468859] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [13989.482030] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [13989.498093] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [13989.507378] RIP: 0010:[] [] osd_scrub_exec+0xf7/0x5b0 [osd_ldiskfs] [13989.518936] RSP: 0018:ffff880703603c78 EFLAGS: 00000246 [13989.525790] RAX: 0000000000280001 RBX: ffff880703603c37 RCX: 0000000000000000 [13989.534692] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [13989.543588] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [13989.552490] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [13989.561387] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [13989.570284] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [13989.580257] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13989.587612] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [13989.596522] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13989.605426] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [13989.614314] Stack: [13989.617480] ffff880703603df0 000000009f8f2001 ffffea00027e3c80 000000009f8f2001 [13989.626731] ffff880703603d57 ffff880703603d57 ffff8800b2a37468 0000000000000000 [13989.636002] ffff880703603d78 ffff880419dc3a00 ffff88042bd79800 ffffffffffffff10 [13989.645255] Call Trace: [13989.648937] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [13989.658066] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [13989.667192] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [13989.676027] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [13989.684675] [] ? wake_up_state+0x20/0x20 [13989.691859] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [13989.701168] [] kthread+0xcf/0xe0 [13989.707566] [] ? kthread_create_on_node+0x140/0x140 [13989.715814] [] ret_from_fork+0x58/0x90 [13989.722788] [] ? kthread_create_on_node+0x140/0x140 [13989.731026] Code: 8b 4d 88 48 85 db 74 23 f6 83 18 0a 00 00 08 74 1a 4d 85 e4 74 15 41 8b 87 6c 14 00 00 41 39 84 24 0c 0a 00 00 0f 82 d9 00 00 00 <31> c0 41 f6 87 74 14 00 00 04 0f 85 6e ff ff ff 41 83 f9 02 0f [14017.330775] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [14017.340428] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [14017.472161] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [14017.485453] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [14017.501705] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [14017.511085] RIP: 0010:[] [] osd_scrub_exec+0x96/0x5b0 [osd_ldiskfs] [14017.522726] RSP: 0018:ffff880703603d10 EFLAGS: 00000292 [14017.529652] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000 [14017.538622] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [14017.547580] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [14017.556526] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffff880703603d57 [14017.565469] R13: 0000000000000004 R14: 000000009f8f2001 R15: 0000000000000018 [14017.574421] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [14017.584439] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14017.591841] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [14017.600796] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14017.609740] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [14017.618671] Stack: [14017.621869] ffffffffa121d1e9 ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 [14017.631129] 00000000810d354f ffff8803162f6000 ffff8800b2a37468 0000000020000000 [14017.640417] 010000000000000c 0000000000000000 0000000000000000 ffff8800b2a36000 [14017.649696] Call Trace: [14017.653413] [] ? osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [14017.662771] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [14017.671933] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [14017.680794] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [14017.689457] [] ? wake_up_state+0x20/0x20 [14017.696673] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [14017.706021] [] kthread+0xcf/0xe0 [14017.712457] [] ? kthread_create_on_node+0x140/0x140 [14017.720742] [] ret_from_fork+0x58/0x90 [14017.727751] [] ? kthread_create_on_node+0x140/0x140 [14017.736019] Code: 00 00 41 80 a7 74 14 00 00 fe 48 8b 4d d0 65 48 33 0c 25 28 00 00 00 0f 85 b1 03 00 00 48 83 c4 68 5b 41 5c 41 5d 41 5e 41 5f 5d 66 0f 1f 84 00 00 00 00 00 4c 8d ae f0 10 00 00 44 89 4d 88 [14045.331751] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [14045.341410] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [14045.473342] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [14045.486652] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [14045.502904] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [14045.512294] RIP: 0010:[] [] osd_scrub_next+0x3d/0x3c0 [osd_ldiskfs] [14045.523949] RSP: 0018:ffff880703603c70 EFLAGS: 00000246 [14045.530893] RAX: 0000000000000000 RBX: ffff880703603c37 RCX: ffff880703603d98 [14045.539861] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [14045.548825] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [14045.557778] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [14045.566730] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [14045.575679] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [14045.585717] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14045.593132] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [14045.602113] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14045.611072] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [14045.620028] Stack: [14045.623264] 000000009f8f2001 0000000000000004 0000000000000001 ffffffffa1217a20 [14045.632574] ffff880703603d98 ffff880703603d57 ffff8800b2a36000 ffff8803162f6000 [14045.641873] ffffffffffffff10 ffffffffa1217a3b 0000000000000010 0000000000000287 [14045.651185] Call Trace: [14045.654926] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [14045.663807] [] ? osd_scrub_next+0x1b/0x3c0 [osd_ldiskfs] [14045.672606] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [14045.681779] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [14045.690948] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [14045.699814] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [14045.708497] [] ? wake_up_state+0x20/0x20 [14045.715708] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [14045.725070] [] kthread+0xcf/0xe0 [14045.731515] [] ? kthread_create_on_node+0x140/0x140 [14045.739814] [] ret_from_fork+0x58/0x90 [14045.746834] [] ? kthread_create_on_node+0x140/0x140 [14045.755123] Code: 49 89 fd 41 54 49 89 d4 53 48 89 cb 48 83 ec 70 65 48 8b 04 25 28 00 00 00 48 89 45 d0 31 c0 48 8b 05 f8 59 9c ff 48 85 c0 75 7b <41> f6 87 58 10 00 00 08 0f 84 62 03 00 00 4d 8d b7 e0 10 00 00 [14073.332724] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [14073.342395] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [14073.473932] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [14073.487185] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [14073.503388] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [14073.512743] RIP: 0010:[] [] osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [14073.524765] RSP: 0018:ffff880703603d10 EFLAGS: 00000293 [14073.531691] RAX: ffffffffa121b990 RBX: 000000009f8f2001 RCX: 0000000000000000 [14073.540656] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [14073.549615] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [14073.558577] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffff880703603d57 [14073.567543] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [14073.576500] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [14073.586541] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14073.593947] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [14073.602908] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14073.611862] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [14073.620813] Stack: [14073.624040] ffffffffa121d1e9 ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 [14073.633334] 00000000810d354f ffff8803162f6000 ffff8800b2a37468 0000000020000000 [14073.642635] 010000000000000c 0000000000000000 0000000000000000 ffff8800b2a36000 [14073.651932] Call Trace: [14073.655654] [] ? osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [14073.665012] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [14073.674169] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [14073.683035] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [14073.691712] [] ? wake_up_state+0x20/0x20 [14073.698927] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [14073.708284] [] kthread+0xcf/0xe0 [14073.714723] [] ? kthread_create_on_node+0x140/0x140 [14073.723016] [] ret_from_fork+0x58/0x90 [14073.730039] [] ? kthread_create_on_node+0x140/0x140 [14073.738317] Code: b5 07 00 00 c7 05 85 a5 02 00 00 00 02 00 89 54 24 08 48 8b 40 40 ba 07 00 00 00 48 89 04 24 31 c0 e8 a5 03 98 ff e9 c7 fd ff ff <66> 66 66 66 90 55 48 89 e5 41 57 49 89 f7 41 56 41 55 41 54 45 [14101.333701] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [14101.343380] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [14101.475521] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [14101.488836] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [14101.505119] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [14101.514518] RIP: 0010:[] [] osd_scrub_exec+0xd4/0x5b0 [osd_ldiskfs] [14101.526185] RSP: 0018:ffff880703603c78 EFLAGS: 00000286 [14101.533140] RAX: ffff8803162f1008 RBX: ffff880703603c37 RCX: 0000000000000000 [14101.542133] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [14101.551106] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [14101.560061] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [14101.569024] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [14101.577988] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [14101.588013] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14101.595424] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [14101.604409] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14101.613372] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [14101.622327] Stack: [14101.625557] ffffea00027e3c80 000000009f8f2001 ffff880703603df0 0000000000000001 [14101.634860] ffff880703603d57 000000009f8f2001 0000000000000004 ffff880703603d57 [14101.644154] ffffffffa121b990 0000000000000000 ffff880703603d78 ffff8800b2a36000 [14101.653456] Call Trace: [14101.657176] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [14101.666343] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [14101.675504] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [14101.684656] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [14101.693515] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [14101.702181] [] ? wake_up_state+0x20/0x20 [14101.709388] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [14101.718727] [] kthread+0xcf/0xe0 [14101.725149] [] ? kthread_create_on_node+0x140/0x140 [14101.733439] [] ret_from_fork+0x58/0x90 [14101.740446] [] ? kthread_create_on_node+0x140/0x140 [14101.748707] Code: 4d 88 4c 89 ef e8 1d fb 46 e0 41 83 87 68 14 00 00 01 49 83 87 88 11 00 00 01 4c 89 ef e8 f5 9f e9 df 44 8b 4d 88 48 85 db 74 23 83 18 0a 00 00 08 74 1a 4d 85 e4 74 15 41 8b 87 6c 14 00 00 [14129.334675] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [14129.344350] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [14129.475910] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [14129.489170] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [14129.505369] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [14129.514732] RIP: 0010:[] [] osd_scrub_exec+0xe9/0x5b0 [osd_ldiskfs] [14129.526365] RSP: 0018:ffff880703603c78 EFLAGS: 00000282 [14129.533294] RAX: 0000000000280001 RBX: ffff880703603c37 RCX: 0000000000000000 [14129.542259] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [14129.551216] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [14129.560163] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [14129.569096] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [14129.578052] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [14129.588080] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14129.595481] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [14129.604434] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14129.613386] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [14129.622320] Stack: [14129.625527] 0000000000000004 0000000000000001 ffff880703603df0 ffff880703603d98 [14129.634820] ffff880703603d57 000000009f8f2001 0000000000000004 ffff880703603d57 [14129.644107] ffff8800b2a37468 0000000000000000 ffff880703603d78 ffff8800b2a36000 [14129.653381] Call Trace: [14129.657096] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [14129.666253] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [14129.675400] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [14129.684257] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [14129.692923] [] ? wake_up_state+0x20/0x20 [14129.700126] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [14129.709469] [] kthread+0xcf/0xe0 [14129.715887] [] ? kthread_create_on_node+0x140/0x140 [14129.724183] [] ret_from_fork+0x58/0x90 [14129.731194] [] ? kthread_create_on_node+0x140/0x140 [14129.739459] Code: 88 11 00 00 01 4c 89 ef e8 f5 9f e9 df 44 8b 4d 88 48 85 db 74 23 f6 83 18 0a 00 00 08 74 1a 4d 85 e4 74 15 41 8b 87 6c 14 00 00 <41> 39 84 24 0c 0a 00 00 0f 82 d9 00 00 00 31 c0 41 f6 87 74 14 [14143.949185] INFO: rcu_sched self-detected stall on CPU { 24} (t=6720132 jiffies g=907112 c=907111 q=21876797) [14143.961685] Task dump for CPU 24: [14143.966412] OI_scrub R running task 0 5551 2 0x00000088 [14143.975322] ffff88083fde6dd0 00000000fb6db880 ffff88082da03db0 ffffffff810c47e8 [14143.984639] 0000000000000018 ffffffff81a1e7c0 ffff88082da03dc8 ffffffff810c8079 [14143.993933] 0000000000000009 ffff88082da03df8 ffffffff811378e0 ffff88082da101c0 [14144.003202] Call Trace: [14144.006869] [] sched_show_task+0xa8/0x110 [14144.014844] [] dump_cpu_task+0x39/0x70 [14144.021797] [] rcu_dump_cpu_stacks+0x90/0xd0 [14144.029310] [] rcu_check_callbacks+0x442/0x720 [14144.036996] [] ? tick_sched_handle.isra.13+0x60/0x60 [14144.045242] [] update_process_times+0x47/0x80 [14144.052792] [] tick_sched_handle.isra.13+0x25/0x60 [14144.060808] [] tick_sched_timer+0x41/0x70 [14144.067933] [] __hrtimer_run_queues+0xd2/0x260 [14144.075529] [] hrtimer_interrupt+0xb0/0x1e0 [14144.082813] [] ? call_softirq+0x1c/0x30 [14144.089689] [] local_apic_timer_interrupt+0x37/0x60 [14144.097717] [] smp_apic_timer_interrupt+0x3f/0x60 [14144.105541] [] apic_timer_interrupt+0x6d/0x80 [14144.112964] [] ? osd_scrub_next+0x149/0x3c0 [osd_ldiskfs] [14144.122291] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [14144.130900] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [14144.139801] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [14144.148705] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [14144.157308] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [14144.165702] [] ? wake_up_state+0x20/0x20 [14144.172637] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [14144.181722] [] kthread+0xcf/0xe0 [14144.187892] [] ? kthread_create_on_node+0x140/0x140 [14144.195904] [] ret_from_fork+0x58/0x90 [14144.202644] [] ? kthread_create_on_node+0x140/0x140 [14169.336075] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [14169.345473] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [14169.475607] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [14169.488783] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [14169.504823] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [14169.514103] RIP: 0010:[] [] osd_inode_iteration+0x2c0/0xcc0 [osd_ldiskfs] [14169.526251] RSP: 0018:ffff880703603d18 EFLAGS: 00000297 [14169.533114] RAX: 0000000000000004 RBX: 000000009f8f2001 RCX: ffff880703603d98 [14169.542016] RDX: ffff880703603d78 RSI: 0000000000000000 RDI: ffff8803162f6000 [14169.550919] RBP: ffff880703603df0 R08: 0000000000000001 R09: 0000000000000004 [14169.559819] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000001 [14169.568710] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [14169.577596] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [14169.587558] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14169.594899] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [14169.603813] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14169.612717] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [14169.621612] Stack: [14169.624785] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [14169.634025] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [14169.643279] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [14169.652517] Call Trace: [14169.656205] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [14169.665351] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [14169.674189] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [14169.682842] [] ? wake_up_state+0x20/0x20 [14169.690022] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [14169.699351] [] kthread+0xcf/0xe0 [14169.705746] [] ? kthread_create_on_node+0x140/0x140 [14169.714007] [] ret_from_fork+0x58/0x90 [14169.720989] [] ? kthread_create_on_node+0x140/0x140 [14169.729238] Code: 38 ff ff ff 48 8b bd 48 ff ff ff 48 8b 85 30 ff ff ff ff d0 83 f8 03 41 89 c5 0f 84 4b 02 00 00 0f 8e 15 01 00 00 83 f8 05 66 90 <0f> 84 8a 02 00 00 83 f8 06 0f 85 a1 01 00 00 48 8b 7d 90 48 85 [14197.337049] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [14197.346643] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [14197.478574] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [14197.491882] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [14197.508175] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [14197.517579] RIP: 0010:[] [] osd_scrub_exec+0xf7/0x5b0 [osd_ldiskfs] [14197.529244] RSP: 0018:ffff880703603c78 EFLAGS: 00000246 [14197.536205] RAX: 0000000000280001 RBX: ffff880703603c37 RCX: 0000000000000000 [14197.545192] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [14197.554178] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [14197.563147] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [14197.572099] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [14197.581048] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [14197.591076] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14197.598487] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [14197.607460] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14197.616431] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [14197.625393] Stack: [14197.628615] ffff880703603d08 ffff880703603d57 ffff880703603df0 000000009f8f2001 [14197.637902] ffff880703603d57 000000009f8f2001 0000000000000004 ffff880703603d57 [14197.647198] 0000000000000000 0000000000000000 ffff880703603d78 ffff8800b2a36000 [14197.656493] Call Trace: [14197.660216] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [14197.669390] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [14197.678561] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [14197.687433] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [14197.696120] [] ? wake_up_state+0x20/0x20 [14197.703345] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [14197.712720] [] kthread+0xcf/0xe0 [14197.719156] [] ? kthread_create_on_node+0x140/0x140 [14197.727453] [] ret_from_fork+0x58/0x90 [14197.734478] [] ? kthread_create_on_node+0x140/0x140 [14197.742760] Code: 8b 4d 88 48 85 db 74 23 f6 83 18 0a 00 00 08 74 1a 4d 85 e4 74 15 41 8b 87 6c 14 00 00 41 39 84 24 0c 0a 00 00 0f 82 d9 00 00 00 <31> c0 41 f6 87 74 14 00 00 04 0f 85 6e ff ff ff 41 83 f9 02 0f [14225.338019] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [14225.347633] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [14225.479386] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [14225.492674] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [14225.508936] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [14225.518320] RIP: 0010:[] [] osd_inode_iteration+0x282/0xcc0 [osd_ldiskfs] [14225.530553] RSP: 0018:ffff880703603d18 EFLAGS: 00000287 [14225.537494] RAX: ffff88042bd78800 RBX: ffff880703603d57 RCX: ffff88042bd78800 [14225.546482] RDX: 00000000000072cf RSI: ffff880419dc3a00 RDI: ffff88042bd79800 [14225.555448] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [14225.564408] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [14225.573359] R13: ffffea00027e3c80 R14: 000000009f8f2001 R15: 0000000000000018 [14225.582310] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [14225.592340] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14225.599748] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [14225.608719] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14225.617680] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [14225.626624] Stack: [14225.629840] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [14225.639132] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [14225.648434] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [14225.657739] Call Trace: [14225.661461] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [14225.670623] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [14225.679493] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [14225.688167] [] ? wake_up_state+0x20/0x20 [14225.695373] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [14225.704734] [] kthread+0xcf/0xe0 [14225.711172] [] ? kthread_create_on_node+0x140/0x140 [14225.719455] [] ret_from_fork+0x58/0x90 [14225.726467] [] ? kthread_create_on_node+0x140/0x140 [14225.734745] Code: 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 e8 77 5b f4 ff 42 8d 14 38 48 8b 45 88 48 8b 80 a8 03 00 00 48 3b 50 20 0f 83 57 01 00 00 <44> 0f b6 85 67 ff ff ff 48 8d 4d a8 48 8d 55 88 48 8b b5 38 ff [14253.339001] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [14253.348632] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [14253.481082] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [14253.494396] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [14253.510694] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [14253.520104] RIP: 0010:[] [] osd_inode_iteration+0x251/0xcc0 [osd_ldiskfs] [14253.532371] RSP: 0018:ffff880703603d18 EFLAGS: 00000286 [14253.539334] RAX: ffff8800b2a37468 RBX: 000000009f8f2001 RCX: 0000000000000000 [14253.548330] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [14253.557320] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [14253.566291] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000001 [14253.575249] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [14253.584201] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [14253.594233] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14253.601641] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [14253.610618] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14253.619585] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [14253.628537] Stack: [14253.631754] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [14253.641058] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [14253.650337] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [14253.659634] Call Trace: [14253.663351] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [14253.672521] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [14253.681398] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [14253.690078] [] ? wake_up_state+0x20/0x20 [14253.697296] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [14253.706657] [] kthread+0xcf/0xe0 [14253.713091] [] ? kthread_create_on_node+0x140/0x140 [14253.721392] [] ret_from_fork+0x58/0x90 [14253.728410] [] ? kthread_create_on_node+0x140/0x140 [14253.736684] Code: b0 00 00 00 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 00 8b 75 98 48 8b 7d 88 e8 dc 2d f5 ff 48 85 c0 48 89 45 90 0f 84 b3 08 00 00 <44> 8b 7d a0 48 8b 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 e8 77 [14281.339977] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [14281.349598] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [14281.481457] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [14281.494765] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [14281.511067] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [14281.520472] RIP: 0010:[] [] osd_inode_iteration+0x299/0xcc0 [osd_ldiskfs] [14281.532732] RSP: 0018:ffff880703603d18 EFLAGS: 00000287 [14281.539697] RAX: ffff88042bd78800 RBX: 000000009f8f2001 RCX: ffff880703603d98 [14281.548690] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff88042bd79800 [14281.557673] RBP: ffff880703603df0 R08: 0000000000000001 R09: 0000000000000004 [14281.566640] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [14281.575596] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [14281.584549] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [14281.594576] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14281.601994] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [14281.610967] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14281.619937] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [14281.628892] Stack: [14281.632112] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [14281.641409] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [14281.650698] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [14281.659996] Call Trace: [14281.663724] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [14281.672897] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [14281.681779] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [14281.690461] [] ? wake_up_state+0x20/0x20 [14281.697683] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [14281.707041] [] kthread+0xcf/0xe0 [14281.713471] [] ? kthread_create_on_node+0x140/0x140 [14281.721756] [] ret_from_fork+0x58/0x90 [14281.728783] [] ? kthread_create_on_node+0x140/0x140 [14281.737070] Code: 8b 45 88 48 8b 80 a8 03 00 00 48 3b 50 20 0f 83 57 01 00 00 44 0f b6 85 67 ff ff ff 48 8d 4d a8 48 8d 55 88 48 8b b5 38 ff ff ff <48> 8b bd 48 ff ff ff 48 8b 85 30 ff ff ff ff d0 83 f8 03 41 89 [14309.340950] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [14309.350596] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [14309.482796] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [14309.496111] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [14309.512426] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [14309.521836] RIP: 0010:[] [] osd_scrub_exec+0xf7/0x5b0 [osd_ldiskfs] [14309.533519] RSP: 0018:ffff880703603c78 EFLAGS: 00000246 [14309.540479] RAX: 0000000000280001 RBX: ffff880703603c37 RCX: 0000000000000000 [14309.549474] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [14309.558454] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [14309.567424] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [14309.576384] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [14309.585340] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [14309.595372] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14309.602779] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [14309.611749] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14309.620712] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [14309.629659] Stack: [14309.632878] ffff880703603d08 000000009f8f2001 ffffea00027e3c80 000000009f8f2001 [14309.642166] ffff880703603d57 0000000000000001 ffffffffa1217a20 ffff880703603d98 [14309.651456] ffff880703603d78 ffff8800b2a36000 ffff8803162f6000 ffffffffffffff10 [14309.660754] Call Trace: [14309.664485] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [14309.673361] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [14309.682531] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [14309.691696] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [14309.700562] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [14309.709234] [] ? wake_up_state+0x20/0x20 [14309.716441] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [14309.725800] [] kthread+0xcf/0xe0 [14309.732230] [] ? kthread_create_on_node+0x140/0x140 [14309.740511] [] ret_from_fork+0x58/0x90 [14309.747519] [] ? kthread_create_on_node+0x140/0x140 [14309.755788] Code: 8b 4d 88 48 85 db 74 23 f6 83 18 0a 00 00 08 74 1a 4d 85 e4 74 15 41 8b 87 6c 14 00 00 41 39 84 24 0c 0a 00 00 0f 82 d9 00 00 00 <31> c0 41 f6 87 74 14 00 00 04 0f 85 6e ff ff ff 41 83 f9 02 0f [14323.958461] INFO: rcu_sched self-detected stall on CPU { 24} (t=6900135 jiffies g=907112 c=907111 q=22047866) [14323.970884] Task dump for CPU 24: [14323.975601] OI_scrub R running task 0 5551 2 0x00000088 [14323.984513] ffff88083fde6dd0 00000000fb6db880 ffff88082da03db0 ffffffff810c47e8 [14323.993812] 0000000000000018 ffffffff81a1e7c0 ffff88082da03dc8 ffffffff810c8079 [14324.003082] 0000000000000009 ffff88082da03df8 ffffffff811378e0 ffff88082da101c0 [14324.012325] Call Trace: [14324.015997] [] sched_show_task+0xa8/0x110 [14324.023953] [] dump_cpu_task+0x39/0x70 [14324.030891] [] rcu_dump_cpu_stacks+0x90/0xd0 [14324.038389] [] rcu_check_callbacks+0x442/0x720 [14324.046059] [] ? tick_sched_handle.isra.13+0x60/0x60 [14324.054289] [] update_process_times+0x47/0x80 [14324.061810] [] tick_sched_handle.isra.13+0x25/0x60 [14324.069805] [] tick_sched_timer+0x41/0x70 [14324.076905] [] __hrtimer_run_queues+0xd2/0x260 [14324.084478] [] hrtimer_interrupt+0xb0/0x1e0 [14324.091747] [] ? call_softirq+0x1c/0x30 [14324.098609] [] local_apic_timer_interrupt+0x37/0x60 [14324.106614] [] smp_apic_timer_interrupt+0x3f/0x60 [14324.114437] [] apic_timer_interrupt+0x6d/0x80 [14324.121863] [] ? osd_scrub_exec+0x95/0x5b0 [osd_ldiskfs] [14324.131101] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [14324.140008] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [14324.148913] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [14324.157513] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [14324.165910] [] ? wake_up_state+0x20/0x20 [14324.172850] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [14324.181939] [] kthread+0xcf/0xe0 [14324.188111] [] ? kthread_create_on_node+0x140/0x140 [14324.196124] [] ret_from_fork+0x58/0x90 [14324.202867] [] ? kthread_create_on_node+0x140/0x140 [14349.342345] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [14349.351714] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [14349.481628] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [14349.494800] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [14349.510864] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [14349.520161] RIP: 0010:[] [] osd_scrub_next+0x350/0x3c0 [osd_ldiskfs] [14349.531818] RSP: 0018:ffff880703603c70 EFLAGS: 00000246 [14349.538676] RAX: 0000000000000000 RBX: ffff880703603c37 RCX: ffff880703603d98 [14349.547570] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [14349.556467] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [14349.565369] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [14349.574268] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [14349.583163] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [14349.593124] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14349.600453] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [14349.609361] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14349.618267] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [14349.627168] Stack: [14349.630338] 000000009f8f2001 ffff880703603df0 000000009f8f2001 ffff880703603df0 [14349.639572] 000000009f8f2001 ffff880703603d57 000000009f8f2001 0000000000000004 [14349.648808] ffff880703603d57 ffff8800b2a37468 0000000000000000 ffff880703603d78 [14349.658057] Call Trace: [14349.661738] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [14349.670864] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [14349.679991] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [14349.688819] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [14349.697452] [] ? wake_up_state+0x20/0x20 [14349.704624] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [14349.713934] [] kthread+0xcf/0xe0 [14349.720331] [] ? kthread_create_on_node+0x140/0x140 [14349.728566] [] ret_from_fork+0x58/0x90 [14349.735539] [] ? kthread_create_on_node+0x140/0x140 [14349.743780] Code: 97 ff 48 8b 55 90 48 8b 4d 88 44 8b 85 74 ff ff ff e9 1d ff ff ff 48 89 d7 44 89 45 90 e8 f9 67 47 e0 44 8b 45 90 0f 1f 44 00 00 <45> 84 c0 b8 04 00 00 00 0f 85 eb fd ff ff 49 8d b7 6c 14 00 00 [14377.343322] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [14377.352942] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [14377.484694] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [14377.497974] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [14377.514216] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [14377.523601] RIP: 0010:[] [] osd_scrub_next+0x12/0x3c0 [osd_ldiskfs] [14377.535246] RSP: 0018:ffff880703603cf0 EFLAGS: 00000287 [14377.542178] RAX: ffffffffa1217a20 RBX: 000000009f8f2001 RCX: ffff880703603d98 [14377.551148] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [14377.560105] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [14377.569055] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffff880703603c78 [14377.578006] R13: 0000000000000296 R14: 0000000000000010 R15: ffffffffa121b9ac [14377.586948] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [14377.596967] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14377.604364] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [14377.613317] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14377.622260] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [14377.631191] Stack: [14377.634389] 0000000000000000 00000000ffffffff 0000000000000001 ffff880703603df0 [14377.643674] ffffffffa121cff9 ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 [14377.652968] 00000000810d354f ffff8803162f6000 ffff8800b2a37468 0000000020000000 [14377.662266] Call Trace: [14377.665977] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [14377.675144] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [14377.684304] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [14377.693168] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [14377.701836] [] ? wake_up_state+0x20/0x20 [14377.709044] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [14377.718390] [] kthread+0xcf/0xe0 [14377.724826] [] ? kthread_create_on_node+0x140/0x140 [14377.733104] [] ret_from_fork+0x58/0x90 [14377.740114] [] ? kthread_create_on_node+0x140/0x140 [14377.748381] Code: 25 fc ff ff 48 83 c4 08 5b 41 5c 41 5d 41 5e 5d c3 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 89 e5 41 57 49 89 f7 41 56 41 55 <49> 89 fd 41 54 49 89 d4 53 48 89 cb 48 83 ec 70 65 48 8b 04 25 [14405.344290] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [14405.353949] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [14405.485908] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [14405.499194] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [14405.515458] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [14405.524852] RIP: 0010:[] [] osd_inode_iteration+0x264/0xcc0 [osd_ldiskfs] [14405.537094] RSP: 0018:ffff880703603d18 EFLAGS: 00000286 [14405.544050] RAX: ffff8800b2a37468 RBX: 000000009f8f2001 RCX: 0000000000000000 [14405.553032] RDX: ffff880703603d78 RSI: ffff880419dc3a00 RDI: ffff88042bd79800 [14405.561997] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [14405.570952] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000001 [14405.579903] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [14405.588859] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [14405.598893] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14405.606294] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [14405.615265] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14405.624221] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [14405.633165] Stack: [14405.636384] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [14405.645679] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [14405.654980] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [14405.664265] Call Trace: [14405.667991] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [14405.677162] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [14405.686039] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [14405.694710] [] ? wake_up_state+0x20/0x20 [14405.701913] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [14405.711262] [] kthread+0xcf/0xe0 [14405.717693] [] ? kthread_create_on_node+0x140/0x140 [14405.725977] [] ret_from_fork+0x58/0x90 [14405.732988] [] ? kthread_create_on_node+0x140/0x140 [14405.741265] Code: 75 98 48 8b 7d 88 e8 dc 2d f5 ff 48 85 c0 48 89 45 90 0f 84 b3 08 00 00 44 8b 7d a0 48 8b 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 77 5b f4 ff 42 8d 14 38 48 8b 45 88 48 8b 80 a8 03 00 00 48 [14433.345272] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [14433.354969] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [14433.486867] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [14433.500179] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [14433.516466] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [14433.525873] RIP: 0010:[] [] osd_inode_iteration+0x255/0xcc0 [osd_ldiskfs] [14433.538132] RSP: 0018:ffff880703603d18 EFLAGS: 00000286 [14433.545101] RAX: ffff8800b2a37468 RBX: 000000009f8f2001 RCX: 0000000000000000 [14433.554096] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [14433.563076] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [14433.572040] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [14433.580990] R13: ffff880703603d08 R14: 000000009f8f2001 R15: ffff880703603d08 [14433.589943] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [14433.599967] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14433.607373] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [14433.616346] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14433.625313] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [14433.634270] Stack: [14433.637501] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [14433.646783] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [14433.656078] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [14433.665376] Call Trace: [14433.669103] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [14433.678262] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [14433.687139] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [14433.695829] [] ? wake_up_state+0x20/0x20 [14433.703046] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [14433.712398] [] kthread+0xcf/0xe0 [14433.718835] [] ? kthread_create_on_node+0x140/0x140 [14433.727117] [] ret_from_fork+0x58/0x90 [14433.734132] [] ? kthread_create_on_node+0x140/0x140 [14433.742407] Code: 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 00 8b 75 98 48 8b 7d 88 e8 dc 2d f5 ff 48 85 c0 48 89 45 90 0f 84 b3 08 00 00 44 8b 7d a0 <48> 8b 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 e8 77 5b f4 ff 42 [14461.346255] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [14461.355930] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [14461.487955] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [14461.501282] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [14461.517585] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [14461.527000] RIP: 0010:[] [] osd_scrub_next+0x12/0x3c0 [osd_ldiskfs] [14461.538669] RSP: 0018:ffff880703603cf0 EFLAGS: 00000287 [14461.545641] RAX: ffffffffa1217a20 RBX: 000000009f8f2001 RCX: ffff880703603d98 [14461.554637] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [14461.563619] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [14461.572588] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffff880703603c78 [14461.581543] R13: ffff880703603d08 R14: 0000000000000010 R15: ffffffffa121ba87 [14461.590487] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [14461.600516] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14461.607936] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [14461.616915] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14461.625888] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [14461.634852] Stack: [14461.638067] 0000000000000000 00000000ffffffff 0000000000000001 ffff880703603df0 [14461.647341] ffffffffa121cff9 ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 [14461.656636] 00000000810d354f ffff8803162f6000 ffff8800b2a37468 0000000020000000 [14461.665943] Call Trace: [14461.669682] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [14461.678855] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [14461.688030] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [14461.696917] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [14461.705592] [] ? wake_up_state+0x20/0x20 [14461.712811] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [14461.722177] [] kthread+0xcf/0xe0 [14461.728620] [] ? kthread_create_on_node+0x140/0x140 [14461.736902] [] ret_from_fork+0x58/0x90 [14461.743919] [] ? kthread_create_on_node+0x140/0x140 [14461.752203] Code: 25 fc ff ff 48 83 c4 08 5b 41 5c 41 5d 41 5e 5d c3 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 89 e5 41 57 49 89 f7 41 56 41 55 <49> 89 fd 41 54 49 89 d4 53 48 89 cb 48 83 ec 70 65 48 8b 04 25 [14489.347216] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [14489.356851] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [14489.488705] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [14489.501998] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [14489.518268] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [14489.527665] RIP: 0010:[] [] osd_scrub_exec+0x1c/0x5b0 [osd_ldiskfs] [14489.539335] RSP: 0018:ffff880703603c78 EFLAGS: 00000296 [14489.546276] RAX: ffffffffa121b990 RBX: ffff880703603c37 RCX: 0000000000000000 [14489.555263] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [14489.564232] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [14489.573197] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [14489.582157] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [14489.591119] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [14489.601152] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14489.608570] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [14489.617539] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14489.626498] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [14489.635451] Stack: [14489.638666] ffffea00027e3c80 000000009f8f2001 0000000000000004 0000000000000001 [14489.647960] ffff880703603d57 ffff880703603d98 ffff880703603d78 ffff8800b2a36000 [14489.657279] ffff8803162f6000 ffffffffffffff10 ffffffffa1217a32 0000000000000010 [14489.666589] Call Trace: [14489.670324] [] ? osd_scrub_next+0x12/0x3c0 [osd_ldiskfs] [14489.679110] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [14489.688277] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [14489.697437] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [14489.706296] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [14489.714969] [] ? wake_up_state+0x20/0x20 [14489.722177] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [14489.731534] [] kthread+0xcf/0xe0 [14489.737971] [] ? kthread_create_on_node+0x140/0x140 [14489.746256] [] ret_from_fork+0x58/0x90 [14489.753263] [] ? kthread_create_on_node+0x140/0x140 [14489.761533] Code: 89 04 24 31 c0 e8 a5 03 98 ff e9 c7 fd ff ff 66 66 66 66 90 55 48 89 e5 41 57 49 89 f7 41 56 41 55 41 54 45 31 e4 53 48 83 ec 68 <48> 8b 9e 08 10 00 00 4c 89 45 90 65 48 8b 04 25 28 00 00 00 48 [14503.967727] INFO: rcu_sched self-detected stall on CPU { 24} (t=7080138 jiffies g=907112 c=907111 q=22723963) [14503.980179] Task dump for CPU 24: [14503.984897] OI_scrub R running task 0 5551 2 0x00000088 [14503.993812] ffff88083fde6dd0 00000000fb6db880 ffff88082da03db0 ffffffff810c47e8 [14504.003106] 0000000000000018 ffffffff81a1e7c0 ffff88082da03dc8 ffffffff810c8079 [14504.012381] 0000000000000009 ffff88082da03df8 ffffffff811378e0 ffff88082da101c0 [14504.021660] Call Trace: [14504.025340] [] sched_show_task+0xa8/0x110 [14504.033304] [] dump_cpu_task+0x39/0x70 [14504.040244] [] rcu_dump_cpu_stacks+0x90/0xd0 [14504.047742] [] rcu_check_callbacks+0x442/0x720 [14504.055413] [] ? tick_sched_handle.isra.13+0x60/0x60 [14504.063658] [] update_process_times+0x47/0x80 [14504.071188] [] tick_sched_handle.isra.13+0x25/0x60 [14504.079193] [] tick_sched_timer+0x41/0x70 [14504.086301] [] __hrtimer_run_queues+0xd2/0x260 [14504.093867] [] hrtimer_interrupt+0xb0/0x1e0 [14504.101134] [] ? call_softirq+0x1c/0x30 [14504.108012] [] local_apic_timer_interrupt+0x37/0x60 [14504.116028] [] smp_apic_timer_interrupt+0x3f/0x60 [14504.123860] [] apic_timer_interrupt+0x6d/0x80 [14504.131287] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [14504.140914] [] ? osd_scrub_exec+0x23/0x5b0 [osd_ldiskfs] [14504.149433] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [14504.158343] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [14504.167239] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [14504.175845] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [14504.184255] [] ? wake_up_state+0x20/0x20 [14504.191211] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [14504.200312] [] kthread+0xcf/0xe0 [14504.206491] [] ? kthread_create_on_node+0x140/0x140 [14504.214504] [] ret_from_fork+0x58/0x90 [14504.221257] [] ? kthread_create_on_node+0x140/0x140 [14529.348616] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [14529.358036] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [14529.488094] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [14529.501273] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [14529.517317] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [14529.526612] RIP: 0010:[] [] osd_scrub_next+0x149/0x3c0 [osd_ldiskfs] [14529.538276] RSP: 0018:ffff880703603c70 EFLAGS: 00000202 [14529.545146] RAX: 0000000000000004 RBX: ffff880703603c37 RCX: ffff880703603d98 [14529.554047] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [14529.562949] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [14529.571849] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [14529.580746] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [14529.589644] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [14529.599611] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14529.606960] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [14529.615874] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14529.624785] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [14529.633699] Stack: [14529.636879] 000000009f8f2001 ffff880703603d08 ffff880703603d57 ffff880703603df0 [14529.646129] 000000009f8f2001 ffff880703603d57 000000009f8f2001 0000000000000004 [14529.655382] ffff880703603d57 ffff8800b2a37468 0000000000000000 ffff880703603d78 [14529.664641] Call Trace: [14529.668345] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [14529.677485] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [14529.686621] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [14529.695468] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [14529.704112] [] ? wake_up_state+0x20/0x20 [14529.711291] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [14529.720611] [] kthread+0xcf/0xe0 [14529.727014] [] ? kthread_create_on_node+0x140/0x140 [14529.735277] [] ret_from_fork+0x58/0x90 [14529.742260] [] ? kthread_create_on_node+0x140/0x140 [14529.750507] Code: 45 90 74 49 49 8d 9f 10 11 00 00 48 89 df e8 df 6a 47 e0 41 c7 87 58 10 00 00 02 00 00 00 48 89 df e8 fc 69 47 e0 b8 05 00 00 00 <48> 8b 75 d0 65 48 33 34 25 28 00 00 00 0f 85 4c 02 00 00 48 83 [14557.349584] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [14557.359196] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [14557.491054] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [14557.504344] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [14557.520614] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [14557.530008] RIP: 0010:[] [] osd_scrub_exec+0x30/0x5b0 [osd_ldiskfs] [14557.541666] RSP: 0018:ffff880703603c78 EFLAGS: 00000296 [14557.548615] RAX: 00000000fb6db880 RBX: ffff880703603c37 RCX: 0000000000000000 [14557.557596] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [14557.566566] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [14557.575521] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [14557.584476] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [14557.593424] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [14557.603458] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14557.610868] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [14557.619839] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14557.628790] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [14557.637737] Stack: [14557.640940] ffff880703603d08 000000009f8f2001 ffffea00027e3c80 000000009f8f2001 [14557.650243] ffff880703603d57 ffff880703603d57 0000000000000000 0000000000000000 [14557.659546] ffff880703603d78 ffff8800b2a36000 ffff8803162f6000 ffffffffffffff10 [14557.668856] Call Trace: [14557.672572] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [14557.681743] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [14557.690907] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [14557.699773] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [14557.708434] [] ? wake_up_state+0x20/0x20 [14557.715645] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [14557.725004] [] kthread+0xcf/0xe0 [14557.731436] [] ? kthread_create_on_node+0x140/0x140 [14557.739729] [] ret_from_fork+0x58/0x90 [14557.746741] [] ? kthread_create_on_node+0x140/0x140 [14557.755012] Code: 55 48 89 e5 41 57 49 89 f7 41 56 41 55 41 54 45 31 e4 53 48 83 ec 68 48 8b 9e 08 10 00 00 4c 89 45 90 65 48 8b 04 25 28 00 00 00 <48> 89 45 d0 31 c0 48 8d 43 08 48 85 db 4c 0f 45 e0 41 83 f9 04 [14585.350568] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [14585.360283] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [14585.492107] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [14585.505395] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [14585.521652] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [14585.531039] RIP: 0010:[] [] osd_scrub_exec+0xf7/0x5b0 [osd_ldiskfs] [14585.542694] RSP: 0018:ffff880703603c78 EFLAGS: 00000246 [14585.549635] RAX: 0000000000280001 RBX: ffff880703603c37 RCX: 0000000000000000 [14585.558612] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [14585.567581] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [14585.576534] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [14585.585487] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [14585.594438] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [14585.604462] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14585.611864] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [14585.620832] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14585.629788] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [14585.638725] Stack: [14585.641937] 0000000000000004 0000000000000001 ffff880703603df0 ffff880703603d98 [14585.651241] ffff880703603d57 000000009f8f2001 0000000000000004 0000000000000001 [14585.660525] 0000000000000004 ffff880703603d98 ffff880703603d78 0000000000000000 [14585.669828] Call Trace: [14585.673543] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [14585.682713] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [14585.691881] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [14585.700744] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [14585.709415] [] ? wake_up_state+0x20/0x20 [14585.716624] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [14585.725996] [] kthread+0xcf/0xe0 [14585.732434] [] ? kthread_create_on_node+0x140/0x140 [14585.740725] [] ret_from_fork+0x58/0x90 [14585.747732] [] ? kthread_create_on_node+0x140/0x140 [14585.755999] Code: 8b 4d 88 48 85 db 74 23 f6 83 18 0a 00 00 08 74 1a 4d 85 e4 74 15 41 8b 87 6c 14 00 00 41 39 84 24 0c 0a 00 00 0f 82 d9 00 00 00 <31> c0 41 f6 87 74 14 00 00 04 0f 85 6e ff ff ff 41 83 f9 02 0f [14613.351543] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [14613.361201] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [14613.493418] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [14613.506717] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [14613.522993] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [14613.532384] RIP: 0010:[] [] osd_inode_iteration+0x4a6/0xcc0 [osd_ldiskfs] [14613.544634] RSP: 0018:ffff880703603d18 EFLAGS: 00000246 [14613.551590] RAX: 0000000020000000 RBX: ffff880703603d98 RCX: 0000000000000000 [14613.560571] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [14613.569537] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [14613.578501] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000001 [14613.587449] R13: 0000000000000004 R14: 000000009f8f2001 R15: ffffea00027e3c80 [14613.596405] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [14613.606437] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14613.613845] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [14613.622813] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14613.631770] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [14613.640716] Stack: [14613.643928] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [14613.653213] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [14613.662488] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [14613.671801] Call Trace: [14613.675532] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [14613.684689] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [14613.693557] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [14613.702232] [] ? wake_up_state+0x20/0x20 [14613.709442] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [14613.718779] [] kthread+0xcf/0xe0 [14613.725213] [] ? kthread_create_on_node+0x140/0x140 [14613.733506] [] ret_from_fork+0x58/0x90 [14613.740520] [] ? kthread_create_on_node+0x140/0x140 [14613.748802] Code: 8b 4d a8 48 8d 55 88 48 8b b5 38 ff ff ff 48 8b bd 48 ff ff ff 48 8b 85 28 ff ff ff ff d0 85 c0 41 89 c5 75 1c 8b 85 58 ff ff ff <41> 3b 04 24 72 10 48 8b 85 50 ff ff ff 44 3b 30 0f 87 95 fd ff [14641.352518] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [14641.362160] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [14641.494132] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [14641.507438] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [14641.523721] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [14641.533125] RIP: 0010:[] [] osd_scrub_next+0x14d/0x3c0 [osd_ldiskfs] [14641.544883] RSP: 0018:ffff880703603c70 EFLAGS: 00000202 [14641.551843] RAX: 0000000000000004 RBX: ffff880703603c37 RCX: ffff880703603d98 [14641.560821] RDX: ffff880703603d78 RSI: 00000000fb6db880 RDI: ffff8803162f6000 [14641.569800] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [14641.578768] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [14641.587722] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [14641.596664] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [14641.606691] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14641.614100] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [14641.623070] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14641.632036] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [14641.640990] Stack: [14641.644208] 000000009f8f2001 ffff880703603d08 000000009f8f2001 ffff880703603df0 [14641.653493] 000000009f8f2001 ffff880703603d57 000000009f8f2001 0000000000000004 [14641.662783] 0000000000000001 0000000000000004 ffff880703603d98 ffff880703603d78 [14641.672077] Call Trace: [14641.675801] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [14641.684968] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [14641.694120] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [14641.702989] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [14641.711665] [] ? wake_up_state+0x20/0x20 [14641.718870] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [14641.728213] [] kthread+0xcf/0xe0 [14641.734633] [] ? kthread_create_on_node+0x140/0x140 [14641.742902] [] ret_from_fork+0x58/0x90 [14641.749917] [] ? kthread_create_on_node+0x140/0x140 [14641.758183] Code: 49 8d 9f 10 11 00 00 48 89 df e8 df 6a 47 e0 41 c7 87 58 10 00 00 02 00 00 00 48 89 df e8 fc 69 47 e0 b8 05 00 00 00 48 8b 75 d0 <65> 48 33 34 25 28 00 00 00 0f 85 4c 02 00 00 48 83 c4 70 5b 41 [14669.353493] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [14669.363149] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [14669.495549] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [14669.508845] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [14669.525106] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [14669.534487] RIP: 0010:[] [] osd_scrub_next+0x12/0x3c0 [osd_ldiskfs] [14669.546151] RSP: 0018:ffff880703603cf0 EFLAGS: 00000287 [14669.553085] RAX: ffffffffa1217a20 RBX: 000000009f8f2001 RCX: ffff880703603d98 [14669.562063] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [14669.571028] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [14669.579981] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffff880703603c78 [14669.588940] R13: 0000000000000246 R14: 0000000000000010 R15: ffffffffa121ba87 [14669.597896] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [14669.607913] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14669.615319] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [14669.624284] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14669.633241] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [14669.642180] Stack: [14669.645392] 0000000000000000 00000000ffffffff 0000000000000001 ffff880703603df0 [14669.654672] ffffffffa121cff9 ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 [14669.663962] 00000000810d354f ffff8803162f6000 ffff8800b2a37468 0000000020000000 [14669.673254] Call Trace: [14669.676980] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [14669.686154] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [14669.695320] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [14669.704181] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [14669.712855] [] ? wake_up_state+0x20/0x20 [14669.720071] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [14669.729433] [] kthread+0xcf/0xe0 [14669.735870] [] ? kthread_create_on_node+0x140/0x140 [14669.744163] [] ret_from_fork+0x58/0x90 [14669.751180] [] ? kthread_create_on_node+0x140/0x140 [14669.759460] Code: 25 fc ff ff 48 83 c4 08 5b 41 5c 41 5d 41 5e 5d c3 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 89 e5 41 57 49 89 f7 41 56 41 55 <49> 89 fd 41 54 49 89 d4 53 48 89 cb 48 83 ec 70 65 48 8b 04 25 [14672.578441] Lustre: soaked-MDT0000: Client e211cd09-6b0c-e43b-059c-f657101042a0 (at 192.168.1.117@o2ib100) reconnecting [14672.593356] Lustre: Skipped 1 previous similar message [14672.601615] Lustre: soaked-MDT0000: Connection restored to e211cd09-6b0c-e43b-059c-f657101042a0 (at 192.168.1.117@o2ib100) [14672.616475] Lustre: Skipped 1 previous similar message [14683.976997] INFO: rcu_sched self-detected stall on CPU { 24} (t=7260141 jiffies g=907112 c=907111 q=22990120) [14683.989488] Task dump for CPU 24: [14683.994222] OI_scrub R running task 0 5551 2 0x00000088 [14684.003151] ffff88083fde6dd0 00000000fb6db880 ffff88082da03db0 ffffffff810c47e8 [14684.012464] 0000000000000018 ffffffff81a1e7c0 ffff88082da03dc8 ffffffff810c8079 [14684.021745] 0000000000000009 ffff88082da03df8 ffffffff811378e0 ffff88082da101c0 [14684.031025] Call Trace: [14684.034697] [] sched_show_task+0xa8/0x110 [14684.042657] [] dump_cpu_task+0x39/0x70 [14684.049594] [] rcu_dump_cpu_stacks+0x90/0xd0 [14684.057092] [] rcu_check_callbacks+0x442/0x720 [14684.064770] [] ? tick_sched_handle.isra.13+0x60/0x60 [14684.073018] [] update_process_times+0x47/0x80 [14684.080556] [] tick_sched_handle.isra.13+0x25/0x60 [14684.088565] [] tick_sched_timer+0x41/0x70 [14684.095693] [] __hrtimer_run_queues+0xd2/0x260 [14684.103284] [] hrtimer_interrupt+0xb0/0x1e0 [14684.110587] [] ? call_softirq+0x1c/0x30 [14684.117501] [] local_apic_timer_interrupt+0x37/0x60 [14684.125572] [] smp_apic_timer_interrupt+0x3f/0x60 [14684.133446] [] apic_timer_interrupt+0x6d/0x80 [14684.140937] [] ? osd_scrub_next+0x45/0x3c0 [osd_ldiskfs] [14684.150233] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [14684.158890] [] ? osd_scrub_next+0x1b/0x3c0 [osd_ldiskfs] [14684.167448] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [14684.176398] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [14684.185345] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [14684.194000] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [14684.202463] [] ? wake_up_state+0x20/0x20 [14684.209466] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [14684.218597] [] kthread+0xcf/0xe0 [14684.224813] [] ? kthread_create_on_node+0x140/0x140 [14684.232861] [] ret_from_fork+0x58/0x90 [14684.239652] [] ? kthread_create_on_node+0x140/0x140 [14709.354887] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [14709.364386] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [14709.494532] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [14709.507688] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [14709.523685] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [14709.532949] RIP: 0010:[] [] osd_scrub_next+0x14d/0x3c0 [osd_ldiskfs] [14709.544594] RSP: 0018:ffff880703603c70 EFLAGS: 00000202 [14709.551443] RAX: 0000000000000004 RBX: ffff880703603c37 RCX: ffff880703603d98 [14709.560337] RDX: ffff880703603d78 RSI: 00000000fb6db880 RDI: ffff8803162f6000 [14709.569233] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [14709.578129] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [14709.587026] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [14709.595912] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [14709.605875] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14709.613223] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [14709.622133] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14709.631038] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [14709.639935] Stack: [14709.643122] 000000009f8f2001 ffffea00027e3c80 000000009f8f2001 ffff880703603df0 [14709.652365] 0000000000000001 ffff880703603d57 000000009f8f2001 0000000000000004 [14709.661598] ffff880703603d57 ffff88042bd78800 ffff88042bd78800 00000000000072cf [14709.670842] Call Trace: [14709.674525] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [14709.683666] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [14709.692790] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [14709.701627] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [14709.710254] [] ? wake_up_state+0x20/0x20 [14709.717425] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [14709.726734] [] kthread+0xcf/0xe0 [14709.733129] [] ? kthread_create_on_node+0x140/0x140 [14709.741382] [] ret_from_fork+0x58/0x90 [14709.748361] [] ? kthread_create_on_node+0x140/0x140 [14709.756600] Code: 49 8d 9f 10 11 00 00 48 89 df e8 df 6a 47 e0 41 c7 87 58 10 00 00 02 00 00 00 48 89 df e8 fc 69 47 e0 b8 05 00 00 00 48 8b 75 d0 <65> 48 33 34 25 28 00 00 00 0f 85 4c 02 00 00 48 83 c4 70 5b 41 [14737.355856] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [14737.365496] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [14737.497481] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [14737.510766] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [14737.527016] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [14737.536407] RIP: 0010:[] [] osd_scrub_exec+0xf9/0x5b0 [osd_ldiskfs] [14737.548063] RSP: 0018:ffff880703603c78 EFLAGS: 00000246 [14737.555000] RAX: 0000000000000000 RBX: ffff880703603c37 RCX: 0000000000000000 [14737.563975] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [14737.572949] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [14737.581913] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [14737.590867] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [14737.599823] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [14737.609856] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14737.617259] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [14737.626221] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14737.635172] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [14737.644111] Stack: [14737.647322] ffff880703603d08 0000000000000001 ffff880703603df0 000000009f8f2001 [14737.656626] ffff880703603d57 000000009f8f2001 0000000000000004 ffff880703603d57 [14737.665924] 0000000000000000 0000000000000000 ffff880703603d78 ffff8800b2a36000 [14737.675211] Call Trace: [14737.678939] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [14737.688103] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [14737.697263] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [14737.706129] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [14737.714789] [] ? wake_up_state+0x20/0x20 [14737.721995] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [14737.731347] [] kthread+0xcf/0xe0 [14737.737787] [] ? kthread_create_on_node+0x140/0x140 [14737.746063] [] ret_from_fork+0x58/0x90 [14737.753070] [] ? kthread_create_on_node+0x140/0x140 [14737.761335] Code: 88 48 85 db 74 23 f6 83 18 0a 00 00 08 74 1a 4d 85 e4 74 15 41 8b 87 6c 14 00 00 41 39 84 24 0c 0a 00 00 0f 82 d9 00 00 00 31 c0 <41> f6 87 74 14 00 00 04 0f 85 6e ff ff ff 41 83 f9 02 0f 84 64 [14765.356831] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [14765.366532] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [14765.498444] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [14765.511729] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [14765.527975] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [14765.537363] RIP: 0010:[] [] osd_inode_iteration+0x251/0xcc0 [osd_ldiskfs] [14765.549598] RSP: 0018:ffff880703603d18 EFLAGS: 00000286 [14765.556544] RAX: ffff8800b2a37468 RBX: 000000009f8f2001 RCX: 0000000000000000 [14765.565528] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [14765.574498] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [14765.583451] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000001 [14765.592405] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [14765.601351] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [14765.611373] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14765.618764] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [14765.627730] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14765.636694] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [14765.645630] Stack: [14765.648845] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [14765.658138] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [14765.667434] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [14765.676725] Call Trace: [14765.680436] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [14765.689593] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [14765.698459] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [14765.707124] [] ? wake_up_state+0x20/0x20 [14765.714327] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [14765.723672] [] kthread+0xcf/0xe0 [14765.730114] [] ? kthread_create_on_node+0x140/0x140 [14765.738411] [] ret_from_fork+0x58/0x90 [14765.745420] [] ? kthread_create_on_node+0x140/0x140 [14765.753696] Code: b0 00 00 00 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 00 8b 75 98 48 8b 7d 88 e8 dc 2d f5 ff 48 85 c0 48 89 45 90 0f 84 b3 08 00 00 <44> 8b 7d a0 48 8b 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 e8 77 [14793.357808] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [14793.367476] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [14793.499468] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [14793.512773] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [14793.529071] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [14793.538490] RIP: 0010:[] [] osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [14793.550551] RSP: 0018:ffff880703603d10 EFLAGS: 00000293 [14793.557515] RAX: ffffffffa121b990 RBX: 000000009f8f2001 RCX: 0000000000000000 [14793.566499] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [14793.575487] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [14793.584450] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffff880703603d57 [14793.593399] R13: ffff880703603d08 R14: 000000009f8f2001 R15: ffffea00027e3c80 [14793.602348] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [14793.612382] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14793.619796] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [14793.628772] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14793.637739] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [14793.646706] Stack: [14793.649928] ffffffffa121d1e9 ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 [14793.659222] 00000000810d354f ffff8803162f6000 ffff8800b2a37468 0000000020000000 [14793.668518] 010000000000000c 0000000000000000 0000000000000000 ffff8800b2a36000 [14793.677820] Call Trace: [14793.681558] [] ? osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [14793.690926] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [14793.700101] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [14793.708990] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [14793.717668] [] ? wake_up_state+0x20/0x20 [14793.724886] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [14793.734243] [] kthread+0xcf/0xe0 [14793.740684] [] ? kthread_create_on_node+0x140/0x140 [14793.748971] [] ret_from_fork+0x58/0x90 [14793.755990] [] ? kthread_create_on_node+0x140/0x140 [14793.764266] Code: b5 07 00 00 c7 05 85 a5 02 00 00 00 02 00 89 54 24 08 48 8b 40 40 ba 07 00 00 00 48 89 04 24 31 c0 e8 a5 03 98 ff e9 c7 fd ff ff <66> 66 66 66 90 55 48 89 e5 41 57 49 89 f7 41 56 41 55 41 54 45 [14821.358790] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [14821.368434] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [14821.500464] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [14821.513763] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [14821.530029] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [14821.539429] RIP: 0010:[] [] osd_inode_iteration+0x264/0xcc0 [osd_ldiskfs] [14821.551690] RSP: 0018:ffff880703603d18 EFLAGS: 00000286 [14821.558648] RAX: ffff8800b2a37468 RBX: 000000009f8f2001 RCX: 0000000000000000 [14821.567643] RDX: ffff880703603d78 RSI: ffff880419dc3a00 RDI: ffff88042bd79800 [14821.576624] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [14821.585580] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [14821.594544] R13: ffff880703603df0 R14: 000000009f8f2001 R15: 0000000000000018 [14821.603494] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [14821.613529] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14821.620947] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [14821.629928] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14821.638896] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [14821.647846] Stack: [14821.651061] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [14821.660367] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [14821.669665] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [14821.678965] Call Trace: [14821.682702] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [14821.691877] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [14821.700749] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [14821.709441] [] ? wake_up_state+0x20/0x20 [14821.716655] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [14821.726005] [] kthread+0xcf/0xe0 [14821.732443] [] ? kthread_create_on_node+0x140/0x140 [14821.740729] [] ret_from_fork+0x58/0x90 [14821.747747] [] ? kthread_create_on_node+0x140/0x140 [14821.756027] Code: 75 98 48 8b 7d 88 e8 dc 2d f5 ff 48 85 c0 48 89 45 90 0f 84 b3 08 00 00 44 8b 7d a0 48 8b 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 77 5b f4 ff 42 8d 14 38 48 8b 45 88 48 8b 80 a8 03 00 00 48 [14849.359756] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [14849.369381] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [14849.501612] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [14849.514918] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [14849.531219] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [14849.540628] RIP: 0010:[] [] osd_scrub_exec+0x1c/0x5b0 [osd_ldiskfs] [14849.552304] RSP: 0018:ffff880703603c78 EFLAGS: 00000296 [14849.559277] RAX: ffffffffa121b990 RBX: ffff880703603c37 RCX: 0000000000000000 [14849.568269] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [14849.577251] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [14849.586218] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [14849.595175] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [14849.604129] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [14849.614162] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14849.621574] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [14849.630544] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14849.639512] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [14849.648471] Stack: [14849.651689] ffff880703603d08 0000000000000001 ffff880703603df0 000000009f8f2001 [14849.660985] ffff880703603d57 000000009f8f2001 0000000000000004 ffff880703603d57 [14849.670284] 0000000000000004 ffff880703603d98 ffff880703603d78 0000000000000000 [14849.679594] Call Trace: [14849.683338] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [14849.692518] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [14849.701679] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [14849.710556] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [14849.719243] [] ? wake_up_state+0x20/0x20 [14849.726457] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [14849.735823] [] kthread+0xcf/0xe0 [14849.742250] [] ? kthread_create_on_node+0x140/0x140 [14849.750545] [] ret_from_fork+0x58/0x90 [14849.757559] [] ? kthread_create_on_node+0x140/0x140 [14849.765838] Code: 89 04 24 31 c0 e8 a5 03 98 ff e9 c7 fd ff ff 66 66 66 66 90 55 48 89 e5 41 57 49 89 f7 41 56 41 55 41 54 45 31 e4 53 48 83 ec 68 <48> 8b 9e 08 10 00 00 4c 89 45 90 65 48 8b 04 25 28 00 00 00 48 [14863.986281] INFO: rcu_sched self-detected stall on CPU { 24} (t=7440144 jiffies g=907112 c=907111 q=23642993) [14863.998731] Task dump for CPU 24: [14864.003463] OI_scrub R running task 0 5551 2 0x00000088 [14864.012388] ffff88083fde6dd0 00000000fb6db880 ffff88082da03db0 ffffffff810c47e8 [14864.021719] 0000000000000018 ffffffff81a1e7c0 ffff88082da03dc8 ffffffff810c8079 [14864.031020] 0000000000000009 ffff88082da03df8 ffffffff811378e0 ffff88082da101c0 [14864.040330] Call Trace: [14864.044012] [] sched_show_task+0xa8/0x110 [14864.052002] [] dump_cpu_task+0x39/0x70 [14864.058957] [] rcu_dump_cpu_stacks+0x90/0xd0 [14864.066479] [] rcu_check_callbacks+0x442/0x720 [14864.074168] [] ? tick_sched_handle.isra.13+0x60/0x60 [14864.082427] [] update_process_times+0x47/0x80 [14864.089976] [] tick_sched_handle.isra.13+0x25/0x60 [14864.097995] [] tick_sched_timer+0x41/0x70 [14864.105117] [] __hrtimer_run_queues+0xd2/0x260 [14864.112702] [] hrtimer_interrupt+0xb0/0x1e0 [14864.119990] [] ? call_softirq+0x1c/0x30 [14864.126872] [] local_apic_timer_interrupt+0x37/0x60 [14864.134912] [] smp_apic_timer_interrupt+0x3f/0x60 [14864.142732] [] apic_timer_interrupt+0x6d/0x80 [14864.150163] [] ? osd_scrub_exec+0x95/0x5b0 [osd_ldiskfs] [14864.159391] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [14864.168296] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [14864.177206] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [14864.185820] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [14864.194210] [] ? wake_up_state+0x20/0x20 [14864.201152] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [14864.210234] [] kthread+0xcf/0xe0 [14864.216406] [] ? kthread_create_on_node+0x140/0x140 [14864.224439] [] ret_from_fork+0x58/0x90 [14864.231192] [] ? kthread_create_on_node+0x140/0x140 [14889.361151] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [14889.370601] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [14889.500301] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [14889.513469] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [14889.529526] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [14889.538814] RIP: 0010:[] [] osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [14889.550761] RSP: 0018:ffff880703603d10 EFLAGS: 00000293 [14889.557622] RAX: ffffffffa121b990 RBX: 000000009f8f2001 RCX: 0000000000000000 [14889.566521] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [14889.575414] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [14889.584308] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000001 [14889.593205] R13: ffff880703603df0 R14: 000000009f8f2001 R15: 0000000000000018 [14889.602102] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [14889.612065] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14889.619399] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [14889.628301] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14889.637203] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [14889.646092] Stack: [14889.649259] ffffffffa121d1e9 ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 [14889.658497] 00000000810d354f ffff8803162f6000 ffff8800b2a37468 0000000020000000 [14889.667741] 010000000000000c 0000000000000000 0000000000000000 ffff8800b2a36000 [14889.676993] Call Trace: [14889.680679] [] ? osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [14889.690009] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [14889.699137] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [14889.707986] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [14889.716637] [] ? wake_up_state+0x20/0x20 [14889.723825] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [14889.733144] [] kthread+0xcf/0xe0 [14889.739552] [] ? kthread_create_on_node+0x140/0x140 [14889.747792] [] ret_from_fork+0x58/0x90 [14889.754781] [] ? kthread_create_on_node+0x140/0x140 [14889.763024] Code: b5 07 00 00 c7 05 85 a5 02 00 00 00 02 00 89 54 24 08 48 8b 40 40 ba 07 00 00 00 48 89 04 24 31 c0 e8 a5 03 98 ff e9 c7 fd ff ff <66> 66 66 66 90 55 48 89 e5 41 57 49 89 f7 41 56 41 55 41 54 45 [14917.362133] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [14917.371761] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [14917.503902] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [14917.517193] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [14917.533471] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [14917.542865] RIP: 0010:[] [] osd_scrub_next+0x3d/0x3c0 [osd_ldiskfs] [14917.554533] RSP: 0018:ffff880703603c70 EFLAGS: 00000246 [14917.561476] RAX: 0000000000000000 RBX: ffff880703603c37 RCX: ffff880703603d98 [14917.570462] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [14917.579429] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [14917.588391] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [14917.597344] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [14917.606316] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [14917.616345] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14917.623761] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [14917.632740] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14917.641710] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [14917.650673] Stack: [14917.653895] 000000009f8f2001 ffffea00027e3c80 000000009f8f2001 ffff880703603df0 [14917.663212] ffff880703603d57 ffff880703603d57 000000009f8f2001 0000000000000004 [14917.672523] ffff880703603d57 ffff88042bd78800 ffff88042bd78800 00000000000072cf [14917.681819] Call Trace: [14917.685554] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [14917.694718] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [14917.703886] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [14917.712757] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [14917.721424] [] ? wake_up_state+0x20/0x20 [14917.728627] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [14917.737998] [] kthread+0xcf/0xe0 [14917.744424] [] ? kthread_create_on_node+0x140/0x140 [14917.752724] [] ret_from_fork+0x58/0x90 [14917.759741] [] ? kthread_create_on_node+0x140/0x140 [14917.768013] Code: 49 89 fd 41 54 49 89 d4 53 48 89 cb 48 83 ec 70 65 48 8b 04 25 28 00 00 00 48 89 45 d0 31 c0 48 8b 05 f8 59 9c ff 48 85 c0 75 7b <41> f6 87 58 10 00 00 08 0f 84 62 03 00 00 4d 8d b7 e0 10 00 00 [14945.363101] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [14945.372754] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [14945.505026] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [14945.518323] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [14945.534588] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [14945.543968] RIP: 0010:[] [] osd_scrub_exec+0xf9/0x5b0 [osd_ldiskfs] [14945.555625] RSP: 0018:ffff880703603c78 EFLAGS: 00000246 [14945.562559] RAX: 0000000000000000 RBX: ffff880703603c37 RCX: 0000000000000000 [14945.571532] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [14945.580494] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [14945.589442] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [14945.598395] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [14945.607356] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [14945.617388] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14945.624794] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [14945.633754] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14945.642717] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [14945.651669] Stack: [14945.654887] ffff880703603d08 000000009f8f2001 ffff880703603df0 000000009f8f2001 [14945.664172] ffff880703603d57 000000009f8f2001 0000000000000004 ffff880703603d57 [14945.673471] 0000000000000004 0000000000000000 ffff880703603d78 ffff8800b2a36000 [14945.682759] Call Trace: [14945.686483] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [14945.695640] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [14945.704805] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [14945.713664] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [14945.722331] [] ? wake_up_state+0x20/0x20 [14945.729534] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [14945.738886] [] kthread+0xcf/0xe0 [14945.745312] [] ? kthread_create_on_node+0x140/0x140 [14945.753584] [] ret_from_fork+0x58/0x90 [14945.760594] [] ? kthread_create_on_node+0x140/0x140 [14945.768866] Code: 88 48 85 db 74 23 f6 83 18 0a 00 00 08 74 1a 4d 85 e4 74 15 41 8b 87 6c 14 00 00 41 39 84 24 0c 0a 00 00 0f 82 d9 00 00 00 31 c0 <41> f6 87 74 14 00 00 04 0f 85 6e ff ff ff 41 83 f9 02 0f 84 64 [14973.364076] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [14973.373687] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [14973.505555] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [14973.518837] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [14973.535069] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [14973.544450] RIP: 0010:[] [] osd_scrub_exec+0x14/0x5b0 [osd_ldiskfs] [14973.556110] RSP: 0018:ffff880703603ce8 EFLAGS: 00000293 [14973.563053] RAX: ffffffffa121b990 RBX: ffff880703603c78 RCX: 0000000000000000 [14973.572032] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [14973.580999] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [14973.589954] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000010 [14973.598887] R13: ffffffffa121b9ac R14: ffffffffffffff10 R15: ffff8803162f6000 [14973.607836] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [14973.617865] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14973.625271] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [14973.634236] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14973.643189] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [14973.652131] Stack: [14973.655338] ffff8800b2a3746c 0000000000000004 00000000ffffffff 0000000000000001 [14973.664615] ffff880703603df0 ffffffffa121d1e9 ffffffffa121b990 ffffffffa1217a20 [14973.673922] ffff8800b2a36000 00000000810d354f ffff8803162f6000 ffff8800b2a37468 [14973.683215] Call Trace: [14973.686941] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [14973.696111] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [14973.705269] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [14973.714140] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [14973.722805] [] ? wake_up_state+0x20/0x20 [14973.730019] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [14973.739386] [] kthread+0xcf/0xe0 [14973.745821] [] ? kthread_create_on_node+0x140/0x140 [14973.754109] [] ret_from_fork+0x58/0x90 [14973.761130] [] ? kthread_create_on_node+0x140/0x140 [14973.769405] Code: 40 40 ba 07 00 00 00 48 89 04 24 31 c0 e8 a5 03 98 ff e9 c7 fd ff ff 66 66 66 66 90 55 48 89 e5 41 57 49 89 f7 41 56 41 55 41 54 <45> 31 e4 53 48 83 ec 68 48 8b 9e 08 10 00 00 4c 89 45 90 65 48 [15001.365056] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [15001.374698] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [15001.506578] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [15001.519876] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [15001.536144] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [15001.545527] RIP: 0010:[] [] osd_inode_iteration+0x4ac/0xcc0 [osd_ldiskfs] [15001.557770] RSP: 0018:ffff880703603d18 EFLAGS: 00000216 [15001.564719] RAX: 0000000020000000 RBX: 0000000000000000 RCX: 0000000000000000 [15001.573703] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [15001.582677] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [15001.591639] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffff880703603d57 [15001.600597] R13: 0000000000000004 R14: 000000009f8f2001 R15: ffffea00027e3c80 [15001.609555] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [15001.619594] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15001.627016] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [15001.635989] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15001.644957] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [15001.653908] Stack: [15001.657124] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [15001.666433] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [15001.675721] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [15001.685015] Call Trace: [15001.688746] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [15001.697917] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [15001.706792] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [15001.715460] [] ? wake_up_state+0x20/0x20 [15001.722671] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [15001.732033] [] kthread+0xcf/0xe0 [15001.738471] [] ? kthread_create_on_node+0x140/0x140 [15001.746755] [] ret_from_fork+0x58/0x90 [15001.753773] [] ? kthread_create_on_node+0x140/0x140 [15001.762051] Code: 88 48 8b b5 38 ff ff ff 48 8b bd 48 ff ff ff 48 8b 85 28 ff ff ff ff d0 85 c0 41 89 c5 75 1c 8b 85 58 ff ff ff 41 3b 04 24 72 10 <48> 8b 85 50 ff ff ff 44 3b 30 0f 87 95 fd ff ff 45 31 ff e9 a2 [15029.366035] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [15029.375672] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [15029.508014] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [15029.521315] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [15029.537628] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [15029.547026] RIP: 0010:[] [] osd_scrub_exec+0xdb/0x5b0 [osd_ldiskfs] [15029.558713] RSP: 0018:ffff880703603c78 EFLAGS: 00000202 [15029.565672] RAX: ffff8803162f1008 RBX: ffff880703603c37 RCX: 0000000000000000 [15029.574661] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [15029.583641] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [15029.592602] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [15029.601555] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [15029.610498] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [15029.620526] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15029.627927] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [15029.636900] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15029.645862] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [15029.654813] Stack: [15029.658028] ffff880703603d08 0000000000000001 ffffea00027e3c80 000000009f8f2001 [15029.667319] ffff880703603d57 ffff880703603d57 0000000000000000 ffff88042bd78800 [15029.676625] 00000000000072ce ffff880419dc3a00 ffff88042bd79800 ffffffffffffff10 [15029.685913] Call Trace: [15029.689642] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [15029.698820] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [15029.707982] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [15029.716857] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [15029.725531] [] ? wake_up_state+0x20/0x20 [15029.732734] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [15029.742098] [] kthread+0xcf/0xe0 [15029.748536] [] ? kthread_create_on_node+0x140/0x140 [15029.756838] [] ret_from_fork+0x58/0x90 [15029.763849] [] ? kthread_create_on_node+0x140/0x140 [15029.772127] Code: fb 46 e0 41 83 87 68 14 00 00 01 49 83 87 88 11 00 00 01 4c 89 ef e8 f5 9f e9 df 44 8b 4d 88 48 85 db 74 23 f6 83 18 0a 00 00 08 <74> 1a 4d 85 e4 74 15 41 8b 87 6c 14 00 00 41 39 84 24 0c 0a 00 [15043.995544] INFO: rcu_sched self-detected stall on CPU { 24} (t=7620147 jiffies g=907112 c=907111 q=23979408) [15044.008013] Task dump for CPU 24: [15044.012739] OI_scrub R running task 0 5551 2 0x00000088 [15044.021675] ffff88083fde6dd0 00000000fb6db880 ffff88082da03db0 ffffffff810c47e8 [15044.031012] 0000000000000018 ffffffff81a1e7c0 ffff88082da03dc8 ffffffff810c8079 [15044.040305] 0000000000000009 ffff88082da03df8 ffffffff811378e0 ffff88082da101c0 [15044.049583] Call Trace: [15044.053263] [] sched_show_task+0xa8/0x110 [15044.061240] [] dump_cpu_task+0x39/0x70 [15044.068208] [] rcu_dump_cpu_stacks+0x90/0xd0 [15044.075729] [] rcu_check_callbacks+0x442/0x720 [15044.083422] [] ? tick_sched_handle.isra.13+0x60/0x60 [15044.091686] [] update_process_times+0x47/0x80 [15044.099242] [] tick_sched_handle.isra.13+0x25/0x60 [15044.107254] [] tick_sched_timer+0x41/0x70 [15044.114375] [] __hrtimer_run_queues+0xd2/0x260 [15044.121960] [] hrtimer_interrupt+0xb0/0x1e0 [15044.129254] [] ? call_softirq+0x1c/0x30 [15044.136141] [] local_apic_timer_interrupt+0x37/0x60 [15044.144178] [] smp_apic_timer_interrupt+0x3f/0x60 [15044.152021] [] apic_timer_interrupt+0x6d/0x80 [15044.159456] [] ? osd_inode_iteration+0x299/0xcc0 [osd_ldiskfs] [15044.169288] [] ? osd_inode_iteration+0x269/0xcc0 [osd_ldiskfs] [15044.178395] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [15044.187301] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [15044.195928] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [15044.204348] [] ? wake_up_state+0x20/0x20 [15044.211302] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [15044.220420] [] kthread+0xcf/0xe0 [15044.226598] [] ? kthread_create_on_node+0x140/0x140 [15044.234630] [] ret_from_fork+0x58/0x90 [15044.241398] [] ? kthread_create_on_node+0x140/0x140 [15046.650465] Lustre: soaked-MDT0000: Client 40e8ebdf-e84f-0dfe-7f82-ddc39c67a41e (at 192.168.1.132@o2ib100) reconnecting [15046.663512] Lustre: soaked-MDT0000: Connection restored to 7eb6195d-0392-6b9e-dc88-22a505b06cf8 (at 192.168.1.132@o2ib100) [15069.367430] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [15069.376909] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [15069.506938] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [15069.520136] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [15069.536240] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [15069.545545] RIP: 0010:[] [] osd_inode_iteration+0x282/0xcc0 [osd_ldiskfs] [15069.557721] RSP: 0018:ffff880703603d18 EFLAGS: 00000287 [15069.564621] RAX: ffff88042bd78800 RBX: ffff880703603d98 RCX: ffff88042bd78800 [15069.573567] RDX: 00000000000072cf RSI: ffff880419dc3a00 RDI: ffff88042bd79800 [15069.582516] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [15069.591451] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000001 [15069.600379] R13: 0000000000000004 R14: 000000009f8f2001 R15: ffffea00027e3c80 [15069.609304] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [15069.619311] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15069.626702] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [15069.635652] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15069.644599] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [15069.653533] Stack: [15069.656739] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [15069.666041] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [15069.675330] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [15069.684622] Call Trace: [15069.688353] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [15069.697524] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [15069.706401] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [15069.715084] [] ? wake_up_state+0x20/0x20 [15069.722302] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [15069.731665] [] kthread+0xcf/0xe0 [15069.738104] [] ? kthread_create_on_node+0x140/0x140 [15069.746392] [] ret_from_fork+0x58/0x90 [15069.753404] [] ? kthread_create_on_node+0x140/0x140 [15069.761685] Code: 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 e8 77 5b f4 ff 42 8d 14 38 48 8b 45 88 48 8b 80 a8 03 00 00 48 3b 50 20 0f 83 57 01 00 00 <44> 0f b6 85 67 ff ff ff 48 8d 4d a8 48 8d 55 88 48 8b b5 38 ff [15097.368403] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [15097.378051] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [15097.510234] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [15097.523546] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [15097.539859] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [15097.549268] RIP: 0010:[] [] osd_scrub_next+0x9/0x3c0 [osd_ldiskfs] [15097.560857] RSP: 0018:ffff880703603d08 EFLAGS: 00000287 [15097.567821] RAX: ffffffffa1217a20 RBX: 000000009f8f2001 RCX: ffff880703603d98 [15097.576810] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [15097.585793] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [15097.594765] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [15097.603719] R13: 0000000000000018 R14: ffff880703603c78 R15: 0000000000000296 [15097.612669] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [15097.622686] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15097.630088] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [15097.639051] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15097.648017] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [15097.656978] Stack: [15097.660204] ffff880703603df0 ffffffffa121cff9 ffffffffa121b990 ffffffffa1217a20 [15097.669493] ffff8800b2a36000 00000000810d354f ffff8803162f6000 ffff8800b2a37468 [15097.678779] 0000000020000000 010000000000000c 0000000000000000 0000000000000000 [15097.688058] Call Trace: [15097.691783] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [15097.700955] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [15097.710110] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [15097.718981] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [15097.727659] [] ? wake_up_state+0x20/0x20 [15097.734866] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [15097.744234] [] kthread+0xcf/0xe0 [15097.750662] [] ? kthread_create_on_node+0x140/0x140 [15097.758945] [] ret_from_fork+0x58/0x90 [15097.765953] [] ? kthread_create_on_node+0x140/0x140 [15097.774220] Code: 48 8d 4c 02 10 48 01 c2 e8 25 fc ff ff 48 83 c4 08 5b 41 5c 41 5d 41 5e 5d c3 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 89 e5 <41> 57 49 89 f7 41 56 41 55 49 89 fd 41 54 49 89 d4 53 48 89 cb [15125.369373] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [15125.379046] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [15125.511017] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [15125.524311] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [15125.540544] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [15125.549920] RIP: 0010:[] [] ldiskfs_itable_unused_count+0x6/0x30 [ldiskfs] [15125.562246] RSP: 0018:ffff880703603d08 EFLAGS: 00000286 [15125.569179] RAX: ffff8800b2a37468 RBX: 000000009f8f2001 RCX: 0000000000000000 [15125.578157] RDX: ffff880703603d78 RSI: ffff880419dc3a00 RDI: ffff88042bd79800 [15125.587123] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [15125.596072] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [15125.605016] R13: 0000000000000018 R14: ffff880703603c78 R15: 0000000000000202 [15125.613964] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [15125.623994] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15125.631398] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [15125.640367] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15125.649319] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [15125.658257] Stack: [15125.661464] ffff880703603df0 ffffffffa121cfb9 ffffffffa121b990 ffffffffa1217a20 [15125.670749] ffff8800b2a36000 00000000810d354f ffff8803162f6000 ffff8800b2a37468 [15125.680044] 0000000020000000 010000000000000c 0000000000000000 0000000000000000 [15125.689342] Call Trace: [15125.693077] [] ? osd_inode_iteration+0x269/0xcc0 [osd_ldiskfs] [15125.702442] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [15125.711611] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [15125.720471] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [15125.729148] [] ? wake_up_state+0x20/0x20 [15125.736356] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [15125.745701] [] kthread+0xcf/0xe0 [15125.752136] [] ? kthread_create_on_node+0x140/0x140 [15125.760418] [] ret_from_fork+0x58/0x90 [15125.767430] [] ? kthread_create_on_node+0x140/0x140 [15125.775704] Code: 00 00 31 c0 0f b7 56 10 48 89 e5 48 83 39 3f 76 07 0f b7 46 30 c1 e0 10 09 d0 5d c3 66 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 <48> 8b 8f a8 03 00 00 31 c0 0f b7 56 1c 48 89 e5 48 83 39 3f 76 [15153.370348] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [15153.380017] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [15153.511951] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [15153.525238] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [15153.541496] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [15153.550891] RIP: 0010:[] [] osd_scrub_next+0x15/0x3c0 [osd_ldiskfs] [15153.562549] RSP: 0018:ffff880703603cf0 EFLAGS: 00000287 [15153.569494] RAX: ffffffffa1217a20 RBX: 000000009f8f2001 RCX: ffff880703603d98 [15153.578465] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [15153.587434] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [15153.596390] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffff880703603c78 [15153.605343] R13: 0000000000000246 R14: 0000000000000010 R15: ffffffffa121ba12 [15153.614296] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [15153.624326] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15153.631734] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [15153.640705] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15153.649674] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [15153.658631] Stack: [15153.661856] 0000000000000000 00000000ffffffff 0000000000000001 ffff880703603df0 [15153.671147] ffffffffa121cff9 ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 [15153.680447] 00000000810d354f ffff8803162f6000 ffff8800b2a37468 0000000020000000 [15153.689732] Call Trace: [15153.693472] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [15153.702655] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [15153.711828] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [15153.720703] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [15153.729385] [] ? wake_up_state+0x20/0x20 [15153.736592] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [15153.745951] [] kthread+0xcf/0xe0 [15153.752394] [] ? kthread_create_on_node+0x140/0x140 [15153.760678] [] ret_from_fork+0x58/0x90 [15153.767699] [] ? kthread_create_on_node+0x140/0x140 [15153.775982] Code: ff 48 83 c4 08 5b 41 5c 41 5d 41 5e 5d c3 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 89 e5 41 57 49 89 f7 41 56 41 55 49 89 fd <41> 54 49 89 d4 53 48 89 cb 48 83 ec 70 65 48 8b 04 25 28 00 00 [15181.371329] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [15181.380979] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [15181.513303] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [15181.526588] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [15181.542859] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [15181.552249] RIP: 0010:[] [] osd_scrub_exec+0x8d/0x5b0 [osd_ldiskfs] [15181.563916] RSP: 0018:ffff880703603ce8 EFLAGS: 00000292 [15181.570861] RAX: 0000000000000000 RBX: ffff880703603c78 RCX: 0000000000000000 [15181.579852] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [15181.588822] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [15181.597780] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000010 [15181.606731] R13: ffffffffa121b9c0 R14: ffffffffffffff10 R15: ffff8803162f6000 [15181.615677] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [15181.625700] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15181.633106] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [15181.642080] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15181.651039] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [15181.659986] Stack: [15181.663213] ffff8800b2a3746c 0000000000000004 00000000ffffffff 0000000000000001 [15181.672504] ffff880703603df0 ffffffffa121d1e9 ffffffffa121b990 ffffffffa1217a20 [15181.681803] ffff8800b2a36000 00000000810d354f ffff8803162f6000 ffff8800b2a37468 [15181.691097] Call Trace: [15181.694837] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [15181.704012] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [15181.713176] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [15181.722042] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [15181.730708] [] ? wake_up_state+0x20/0x20 [15181.737919] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [15181.747284] [] kthread+0xcf/0xe0 [15181.753725] [] ? kthread_create_on_node+0x140/0x140 [15181.762021] [] ret_from_fork+0x58/0x90 [15181.769035] [] ? kthread_create_on_node+0x140/0x140 [15181.777315] Code: 85 c0 41 89 c1 0f 84 a3 01 00 00 41 80 a7 74 14 00 00 fe 48 8b 4d d0 65 48 33 0c 25 28 00 00 00 0f 85 b1 03 00 00 48 83 c4 68 5b <41> 5c 41 5d 41 5e 41 5f 5d c3 66 0f 1f 84 00 00 00 00 00 4c 8d [15209.372305] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [15209.381958] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [15209.513903] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [15209.527196] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [15209.543449] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [15209.552838] RIP: 0010:[] [] osd_inode_iteration+0x251/0xcc0 [osd_ldiskfs] [15209.565084] RSP: 0018:ffff880703603d18 EFLAGS: 00000286 [15209.572024] RAX: ffff8800b2a37468 RBX: ffff880703603d98 RCX: 0000000000000000 [15209.581007] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [15209.589971] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [15209.598922] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000001 [15209.607875] R13: 0000000000000004 R14: 000000009f8f2001 R15: 0000000000000018 [15209.616828] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [15209.626860] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15209.634259] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [15209.643228] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15209.652188] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [15209.661122] Stack: [15209.664330] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [15209.673597] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [15209.682896] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [15209.692193] Call Trace: [15209.695915] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [15209.705066] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [15209.713925] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [15209.722583] [] ? wake_up_state+0x20/0x20 [15209.729782] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [15209.739128] [] kthread+0xcf/0xe0 [15209.745547] [] ? kthread_create_on_node+0x140/0x140 [15209.753821] [] ret_from_fork+0x58/0x90 [15209.760829] [] ? kthread_create_on_node+0x140/0x140 [15209.769090] Code: b0 00 00 00 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 00 8b 75 98 48 8b 7d 88 e8 dc 2d f5 ff 48 85 c0 48 89 45 90 0f 84 b3 08 00 00 <44> 8b 7d a0 48 8b 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 e8 77 [15224.004822] INFO: rcu_sched self-detected stall on CPU { 24} (t=7800150 jiffies g=907112 c=907111 q=24629680) [15224.017296] Task dump for CPU 24: [15224.022034] OI_scrub R running task 0 5551 2 0x00000088 [15224.031003] ffff88083fde6dd0 00000000fb6db880 ffff88082da03db0 ffffffff810c47e8 [15224.040305] 0000000000000018 ffffffff81a1e7c0 ffff88082da03dc8 ffffffff810c8079 [15224.049604] 0000000000000009 ffff88082da03df8 ffffffff811378e0 ffff88082da101c0 [15224.058897] Call Trace: [15224.062594] [] sched_show_task+0xa8/0x110 [15224.070581] [] dump_cpu_task+0x39/0x70 [15224.077551] [] rcu_dump_cpu_stacks+0x90/0xd0 [15224.085090] [] rcu_check_callbacks+0x442/0x720 [15224.092786] [] ? tick_sched_handle.isra.13+0x60/0x60 [15224.101046] [] update_process_times+0x47/0x80 [15224.108602] [] tick_sched_handle.isra.13+0x25/0x60 [15224.116622] [] tick_sched_timer+0x41/0x70 [15224.123752] [] __hrtimer_run_queues+0xd2/0x260 [15224.131356] [] hrtimer_interrupt+0xb0/0x1e0 [15224.138641] [] ? call_softirq+0x1c/0x30 [15224.145522] [] local_apic_timer_interrupt+0x37/0x60 [15224.153549] [] smp_apic_timer_interrupt+0x3f/0x60 [15224.161366] [] apic_timer_interrupt+0x6d/0x80 [15224.168786] [] ? osd_inode_iteration+0x2af/0xcc0 [osd_ldiskfs] [15224.178601] [] ? osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [15224.187677] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [15224.196581] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [15224.205188] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [15224.213594] [] ? wake_up_state+0x20/0x20 [15224.220552] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [15224.229669] [] kthread+0xcf/0xe0 [15224.235836] [] ? kthread_create_on_node+0x140/0x140 [15224.243857] [] ret_from_fork+0x58/0x90 [15224.250613] [] ? kthread_create_on_node+0x140/0x140 [15249.373692] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [15249.383143] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [15249.512931] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [15249.526119] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [15249.542189] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [15249.551489] RIP: 0010:[] [] ldiskfs_itable_unused_count+0x23/0x30 [ldiskfs] [15249.563816] RSP: 0018:ffff880703603d08 EFLAGS: 00000297 [15249.570685] RAX: 0000000000000000 RBX: ffff880703603d57 RCX: ffff88042bd78800 [15249.579598] RDX: 00000000000072ce RSI: ffff880419dc3a00 RDI: ffff88042bd79800 [15249.588497] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [15249.597410] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [15249.606307] R13: 0000000000000018 R14: ffff880703603c78 R15: 0000000000000296 [15249.615194] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [15249.625153] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15249.632487] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [15249.641380] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15249.650289] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [15249.659191] Stack: [15249.662357] ffff880703603df0 ffffffffa121cfb9 ffffffffa121b990 ffffffffa1217a20 [15249.671612] ffff8800b2a36000 00000000810d354f ffff8803162f6000 ffff8800b2a37468 [15249.680860] 0000000020000000 010000000000000c 0000000000000000 0000000000000000 [15249.690115] Call Trace: [15249.693821] [] osd_inode_iteration+0x269/0xcc0 [osd_ldiskfs] [15249.702952] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [15249.712079] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [15249.720916] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [15249.729579] [] ? wake_up_state+0x20/0x20 [15249.736755] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [15249.746069] [] kthread+0xcf/0xe0 [15249.752458] [] ? kthread_create_on_node+0x140/0x140 [15249.760703] [] ret_from_fork+0x58/0x90 [15249.767680] [] ? kthread_create_on_node+0x140/0x140 [15249.775922] Code: 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 8b 8f a8 03 00 00 31 c0 0f b7 56 1c 48 89 e5 48 83 39 3f 76 07 0f b7 46 32 c1 e0 10 <09> d0 5d c3 66 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 89 16 [15277.374668] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [15277.384282] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [15277.516259] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [15277.529546] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [15277.545793] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [15277.555171] RIP: 0010:[] [] osd_scrub_next+0x9/0x3c0 [osd_ldiskfs] [15277.566730] RSP: 0018:ffff880703603d08 EFLAGS: 00000287 [15277.573668] RAX: ffffffffa1217a20 RBX: 0000000000000001 RCX: ffff880703603d98 [15277.582635] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [15277.591597] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [15277.600546] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [15277.609481] R13: ffffea00027e3c80 R14: ffff880703603c70 R15: ffff880703603d08 [15277.618432] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [15277.628454] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15277.635852] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [15277.644810] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15277.653768] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [15277.662717] Stack: [15277.665946] ffff880703603df0 ffffffffa121cff9 ffffffffa121b990 ffffffffa1217a20 [15277.675252] ffff8800b2a36000 00000000810d354f ffff8803162f6000 ffff8800b2a37468 [15277.684553] 0000000020000000 010000000000000c 0000000000000000 0000000000000000 [15277.693844] Call Trace: [15277.697574] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [15277.706748] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [15277.715912] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [15277.724775] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [15277.733450] [] ? wake_up_state+0x20/0x20 [15277.740660] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [15277.750004] [] kthread+0xcf/0xe0 [15277.756438] [] ? kthread_create_on_node+0x140/0x140 [15277.764708] [] ret_from_fork+0x58/0x90 [15277.771713] [] ? kthread_create_on_node+0x140/0x140 [15277.779991] Code: 48 8d 4c 02 10 48 01 c2 e8 25 fc ff ff 48 83 c4 08 5b 41 5c 41 5d 41 5e 5d c3 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 89 e5 <41> 57 49 89 f7 41 56 41 55 49 89 fd 41 54 49 89 d4 53 48 89 cb [15305.375647] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [15305.385316] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [15305.517425] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [15305.530716] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [15305.546981] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [15305.556368] RIP: 0010:[] [] osd_inode_iteration+0x282/0xcc0 [osd_ldiskfs] [15305.568605] RSP: 0018:ffff880703603d18 EFLAGS: 00000287 [15305.575550] RAX: ffff88042bd78800 RBX: ffff880703603d98 RCX: ffff88042bd78800 [15305.584535] RDX: 00000000000072cf RSI: ffff880419dc3a00 RDI: ffff88042bd79800 [15305.593505] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [15305.602466] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000001 [15305.611421] R13: 0000000000000004 R14: 000000009f8f2001 R15: 0000000000000018 [15305.620375] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [15305.630403] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15305.637809] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [15305.646785] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15305.655747] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [15305.664684] Stack: [15305.667891] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [15305.677160] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [15305.686455] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [15305.695737] Call Trace: [15305.699461] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [15305.708617] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [15305.717488] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [15305.726167] [] ? wake_up_state+0x20/0x20 [15305.733372] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [15305.742758] [] kthread+0xcf/0xe0 [15305.749197] [] ? kthread_create_on_node+0x140/0x140 [15305.757496] [] ret_from_fork+0x58/0x90 [15305.764509] [] ? kthread_create_on_node+0x140/0x140 [15305.772787] Code: 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 e8 77 5b f4 ff 42 8d 14 38 48 8b 45 88 48 8b 80 a8 03 00 00 48 3b 50 20 0f 83 57 01 00 00 <44> 0f b6 85 67 ff ff ff 48 8d 4d a8 48 8d 55 88 48 8b b5 38 ff [15333.376618] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [15333.386257] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [15333.518256] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [15333.531558] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [15333.547853] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [15333.557252] RIP: 0010:[] [] osd_inode_iteration+0x4a0/0xcc0 [osd_ldiskfs] [15333.569502] RSP: 0018:ffff880703603d18 EFLAGS: 00000246 [15333.576459] RAX: 0000000000000000 RBX: 000000009f8f2001 RCX: 0000000000000000 [15333.585441] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [15333.594425] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [15333.603390] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [15333.612340] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [15333.621286] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [15333.631314] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15333.638715] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [15333.647683] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15333.656645] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [15333.665596] Stack: [15333.668810] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [15333.678094] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [15333.687384] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [15333.696688] Call Trace: [15333.700413] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [15333.709580] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [15333.718453] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [15333.727144] [] ? wake_up_state+0x20/0x20 [15333.734375] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [15333.743743] [] kthread+0xcf/0xe0 [15333.750176] [] ? kthread_create_on_node+0x140/0x140 [15333.758461] [] ret_from_fork+0x58/0x90 [15333.765468] [] ? kthread_create_on_node+0x140/0x140 [15333.773749] Code: 85 67 ff ff ff 48 8b 4d a8 48 8d 55 88 48 8b b5 38 ff ff ff 48 8b bd 48 ff ff ff 48 8b 85 28 ff ff ff ff d0 85 c0 41 89 c5 75 1c <8b> 85 58 ff ff ff 41 3b 04 24 72 10 48 8b 85 50 ff ff ff 44 3b [15361.377593] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [15361.387244] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [15361.519116] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [15361.532433] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [15361.548723] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [15361.558122] RIP: 0010:[] [] osd_scrub_exec+0x8d/0x5b0 [osd_ldiskfs] [15361.569815] RSP: 0018:ffff880703603ce8 EFLAGS: 00000292 [15361.576778] RAX: 0000000000000000 RBX: ffff880703603c78 RCX: 0000000000000000 [15361.585761] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [15361.594737] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [15361.603695] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000010 [15361.612655] R13: ffffffffa121ba09 R14: ffffffffffffff10 R15: ffff8803162f6000 [15361.621596] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [15361.631625] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15361.639028] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [15361.647977] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15361.656934] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [15361.665876] Stack: [15361.669089] ffff8800b2a3746c 0000000000000004 00000000ffffffff 0000000000000001 [15361.678377] ffff880703603df0 ffffffffa121d1e9 ffffffffa121b990 ffffffffa1217a20 [15361.687665] ffff8800b2a36000 00000000810d354f ffff8803162f6000 ffff8800b2a37468 [15361.696955] Call Trace: [15361.700684] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [15361.709848] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [15361.719014] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [15361.727881] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [15361.736539] [] ? wake_up_state+0x20/0x20 [15361.743740] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [15361.753095] [] kthread+0xcf/0xe0 [15361.759526] [] ? kthread_create_on_node+0x140/0x140 [15361.767819] [] ret_from_fork+0x58/0x90 [15361.774825] [] ? kthread_create_on_node+0x140/0x140 [15361.783095] Code: 85 c0 41 89 c1 0f 84 a3 01 00 00 41 80 a7 74 14 00 00 fe 48 8b 4d d0 65 48 33 0c 25 28 00 00 00 0f 85 b1 03 00 00 48 83 c4 68 5b <41> 5c 41 5d 41 5e 41 5f 5d c3 66 0f 1f 84 00 00 00 00 00 4c 8d [15389.378576] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [15389.388210] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [15389.520525] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [15389.533818] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [15389.550084] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [15389.559475] RIP: 0010:[] [] osd_inode_iteration+0x255/0xcc0 [osd_ldiskfs] [15389.571716] RSP: 0018:ffff880703603d18 EFLAGS: 00000286 [15389.578658] RAX: ffff8800b2a37468 RBX: 000000009f8f2001 RCX: 0000000000000000 [15389.587643] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [15389.596608] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [15389.605567] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000001 [15389.614522] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [15389.623475] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [15389.633508] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15389.640913] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [15389.649891] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15389.658852] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [15389.667808] Stack: [15389.671030] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [15389.680330] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [15389.689635] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [15389.698932] Call Trace: [15389.702660] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [15389.711826] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [15389.720691] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [15389.729370] [] ? wake_up_state+0x20/0x20 [15389.736576] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [15389.745935] [] kthread+0xcf/0xe0 [15389.752368] [] ? kthread_create_on_node+0x140/0x140 [15389.760641] [] ret_from_fork+0x58/0x90 [15389.767659] [] ? kthread_create_on_node+0x140/0x140 [15389.775934] Code: 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 00 8b 75 98 48 8b 7d 88 e8 dc 2d f5 ff 48 85 c0 48 89 45 90 0f 84 b3 08 00 00 44 8b 7d a0 <48> 8b 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 e8 77 5b f4 ff 42 [15404.014084] INFO: rcu_sched self-detected stall on CPU { 24} (t=7980153 jiffies g=907112 c=907111 q=25014902) [15404.026520] Task dump for CPU 24: [15404.031265] OI_scrub R running task 0 5551 2 0x00000088 [15404.040207] ffff88083fde6dd0 00000000fb6db880 ffff88082da03db0 ffffffff810c47e8 [15404.049544] 0000000000000018 ffffffff81a1e7c0 ffff88082da03dc8 ffffffff810c8079 [15404.058866] 0000000000000009 ffff88082da03df8 ffffffff811378e0 ffff88082da101c0 [15404.068176] Call Trace: [15404.071883] [] sched_show_task+0xa8/0x110 [15404.079878] [] dump_cpu_task+0x39/0x70 [15404.086846] [] rcu_dump_cpu_stacks+0x90/0xd0 [15404.094384] [] rcu_check_callbacks+0x442/0x720 [15404.102104] [] ? tick_sched_handle.isra.13+0x60/0x60 [15404.110395] [] update_process_times+0x47/0x80 [15404.117965] [] tick_sched_handle.isra.13+0x25/0x60 [15404.126006] [] tick_sched_timer+0x41/0x70 [15404.133155] [] __hrtimer_run_queues+0xd2/0x260 [15404.140762] [] hrtimer_interrupt+0xb0/0x1e0 [15404.148070] [] ? call_softirq+0x1c/0x30 [15404.154965] [] local_apic_timer_interrupt+0x37/0x60 [15404.163013] [] smp_apic_timer_interrupt+0x3f/0x60 [15404.170858] [] apic_timer_interrupt+0x6d/0x80 [15404.178295] [] ? osd_scrub_exec+0xe9/0x5b0 [osd_ldiskfs] [15404.187530] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [15404.196429] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [15404.205338] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [15404.213951] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [15404.222368] [] ? wake_up_state+0x20/0x20 [15404.229322] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [15404.238407] [] kthread+0xcf/0xe0 [15404.244583] [] ? kthread_create_on_node+0x140/0x140 [15404.252607] [] ret_from_fork+0x58/0x90 [15404.259375] [] ? kthread_create_on_node+0x140/0x140 [15429.379961] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [15429.389347] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [15429.519050] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [15429.532215] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [15429.548280] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [15429.557578] RIP: 0010:[] [] osd_scrub_exec+0xf7/0x5b0 [osd_ldiskfs] [15429.569136] RSP: 0018:ffff880703603c78 EFLAGS: 00000246 [15429.576003] RAX: 0000000000280001 RBX: ffff880703603c37 RCX: 0000000000000000 [15429.584908] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [15429.593805] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [15429.602701] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [15429.611600] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [15429.620495] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [15429.630463] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15429.637808] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [15429.646715] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15429.655621] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [15429.664521] Stack: [15429.667689] ffff880703603df0 000000009f8f2001 ffffea00027e3c80 000000009f8f2001 [15429.676937] ffff880703603d57 ffff880703603d57 ffff8800b2a37468 0000000000000000 [15429.686188] ffff880703603d78 ffff880419dc3a00 ffff88042bd79800 ffffffffffffff10 [15429.695437] Call Trace: [15429.699116] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [15429.708241] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [15429.717370] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [15429.726208] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [15429.734841] [] ? wake_up_state+0x20/0x20 [15429.742027] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [15429.751336] [] kthread+0xcf/0xe0 [15429.757728] [] ? kthread_create_on_node+0x140/0x140 [15429.765973] [] ret_from_fork+0x58/0x90 [15429.772945] [] ? kthread_create_on_node+0x140/0x140 [15429.781179] Code: 8b 4d 88 48 85 db 74 23 f6 83 18 0a 00 00 08 74 1a 4d 85 e4 74 15 41 8b 87 6c 14 00 00 41 39 84 24 0c 0a 00 00 0f 82 d9 00 00 00 <31> c0 41 f6 87 74 14 00 00 04 0f 85 6e ff ff ff 41 83 f9 02 0f [15457.380943] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [15457.390548] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [15457.522794] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [15457.536084] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [15457.552356] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [15457.561741] RIP: 0010:[] [] ldiskfs_itable_unused_count+0x6/0x30 [ldiskfs] [15457.574079] RSP: 0018:ffff880703603d08 EFLAGS: 00000286 [15457.581024] RAX: ffff8800b2a37468 RBX: 0000000000000001 RCX: 0000000000000000 [15457.590009] RDX: ffff880703603d78 RSI: ffff880419dc3a00 RDI: ffff88042bd79800 [15457.598977] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [15457.607933] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [15457.616885] R13: 0000000000000018 R14: ffff880703603c70 R15: 0000000000000202 [15457.625834] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [15457.635857] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15457.643257] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [15457.652218] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15457.661172] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [15457.670103] Stack: [15457.673306] ffff880703603df0 ffffffffa121cfb9 ffffffffa121b990 ffffffffa1217a20 [15457.682594] ffff8800b2a36000 00000000810d354f ffff8803162f6000 ffff8800b2a37468 [15457.691880] 0000000020000000 010000000000000c 0000000000000000 0000000000000000 [15457.701157] Call Trace: [15457.704898] [] ? osd_inode_iteration+0x269/0xcc0 [osd_ldiskfs] [15457.714254] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [15457.723412] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [15457.732271] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [15457.740939] [] ? wake_up_state+0x20/0x20 [15457.748149] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [15457.757512] [] kthread+0xcf/0xe0 [15457.763952] [] ? kthread_create_on_node+0x140/0x140 [15457.772248] [] ret_from_fork+0x58/0x90 [15457.779260] [] ? kthread_create_on_node+0x140/0x140 [15457.787531] Code: 00 00 31 c0 0f b7 56 10 48 89 e5 48 83 39 3f 76 07 0f b7 46 30 c1 e0 10 09 d0 5d c3 66 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 <48> 8b 8f a8 03 00 00 31 c0 0f b7 56 1c 48 89 e5 48 83 39 3f 76 [15485.381919] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [15485.391564] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [15485.523569] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [15485.536845] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [15485.553104] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [15485.562498] RIP: 0010:[] [] osd_inode_iteration+0x251/0xcc0 [osd_ldiskfs] [15485.574750] RSP: 0018:ffff880703603d18 EFLAGS: 00000286 [15485.581698] RAX: ffff8800b2a37468 RBX: ffff880703603d98 RCX: 0000000000000000 [15485.590684] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [15485.599655] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [15485.608612] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000001 [15485.617563] R13: 0000000000000004 R14: 000000009f8f2001 R15: ffffea00027e3c80 [15485.626511] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [15485.636537] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15485.643942] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [15485.652909] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15485.661871] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [15485.670825] Stack: [15485.674031] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [15485.683333] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [15485.692639] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [15485.701943] Call Trace: [15485.705673] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [15485.714842] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [15485.723720] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [15485.732398] [] ? wake_up_state+0x20/0x20 [15485.739609] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [15485.748970] [] kthread+0xcf/0xe0 [15485.755402] [] ? kthread_create_on_node+0x140/0x140 [15485.763698] [] ret_from_fork+0x58/0x90 [15485.770713] [] ? kthread_create_on_node+0x140/0x140 [15485.778995] Code: b0 00 00 00 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 00 8b 75 98 48 8b 7d 88 e8 dc 2d f5 ff 48 85 c0 48 89 45 90 0f 84 b3 08 00 00 <44> 8b 7d a0 48 8b 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 e8 77 [15513.382894] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [15513.392520] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [15513.524687] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [15513.537987] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [15513.554279] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [15513.563676] RIP: 0010:[] [] osd_scrub_next+0x353/0x3c0 [osd_ldiskfs] [15513.575440] RSP: 0018:ffff880703603c70 EFLAGS: 00000202 [15513.582407] RAX: 0000000000000000 RBX: ffff880703603c37 RCX: ffff880703603d98 [15513.591401] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [15513.600380] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [15513.609342] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [15513.618296] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [15513.627245] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [15513.637272] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15513.644682] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [15513.653652] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15513.662626] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [15513.671587] Stack: [15513.674808] 000000009f8f2001 ffff880703603d08 ffff880703603d57 ffffea00027e3c80 [15513.684094] 000000009f8f2001 ffff880703603d57 ffff880703603d57 0000000000000000 [15513.693379] 0000000000000000 ffff880703603d78 ffff8800b2a36000 ffff8803162f6000 [15513.702647] Call Trace: [15513.706366] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [15513.715529] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [15513.724686] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [15513.733553] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [15513.742228] [] ? wake_up_state+0x20/0x20 [15513.749429] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [15513.758780] [] kthread+0xcf/0xe0 [15513.765203] [] ? kthread_create_on_node+0x140/0x140 [15513.773487] [] ret_from_fork+0x58/0x90 [15513.780496] [] ? kthread_create_on_node+0x140/0x140 [15513.788766] Code: 8b 55 90 48 8b 4d 88 44 8b 85 74 ff ff ff e9 1d ff ff ff 48 89 d7 44 89 45 90 e8 f9 67 47 e0 44 8b 45 90 0f 1f 44 00 00 45 84 c0 04 00 00 00 0f 85 eb fd ff ff 49 8d b7 6c 14 00 00 4c 89 e7 [15541.383870] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [15541.393528] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [15541.525682] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [15541.538971] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [15541.555205] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [15541.564585] RIP: 0010:[] [] osd_inode_iteration+0x490/0xcc0 [osd_ldiskfs] [15541.576832] RSP: 0018:ffff880703603d18 EFLAGS: 00000293 [15541.583760] RAX: 0000000000000004 RBX: 000000009f8f2001 RCX: 0000000000000000 [15541.592740] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [15541.601702] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [15541.610659] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffff880703603d57 [15541.619607] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [15541.628557] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [15541.638585] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15541.645981] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [15541.654958] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15541.663916] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [15541.672858] Stack: [15541.676063] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [15541.685347] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [15541.694637] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [15541.703926] Call Trace: [15541.707647] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [15541.716817] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [15541.725697] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [15541.734376] [] ? wake_up_state+0x20/0x20 [15541.741590] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [15541.750967] [] kthread+0xcf/0xe0 [15541.757408] [] ? kthread_create_on_node+0x140/0x140 [15541.765710] [] ret_from_fork+0x58/0x90 [15541.772731] [] ? kthread_create_on_node+0x140/0x140 [15541.781013] Code: 0f fc ff ff 0f 1f 80 00 00 00 00 45 89 e9 4c 8d 85 67 ff ff ff 48 8b 4d a8 48 8d 55 88 48 8b b5 38 ff ff ff 48 8b bd 48 ff ff ff <48> 8b 85 28 ff ff ff ff d0 85 c0 41 89 c5 75 1c 8b 85 58 ff ff [15569.384853] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [15569.394539] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [15569.526559] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [15569.539874] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [15569.556173] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [15569.565586] RIP: 0010:[] [] osd_scrub_exec+0x8f/0x5b0 [osd_ldiskfs] [15569.577263] RSP: 0018:ffff880703603cf0 EFLAGS: 00000292 [15569.584227] RAX: 0000000000000000 RBX: 000000009f8f2001 RCX: 0000000000000000 [15569.593224] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [15569.602215] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [15569.611189] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffff880703603c78 [15569.620146] R13: ffff880703603d08 R14: 0000000000000010 R15: ffffffffa121ba79 [15569.629098] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [15569.639127] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15569.646536] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [15569.655506] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15569.664471] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [15569.673424] Stack: [15569.676644] 0000000000000004 00000000ffffffff 0000000000000001 ffff880703603df0 [15569.685940] ffffffffa121d1e9 ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 [15569.695230] 00000000810d354f ffff8803162f6000 ffff8800b2a37468 0000000020000000 [15569.704536] Call Trace: [15569.708272] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [15569.717443] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [15569.726611] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [15569.735486] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [15569.744150] [] ? wake_up_state+0x20/0x20 [15569.751352] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [15569.760729] [] kthread+0xcf/0xe0 [15569.767162] [] ? kthread_create_on_node+0x140/0x140 [15569.775452] [] ret_from_fork+0x58/0x90 [15569.782466] [] ? kthread_create_on_node+0x140/0x140 [15569.790737] Code: 41 89 c1 0f 84 a3 01 00 00 41 80 a7 74 14 00 00 fe 48 8b 4d d0 65 48 33 0c 25 28 00 00 00 0f 85 b1 03 00 00 48 83 c4 68 5b 41 5c <41> 5d 41 5e 41 5f 5d c3 66 0f 1f 84 00 00 00 00 00 4c 8d ae f0 [15584.023363] INFO: rcu_sched self-detected stall on CPU { 24} (t=8160156 jiffies g=907112 c=907111 q=25451775) [15584.035815] Task dump for CPU 24: [15584.040556] OI_scrub R running task 0 5551 2 0x00000088 [15584.049491] ffff88083fde6dd0 00000000fb6db880 ffff88082da03db0 ffffffff810c47e8 [15584.058833] 0000000000000018 ffffffff81a1e7c0 ffff88082da03dc8 ffffffff810c8079 [15584.068112] 0000000000000009 ffff88082da03df8 ffffffff811378e0 ffff88082da101c0 [15584.077390] Call Trace: [15584.081080] [] sched_show_task+0xa8/0x110 [15584.089069] [] dump_cpu_task+0x39/0x70 [15584.096032] [] rcu_dump_cpu_stacks+0x90/0xd0 [15584.103549] [] rcu_check_callbacks+0x442/0x720 [15584.111249] [] ? tick_sched_handle.isra.13+0x60/0x60 [15584.119510] [] update_process_times+0x47/0x80 [15584.127074] [] tick_sched_handle.isra.13+0x25/0x60 [15584.135081] [] tick_sched_timer+0x41/0x70 [15584.142216] [] __hrtimer_run_queues+0xd2/0x260 [15584.149814] [] hrtimer_interrupt+0xb0/0x1e0 [15584.157091] [] ? call_softirq+0x1c/0x30 [15584.163973] [] local_apic_timer_interrupt+0x37/0x60 [15584.172003] [] smp_apic_timer_interrupt+0x3f/0x60 [15584.179849] [] apic_timer_interrupt+0x6d/0x80 [15584.187275] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [15584.196895] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [15584.205803] [] ? osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [15584.214905] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [15584.223822] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [15584.232437] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [15584.240834] [] ? wake_up_state+0x20/0x20 [15584.247780] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [15584.256884] [] kthread+0xcf/0xe0 [15584.263068] [] ? kthread_create_on_node+0x140/0x140 [15584.271087] [] ret_from_fork+0x58/0x90 [15584.277846] [] ? kthread_create_on_node+0x140/0x140 [15609.386240] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [15609.395666] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [15609.525368] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [15609.538543] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [15609.554580] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [15609.563867] RIP: 0010:[] [] osd_scrub_next+0x45/0x3c0 [osd_ldiskfs] [15609.575423] RSP: 0018:ffff880703603c70 EFLAGS: 00000202 [15609.582282] RAX: 0000000000000000 RBX: ffff880703603c37 RCX: ffff880703603d98 [15609.591183] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [15609.600088] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [15609.608992] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [15609.617885] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [15609.626785] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [15609.636750] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15609.644100] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [15609.653011] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15609.661924] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [15609.670825] Stack: [15609.673993] 000000009f8f2001 ffffea00027e3c80 000000009f8f2001 0000000000000004 [15609.683247] ffff880703603d57 ffff880703603d57 0000000000000000 ffff880703603d78 [15609.692506] ffff8800b2a36000 ffff8803162f6000 ffffffffffffff10 ffffffffa121b9a2 [15609.701757] Call Trace: [15609.705456] [] ? osd_scrub_exec+0x12/0x5b0 [osd_ldiskfs] [15609.714205] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [15609.723353] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [15609.732500] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [15609.741345] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [15609.749991] [] ? wake_up_state+0x20/0x20 [15609.757177] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [15609.766500] [] kthread+0xcf/0xe0 [15609.772901] [] ? kthread_create_on_node+0x140/0x140 [15609.781149] [] ret_from_fork+0x58/0x90 [15609.788130] [] ? kthread_create_on_node+0x140/0x140 [15609.796366] Code: 53 48 89 cb 48 83 ec 70 65 48 8b 04 25 28 00 00 00 48 89 45 d0 31 c0 48 8b 05 f8 59 9c ff 48 85 c0 75 7b 41 f6 87 58 10 00 00 08 <0f> 84 62 03 00 00 4d 8d b7 e0 10 00 00 4d 3b b7 e0 10 00 00 0f [15637.387209] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [15637.396857] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [15637.528569] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [15637.541839] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [15637.558070] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [15637.567440] RIP: 0010:[] [] osd_inode_iteration+0x269/0xcc0 [osd_ldiskfs] [15637.579677] RSP: 0018:ffff880703603d18 EFLAGS: 00000202 [15637.586620] RAX: 00000000000072ce RBX: 0000000000000001 RCX: ffff88042bd78800 [15637.595588] RDX: 00000000000072ce RSI: ffff880419dc3a00 RDI: ffff88042bd79800 [15637.604542] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [15637.613500] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [15637.622457] R13: ffffea00027e3c80 R14: 000000009f8f2001 R15: 0000000000000018 [15637.631409] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [15637.641435] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15637.648844] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [15637.657807] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15637.666764] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [15637.675715] Stack: [15637.678933] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [15637.688233] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [15637.697525] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [15637.706810] Call Trace: [15637.710532] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [15637.719697] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [15637.728557] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [15637.737224] [] ? wake_up_state+0x20/0x20 [15637.744426] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [15637.753777] [] kthread+0xcf/0xe0 [15637.760203] [] ? kthread_create_on_node+0x140/0x140 [15637.768478] [] ret_from_fork+0x58/0x90 [15637.775493] [] ? kthread_create_on_node+0x140/0x140 [15637.783767] Code: 88 e8 dc 2d f5 ff 48 85 c0 48 89 45 90 0f 84 b3 08 00 00 44 8b 7d a0 48 8b 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 e8 77 5b f4 ff <42> 8d 14 38 48 8b 45 88 48 8b 80 a8 03 00 00 48 3b 50 20 0f 83 [15665.388183] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [15665.397822] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [15665.529995] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [15665.543307] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [15665.559588] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [15665.568987] RIP: 0010:[] [] osd_scrub_exec+0x79/0x5b0 [osd_ldiskfs] [15665.580657] RSP: 0018:ffff880703603c78 EFLAGS: 00000202 [15665.587618] RAX: 0000000000000000 RBX: ffff880703603c37 RCX: 00000000fb6db880 [15665.596605] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [15665.605586] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [15665.614544] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [15665.623492] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [15665.632437] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [15665.642464] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15665.649876] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [15665.658839] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15665.667808] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [15665.676762] Stack: [15665.679973] ffffea00027e3c80 000000009f8f2001 ffff880703603df0 0000000000000001 [15665.689262] ffff880703603d57 000000009f8f2001 0000000000000004 0000000000000001 [15665.698558] 0000000000000004 ffff880703603d98 ffff880703603d78 0000000000000000 [15665.707833] Call Trace: [15665.711565] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [15665.720728] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [15665.729894] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [15665.738760] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [15665.747430] [] ? wake_up_state+0x20/0x20 [15665.754640] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [15665.763992] [] kthread+0xcf/0xe0 [15665.770423] [] ? kthread_create_on_node+0x140/0x140 [15665.778702] [] ret_from_fork+0x58/0x90 [15665.785705] [] ? kthread_create_on_node+0x140/0x140 [15665.793968] Code: 07 74 4f 41 83 f9 02 74 78 48 89 ca 44 89 c9 e8 0e c6 ff ff 85 c0 41 89 c1 0f 84 a3 01 00 00 41 80 a7 74 14 00 00 fe 48 8b 4d d0 <65> 48 33 0c 25 28 00 00 00 0f 85 b1 03 00 00 48 83 c4 68 5b 41 [15693.389164] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [15693.398829] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [15693.531039] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [15693.544325] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [15693.560578] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [15693.569966] RIP: 0010:[] [] osd_inode_iteration+0x282/0xcc0 [osd_ldiskfs] [15693.582204] RSP: 0018:ffff880703603d18 EFLAGS: 00000287 [15693.589132] RAX: ffff88042bd78800 RBX: ffff880703603d98 RCX: ffff88042bd78800 [15693.598105] RDX: 00000000000072cf RSI: ffff880419dc3a00 RDI: ffff88042bd79800 [15693.607072] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [15693.616032] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000001 [15693.624984] R13: 0000000000000004 R14: 000000009f8f2001 R15: 0000000000000018 [15693.633932] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [15693.643954] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15693.651356] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [15693.660321] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15693.669271] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [15693.678215] Stack: [15693.681424] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [15693.690686] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [15693.699982] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [15693.709237] Call Trace: [15693.712954] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [15693.722111] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [15693.730980] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [15693.739650] [] ? wake_up_state+0x20/0x20 [15693.746854] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [15693.756208] [] kthread+0xcf/0xe0 [15693.762646] [] ? kthread_create_on_node+0x140/0x140 [15693.770945] [] ret_from_fork+0x58/0x90 [15693.777956] [] ? kthread_create_on_node+0x140/0x140 [15693.786217] Code: 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 e8 77 5b f4 ff 42 8d 14 38 48 8b 45 88 48 8b 80 a8 03 00 00 48 3b 50 20 0f 83 57 01 00 00 <44> 0f b6 85 67 ff ff ff 48 8d 4d a8 48 8d 55 88 48 8b b5 38 ff [15721.390133] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [15721.399817] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [15721.531823] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [15721.545142] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [15721.561417] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [15721.570813] RIP: 0010:[] [] osd_scrub_next+0x12/0x3c0 [osd_ldiskfs] [15721.582477] RSP: 0018:ffff880703603cf0 EFLAGS: 00000287 [15721.589427] RAX: ffffffffa1217a20 RBX: 000000009f8f2001 RCX: ffff880703603d98 [15721.598411] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [15721.607379] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [15721.616335] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffff880703603c70 [15721.625273] R13: ffff880703603d08 R14: 0000000000000010 R15: ffffffffa1217d70 [15721.634207] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [15721.644236] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15721.651633] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [15721.660591] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15721.669549] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [15721.678510] Stack: [15721.681723] 0000000000000000 00000000ffffffff 0000000000000001 ffff880703603df0 [15721.691021] ffffffffa121cff9 ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 [15721.700310] 00000000810d354f ffff8803162f6000 ffff8800b2a37468 0000000020000000 [15721.709626] Call Trace: [15721.713358] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [15721.722530] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [15721.731698] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [15721.740574] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [15721.749247] [] ? wake_up_state+0x20/0x20 [15721.756465] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [15721.765822] [] kthread+0xcf/0xe0 [15721.772255] [] ? kthread_create_on_node+0x140/0x140 [15721.780550] [] ret_from_fork+0x58/0x90 [15721.787572] [] ? kthread_create_on_node+0x140/0x140 [15721.795847] Code: 25 fc ff ff 48 83 c4 08 5b 41 5c 41 5d 41 5e 5d c3 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 89 e5 41 57 49 89 f7 41 56 41 55 <49> 89 fd 41 54 49 89 d4 53 48 89 cb 48 83 ec 70 65 48 8b 04 25 [15749.391115] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [15749.400768] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [15749.533019] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [15749.546326] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [15749.562621] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [15749.572015] RIP: 0010:[] [] osd_scrub_exec+0xf7/0x5b0 [osd_ldiskfs] [15749.583685] RSP: 0018:ffff880703603c78 EFLAGS: 00000246 [15749.590633] RAX: 0000000000280001 RBX: ffff880703603c37 RCX: 0000000000000000 [15749.599615] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [15749.608590] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [15749.617554] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [15749.626513] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [15749.635474] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [15749.645513] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15749.652930] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [15749.661906] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15749.670872] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [15749.679833] Stack: [15749.683060] ffff880703603d08 0000000000000001 ffff880703603df0 000000009f8f2001 [15749.692376] ffff880703603d57 000000009f8f2001 0000000000000004 ffff880703603d57 [15749.701682] 0000000020000000 0000000000000000 ffff880703603d78 ffff8800b2a36000 [15749.710978] Call Trace: [15749.714725] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [15749.723903] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [15749.733061] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [15749.741920] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [15749.750592] [] ? wake_up_state+0x20/0x20 [15749.757804] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [15749.767184] [] kthread+0xcf/0xe0 [15749.773627] [] ? kthread_create_on_node+0x140/0x140 [15749.781923] [] ret_from_fork+0x58/0x90 [15749.788939] [] ? kthread_create_on_node+0x140/0x140 [15749.797222] Code: 8b 4d 88 48 85 db 74 23 f6 83 18 0a 00 00 08 74 1a 4d 85 e4 74 15 41 8b 87 6c 14 00 00 41 39 84 24 0c 0a 00 00 0f 82 d9 00 00 00 <31> c0 41 f6 87 74 14 00 00 04 0f 85 6e ff ff ff 41 83 f9 02 0f [15764.032626] INFO: rcu_sched self-detected stall on CPU { 24} (t=8340159 jiffies g=907112 c=907111 q=26089096) [15764.045076] Task dump for CPU 24: [15764.049809] OI_scrub R running task 0 5551 2 0x00000088 [15764.058739] ffff88083fde6dd0 00000000fb6db880 ffff88082da03db0 ffffffff810c47e8 [15764.068061] 0000000000000018 ffffffff81a1e7c0 ffff88082da03dc8 ffffffff810c8079 [15764.077364] 0000000000000009 ffff88082da03df8 ffffffff811378e0 ffff88082da101c0 [15764.086638] Call Trace: [15764.090319] [] sched_show_task+0xa8/0x110 [15764.098303] [] dump_cpu_task+0x39/0x70 [15764.105265] [] rcu_dump_cpu_stacks+0x90/0xd0 [15764.112781] [] rcu_check_callbacks+0x442/0x720 [15764.120472] [] ? tick_sched_handle.isra.13+0x60/0x60 [15764.128726] [] update_process_times+0x47/0x80 [15764.136277] [] tick_sched_handle.isra.13+0x25/0x60 [15764.144295] [] tick_sched_timer+0x41/0x70 [15764.151415] [] __hrtimer_run_queues+0xd2/0x260 [15764.159009] [] hrtimer_interrupt+0xb0/0x1e0 [15764.166293] [] ? call_softirq+0x1c/0x30 [15764.173172] [] local_apic_timer_interrupt+0x37/0x60 [15764.181200] [] smp_apic_timer_interrupt+0x3f/0x60 [15764.189032] [] apic_timer_interrupt+0x6d/0x80 [15764.196468] [] ? osd_inode_iteration+0x47a/0xcc0 [osd_ldiskfs] [15764.206285] [] ? osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [15764.215391] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [15764.224308] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [15764.232931] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [15764.241342] [] ? wake_up_state+0x20/0x20 [15764.248295] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [15764.257385] [] kthread+0xcf/0xe0 [15764.263550] [] ? kthread_create_on_node+0x140/0x140 [15764.271575] [] ret_from_fork+0x58/0x90 [15764.278340] [] ? kthread_create_on_node+0x140/0x140 [15789.392507] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [15789.401894] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [15789.531500] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [15789.544673] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [15789.560744] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [15789.570032] RIP: 0010:[] [] osd_scrub_next+0x45/0x3c0 [osd_ldiskfs] [15789.581606] RSP: 0018:ffff880703603c70 EFLAGS: 00000202 [15789.588463] RAX: 0000000000000000 RBX: ffff880703603c37 RCX: ffff880703603d98 [15789.597373] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [15789.606279] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [15789.615183] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [15789.624091] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [15789.632993] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [15789.642967] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15789.650320] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [15789.659242] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15789.668158] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [15789.677064] Stack: [15789.680240] 000000009f8f2001 ffff880703603d08 0000000000000001 ffff880703603df0 [15789.689489] 000000009f8f2001 ffff880703603d57 000000009f8f2001 0000000000000004 [15789.698731] ffff880703603d57 ffff88042bd78800 ffff88042bd78800 00000000000072cf [15789.707980] Call Trace: [15789.711676] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [15789.720816] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [15789.729936] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [15789.738773] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [15789.747418] [] ? wake_up_state+0x20/0x20 [15789.754618] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [15789.763937] [] kthread+0xcf/0xe0 [15789.770340] [] ? kthread_create_on_node+0x140/0x140 [15789.778586] [] ret_from_fork+0x58/0x90 [15789.785574] [] ? kthread_create_on_node+0x140/0x140 [15789.793824] Code: 53 48 89 cb 48 83 ec 70 65 48 8b 04 25 28 00 00 00 48 89 45 d0 31 c0 48 8b 05 f8 59 9c ff 48 85 c0 75 7b 41 f6 87 58 10 00 00 08 <0f> 84 62 03 00 00 4d 8d b7 e0 10 00 00 4d 3b b7 e0 10 00 00 0f [15817.393478] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [15817.403102] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [15817.534833] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [15817.548122] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [15817.564380] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [15817.573760] RIP: 0010:[] [] osd_inode_iteration+0x489/0xcc0 [osd_ldiskfs] [15817.585989] RSP: 0018:ffff880703603d18 EFLAGS: 00000293 [15817.592923] RAX: 0000000000000004 RBX: 000000009f8f2001 RCX: 0000000000000000 [15817.601902] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [15817.610871] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [15817.619827] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [15817.628774] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [15817.637723] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [15817.647756] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15817.655159] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [15817.664125] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15817.673075] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [15817.682013] Stack: [15817.685212] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [15817.694486] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [15817.703798] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [15817.713085] Call Trace: [15817.716795] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [15817.725957] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [15817.734827] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [15817.743500] [] ? wake_up_state+0x20/0x20 [15817.750704] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [15817.760066] [] kthread+0xcf/0xe0 [15817.766496] [] ? kthread_create_on_node+0x140/0x140 [15817.774782] [] ret_from_fork+0x58/0x90 [15817.781793] [] ? kthread_create_on_node+0x140/0x140 [15817.790063] Code: 00 e8 7c eb 97 ff e9 0f fc ff ff 0f 1f 80 00 00 00 00 45 89 e9 4c 8d 85 67 ff ff ff 48 8b 4d a8 48 8d 55 88 48 8b b5 38 ff ff ff <48> 8b bd 48 ff ff ff 48 8b 85 28 ff ff ff ff d0 85 c0 41 89 c5 [15845.394461] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [15845.404130] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [15845.536604] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [15845.549924] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [15845.566223] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [15845.575626] RIP: 0010:[] [] osd_inode_iteration+0x2be/0xcc0 [osd_ldiskfs] [15845.587903] RSP: 0018:ffff880703603d18 EFLAGS: 00000297 [15845.594870] RAX: 0000000000000004 RBX: 0000000000000000 RCX: ffff880703603d98 [15845.603863] RDX: ffff880703603d78 RSI: 0000000000000000 RDI: ffff8803162f6000 [15845.612846] RBP: ffff880703603df0 R08: 0000000000000001 R09: 0000000000000004 [15845.621812] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffff880703603d57 [15845.630769] R13: 0000000000000004 R14: 000000009f8f2001 R15: ffffea00027e3c80 [15845.639719] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [15845.649742] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15845.657145] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [15845.666107] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15845.675073] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [15845.684020] Stack: [15845.687238] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [15845.696524] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [15845.705807] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [15845.715112] Call Trace: [15845.718839] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [15845.727998] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [15845.736864] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [15845.745543] [] ? wake_up_state+0x20/0x20 [15845.752756] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [15845.762117] [] kthread+0xcf/0xe0 [15845.768552] [] ? kthread_create_on_node+0x140/0x140 [15845.776836] [] ret_from_fork+0x58/0x90 [15845.783848] [] ? kthread_create_on_node+0x140/0x140 [15845.792119] Code: 8b b5 38 ff ff ff 48 8b bd 48 ff ff ff 48 8b 85 30 ff ff ff ff d0 83 f8 03 41 89 c5 0f 84 4b 02 00 00 0f 8e 15 01 00 00 83 f8 05 <66> 90 0f 84 8a 02 00 00 83 f8 06 0f 85 a1 01 00 00 48 8b 7d 90 [15873.395434] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [15873.405058] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [15873.536902] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [15873.550202] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [15873.566491] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [15873.575886] RIP: 0010:[] [] osd_scrub_exec+0x1c/0x5b0 [osd_ldiskfs] [15873.587553] RSP: 0018:ffff880703603c78 EFLAGS: 00000296 [15873.594520] RAX: ffffffffa121b990 RBX: ffff880703603c37 RCX: 0000000000000000 [15873.603513] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [15873.612499] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [15873.621475] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [15873.630425] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [15873.639374] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [15873.649395] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15873.656805] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [15873.665779] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15873.674744] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [15873.683696] Stack: [15873.686916] ffff880703603d08 0000000000000001 ffff880703603df0 000000009f8f2001 [15873.696206] ffff880703603d57 000000009f8f2001 0000000000000004 ffff880703603d57 [15873.705486] ffff88042bd78800 ffff88042bd78800 00000000000072cf ffff880419dc3a00 [15873.714781] Call Trace: [15873.718524] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [15873.727692] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [15873.736869] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [15873.745743] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [15873.754410] [] ? wake_up_state+0x20/0x20 [15873.761617] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [15873.770961] [] kthread+0xcf/0xe0 [15873.777388] [] ? kthread_create_on_node+0x140/0x140 [15873.785663] [] ret_from_fork+0x58/0x90 [15873.792679] [] ? kthread_create_on_node+0x140/0x140 [15873.800948] Code: 89 04 24 31 c0 e8 a5 03 98 ff e9 c7 fd ff ff 66 66 66 66 90 55 48 89 e5 41 57 49 89 f7 41 56 41 55 41 54 45 31 e4 53 48 83 ec 68 <48> 8b 9e 08 10 00 00 4c 89 45 90 65 48 8b 04 25 28 00 00 00 48 [15901.396410] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [15901.406036] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [15901.537684] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [15901.550981] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [15901.567232] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [15901.576611] RIP: 0010:[] [] osd_scrub_exec+0xe9/0x5b0 [osd_ldiskfs] [15901.588249] RSP: 0018:ffff880703603c78 EFLAGS: 00000282 [15901.595189] RAX: 0000000000280001 RBX: ffff880703603c37 RCX: 0000000000000000 [15901.604167] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [15901.613135] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [15901.622089] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [15901.631040] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [15901.639992] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [15901.650025] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15901.657433] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [15901.666400] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15901.675359] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [15901.684303] Stack: [15901.687515] ffff880703603d08 0000000000000001 ffffea00027e3c80 000000009f8f2001 [15901.696795] ffff880703603d57 0000000000000001 ffffffffa1217a20 ffff880703603d98 [15901.706090] ffff880703603d78 ffff8800b2a36000 ffff8803162f6000 ffffffffffffff10 [15901.715380] Call Trace: [15901.719087] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [15901.727959] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [15901.737126] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [15901.746282] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [15901.755147] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [15901.763817] [] ? wake_up_state+0x20/0x20 [15901.771024] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [15901.780368] [] kthread+0xcf/0xe0 [15901.786799] [] ? kthread_create_on_node+0x140/0x140 [15901.795079] [] ret_from_fork+0x58/0x90 [15901.802078] [] ? kthread_create_on_node+0x140/0x140 [15901.810347] Code: 88 11 00 00 01 4c 89 ef e8 f5 9f e9 df 44 8b 4d 88 48 85 db 74 23 f6 83 18 0a 00 00 08 74 1a 4d 85 e4 74 15 41 8b 87 6c 14 00 00 <41> 39 84 24 0c 0a 00 00 0f 82 d9 00 00 00 31 c0 41 f6 87 74 14 [15929.397381] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [15929.407029] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [15929.538642] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [15929.551905] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [15929.568111] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [15929.577472] RIP: 0010:[] [] osd_inode_iteration+0x28a/0xcc0 [osd_ldiskfs] [15929.589677] RSP: 0018:ffff880703603d18 EFLAGS: 00000287 [15929.596611] RAX: ffff88042bd78800 RBX: 000000009f8f2001 RCX: ffff88042bd78800 [15929.605573] RDX: 00000000000072cf RSI: ffff880419dc3a00 RDI: ffff88042bd79800 [15929.614523] RBP: ffff880703603df0 R08: 0000000000000001 R09: 0000000000000004 [15929.623473] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffff880703603d57 [15929.632439] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [15929.641396] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [15929.651426] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15929.658826] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [15929.667800] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15929.676753] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [15929.685695] Stack: [15929.688915] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [15929.698225] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [15929.707504] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [15929.716793] Call Trace: [15929.720514] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [15929.729683] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [15929.738554] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [15929.747227] [] ? wake_up_state+0x20/0x20 [15929.754439] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [15929.763802] [] kthread+0xcf/0xe0 [15929.770241] [] ? kthread_create_on_node+0x140/0x140 [15929.778535] [] ret_from_fork+0x58/0x90 [15929.785554] [] ? kthread_create_on_node+0x140/0x140 [15929.793834] Code: a8 00 00 00 00 e8 77 5b f4 ff 42 8d 14 38 48 8b 45 88 48 8b 80 a8 03 00 00 48 3b 50 20 0f 83 57 01 00 00 44 0f b6 85 67 ff ff ff <48> 8d 4d a8 48 8d 55 88 48 8b b5 38 ff ff ff 48 8b bd 48 ff ff [15944.041896] INFO: rcu_sched self-detected stall on CPU { 24} (t=8520162 jiffies g=907112 c=907111 q=26370838) [15944.054361] Task dump for CPU 24: [15944.059123] OI_scrub R running task 0 5551 2 0x00000088 [15944.068091] ffff88083fde6dd0 00000000fb6db880 ffff88082da03db0 ffffffff810c47e8 [15944.077436] 0000000000000018 ffffffff81a1e7c0 ffff88082da03dc8 ffffffff810c8079 [15944.086756] 0000000000000009 ffff88082da03df8 ffffffff811378e0 ffff88082da101c0 [15944.096058] Call Trace: [15944.099772] [] sched_show_task+0xa8/0x110 [15944.107806] [] dump_cpu_task+0x39/0x70 [15944.114804] [] rcu_dump_cpu_stacks+0x90/0xd0 [15944.122367] [] rcu_check_callbacks+0x442/0x720 [15944.130087] [] ? tick_sched_handle.isra.13+0x60/0x60 [15944.138390] [] update_process_times+0x47/0x80 [15944.145971] [] tick_sched_handle.isra.13+0x25/0x60 [15944.154011] [] tick_sched_timer+0x41/0x70 [15944.161177] [] __hrtimer_run_queues+0xd2/0x260 [15944.168793] [] hrtimer_interrupt+0xb0/0x1e0 [15944.176102] [] ? call_softirq+0x1c/0x30 [15944.183002] [] local_apic_timer_interrupt+0x37/0x60 [15944.191063] [] smp_apic_timer_interrupt+0x3f/0x60 [15944.198907] [] apic_timer_interrupt+0x6d/0x80 [15944.206349] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [15944.215700] [] ? osd_scrub_next+0x1b/0x3c0 [osd_ldiskfs] [15944.224227] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [15944.233141] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [15944.242047] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [15944.250661] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [15944.259077] [] ? wake_up_state+0x20/0x20 [15944.266027] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [15944.275131] [] kthread+0xcf/0xe0 [15944.281303] [] ? kthread_create_on_node+0x140/0x140 [15944.289337] [] ret_from_fork+0x58/0x90 [15944.296102] [] ? kthread_create_on_node+0x140/0x140 [15969.398773] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [15969.408190] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [15969.537894] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [15969.551065] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [15969.567103] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [15969.576384] RIP: 0010:[] [] osd_inode_iteration+0x4a6/0xcc0 [osd_ldiskfs] [15969.588518] RSP: 0018:ffff880703603d18 EFLAGS: 00000246 [15969.595374] RAX: 0000000020000000 RBX: 000000009f8f2001 RCX: 0000000000000000 [15969.604276] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [15969.613172] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [15969.622069] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [15969.630965] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [15969.639862] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [15969.649829] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15969.657178] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [15969.666091] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15969.675002] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [15969.683904] Stack: [15969.687069] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [15969.696319] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [15969.705575] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [15969.714838] Call Trace: [15969.718534] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [15969.727675] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [15969.736518] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [15969.745172] [] ? wake_up_state+0x20/0x20 [15969.752355] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [15969.761670] [] kthread+0xcf/0xe0 [15969.768075] [] ? kthread_create_on_node+0x140/0x140 [15969.776330] [] ret_from_fork+0x58/0x90 [15969.783312] [] ? kthread_create_on_node+0x140/0x140 [15969.791563] Code: 8b 4d a8 48 8d 55 88 48 8b b5 38 ff ff ff 48 8b bd 48 ff ff ff 48 8b 85 28 ff ff ff ff d0 85 c0 41 89 c5 75 1c 8b 85 58 ff ff ff <41> 3b 04 24 72 10 48 8b 85 50 ff ff ff 44 3b 30 0f 87 95 fd ff [15997.399754] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [15997.409367] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [15997.541765] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [15997.555079] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [15997.571374] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [15997.580783] RIP: 0010:[] [] osd_inode_iteration+0x299/0xcc0 [osd_ldiskfs] [15997.593042] RSP: 0018:ffff880703603d18 EFLAGS: 00000287 [15997.600009] RAX: ffff88042bd78800 RBX: 000000009f8f2001 RCX: ffff880703603d98 [15997.609005] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff88042bd79800 [15997.617998] RBP: ffff880703603df0 R08: 0000000000000001 R09: 0000000000000004 [15997.626968] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000001 [15997.635926] R13: ffff880703603d08 R14: 000000009f8f2001 R15: ffffea00027e3c80 [15997.644889] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [15997.654924] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15997.662337] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [15997.671309] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [15997.680283] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [15997.689237] Stack: [15997.692462] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [15997.701753] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [15997.711044] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [15997.720351] Call Trace: [15997.724094] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [15997.733266] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [15997.742149] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [15997.750843] [] ? wake_up_state+0x20/0x20 [15997.758062] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [15997.767432] [] kthread+0xcf/0xe0 [15997.773877] [] ? kthread_create_on_node+0x140/0x140 [15997.782169] [] ret_from_fork+0x58/0x90 [15997.789189] [] ? kthread_create_on_node+0x140/0x140 [15997.797475] Code: 8b 45 88 48 8b 80 a8 03 00 00 48 3b 50 20 0f 83 57 01 00 00 44 0f b6 85 67 ff ff ff 48 8d 4d a8 48 8d 55 88 48 8b b5 38 ff ff ff <48> 8b bd 48 ff ff ff 48 8b 85 30 ff ff ff ff d0 83 f8 03 41 89 [16025.400730] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [16025.410363] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [16025.542469] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [16025.555785] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [16025.572095] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [16025.581509] RIP: 0010:[] [] osd_inode_iteration+0x2ac/0xcc0 [osd_ldiskfs] [16025.593767] RSP: 0018:ffff880703603d18 EFLAGS: 00000202 [16025.600747] RAX: 0000000000000004 RBX: 000000009f8f2001 RCX: ffff880703603d98 [16025.609747] RDX: ffff880703603d78 RSI: 0000000000000000 RDI: ffff8803162f6000 [16025.618743] RBP: ffff880703603df0 R08: 0000000000000001 R09: 0000000000000004 [16025.627712] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000001 [16025.636669] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [16025.645619] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [16025.655646] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16025.663047] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [16025.672014] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16025.680978] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [16025.689927] Stack: [16025.693134] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [16025.702426] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [16025.711714] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [16025.721010] Call Trace: [16025.724723] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [16025.733883] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [16025.742755] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [16025.751437] [] ? wake_up_state+0x20/0x20 [16025.758642] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [16025.767999] [] kthread+0xcf/0xe0 [16025.774429] [] ? kthread_create_on_node+0x140/0x140 [16025.782696] [] ret_from_fork+0x58/0x90 [16025.789714] [] ? kthread_create_on_node+0x140/0x140 [16025.797985] Code: 00 44 0f b6 85 67 ff ff ff 48 8d 4d a8 48 8d 55 88 48 8b b5 38 ff ff ff 48 8b bd 48 ff ff ff 48 8b 85 30 ff ff ff ff d0 83 f8 03 <41> 89 c5 0f 84 4b 02 00 00 0f 8e 15 01 00 00 83 f8 05 66 90 0f [16053.401699] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [16053.411337] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [16053.543296] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [16053.556597] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [16053.572890] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [16053.582297] RIP: 0010:[] [] osd_scrub_next+0x149/0x3c0 [osd_ldiskfs] [16053.594067] RSP: 0018:ffff880703603c70 EFLAGS: 00000202 [16053.601028] RAX: 0000000000000004 RBX: ffff880703603c37 RCX: ffff880703603d98 [16053.610016] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [16053.618995] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [16053.627956] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [16053.636910] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [16053.645860] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [16053.655886] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16053.663298] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [16053.672263] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16053.681233] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [16053.690189] Stack: [16053.693408] 000000009f8f2001 0000000000000004 ffff880703603d57 ffff880703603df0 [16053.702705] 0000000000000000 ffff880703603d57 000000009f8f2001 0000000000000004 [16053.711993] ffff880703603d57 0000000000000000 0000000000000000 ffff880703603d78 [16053.721295] Call Trace: [16053.725039] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [16053.734216] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [16053.743390] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [16053.752271] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [16053.760950] [] ? wake_up_state+0x20/0x20 [16053.768168] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [16053.777529] [] kthread+0xcf/0xe0 [16053.783959] [] ? kthread_create_on_node+0x140/0x140 [16053.792251] [] ret_from_fork+0x58/0x90 [16053.799272] [] ? kthread_create_on_node+0x140/0x140 [16053.807556] Code: 45 90 74 49 49 8d 9f 10 11 00 00 48 89 df e8 df 6a 47 e0 41 c7 87 58 10 00 00 02 00 00 00 48 89 df e8 fc 69 47 e0 b8 05 00 00 00 <48> 8b 75 d0 65 48 33 34 25 28 00 00 00 0f 85 4c 02 00 00 48 83 [16078.645768] Lustre: soaked-MDT0000: Client e211cd09-6b0c-e43b-059c-f657101042a0 (at 192.168.1.117@o2ib100) reconnecting [16078.660509] Lustre: soaked-MDT0000: Connection restored to e211cd09-6b0c-e43b-059c-f657101042a0 (at 192.168.1.117@o2ib100) [16081.402676] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [16081.412372] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [16081.544210] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [16081.557492] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [16081.573751] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [16081.583139] RIP: 0010:[] [] ldiskfs_itable_unused_count+0x16/0x30 [ldiskfs] [16081.595579] RSP: 0018:ffff880703603d08 EFLAGS: 00000246 [16081.602542] RAX: 0000000000000000 RBX: 000000009f8f2001 RCX: ffff88042bd78800 [16081.611543] RDX: 00000000000072ce RSI: ffff880419dc3a00 RDI: ffff88042bd79800 [16081.620540] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [16081.629532] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [16081.638530] R13: 0000000000000018 R14: ffff880703603c78 R15: 0000000000000246 [16081.647523] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [16081.657582] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16081.665014] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [16081.674014] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16081.682993] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [16081.691966] Stack: [16081.695225] ffff880703603df0 ffffffffa121cfb9 ffffffffa121b990 ffffffffa1217a20 [16081.704545] ffff8800b2a36000 00000000810d354f ffff8803162f6000 ffff8800b2a37468 [16081.713867] 0000000020000000 010000000000000c 0000000000000000 0000000000000000 [16081.723171] Call Trace: [16081.726926] [] osd_inode_iteration+0x269/0xcc0 [osd_ldiskfs] [16081.736115] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [16081.745308] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [16081.754199] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [16081.762893] [] ? wake_up_state+0x20/0x20 [16081.770136] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [16081.779505] [] kthread+0xcf/0xe0 [16081.785963] [] ? kthread_create_on_node+0x140/0x140 [16081.794267] [] ret_from_fork+0x58/0x90 [16081.801305] [] ? kthread_create_on_node+0x140/0x140 [16081.809612] Code: 07 0f b7 46 30 c1 e0 10 09 d0 5d c3 66 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 8b 8f a8 03 00 00 31 c0 0f b7 56 1c 48 89 e5 <48> 83 39 3f 76 07 0f b7 46 32 c1 e0 10 09 d0 5d c3 66 0f 1f 84 [16109.403650] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [16109.413366] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [16109.545151] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [16109.558432] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [16109.574685] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [16109.584061] RIP: 0010:[] [] osd_inode_iteration+0x251/0xcc0 [osd_ldiskfs] [16109.596289] RSP: 0018:ffff880703603d18 EFLAGS: 00000286 [16109.603215] RAX: ffff8800b2a37468 RBX: 0000000000000000 RCX: 0000000000000000 [16109.612189] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [16109.621147] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [16109.630090] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffff880703603d57 [16109.639038] R13: 0000000000000004 R14: 000000009f8f2001 R15: 0000000000000018 [16109.647990] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [16109.658014] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16109.665410] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [16109.674376] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16109.683331] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [16109.692272] Stack: [16109.695499] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [16109.704787] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [16109.714059] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [16109.723367] Call Trace: [16109.727082] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [16109.736251] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [16109.745127] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [16109.753793] [] ? wake_up_state+0x20/0x20 [16109.760996] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [16109.770366] [] kthread+0xcf/0xe0 [16109.776808] [] ? kthread_create_on_node+0x140/0x140 [16109.785084] [] ret_from_fork+0x58/0x90 [16109.792102] [] ? kthread_create_on_node+0x140/0x140 [16109.800373] Code: b0 00 00 00 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 00 8b 75 98 48 8b 7d 88 e8 dc 2d f5 ff 48 85 c0 48 89 45 90 0f 84 b3 08 00 00 <44> 8b 7d a0 48 8b 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 e8 77 [16124.051161] INFO: rcu_sched self-detected stall on CPU { 24} (t=8700165 jiffies g=907112 c=907111 q=26715744) [16124.063636] Task dump for CPU 24: [16124.068400] OI_scrub R running task 0 5551 2 0x00000088 [16124.077348] ffff88083fde6dd0 00000000fb6db880 ffff88082da03db0 ffffffff810c47e8 [16124.086690] 0000000000000018 ffffffff81a1e7c0 ffff88082da03dc8 ffffffff810c8079 [16124.096027] 0000000000000009 ffff88082da03df8 ffffffff811378e0 ffff88082da101c0 [16124.105316] Call Trace: [16124.109024] [] sched_show_task+0xa8/0x110 [16124.117022] [] dump_cpu_task+0x39/0x70 [16124.124001] [] rcu_dump_cpu_stacks+0x90/0xd0 [16124.131543] [] rcu_check_callbacks+0x442/0x720 [16124.139262] [] ? tick_sched_handle.isra.13+0x60/0x60 [16124.147546] [] update_process_times+0x47/0x80 [16124.155126] [] tick_sched_handle.isra.13+0x25/0x60 [16124.163162] [] tick_sched_timer+0x41/0x70 [16124.170311] [] __hrtimer_run_queues+0xd2/0x260 [16124.177916] [] hrtimer_interrupt+0xb0/0x1e0 [16124.185216] [] ? call_softirq+0x1c/0x30 [16124.192097] [] local_apic_timer_interrupt+0x37/0x60 [16124.200145] [] smp_apic_timer_interrupt+0x3f/0x60 [16124.207971] [] apic_timer_interrupt+0x6d/0x80 [16124.215394] [] ? osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [16124.225205] [] ? osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [16124.234300] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [16124.243205] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [16124.251816] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [16124.260227] [] ? wake_up_state+0x20/0x20 [16124.267182] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [16124.276276] [] kthread+0xcf/0xe0 [16124.282445] [] ? kthread_create_on_node+0x140/0x140 [16124.290464] [] ret_from_fork+0x58/0x90 [16124.297224] [] ? kthread_create_on_node+0x140/0x140 [16149.405043] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [16149.414385] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [16149.544062] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [16149.557241] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [16149.573310] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [16149.582605] RIP: 0010:[] [] osd_inode_iteration+0x282/0xcc0 [osd_ldiskfs] [16149.594766] RSP: 0018:ffff880703603d18 EFLAGS: 00000287 [16149.601636] RAX: ffff88042bd78800 RBX: 000000009f8f2001 RCX: ffff88042bd78800 [16149.610551] RDX: 00000000000072cf RSI: ffff880419dc3a00 RDI: ffff88042bd79800 [16149.619450] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [16149.628360] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000001 [16149.637263] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [16149.646157] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [16149.656122] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16149.663477] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [16149.672391] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16149.681304] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [16149.690204] Stack: [16149.693379] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [16149.702644] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [16149.711904] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [16149.721162] Call Trace: [16149.724859] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [16149.734002] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [16149.742847] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [16149.751510] [] ? wake_up_state+0x20/0x20 [16149.758710] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [16149.768044] [] kthread+0xcf/0xe0 [16149.774456] [] ? kthread_create_on_node+0x140/0x140 [16149.782725] [] ret_from_fork+0x58/0x90 [16149.789711] [] ? kthread_create_on_node+0x140/0x140 [16149.797957] Code: 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 e8 77 5b f4 ff 42 8d 14 38 48 8b 45 88 48 8b 80 a8 03 00 00 48 3b 50 20 0f 83 57 01 00 00 <44> 0f b6 85 67 ff ff ff 48 8d 4d a8 48 8d 55 88 48 8b b5 38 ff [16177.406024] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [16177.415667] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [16177.547621] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [16177.560949] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [16177.577247] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [16177.586659] RIP: 0010:[] [] osd_scrub_exec+0xd4/0x5b0 [osd_ldiskfs] [16177.598340] RSP: 0018:ffff880703603c78 EFLAGS: 00000286 [16177.605302] RAX: ffff8803162f1008 RBX: ffff880703603c37 RCX: 0000000000000000 [16177.614289] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [16177.623267] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [16177.632229] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [16177.641181] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [16177.650140] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [16177.660173] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16177.667582] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [16177.676554] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16177.685514] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [16177.694462] Stack: [16177.697689] ffff880703603d08 ffff880703603d57 ffff880703603df0 000000009f8f2001 [16177.706978] ffff880703603d57 000000009f8f2001 0000000000000004 ffff880703603d57 [16177.716272] ffff88042bd78800 ffff88042bd78800 00000000000072cf ffff880419dc3a00 [16177.725583] Call Trace: [16177.729306] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [16177.738472] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [16177.747625] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [16177.756495] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [16177.765169] [] ? wake_up_state+0x20/0x20 [16177.772380] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [16177.781737] [] kthread+0xcf/0xe0 [16177.788169] [] ? kthread_create_on_node+0x140/0x140 [16177.796452] [] ret_from_fork+0x58/0x90 [16177.803460] [] ? kthread_create_on_node+0x140/0x140 [16177.811734] Code: 4d 88 4c 89 ef e8 1d fb 46 e0 41 83 87 68 14 00 00 01 49 83 87 88 11 00 00 01 4c 89 ef e8 f5 9f e9 df 44 8b 4d 88 48 85 db 74 23 83 18 0a 00 00 08 74 1a 4d 85 e4 74 15 41 8b 87 6c 14 00 00 [16205.407000] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [16205.416638] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [16205.548557] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [16205.561843] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [16205.578099] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [16205.587495] RIP: 0010:[] [] osd_scrub_next+0x9/0x3c0 [osd_ldiskfs] [16205.599058] RSP: 0018:ffff880703603d08 EFLAGS: 00000287 [16205.606004] RAX: ffffffffa1217a20 RBX: ffff880703603d57 RCX: ffff880703603d98 [16205.614986] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [16205.623948] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [16205.632897] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [16205.641848] R13: ffffea00027e3c80 R14: ffff880703603c78 R15: ffff880703603d08 [16205.650803] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [16205.660835] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16205.668239] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [16205.677209] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16205.686158] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [16205.695108] Stack: [16205.698311] ffff880703603df0 ffffffffa121cff9 ffffffffa121b990 ffffffffa1217a20 [16205.707589] ffff8800b2a36000 00000000810d354f ffff8803162f6000 ffff8800b2a37468 [16205.716891] 0000000020000000 010000000000000c 0000000000000000 0000000000000000 [16205.726183] Call Trace: [16205.729918] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [16205.739095] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [16205.748253] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [16205.757106] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [16205.765774] [] ? wake_up_state+0x20/0x20 [16205.772986] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [16205.782349] [] kthread+0xcf/0xe0 [16205.788785] [] ? kthread_create_on_node+0x140/0x140 [16205.797068] [] ret_from_fork+0x58/0x90 [16205.804083] [] ? kthread_create_on_node+0x140/0x140 [16205.812356] Code: 48 8d 4c 02 10 48 01 c2 e8 25 fc ff ff 48 83 c4 08 5b 41 5c 41 5d 41 5e 5d c3 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 89 e5 <41> 57 49 89 f7 41 56 41 55 49 89 fd 41 54 49 89 d4 53 48 89 cb [16233.407970] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [16233.417671] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [16233.549536] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [16233.562822] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [16233.579073] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [16233.588456] RIP: 0010:[] [] osd_inode_iteration+0x28a/0xcc0 [osd_ldiskfs] [16233.600689] RSP: 0018:ffff880703603d18 EFLAGS: 00000287 [16233.607635] RAX: ffff88042bd78800 RBX: 000000009f8f2001 RCX: ffff88042bd78800 [16233.616613] RDX: 00000000000072cf RSI: ffff880419dc3a00 RDI: ffff88042bd79800 [16233.625579] RBP: ffff880703603df0 R08: 0000000000000001 R09: 0000000000000004 [16233.634538] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000001 [16233.643492] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [16233.652451] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [16233.662481] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16233.669898] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [16233.678874] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16233.687829] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [16233.696774] Stack: [16233.700001] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [16233.709292] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [16233.718597] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [16233.727874] Call Trace: [16233.731602] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [16233.740759] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [16233.749632] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [16233.758317] [] ? wake_up_state+0x20/0x20 [16233.765523] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [16233.774899] [] kthread+0xcf/0xe0 [16233.781334] [] ? kthread_create_on_node+0x140/0x140 [16233.789624] [] ret_from_fork+0x58/0x90 [16233.796637] [] ? kthread_create_on_node+0x140/0x140 [16233.804919] Code: a8 00 00 00 00 e8 77 5b f4 ff 42 8d 14 38 48 8b 45 88 48 8b 80 a8 03 00 00 48 3b 50 20 0f 83 57 01 00 00 44 0f b6 85 67 ff ff ff <48> 8d 4d a8 48 8d 55 88 48 8b b5 38 ff ff ff 48 8b bd 48 ff ff [16261.408947] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [16261.418621] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [16261.550701] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [16261.564014] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [16261.580316] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [16261.589732] RIP: 0010:[] [] osd_inode_iteration+0x264/0xcc0 [osd_ldiskfs] [16261.601993] RSP: 0018:ffff880703603d18 EFLAGS: 00000286 [16261.608965] RAX: ffff8800b2a37468 RBX: 000000009f8f2001 RCX: 0000000000000000 [16261.617959] RDX: ffff880703603d78 RSI: ffff880419dc3a00 RDI: ffff88042bd79800 [16261.626944] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [16261.635914] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000001 [16261.644869] R13: ffff880703603d08 R14: 000000009f8f2001 R15: ffffea00027e3c80 [16261.653811] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [16261.663840] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16261.671247] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [16261.680215] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16261.689181] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [16261.698138] Stack: [16261.701353] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [16261.710641] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [16261.719929] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [16261.729242] Call Trace: [16261.732970] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [16261.742128] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [16261.751004] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [16261.759696] [] ? wake_up_state+0x20/0x20 [16261.766918] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [16261.776267] [] kthread+0xcf/0xe0 [16261.782707] [] ? kthread_create_on_node+0x140/0x140 [16261.790989] [] ret_from_fork+0x58/0x90 [16261.798002] [] ? kthread_create_on_node+0x140/0x140 [16261.806283] Code: 75 98 48 8b 7d 88 e8 dc 2d f5 ff 48 85 c0 48 89 45 90 0f 84 b3 08 00 00 44 8b 7d a0 48 8b 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 77 5b f4 ff 42 8d 14 38 48 8b 45 88 48 8b 80 a8 03 00 00 48 [16289.409920] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [16289.419608] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [16289.551684] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [16289.565003] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [16289.581318] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [16289.590728] RIP: 0010:[] [] osd_scrub_exec+0x9/0x5b0 [osd_ldiskfs] [16289.602313] RSP: 0018:ffff880703603d08 EFLAGS: 00000293 [16289.609274] RAX: ffffffffa121b990 RBX: ffff880703603d57 RCX: 0000000000000000 [16289.618261] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [16289.627238] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [16289.636198] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [16289.645149] R13: 0000000000000018 R14: ffff880703603c78 R15: 0000000000000296 [16289.654098] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [16289.664114] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16289.671517] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [16289.680476] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16289.689429] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [16289.698368] Stack: [16289.701585] ffff880703603df0 ffffffffa121d1e9 ffffffffa121b990 ffffffffa1217a20 [16289.710869] ffff8800b2a36000 00000000810d354f ffff8803162f6000 ffff8800b2a37468 [16289.720162] 0000000020000000 010000000000000c 0000000000000000 0000000000000000 [16289.729436] Call Trace: [16289.733160] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [16289.742320] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [16289.751472] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [16289.760342] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [16289.769011] [] ? wake_up_state+0x20/0x20 [16289.776207] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [16289.785552] [] kthread+0xcf/0xe0 [16289.791970] [] ? kthread_create_on_node+0x140/0x140 [16289.800253] [] ret_from_fork+0x58/0x90 [16289.807255] [] ? kthread_create_on_node+0x140/0x140 [16289.815520] Code: 00 00 00 02 00 89 54 24 08 48 8b 40 40 ba 07 00 00 00 48 89 04 24 31 c0 e8 a5 03 98 ff e9 c7 fd ff ff 66 66 66 66 90 55 48 89 e5 <41> 57 49 89 f7 41 56 41 55 41 54 45 31 e4 53 48 83 ec 68 48 8b [16304.060431] INFO: rcu_sched self-detected stall on CPU { 24} (t=8880168 jiffies g=907112 c=907111 q=27253292) [16304.072887] Task dump for CPU 24: [16304.077600] OI_scrub R running task 0 5551 2 0x00000088 [16304.086510] ffff88083fde6dd0 00000000fb6db880 ffff88082da03db0 ffffffff810c47e8 [16304.095835] 0000000000000018 ffffffff81a1e7c0 ffff88082da03dc8 ffffffff810c8079 [16304.105127] 0000000000000009 ffff88082da03df8 ffffffff811378e0 ffff88082da101c0 [16304.114399] Call Trace: [16304.118071] [] sched_show_task+0xa8/0x110 [16304.126042] [] dump_cpu_task+0x39/0x70 [16304.133001] [] rcu_dump_cpu_stacks+0x90/0xd0 [16304.140508] [] rcu_check_callbacks+0x442/0x720 [16304.148189] [] ? tick_sched_handle.isra.13+0x60/0x60 [16304.156430] [] update_process_times+0x47/0x80 [16304.163968] [] tick_sched_handle.isra.13+0x25/0x60 [16304.171977] [] tick_sched_timer+0x41/0x70 [16304.179086] [] __hrtimer_run_queues+0xd2/0x260 [16304.186661] [] hrtimer_interrupt+0xb0/0x1e0 [16304.193928] [] ? call_softirq+0x1c/0x30 [16304.200798] [] local_apic_timer_interrupt+0x37/0x60 [16304.208813] [] smp_apic_timer_interrupt+0x3f/0x60 [16304.216623] [] apic_timer_interrupt+0x6d/0x80 [16304.224047] [] ? osd_scrub_next+0x45/0x3c0 [osd_ldiskfs] [16304.233281] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [16304.242175] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [16304.251067] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [16304.259672] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [16304.268082] [] ? wake_up_state+0x20/0x20 [16304.275030] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [16304.284110] [] kthread+0xcf/0xe0 [16304.290280] [] ? kthread_create_on_node+0x140/0x140 [16304.298296] [] ret_from_fork+0x58/0x90 [16304.305044] [] ? kthread_create_on_node+0x140/0x140 [16329.411321] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [16329.420737] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [16329.550475] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [16329.563643] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [16329.579713] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [16329.589001] RIP: 0010:[] [] osd_inode_iteration+0x264/0xcc0 [osd_ldiskfs] [16329.601145] RSP: 0018:ffff880703603d18 EFLAGS: 00000286 [16329.608006] RAX: ffff8800b2a37468 RBX: 0000000000000001 RCX: 0000000000000000 [16329.616912] RDX: ffff880703603d78 RSI: ffff880419dc3a00 RDI: ffff88042bd79800 [16329.625806] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [16329.634706] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [16329.643614] R13: ffffea00027e3c80 R14: 000000009f8f2001 R15: ffff880703603d08 [16329.652510] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [16329.662474] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16329.669818] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [16329.678719] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16329.687621] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [16329.696530] Stack: [16329.699699] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [16329.708934] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [16329.718174] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [16329.727414] Call Trace: [16329.731099] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [16329.740233] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [16329.749071] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [16329.757724] [] ? wake_up_state+0x20/0x20 [16329.764920] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [16329.774277] [] kthread+0xcf/0xe0 [16329.780683] [] ? kthread_create_on_node+0x140/0x140 [16329.788955] [] ret_from_fork+0x58/0x90 [16329.795944] [] ? kthread_create_on_node+0x140/0x140 [16329.804202] Code: 75 98 48 8b 7d 88 e8 dc 2d f5 ff 48 85 c0 48 89 45 90 0f 84 b3 08 00 00 44 8b 7d a0 48 8b 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 77 5b f4 ff 42 8d 14 38 48 8b 45 88 48 8b 80 a8 03 00 00 48 [16357.412289] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [16357.421950] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [16357.553926] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [16357.567258] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [16357.583562] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [16357.592969] RIP: 0010:[] [] osd_inode_iteration+0x473/0xcc0 [osd_ldiskfs] [16357.605222] RSP: 0018:ffff880703603d18 EFLAGS: 00000293 [16357.612197] RAX: 0000000000000004 RBX: 000000009f8f2001 RCX: ffff880703603d98 [16357.621191] RDX: ffff880703603d78 RSI: 0000000000000000 RDI: ffff8803162f6000 [16357.630173] RBP: ffff880703603df0 R08: 0000000000000001 R09: 0000000000000004 [16357.639153] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffff880703603d57 [16357.648116] R13: ffff880703603d08 R14: 000000009f8f2001 R15: ffffea00027e3c80 [16357.657076] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [16357.667104] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16357.674515] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [16357.683491] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16357.692456] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [16357.701403] Stack: [16357.704632] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [16357.713921] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [16357.723224] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [16357.732534] Call Trace: [16357.736258] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [16357.745431] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [16357.754307] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [16357.762985] [] ? wake_up_state+0x20/0x20 [16357.770200] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [16357.779548] [] kthread+0xcf/0xe0 [16357.785978] [] ? kthread_create_on_node+0x140/0x140 [16357.794256] [] ret_from_fork+0x58/0x90 [16357.801269] [] ? kthread_create_on_node+0x140/0x140 [16357.809541] Code: 00 00 48 c7 05 9b 86 02 00 00 00 00 00 c7 05 89 86 02 00 01 00 00 00 e8 7c eb 97 ff e9 0f fc ff ff 0f 1f 80 00 00 00 00 45 89 e9 <4c> 8d 85 67 ff ff ff 48 8b 4d a8 48 8d 55 88 48 8b b5 38 ff ff [16385.413265] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [16385.422915] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [16385.554787] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [16385.568106] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [16385.584411] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [16385.593817] RIP: 0010:[] [] osd_scrub_next+0xb/0x3c0 [osd_ldiskfs] [16385.605408] RSP: 0018:ffff880703603d00 EFLAGS: 00000287 [16385.612380] RAX: ffffffffa1217a20 RBX: 0000000000000001 RCX: ffff880703603d98 [16385.621365] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [16385.630351] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [16385.639316] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [16385.648268] R13: 0000000000000018 R14: ffff880703603c78 R15: 0000000000000296 [16385.657211] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [16385.667242] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16385.674659] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [16385.683623] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16385.692569] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [16385.701507] Stack: [16385.704712] 0000000000000001 ffff880703603df0 ffffffffa121cff9 ffffffffa121b990 [16385.713988] ffffffffa1217a20 ffff8800b2a36000 00000000810d354f ffff8803162f6000 [16385.723269] ffff8800b2a37468 0000000020000000 010000000000000c 0000000000000000 [16385.732549] Call Trace: [16385.736260] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [16385.745431] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [16385.754588] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [16385.763463] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [16385.772126] [] ? wake_up_state+0x20/0x20 [16385.779325] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [16385.788663] [] kthread+0xcf/0xe0 [16385.795094] [] ? kthread_create_on_node+0x140/0x140 [16385.803364] [] ret_from_fork+0x58/0x90 [16385.810374] [] ? kthread_create_on_node+0x140/0x140 [16385.818634] Code: 4c 02 10 48 01 c2 e8 25 fc ff ff 48 83 c4 08 5b 41 5c 41 5d 41 5e 5d c3 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 89 e5 41 57 <49> 89 f7 41 56 41 55 49 89 fd 41 54 49 89 d4 53 48 89 cb 48 83 [16388.169870] Lustre: soaked-MDT0000: Client 40e8ebdf-e84f-0dfe-7f82-ddc39c67a41e (at 192.168.1.132@o2ib100) reconnecting [16388.184482] Lustre: soaked-MDT0000: Connection restored to 7eb6195d-0392-6b9e-dc88-22a505b06cf8 (at 192.168.1.132@o2ib100) [16413.414240] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [16413.423911] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [16413.555680] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [16413.568962] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [16413.585201] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [16413.594577] RIP: 0010:[] [] osd_scrub_next+0x45/0x3c0 [osd_ldiskfs] [16413.606231] RSP: 0018:ffff880703603c70 EFLAGS: 00000202 [16413.613179] RAX: 0000000000000000 RBX: ffff880703603c37 RCX: ffff880703603d98 [16413.622160] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [16413.631130] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [16413.640107] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [16413.649088] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [16413.658085] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [16413.668146] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16413.675590] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [16413.684582] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16413.693571] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [16413.702551] Stack: [16413.705815] 000000009f8f2001 ffff880703603d08 0000000000000001 ffff880703603df0 [16413.715141] 000000009f8f2001 ffff880703603d57 000000009f8f2001 0000000000000004 [16413.724462] 0000000000000001 0000000000000004 ffff880703603d98 ffff880703603d78 [16413.733787] Call Trace: [16413.737541] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [16413.746738] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [16413.755917] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [16413.764808] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [16413.773504] [] ? wake_up_state+0x20/0x20 [16413.780744] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [16413.790137] [] kthread+0xcf/0xe0 [16413.796595] [] ? kthread_create_on_node+0x140/0x140 [16413.804915] [] ret_from_fork+0x58/0x90 [16413.811952] [] ? kthread_create_on_node+0x140/0x140 [16413.820257] Code: 53 48 89 cb 48 83 ec 70 65 48 8b 04 25 28 00 00 00 48 89 45 d0 31 c0 48 8b 05 f8 59 9c ff 48 85 c0 75 7b 41 f6 87 58 10 00 00 08 <0f> 84 62 03 00 00 4d 8d b7 e0 10 00 00 4d 3b b7 e0 10 00 00 0f [16441.415227] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [16441.424911] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [16441.557107] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [16441.570405] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [16441.586668] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [16441.596056] RIP: 0010:[] [] osd_inode_iteration+0x299/0xcc0 [osd_ldiskfs] [16441.608310] RSP: 0018:ffff880703603d18 EFLAGS: 00000287 [16441.615256] RAX: ffff88042bd78800 RBX: 000000009f8f2001 RCX: ffff880703603d98 [16441.624239] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff88042bd79800 [16441.633210] RBP: ffff880703603df0 R08: 0000000000000001 R09: 0000000000000004 [16441.642168] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffff880703603d57 [16441.651125] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [16441.660085] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [16441.670122] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16441.677537] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [16441.686510] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16441.695478] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [16441.704443] Stack: [16441.707666] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [16441.716962] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [16441.726260] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [16441.735555] Call Trace: [16441.739279] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [16441.748455] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [16441.757338] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [16441.766028] [] ? wake_up_state+0x20/0x20 [16441.773238] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [16441.782588] [] kthread+0xcf/0xe0 [16441.789037] [] ? kthread_create_on_node+0x140/0x140 [16441.797334] [] ret_from_fork+0x58/0x90 [16441.804367] [] ? kthread_create_on_node+0x140/0x140 [16441.812652] Code: 8b 45 88 48 8b 80 a8 03 00 00 48 3b 50 20 0f 83 57 01 00 00 44 0f b6 85 67 ff ff ff 48 8d 4d a8 48 8d 55 88 48 8b b5 38 ff ff ff <48> 8b bd 48 ff ff ff 48 8b 85 30 ff ff ff ff d0 83 f8 03 41 89 [16469.416191] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [16469.425860] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [16469.557882] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [16469.571195] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [16469.587507] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [16469.596922] RIP: 0010:[] [] osd_scrub_next+0x353/0x3c0 [osd_ldiskfs] [16469.608706] RSP: 0018:ffff880703603c70 EFLAGS: 00000202 [16469.615672] RAX: 0000000000000000 RBX: ffff880703603c37 RCX: ffff880703603d98 [16469.624666] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [16469.633659] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [16469.642633] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [16469.651591] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [16469.660552] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [16469.670584] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16469.677997] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [16469.686975] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16469.695948] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [16469.704898] Stack: [16469.708114] 000000009f8f2001 ffff880703603df0 ffff880703603d57 ffff880703603df0 [16469.717415] 000000009f8f2001 ffff880703603d57 000000009f8f2001 0000000000000004 [16469.726711] 0000000000000001 ffff88042bd78800 ffff88042bd78800 00000000000072cf [16469.736000] Call Trace: [16469.739722] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [16469.748893] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [16469.758052] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [16469.766919] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [16469.775581] [] ? wake_up_state+0x20/0x20 [16469.782790] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [16469.792132] [] kthread+0xcf/0xe0 [16469.798555] [] ? kthread_create_on_node+0x140/0x140 [16469.806826] [] ret_from_fork+0x58/0x90 [16469.813829] [] ? kthread_create_on_node+0x140/0x140 [16469.822107] Code: 8b 55 90 48 8b 4d 88 44 8b 85 74 ff ff ff e9 1d ff ff ff 48 89 d7 44 89 45 90 e8 f9 67 47 e0 44 8b 45 90 0f 1f 44 00 00 45 84 c0 04 00 00 00 0f 85 eb fd ff ff 49 8d b7 6c 14 00 00 4c 89 e7 [16484.069702] INFO: rcu_sched self-detected stall on CPU { 24} (t=9060171 jiffies g=907112 c=907111 q=28308206) [16484.082158] Task dump for CPU 24: [16484.086888] OI_scrub R running task 0 5551 2 0x00000088 [16484.095824] ffff88083fde6dd0 00000000fb6db880 ffff88082da03db0 ffffffff810c47e8 [16484.105139] 0000000000000018 ffffffff81a1e7c0 ffff88082da03dc8 ffffffff810c8079 [16484.114438] 0000000000000009 ffff88082da03df8 ffffffff811378e0 ffff88082da101c0 [16484.123708] Call Trace: [16484.127384] [] sched_show_task+0xa8/0x110 [16484.135357] [] dump_cpu_task+0x39/0x70 [16484.142305] [] rcu_dump_cpu_stacks+0x90/0xd0 [16484.149818] [] rcu_check_callbacks+0x442/0x720 [16484.157509] [] ? tick_sched_handle.isra.13+0x60/0x60 [16484.165767] [] update_process_times+0x47/0x80 [16484.173315] [] tick_sched_handle.isra.13+0x25/0x60 [16484.181326] [] tick_sched_timer+0x41/0x70 [16484.188437] [] __hrtimer_run_queues+0xd2/0x260 [16484.196018] [] hrtimer_interrupt+0xb0/0x1e0 [16484.203299] [] ? call_softirq+0x1c/0x30 [16484.210166] [] local_apic_timer_interrupt+0x37/0x60 [16484.218185] [] smp_apic_timer_interrupt+0x3f/0x60 [16484.226004] [] apic_timer_interrupt+0x6d/0x80 [16484.233421] [] ? osd_inode_iteration+0x251/0xcc0 [osd_ldiskfs] [16484.243228] [] ? osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [16484.252322] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [16484.261220] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [16484.269832] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [16484.278251] [] ? wake_up_state+0x20/0x20 [16484.285205] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [16484.294295] [] kthread+0xcf/0xe0 [16484.300461] [] ? kthread_create_on_node+0x140/0x140 [16484.308481] [] ret_from_fork+0x58/0x90 [16484.315235] [] ? kthread_create_on_node+0x140/0x140 [16509.417584] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [16509.427000] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [16509.556822] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [16509.569994] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [16509.586063] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [16509.595371] RIP: 0010:[] [] osd_inode_iteration+0x282/0xcc0 [osd_ldiskfs] [16509.607511] RSP: 0018:ffff880703603d18 EFLAGS: 00000287 [16509.614372] RAX: ffff88042bd78800 RBX: ffff880703603d57 RCX: ffff88042bd78800 [16509.623279] RDX: 00000000000072cf RSI: ffff880419dc3a00 RDI: ffff88042bd79800 [16509.632183] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [16509.641093] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [16509.650001] R13: ffffea00027e3c80 R14: 000000009f8f2001 R15: ffff880703603d08 [16509.658900] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [16509.668879] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16509.676230] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [16509.685146] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16509.694077] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [16509.702979] Stack: [16509.706151] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [16509.715408] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [16509.724664] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [16509.733923] Call Trace: [16509.737620] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [16509.746759] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [16509.755609] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [16509.764259] [] ? wake_up_state+0x20/0x20 [16509.771457] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [16509.780800] [] kthread+0xcf/0xe0 [16509.787210] [] ? kthread_create_on_node+0x140/0x140 [16509.795464] [] ret_from_fork+0x58/0x90 [16509.802448] [] ? kthread_create_on_node+0x140/0x140 [16509.810709] Code: 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 e8 77 5b f4 ff 42 8d 14 38 48 8b 45 88 48 8b 80 a8 03 00 00 48 3b 50 20 0f 83 57 01 00 00 <44> 0f b6 85 67 ff ff ff 48 8d 4d a8 48 8d 55 88 48 8b b5 38 ff [16537.418566] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [16537.428194] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [16537.560709] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [16537.574034] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [16537.590347] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [16537.599776] RIP: 0010:[] [] osd_scrub_exec+0x3d/0x5b0 [osd_ldiskfs] [16537.611468] RSP: 0018:ffff880703603c78 EFLAGS: 00000286 [16537.618441] RAX: ffff8803162f1008 RBX: ffff880703603c37 RCX: 0000000000000000 [16537.627440] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [16537.636428] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [16537.645401] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [16537.654364] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [16537.663322] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [16537.673364] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16537.680771] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [16537.689746] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16537.698721] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [16537.707675] Stack: [16537.710896] ffffea00027e3c80 000000009f8f2001 ffff880703603df0 0000000000000001 [16537.720198] ffff880703603d57 000000009f8f2001 0000000000000004 ffff880703603d57 [16537.729497] ffff8800b2a37468 0000000000000000 ffff880703603d78 ffff8800b2a36000 [16537.738794] Call Trace: [16537.742536] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [16537.751716] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [16537.760880] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [16537.769754] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [16537.778432] [] ? wake_up_state+0x20/0x20 [16537.785645] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [16537.795024] [] kthread+0xcf/0xe0 [16537.801461] [] ? kthread_create_on_node+0x140/0x140 [16537.809749] [] ret_from_fork+0x58/0x90 [16537.816760] [] ? kthread_create_on_node+0x140/0x140 [16537.825031] Code: 41 54 45 31 e4 53 48 83 ec 68 48 8b 9e 08 10 00 00 4c 89 45 90 65 48 8b 04 25 28 00 00 00 48 89 45 d0 31 c0 48 8d 43 08 48 85 db <4c> 0f 45 e0 41 83 f9 04 0f 84 84 00 00 00 41 83 f9 07 74 4f 41 [16565.419541] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [16565.429187] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [16565.561314] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [16565.574598] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [16565.590838] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [16565.600225] RIP: 0010:[] [] osd_inode_iteration+0x47a/0xcc0 [osd_ldiskfs] [16565.612465] RSP: 0018:ffff880703603d18 EFLAGS: 00000293 [16565.619414] RAX: 0000000000000004 RBX: ffff880703603d57 RCX: ffff880703603d98 [16565.628395] RDX: ffff880703603d78 RSI: 0000000000000000 RDI: ffff8803162f6000 [16565.637357] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [16565.646317] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [16565.655270] R13: ffffea00027e3c80 R14: 000000009f8f2001 R15: ffff880703603d08 [16565.664229] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [16565.674266] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16565.681687] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [16565.690668] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16565.699625] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [16565.708573] Stack: [16565.711789] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [16565.721082] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [16565.730392] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [16565.739696] Call Trace: [16565.743426] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [16565.752598] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [16565.761472] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [16565.770159] [] ? wake_up_state+0x20/0x20 [16565.777376] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [16565.786743] [] kthread+0xcf/0xe0 [16565.793192] [] ? kthread_create_on_node+0x140/0x140 [16565.801496] [] ret_from_fork+0x58/0x90 [16565.808520] [] ? kthread_create_on_node+0x140/0x140 [16565.816798] Code: 02 00 00 00 00 00 c7 05 89 86 02 00 01 00 00 00 e8 7c eb 97 ff e9 0f fc ff ff 0f 1f 80 00 00 00 00 45 89 e9 4c 8d 85 67 ff ff ff <48> 8b 4d a8 48 8d 55 88 48 8b b5 38 ff ff ff 48 8b bd 48 ff ff [16593.420520] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [16593.430150] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [16593.562221] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [16593.575539] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [16593.591845] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [16593.601268] RIP: 0010:[] [] osd_inode_iteration+0x251/0xcc0 [osd_ldiskfs] [16593.613595] RSP: 0018:ffff880703603d18 EFLAGS: 00000286 [16593.620563] RAX: ffff8800b2a37468 RBX: ffff880703603d98 RCX: 0000000000000000 [16593.629602] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [16593.638588] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [16593.647558] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000001 [16593.656513] R13: 0000000000000004 R14: 000000009f8f2001 R15: 0000000000000018 [16593.665497] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [16593.675552] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16593.682953] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [16593.691930] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16593.700886] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [16593.709831] Stack: [16593.713037] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [16593.722376] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [16593.731661] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [16593.740952] Call Trace: [16593.744670] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [16593.753890] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [16593.762763] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [16593.771477] [] ? wake_up_state+0x20/0x20 [16593.778684] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [16593.788026] [] kthread+0xcf/0xe0 [16593.794562] [] ? kthread_create_on_node+0x140/0x140 [16593.802841] [] ret_from_fork+0x58/0x90 [16593.809847] [] ? kthread_create_on_node+0x140/0x140 [16593.818120] Code: b0 00 00 00 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 00 8b 75 98 48 8b 7d 88 e8 dc 2d f5 ff 48 85 c0 48 89 45 90 0f 84 b3 08 00 00 <44> 8b 7d a0 48 8b 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 e8 77 [16621.421486] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [16621.431107] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [16621.563027] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [16621.576326] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [16621.592607] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [16621.602003] RIP: 0010:[] [] osd_scrub_next+0x149/0x3c0 [osd_ldiskfs] [16621.613762] RSP: 0018:ffff880703603c70 EFLAGS: 00000202 [16621.620714] RAX: 0000000000000004 RBX: ffff880703603c37 RCX: ffff880703603d98 [16621.629691] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [16621.638664] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [16621.647609] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [16621.656555] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [16621.665505] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [16621.675523] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16621.682923] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [16621.691891] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16621.700853] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [16621.709814] Stack: [16621.713044] 000000009f8f2001 0000000000000004 0000000000000001 ffff880703603df0 [16621.722334] ffff880703603d98 ffff880703603d57 000000009f8f2001 0000000000000004 [16621.731635] ffff880703603d57 ffff8800b2a37468 0000000000000000 ffff880703603d78 [16621.740934] Call Trace: [16621.744665] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [16621.753831] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [16621.763000] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [16621.771869] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [16621.780545] [] ? wake_up_state+0x20/0x20 [16621.787753] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [16621.797099] [] kthread+0xcf/0xe0 [16621.803525] [] ? kthread_create_on_node+0x140/0x140 [16621.811801] [] ret_from_fork+0x58/0x90 [16621.818800] [] ? kthread_create_on_node+0x140/0x140 [16621.827073] Code: 45 90 74 49 49 8d 9f 10 11 00 00 48 89 df e8 df 6a 47 e0 41 c7 87 58 10 00 00 02 00 00 00 48 89 df e8 fc 69 47 e0 b8 05 00 00 00 <48> 8b 75 d0 65 48 33 34 25 28 00 00 00 0f 85 4c 02 00 00 48 83 [16649.422461] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [16649.432106] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [16649.563936] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [16649.577258] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [16649.593508] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [16649.602894] RIP: 0010:[] [] osd_scrub_next+0x149/0x3c0 [osd_ldiskfs] [16649.614644] RSP: 0018:ffff880703603c70 EFLAGS: 00000202 [16649.621583] RAX: 0000000000000004 RBX: ffff880703603c37 RCX: ffff880703603d98 [16649.630568] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [16649.639534] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [16649.648489] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [16649.657438] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [16649.666389] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [16649.676417] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16649.683824] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [16649.692790] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16649.701749] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [16649.710692] Stack: [16649.713900] 000000009f8f2001 0000000000000004 0000000000000001 ffffffffa1217a20 [16649.723195] ffff880703603d98 ffff880703603d57 ffff8800b2a36000 ffff8803162f6000 [16649.732509] ffffffffffffff10 ffffffffa1217a3e 0000000000000010 0000000000000287 [16649.741823] Call Trace: [16649.745549] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [16649.754434] [] ? osd_scrub_next+0x1e/0x3c0 [osd_ldiskfs] [16649.763223] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [16649.772388] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [16649.781544] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [16649.790438] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [16649.799118] [] ? wake_up_state+0x20/0x20 [16649.806334] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [16649.815720] [] kthread+0xcf/0xe0 [16649.822155] [] ? kthread_create_on_node+0x140/0x140 [16649.830437] [] ret_from_fork+0x58/0x90 [16649.837459] [] ? kthread_create_on_node+0x140/0x140 [16649.845736] Code: 45 90 74 49 49 8d 9f 10 11 00 00 48 89 df e8 df 6a 47 e0 41 c7 87 58 10 00 00 02 00 00 00 48 89 df e8 fc 69 47 e0 b8 05 00 00 00 <48> 8b 75 d0 65 48 33 34 25 28 00 00 00 0f 85 4c 02 00 00 48 83 [16664.078993] INFO: rcu_sched self-detected stall on CPU { 24} (t=9240174 jiffies g=907112 c=907111 q=29961699) [16664.091454] Task dump for CPU 24: [16664.096167] OI_scrub R running task 0 5551 2 0x00000088 [16664.105109] ffff88083fde6dd0 00000000fb6db880 ffff88082da03db0 ffffffff810c47e8 [16664.114414] 0000000000000018 ffffffff81a1e7c0 ffff88082da03dc8 ffffffff810c8079 [16664.123689] 0000000000000009 ffff88082da03df8 ffffffff811378e0 ffff88082da101c0 [16664.132922] Call Trace: [16664.136573] [] sched_show_task+0xa8/0x110 [16664.144542] [] dump_cpu_task+0x39/0x70 [16664.151482] [] rcu_dump_cpu_stacks+0x90/0xd0 [16664.158982] [] rcu_check_callbacks+0x442/0x720 [16664.166649] [] ? tick_sched_handle.isra.13+0x60/0x60 [16664.174883] [] update_process_times+0x47/0x80 [16664.182407] [] tick_sched_handle.isra.13+0x25/0x60 [16664.190398] [] tick_sched_timer+0x41/0x70 [16664.197501] [] __hrtimer_run_queues+0xd2/0x260 [16664.205071] [] hrtimer_interrupt+0xb0/0x1e0 [16664.212338] [] ? call_softirq+0x1c/0x30 [16664.219206] [] local_apic_timer_interrupt+0x37/0x60 [16664.227229] [] smp_apic_timer_interrupt+0x3f/0x60 [16664.235067] [] apic_timer_interrupt+0x6d/0x80 [16664.242513] [] ? osd_scrub_exec+0x34/0x5b0 [osd_ldiskfs] [16664.251753] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [16664.260661] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [16664.269566] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [16664.278173] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [16664.286577] [] ? wake_up_state+0x20/0x20 [16664.293522] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [16664.302626] [] kthread+0xcf/0xe0 [16664.308803] [] ? kthread_create_on_node+0x140/0x140 [16664.316839] [] ret_from_fork+0x58/0x90 [16664.323591] [] ? kthread_create_on_node+0x140/0x140 [16689.423869] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [16689.433269] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [16689.562901] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [16689.576069] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [16689.592125] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [16689.601415] RIP: 0010:[] [] osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [16689.613367] RSP: 0018:ffff880703603d10 EFLAGS: 00000293 [16689.620213] RAX: ffffffffa121b990 RBX: ffff880703603d98 RCX: 0000000000000000 [16689.629113] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [16689.638004] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [16689.646900] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000001 [16689.655795] R13: 0000000000000004 R14: 000000009f8f2001 R15: ffffea00027e3c80 [16689.664670] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [16689.674630] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16689.681966] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [16689.690864] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16689.699765] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [16689.708649] Stack: [16689.711809] ffffffffa121d1e9 ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 [16689.721055] 00000000810d354f ffff8803162f6000 ffff8800b2a37468 0000000020000000 [16689.730291] 010000000000000c 0000000000000000 0000000000000000 ffff8800b2a36000 [16689.739535] Call Trace: [16689.743212] [] ? osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [16689.752542] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [16689.761678] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [16689.770514] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [16689.779156] [] ? wake_up_state+0x20/0x20 [16689.786332] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [16689.795678] [] kthread+0xcf/0xe0 [16689.802085] [] ? kthread_create_on_node+0x140/0x140 [16689.810351] [] ret_from_fork+0x58/0x90 [16689.817329] [] ? kthread_create_on_node+0x140/0x140 [16689.825571] Code: b5 07 00 00 c7 05 85 a5 02 00 00 00 02 00 89 54 24 08 48 8b 40 40 ba 07 00 00 00 48 89 04 24 31 c0 e8 a5 03 98 ff e9 c7 fd ff ff <66> 66 66 66 90 55 48 89 e5 41 57 49 89 f7 41 56 41 55 41 54 45 [16717.424831] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [16717.434464] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [16717.566203] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [16717.579498] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [16717.595748] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [16717.605140] RIP: 0010:[] [] osd_scrub_exec+0x95/0x5b0 [osd_ldiskfs] [16717.616787] RSP: 0018:ffff880703603d08 EFLAGS: 00000292 [16717.623728] RAX: 0000000000000000 RBX: 0000000000000001 RCX: 0000000000000000 [16717.632704] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [16717.641675] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [16717.650633] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [16717.659587] R13: 0000000000000018 R14: ffff880703603c78 R15: 0000000000000246 [16717.668530] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [16717.678555] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16717.685953] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [16717.694915] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16717.703883] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [16717.712826] Stack: [16717.716038] ffff880703603df0 ffffffffa121d1e9 ffffffffa121b990 ffffffffa1217a20 [16717.725324] ffff8800b2a36000 00000000810d354f ffff8803162f6000 ffff8800b2a37468 [16717.734617] 0000000020000000 010000000000000c 0000000000000000 0000000000000000 [16717.743926] Call Trace: [16717.747644] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [16717.756807] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [16717.765962] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [16717.774823] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [16717.783483] [] ? wake_up_state+0x20/0x20 [16717.790681] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [16717.800024] [] kthread+0xcf/0xe0 [16717.806445] [] ? kthread_create_on_node+0x140/0x140 [16717.814715] [] ret_from_fork+0x58/0x90 [16717.821717] [] ? kthread_create_on_node+0x140/0x140 [16717.829984] Code: 01 00 00 41 80 a7 74 14 00 00 fe 48 8b 4d d0 65 48 33 0c 25 28 00 00 00 0f 85 b1 03 00 00 48 83 c4 68 5b 41 5c 41 5d 41 5e 41 5f <5d> c3 66 0f 1f 84 00 00 00 00 00 4c 8d ae f0 10 00 00 44 89 4d [16745.425806] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [16745.435489] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [16745.567226] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [16745.580507] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [16745.596738] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [16745.606114] RIP: 0010:[] [] osd_scrub_next+0x9/0x3c0 [osd_ldiskfs] [16745.617656] RSP: 0018:ffff880703603d08 EFLAGS: 00000287 [16745.624587] RAX: ffffffffa1217a20 RBX: 0000000000000001 RCX: ffff880703603d98 [16745.633560] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [16745.642506] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [16745.651458] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [16745.660396] R13: 0000000000000018 R14: ffff880703603c78 R15: 0000000000000296 [16745.669322] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [16745.679333] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16745.686722] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [16745.695675] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16745.704621] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [16745.713553] Stack: [16745.716750] ffff880703603df0 ffffffffa121cff9 ffffffffa121b990 ffffffffa1217a20 [16745.726046] ffff8800b2a36000 00000000810d354f ffff8803162f6000 ffff8800b2a37468 [16745.735340] 0000000020000000 010000000000000c 0000000000000000 0000000000000000 [16745.744627] Call Trace: [16745.748345] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [16745.757511] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [16745.766669] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [16745.775521] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [16745.784179] [] ? wake_up_state+0x20/0x20 [16745.791375] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [16745.800733] [] kthread+0xcf/0xe0 [16745.807148] [] ? kthread_create_on_node+0x140/0x140 [16745.815436] [] ret_from_fork+0x58/0x90 [16745.822436] [] ? kthread_create_on_node+0x140/0x140 [16745.830702] Code: 48 8d 4c 02 10 48 01 c2 e8 25 fc ff ff 48 83 c4 08 5b 41 5c 41 5d 41 5e 5d c3 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 89 e5 <41> 57 49 89 f7 41 56 41 55 49 89 fd 41 54 49 89 d4 53 48 89 cb [16773.426782] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [16773.436507] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [16773.568363] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [16773.581647] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [16773.597885] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [16773.607269] RIP: 0010:[] [] osd_scrub_next+0x160/0x3c0 [osd_ldiskfs] [16773.619035] RSP: 0018:ffff880703603ce0 EFLAGS: 00000282 [16773.625967] RAX: 0000000000000004 RBX: ffff880703603c70 RCX: ffff880703603d98 [16773.634942] RDX: ffff880703603d78 RSI: 0000000000000000 RDI: ffff8803162f6000 [16773.643900] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [16773.652847] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000010 [16773.661774] R13: ffffffffa1217d70 R14: ffffffffffffff10 R15: ffff8803162f6000 [16773.670713] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [16773.680730] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16773.688127] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [16773.697085] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16773.706044] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [16773.714990] Stack: [16773.718196] ffff880419dc3a00 ffff8800b2a3746c 0000000000000000 00000000ffffffff [16773.727504] 0000000000000001 ffff880703603df0 ffffffffa121cff9 ffffffffa121b990 [16773.736801] ffffffffa1217a20 ffff8800b2a36000 00000000810d354f ffff8803162f6000 [16773.746097] Call Trace: [16773.749818] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [16773.758989] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [16773.768149] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [16773.777019] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [16773.785679] [] ? wake_up_state+0x20/0x20 [16773.792900] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [16773.802255] [] kthread+0xcf/0xe0 [16773.808685] [] ? kthread_create_on_node+0x140/0x140 [16773.816972] [] ret_from_fork+0x58/0x90 [16773.823990] [] ? kthread_create_on_node+0x140/0x140 [16773.832260] Code: 10 00 00 02 00 00 00 48 89 df e8 fc 69 47 e0 b8 05 00 00 00 48 8b 75 d0 65 48 33 34 25 28 00 00 00 0f 85 4c 02 00 00 48 83 c4 70 <5b> 41 5c 41 5d 41 5e 41 5f 5d c3 48 8b 05 be 58 9c ff 48 85 c0 [16801.427757] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [16801.437414] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [16801.569357] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [16801.582646] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [16801.598908] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [16801.608298] RIP: 0010:[] [] osd_inode_iteration+0x489/0xcc0 [osd_ldiskfs] [16801.620543] RSP: 0018:ffff880703603d18 EFLAGS: 00000293 [16801.627518] RAX: 0000000000000004 RBX: 000000009f8f2001 RCX: 0000000000000000 [16801.636496] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [16801.645462] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [16801.654417] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000001 [16801.663365] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [16801.672318] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [16801.682346] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16801.689754] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [16801.698727] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16801.707691] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [16801.716627] Stack: [16801.719837] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [16801.729123] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [16801.738446] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [16801.747744] Call Trace: [16801.751497] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [16801.760660] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [16801.769536] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [16801.778226] [] ? wake_up_state+0x20/0x20 [16801.785430] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [16801.794782] [] kthread+0xcf/0xe0 [16801.801215] [] ? kthread_create_on_node+0x140/0x140 [16801.809498] [] ret_from_fork+0x58/0x90 [16801.816511] [] ? kthread_create_on_node+0x140/0x140 [16801.824782] Code: 00 e8 7c eb 97 ff e9 0f fc ff ff 0f 1f 80 00 00 00 00 45 89 e9 4c 8d 85 67 ff ff ff 48 8b 4d a8 48 8d 55 88 48 8b b5 38 ff ff ff <48> 8b bd 48 ff ff ff 48 8b 85 28 ff ff ff ff d0 85 c0 41 89 c5 [16814.825524] Lustre: soaked-MDT0000: Client 91eacee7-b83c-19ec-9326-47c5ebda930d (at 192.168.1.116@o2ib100) reconnecting [16814.838837] Lustre: soaked-MDT0000: Connection restored to 91eacee7-b83c-19ec-9326-47c5ebda930d (at 192.168.1.116@o2ib100) [16829.428739] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [16829.438412] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [16829.570440] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [16829.583752] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [16829.600039] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [16829.609449] RIP: 0010:[] [] osd_scrub_next+0x149/0x3c0 [osd_ldiskfs] [16829.621202] RSP: 0018:ffff880703603c70 EFLAGS: 00000202 [16829.628168] RAX: 0000000000000004 RBX: ffff880703603c37 RCX: ffff880703603d98 [16829.637166] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [16829.646142] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [16829.655130] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [16829.664111] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [16829.673103] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [16829.683166] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16829.690609] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [16829.699609] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16829.708595] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [16829.717587] Stack: [16829.720836] 000000009f8f2001 0000000000000004 ffff880703603d57 ffff880703603df0 [16829.730155] 0000000000000000 ffff880703603d57 000000009f8f2001 0000000000000004 [16829.739498] ffff880703603d57 ffff88042bd78800 ffff88042bd78800 00000000000072cf [16829.748836] Call Trace: [16829.752593] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [16829.761804] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [16829.770998] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [16829.779895] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [16829.788602] [] ? wake_up_state+0x20/0x20 [16829.795847] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [16829.805245] [] kthread+0xcf/0xe0 [16829.811711] [] ? kthread_create_on_node+0x140/0x140 [16829.820020] [] ret_from_fork+0x58/0x90 [16829.827065] [] ? kthread_create_on_node+0x140/0x140 [16829.835377] Code: 45 90 74 49 49 8d 9f 10 11 00 00 48 89 df e8 df 6a 47 e0 41 c7 87 58 10 00 00 02 00 00 00 48 89 df e8 fc 69 47 e0 b8 05 00 00 00 <48> 8b 75 d0 65 48 33 34 25 28 00 00 00 0f 85 4c 02 00 00 48 83 [16844.088243] INFO: rcu_sched self-detected stall on CPU { 24} (t=9420177 jiffies g=907112 c=907111 q=31174035) [16844.100730] Task dump for CPU 24: [16844.105467] OI_scrub R running task 0 5551 2 0x00000088 [16844.114404] ffff88083fde6dd0 00000000fb6db880 ffff88082da03db0 ffffffff810c47e8 [16844.123733] 0000000000000018 ffffffff81a1e7c0 ffff88082da03dc8 ffffffff810c8079 [16844.133031] 0000000000000009 ffff88082da03df8 ffffffff811378e0 ffff88082da101c0 [16844.142313] Call Trace: [16844.145998] [] sched_show_task+0xa8/0x110 [16844.153981] [] dump_cpu_task+0x39/0x70 [16844.160947] [] rcu_dump_cpu_stacks+0x90/0xd0 [16844.168468] [] rcu_check_callbacks+0x442/0x720 [16844.176169] [] ? tick_sched_handle.isra.13+0x60/0x60 [16844.184430] [] update_process_times+0x47/0x80 [16844.191969] [] tick_sched_handle.isra.13+0x25/0x60 [16844.199984] [] tick_sched_timer+0x41/0x70 [16844.207102] [] __hrtimer_run_queues+0xd2/0x260 [16844.214696] [] hrtimer_interrupt+0xb0/0x1e0 [16844.221974] [] ? call_softirq+0x1c/0x30 [16844.228853] [] local_apic_timer_interrupt+0x37/0x60 [16844.236873] [] smp_apic_timer_interrupt+0x3f/0x60 [16844.244695] [] apic_timer_interrupt+0x6d/0x80 [16844.252128] [] ? osd_inode_iteration+0x282/0xcc0 [osd_ldiskfs] [16844.261949] [] ? osd_inode_iteration+0x269/0xcc0 [osd_ldiskfs] [16844.271062] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [16844.279969] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [16844.288585] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [16844.296993] [] ? wake_up_state+0x20/0x20 [16844.304007] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [16844.313097] [] kthread+0xcf/0xe0 [16844.319270] [] ? kthread_create_on_node+0x140/0x140 [16844.327300] [] ret_from_fork+0x58/0x90 [16844.334056] [] ? kthread_create_on_node+0x140/0x140 [16869.430125] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [16869.439545] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [16869.569471] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [16869.582905] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [16869.598980] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [16869.608278] RIP: 0010:[] [] ldiskfs_itable_unused_count+0x16/0x30 [ldiskfs] [16869.620619] RSP: 0018:ffff880703603d08 EFLAGS: 00000246 [16869.627485] RAX: 0000000000000000 RBX: 000000009f8f2001 RCX: ffff88042bd78800 [16869.636393] RDX: 00000000000072ce RSI: ffff880419dc3a00 RDI: ffff88042bd79800 [16869.645291] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [16869.654191] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [16869.663097] R13: 0000000000000018 R14: ffff880703603c78 R15: 0000000000000246 [16869.671989] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [16869.681950] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16869.689288] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [16869.698188] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16869.707087] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [16869.715980] Stack: [16869.719155] ffff880703603df0 ffffffffa121cfb9 ffffffffa121b990 ffffffffa1217a20 [16869.728400] ffff8800b2a36000 00000000810d354f ffff8803162f6000 ffff8800b2a37468 [16869.737665] 0000000020000000 010000000000000c 0000000000000000 0000000000000000 [16869.746903] Call Trace: [16869.750596] [] osd_inode_iteration+0x269/0xcc0 [osd_ldiskfs] [16869.759738] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [16869.768871] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [16869.777710] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [16869.786353] [] ? wake_up_state+0x20/0x20 [16869.793529] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [16869.802842] [] kthread+0xcf/0xe0 [16869.809233] [] ? kthread_create_on_node+0x140/0x140 [16869.817465] [] ret_from_fork+0x58/0x90 [16869.824440] [] ? kthread_create_on_node+0x140/0x140 [16869.832685] Code: 07 0f b7 46 30 c1 e0 10 09 d0 5d c3 66 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 8b 8f a8 03 00 00 31 c0 0f b7 56 1c 48 89 e5 <48> 83 39 3f 76 07 0f b7 46 32 c1 e0 10 09 d0 5d c3 66 0f 1f 84 [16897.431101] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [16897.440751] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [16897.572481] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [16897.585765] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [16897.602011] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [16897.611475] RIP: 0010:[] [] osd_scrub_exec+0x3d/0x5b0 [osd_ldiskfs] [16897.623131] RSP: 0018:ffff880703603c78 EFLAGS: 00000286 [16897.630060] RAX: ffff8803162f1008 RBX: ffff880703603c37 RCX: 0000000000000000 [16897.639035] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [16897.647998] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [16897.656932] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [16897.665871] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [16897.674821] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [16897.684841] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16897.692241] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [16897.701212] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16897.710168] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [16897.719111] Stack: [16897.722323] ffff880703603d08 000000009f8f2001 ffff880703603df0 000000009f8f2001 [16897.731607] ffff880703603d57 000000009f8f2001 0000000000000004 0000000000000001 [16897.740914] 0000000000000004 ffff880703603d98 ffff880703603d78 0000000000000000 [16897.750234] Call Trace: [16897.753952] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [16897.763118] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [16897.772283] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [16897.781208] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [16897.789904] [] ? wake_up_state+0x20/0x20 [16897.797205] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [16897.806550] [] kthread+0xcf/0xe0 [16897.812972] [] ? kthread_create_on_node+0x140/0x140 [16897.821250] [] ret_from_fork+0x58/0x90 [16897.828252] [] ? kthread_create_on_node+0x140/0x140 [16897.836513] Code: 41 54 45 31 e4 53 48 83 ec 68 48 8b 9e 08 10 00 00 4c 89 45 90 65 48 8b 04 25 28 00 00 00 48 89 45 d0 31 c0 48 8d 43 08 48 85 db <4c> 0f 45 e0 41 83 f9 04 0f 84 84 00 00 00 41 83 f9 07 74 4f 41 [16925.432077] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [16925.441759] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [16925.573541] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [16925.586831] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [16925.603063] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [16925.612478] RIP: 0010:[] [] osd_scrub_next+0x161/0x3c0 [osd_ldiskfs] [16925.624225] RSP: 0018:ffff880703603ce8 EFLAGS: 00000282 [16925.631163] RAX: 0000000000000004 RBX: ffff880703603c78 RCX: ffff880703603d98 [16925.640138] RDX: ffff880703603d78 RSI: 0000000000000000 RDI: ffff8803162f6000 [16925.649095] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [16925.658036] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000010 [16925.666984] R13: ffffffffa121ba79 R14: ffffffffffffff10 R15: ffff8803162f6000 [16925.675932] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [16925.685951] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16925.693354] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [16925.702308] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16925.711260] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [16925.720199] Stack: [16925.723397] ffff8800b2a3746c 0000000000000000 00000000ffffffff 0000000000000001 [16925.732699] ffff880703603df0 ffffffffa121cff9 ffffffffa121b990 ffffffffa1217a20 [16925.741984] ffff8800b2a36000 00000000810d354f ffff8803162f6000 ffff8800b2a37468 [16925.751263] Call Trace: [16925.754968] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [16925.764149] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [16925.773309] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [16925.782229] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [16925.790889] [] ? wake_up_state+0x20/0x20 [16925.798097] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [16925.807443] [] kthread+0xcf/0xe0 [16925.813870] [] ? kthread_create_on_node+0x140/0x140 [16925.822147] [] ret_from_fork+0x58/0x90 [16925.829156] [] ? kthread_create_on_node+0x140/0x140 [16925.837423] Code: 00 00 02 00 00 00 48 89 df e8 fc 69 47 e0 b8 05 00 00 00 48 8b 75 d0 65 48 33 34 25 28 00 00 00 0f 85 4c 02 00 00 48 83 c4 70 5b <41> 5c 41 5d 41 5e 41 5f 5d c3 48 8b 05 be 58 9c ff 48 85 c0 0f [16953.433051] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [16953.442734] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [16953.574440] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [16953.587726] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [16953.603988] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [16953.613381] RIP: 0010:[] [] osd_inode_iteration+0x251/0xcc0 [osd_ldiskfs] [16953.625614] RSP: 0018:ffff880703603d18 EFLAGS: 00000286 [16953.632555] RAX: ffff8800b2a37468 RBX: 000000009f8f2001 RCX: 0000000000000000 [16953.641535] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [16953.650504] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [16953.659459] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [16953.668408] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [16953.677357] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [16953.687394] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16953.694806] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [16953.703782] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16953.712740] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [16953.721692] Stack: [16953.724904] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [16953.734208] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [16953.743520] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [16953.752816] Call Trace: [16953.756537] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [16953.765699] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [16953.774573] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [16953.783249] [] ? wake_up_state+0x20/0x20 [16953.790454] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [16953.799803] [] kthread+0xcf/0xe0 [16953.806235] [] ? kthread_create_on_node+0x140/0x140 [16953.814515] [] ret_from_fork+0x58/0x90 [16953.821535] [] ? kthread_create_on_node+0x140/0x140 [16953.829814] Code: b0 00 00 00 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 00 8b 75 98 48 8b 7d 88 e8 dc 2d f5 ff 48 85 c0 48 89 45 90 0f 84 b3 08 00 00 <44> 8b 7d a0 48 8b 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 e8 77 [16981.434027] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [16981.443714] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [16981.575611] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [16981.588919] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [16981.605215] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [16981.614627] RIP: 0010:[] [] osd_scrub_exec+0xe9/0x5b0 [osd_ldiskfs] [16981.626290] RSP: 0018:ffff880703603c78 EFLAGS: 00000282 [16981.633248] RAX: 0000000000280001 RBX: ffff880703603c37 RCX: 0000000000000000 [16981.642235] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [16981.651212] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [16981.660173] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [16981.669134] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [16981.678084] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [16981.688110] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16981.695516] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [16981.704482] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16981.713450] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [16981.722405] Stack: [16981.725612] ffff880703603df0 000000009f8f2001 ffffea00027e3c80 000000009f8f2001 [16981.734909] ffff880703603d57 ffff880703603d57 ffff8800b2a37468 0000000000000000 [16981.744192] ffff880703603d78 ffff880419dc3a00 ffff88042bd79800 ffffffffffffff10 [16981.753487] Call Trace: [16981.757209] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [16981.766378] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [16981.775535] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [16981.784401] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [16981.793063] [] ? wake_up_state+0x20/0x20 [16981.800265] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [16981.809605] [] kthread+0xcf/0xe0 [16981.816036] [] ? kthread_create_on_node+0x140/0x140 [16981.824312] [] ret_from_fork+0x58/0x90 [16981.831316] [] ? kthread_create_on_node+0x140/0x140 [16981.839584] Code: 88 11 00 00 01 4c 89 ef e8 f5 9f e9 df 44 8b 4d 88 48 85 db 74 23 f6 83 18 0a 00 00 08 74 1a 4d 85 e4 74 15 41 8b 87 6c 14 00 00 <41> 39 84 24 0c 0a 00 00 0f 82 d9 00 00 00 31 c0 41 f6 87 74 14 [16987.725936] Lustre: soaked-MDT0000: Client a2eaa722-5c21-8192-3a30-b2265306fd45 (at 192.168.1.120@o2ib100) reconnecting [16987.739262] Lustre: soaked-MDT0000: Connection restored to 91d08a06-6f67-40c2-ceb3-b8c39685af23 (at 192.168.1.120@o2ib100) [17009.435002] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [17009.444709] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [17009.576378] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [17009.589659] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [17009.605932] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [17009.615313] RIP: 0010:[] [] osd_scrub_exec+0xf9/0x5b0 [osd_ldiskfs] [17009.626979] RSP: 0018:ffff880703603c78 EFLAGS: 00000246 [17009.633936] RAX: 0000000000000000 RBX: ffff880703603c37 RCX: 0000000000000000 [17009.642921] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [17009.651910] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [17009.660886] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [17009.669867] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [17009.678856] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [17009.688913] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17009.696341] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [17009.705326] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17009.714350] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [17009.723315] Stack: [17009.726563] ffff880703603df0 0000000000000001 ffff880703603df0 000000009f8f2001 [17009.735867] ffff880703603d57 000000009f8f2001 0000000000000004 ffff880703603d57 [17009.745189] 0000000000000000 0000000000000000 ffff880703603d78 ffff8800b2a36000 [17009.754508] Call Trace: [17009.758259] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [17009.767453] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [17009.776637] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [17009.785531] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [17009.794242] [] ? wake_up_state+0x20/0x20 [17009.801483] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [17009.810864] [] kthread+0xcf/0xe0 [17009.817332] [] ? kthread_create_on_node+0x140/0x140 [17009.825686] [] ret_from_fork+0x58/0x90 [17009.832727] [] ? kthread_create_on_node+0x140/0x140 [17009.841032] Code: 88 48 85 db 74 23 f6 83 18 0a 00 00 08 74 1a 4d 85 e4 74 15 41 8b 87 6c 14 00 00 41 39 84 24 0c 0a 00 00 0f 82 d9 00 00 00 31 c0 <41> f6 87 74 14 00 00 04 0f 85 6e ff ff ff 41 83 f9 02 0f 84 64 [17024.097520] INFO: rcu_sched self-detected stall on CPU { 24} (t=9600180 jiffies g=907112 c=907111 q=33376948) [17024.110023] Task dump for CPU 24: [17024.114766] OI_scrub R running task 0 5551 2 0x00000088 [17024.123690] ffff88083fde6dd0 00000000fb6db880 ffff88082da03db0 ffffffff810c47e8 [17024.133012] 0000000000000018 ffffffff81a1e7c0 ffff88082da03dc8 ffffffff810c8079 [17024.142313] 0000000000000009 ffff88082da03df8 ffffffff811378e0 ffff88082da101c0 [17024.151591] Call Trace: [17024.155270] [] sched_show_task+0xa8/0x110 [17024.163242] [] dump_cpu_task+0x39/0x70 [17024.170197] [] rcu_dump_cpu_stacks+0x90/0xd0 [17024.177714] [] rcu_check_callbacks+0x442/0x720 [17024.185401] [] ? tick_sched_handle.isra.13+0x60/0x60 [17024.193650] [] update_process_times+0x47/0x80 [17024.201202] [] tick_sched_handle.isra.13+0x25/0x60 [17024.209214] [] tick_sched_timer+0x41/0x70 [17024.216336] [] __hrtimer_run_queues+0xd2/0x260 [17024.223916] [] hrtimer_interrupt+0xb0/0x1e0 [17024.231190] [] ? call_softirq+0x1c/0x30 [17024.238063] [] local_apic_timer_interrupt+0x37/0x60 [17024.246085] [] smp_apic_timer_interrupt+0x3f/0x60 [17024.253944] [] apic_timer_interrupt+0x6d/0x80 [17024.261370] [] ? osd_inode_iteration+0x282/0xcc0 [osd_ldiskfs] [17024.271184] [] ? osd_inode_iteration+0x269/0xcc0 [osd_ldiskfs] [17024.280293] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [17024.289207] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [17024.297822] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [17024.306250] [] ? wake_up_state+0x20/0x20 [17024.313203] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [17024.322291] [] kthread+0xcf/0xe0 [17024.328460] [] ? kthread_create_on_node+0x140/0x140 [17024.336484] [] ret_from_fork+0x58/0x90 [17024.343262] [] ? kthread_create_on_node+0x140/0x140 [17049.436396] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [17049.445822] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [17049.575710] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [17049.588929] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [17049.605050] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [17049.614418] RIP: 0010:[] [] osd_scrub_next+0xb/0x3c0 [osd_ldiskfs] [17049.625921] RSP: 0018:ffff880703603d00 EFLAGS: 00000287 [17049.632790] RAX: ffffffffa1217a20 RBX: 000000009f8f2001 RCX: ffff880703603d98 [17049.641698] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [17049.650614] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [17049.659512] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [17049.668414] R13: 0000000000000018 R14: ffff880703603c78 R15: 0000000000000286 [17049.677305] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [17049.687268] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17049.694607] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [17049.703507] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17049.712409] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [17049.721383] Stack: [17049.724619] 0000000000000001 ffff880703603df0 ffffffffa121cff9 ffffffffa121b990 [17049.733877] ffffffffa1217a20 ffff8800b2a36000 00000000810d354f ffff8803162f6000 [17049.743126] ffff8800b2a37468 0000000020000000 010000000000000c 0000000000000000 [17049.752391] Call Trace: [17049.756082] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [17049.765216] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [17049.774409] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [17049.783260] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [17049.791902] [] ? wake_up_state+0x20/0x20 [17049.799143] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [17049.808554] [] kthread+0xcf/0xe0 [17049.815021] [] ? kthread_create_on_node+0x140/0x140 [17049.823257] [] ret_from_fork+0x58/0x90 [17049.830313] [] ? kthread_create_on_node+0x140/0x140 [17049.838550] Code: 4c 02 10 48 01 c2 e8 25 fc ff ff 48 83 c4 08 5b 41 5c 41 5d 41 5e 5d c3 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 89 e5 41 57 <49> 89 f7 41 56 41 55 49 89 fd 41 54 49 89 d4 53 48 89 cb 48 83 [17077.437371] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [17077.447002] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [17077.578721] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [17077.592000] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [17077.608249] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [17077.617625] RIP: 0010:[] [] osd_inode_iteration+0x2ac/0xcc0 [osd_ldiskfs] [17077.629850] RSP: 0018:ffff880703603d18 EFLAGS: 00000202 [17077.636777] RAX: 0000000000000004 RBX: ffff880703603d98 RCX: ffff880703603d98 [17077.645742] RDX: ffff880703603d78 RSI: 0000000000000000 RDI: ffff8803162f6000 [17077.654705] RBP: ffff880703603df0 R08: 0000000000000001 R09: 0000000000000004 [17077.663653] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000001 [17077.672595] R13: 0000000000000004 R14: 000000009f8f2001 R15: ffffea00027e3c80 [17077.681537] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [17077.691555] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17077.698950] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [17077.707908] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17077.716859] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [17077.725810] Stack: [17077.729007] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [17077.738301] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [17077.747599] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [17077.756890] Call Trace: [17077.760604] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [17077.769765] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [17077.778630] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [17077.787311] [] ? wake_up_state+0x20/0x20 [17077.794510] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [17077.803854] [] kthread+0xcf/0xe0 [17077.810291] [] ? kthread_create_on_node+0x140/0x140 [17077.818572] [] ret_from_fork+0x58/0x90 [17077.825579] [] ? kthread_create_on_node+0x140/0x140 [17077.833848] Code: 00 44 0f b6 85 67 ff ff ff 48 8d 4d a8 48 8d 55 88 48 8b b5 38 ff ff ff 48 8b bd 48 ff ff ff 48 8b 85 30 ff ff ff ff d0 83 f8 03 <41> 89 c5 0f 84 4b 02 00 00 0f 8e 15 01 00 00 83 f8 05 66 90 0f [17105.438348] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [17105.448015] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [17105.580201] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [17105.593505] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [17105.609793] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [17105.619195] RIP: 0010:[] [] osd_scrub_next+0x160/0x3c0 [osd_ldiskfs] [17105.630956] RSP: 0018:ffff880703603ce0 EFLAGS: 00000282 [17105.637914] RAX: 0000000000000004 RBX: ffff880703603c78 RCX: ffff880703603d98 [17105.646900] RDX: ffff880703603d78 RSI: 0000000000000000 RDI: ffff8803162f6000 [17105.655876] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [17105.664836] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000010 [17105.673779] R13: ffffffffa121ba87 R14: ffffffffffffff10 R15: ffff8803162f6000 [17105.682720] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [17105.692735] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17105.700139] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [17105.709102] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17105.718062] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [17105.727011] Stack: [17105.730225] ffff880419dc3a00 ffff8800b2a3746c 0000000000000000 00000000ffffffff [17105.739515] 0000000000000001 ffff880703603df0 ffffffffa121cff9 ffffffffa121b990 [17105.748811] ffffffffa1217a20 ffff8800b2a36000 00000000810d354f ffff8803162f6000 [17105.758170] Call Trace: [17105.761914] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [17105.771082] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [17105.780252] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [17105.789133] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [17105.797805] [] ? wake_up_state+0x20/0x20 [17105.805023] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [17105.814371] [] kthread+0xcf/0xe0 [17105.820807] [] ? kthread_create_on_node+0x140/0x140 [17105.829091] [] ret_from_fork+0x58/0x90 [17105.836100] [] ? kthread_create_on_node+0x140/0x140 [17105.844375] Code: 10 00 00 02 00 00 00 48 89 df e8 fc 69 47 e0 b8 05 00 00 00 48 8b 75 d0 65 48 33 34 25 28 00 00 00 0f 85 4c 02 00 00 48 83 c4 70 <5b> 41 5c 41 5d 41 5e 41 5f 5d c3 48 8b 05 be 58 9c ff 48 85 c0 [17133.439329] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [17133.448999] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [17133.580799] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [17133.594078] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [17133.610335] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [17133.619718] RIP: 0010:[] [] osd_scrub_exec+0x30/0x5b0 [osd_ldiskfs] [17133.631392] RSP: 0018:ffff880703603c78 EFLAGS: 00000296 [17133.638333] RAX: 00000000fb6db880 RBX: ffff880703603c37 RCX: 0000000000000000 [17133.647307] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [17133.656272] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [17133.665222] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [17133.674170] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [17133.683117] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [17133.693154] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17133.700565] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [17133.709533] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17133.718493] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [17133.727446] Stack: [17133.730665] ffff880703603df0 000000009f8f2001 ffff880703603df0 000000009f8f2001 [17133.739974] ffff880703603d57 000000009f8f2001 0000000000000004 ffff880703603d57 [17133.749278] 0000000020000000 0000000000000000 ffff880703603d78 ffff8800b2a36000 [17133.758577] Call Trace: [17133.762302] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [17133.771465] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [17133.780621] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [17133.789486] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [17133.798150] [] ? wake_up_state+0x20/0x20 [17133.805361] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [17133.814705] [] kthread+0xcf/0xe0 [17133.821144] [] ? kthread_create_on_node+0x140/0x140 [17133.829429] [] ret_from_fork+0x58/0x90 [17133.836436] [] ? kthread_create_on_node+0x140/0x140 [17133.844699] Code: 55 48 89 e5 41 57 49 89 f7 41 56 41 55 41 54 45 31 e4 53 48 83 ec 68 48 8b 9e 08 10 00 00 4c 89 45 90 65 48 8b 04 25 28 00 00 00 <48> 89 45 d0 31 c0 48 8d 43 08 48 85 db 4c 0f 45 e0 41 83 f9 04 [17154.584325] Lustre: soaked-MDT0000: Client af5c0dbc-bc84-dcde-3672-645b97702b86 (at 192.168.1.122@o2ib100) reconnecting [17154.597698] Lustre: Skipped 1 previous similar message [17154.604530] Lustre: soaked-MDT0000: Connection restored to 050e5cd6-4fff-0145-e167-2fb74af973b4 (at 192.168.1.122@o2ib100) [17154.617921] Lustre: Skipped 1 previous similar message [17161.440299] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [17161.449988] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [17161.581538] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [17161.594805] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [17161.611042] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [17161.620425] RIP: 0010:[] [] ldiskfs_itable_unused_count+0x6/0x30 [ldiskfs] [17161.632769] RSP: 0018:ffff880703603d08 EFLAGS: 00000286 [17161.639733] RAX: ffff8800b2a37468 RBX: 000000009f8f2001 RCX: 0000000000000000 [17161.648740] RDX: ffff880703603d78 RSI: ffff880419dc3a00 RDI: ffff88042bd79800 [17161.657741] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [17161.666746] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [17161.675751] R13: ffff880703603d08 R14: ffff880703603c78 R15: 0000000000000296 [17161.684747] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [17161.694816] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17161.702265] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [17161.711274] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17161.720287] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [17161.729284] Stack: [17161.732544] ffff880703603df0 ffffffffa121cfb9 ffffffffa121b990 ffffffffa1217a20 [17161.741889] ffff8800b2a36000 00000000810d354f ffff8803162f6000 ffff8800b2a37468 [17161.751223] 0000000020000000 010000000000000c 0000000000000000 0000000000000000 [17161.760563] Call Trace: [17161.764337] [] ? osd_inode_iteration+0x269/0xcc0 [osd_ldiskfs] [17161.773752] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [17161.782970] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [17161.791902] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [17161.800628] [] ? wake_up_state+0x20/0x20 [17161.807895] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [17161.817302] [] kthread+0xcf/0xe0 [17161.823791] [] ? kthread_create_on_node+0x140/0x140 [17161.832133] [] ret_from_fork+0x58/0x90 [17161.839202] [] ? kthread_create_on_node+0x140/0x140 [17161.847520] Code: 00 00 31 c0 0f b7 56 10 48 89 e5 48 83 39 3f 76 07 0f b7 46 30 c1 e0 10 09 d0 5d c3 66 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 <48> 8b 8f a8 03 00 00 31 c0 0f b7 56 1c 48 89 e5 48 83 39 3f 76 [17189.441279] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [17189.450964] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [17189.582949] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [17189.596242] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [17189.612514] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [17189.621903] RIP: 0010:[] [] osd_inode_iteration+0x4b3/0xcc0 [osd_ldiskfs] [17189.634156] RSP: 0018:ffff880703603d18 EFLAGS: 00000216 [17189.641102] RAX: ffff8800b2a37468 RBX: 000000009f8f2001 RCX: 0000000000000000 [17189.650096] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [17189.659075] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [17189.668039] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000001 [17189.676989] R13: ffff880703603df0 R14: 000000009f8f2001 R15: 0000000000000018 [17189.685938] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [17189.695972] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17189.703383] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [17189.712359] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17189.721333] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [17189.730286] Stack: [17189.733504] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [17189.742786] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [17189.752085] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [17189.761382] Call Trace: [17189.765113] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [17189.774286] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [17189.783158] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [17189.791828] [] ? wake_up_state+0x20/0x20 [17189.799035] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [17189.808388] [] kthread+0xcf/0xe0 [17189.814827] [] ? kthread_create_on_node+0x140/0x140 [17189.823104] [] ret_from_fork+0x58/0x90 [17189.830105] [] ? kthread_create_on_node+0x140/0x140 [17189.838374] Code: ff 48 8b bd 48 ff ff ff 48 8b 85 28 ff ff ff ff d0 85 c0 41 89 c5 75 1c 8b 85 58 ff ff ff 41 3b 04 24 72 10 48 8b 85 50 ff ff ff <44> 3b 30 0f 87 95 fd ff ff 45 31 ff e9 a2 fc ff ff 0f 1f 40 00 [17204.106783] INFO: rcu_sched self-detected stall on CPU { 24} (t=9780183 jiffies g=907112 c=907111 q=34923589) [17204.119225] Task dump for CPU 24: [17204.123964] OI_scrub R running task 0 5551 2 0x00000088 [17204.132904] ffff88083fde6dd0 00000000fb6db880 ffff88082da03db0 ffffffff810c47e8 [17204.142220] 0000000000000018 ffffffff81a1e7c0 ffff88082da03dc8 ffffffff810c8079 [17204.151532] 0000000000000009 ffff88082da03df8 ffffffff811378e0 ffff88082da101c0 [17204.160828] Call Trace: [17204.164536] [] sched_show_task+0xa8/0x110 [17204.172530] [] dump_cpu_task+0x39/0x70 [17204.179492] [] rcu_dump_cpu_stacks+0x90/0xd0 [17204.187024] [] rcu_check_callbacks+0x442/0x720 [17204.194724] [] ? tick_sched_handle.isra.13+0x60/0x60 [17204.202997] [] update_process_times+0x47/0x80 [17204.210561] [] tick_sched_handle.isra.13+0x25/0x60 [17204.218603] [] tick_sched_timer+0x41/0x70 [17204.225739] [] __hrtimer_run_queues+0xd2/0x260 [17204.233351] [] hrtimer_interrupt+0xb0/0x1e0 [17204.240639] [] ? call_softirq+0x1c/0x30 [17204.247520] [] local_apic_timer_interrupt+0x37/0x60 [17204.255551] [] smp_apic_timer_interrupt+0x3f/0x60 [17204.263381] [] apic_timer_interrupt+0x6d/0x80 [17204.270810] [] ? osd_inode_iteration+0x269/0xcc0 [osd_ldiskfs] [17204.280616] [] ? osd_inode_iteration+0x269/0xcc0 [osd_ldiskfs] [17204.289708] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [17204.298607] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [17204.307224] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [17204.315638] [] ? wake_up_state+0x20/0x20 [17204.322589] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [17204.331675] [] kthread+0xcf/0xe0 [17204.337853] [] ? kthread_create_on_node+0x140/0x140 [17204.345869] [] ret_from_fork+0x58/0x90 [17204.352649] [] ? kthread_create_on_node+0x140/0x140 [17229.442666] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [17229.452086] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [17229.581807] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [17229.594980] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [17229.611057] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [17229.620361] RIP: 0010:[] [] osd_inode_iteration+0x255/0xcc0 [osd_ldiskfs] [17229.632518] RSP: 0018:ffff880703603d18 EFLAGS: 00000286 [17229.639382] RAX: ffff8800b2a37468 RBX: 0000000000000000 RCX: 0000000000000000 [17229.648289] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [17229.657187] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [17229.666088] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffff880703603d57 [17229.674991] R13: 0000000000000004 R14: 000000009f8f2001 R15: 0000000000000018 [17229.683900] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [17229.693869] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17229.701224] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [17229.710135] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17229.719044] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [17229.727940] Stack: [17229.731112] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [17229.740358] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [17229.749614] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [17229.758877] Call Trace: [17229.762553] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [17229.771682] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [17229.780526] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [17229.789171] [] ? wake_up_state+0x20/0x20 [17229.796368] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [17229.805696] [] kthread+0xcf/0xe0 [17229.812102] [] ? kthread_create_on_node+0x140/0x140 [17229.820362] [] ret_from_fork+0x58/0x90 [17229.827350] [] ? kthread_create_on_node+0x140/0x140 [17229.835589] Code: 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 00 8b 75 98 48 8b 7d 88 e8 dc 2d f5 ff 48 85 c0 48 89 45 90 0f 84 b3 08 00 00 44 8b 7d a0 <48> 8b 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 e8 77 5b f4 ff 42 [17257.443641] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [17257.453249] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [17257.585260] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [17257.598565] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [17257.614871] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [17257.624286] RIP: 0010:[] [] osd_scrub_next+0x160/0x3c0 [osd_ldiskfs] [17257.636067] RSP: 0018:ffff880703603ce0 EFLAGS: 00000282 [17257.643032] RAX: 0000000000000004 RBX: ffff880703603c78 RCX: ffff880703603d98 [17257.652031] RDX: ffff880703603d78 RSI: 0000000000000000 RDI: ffff8803162f6000 [17257.661021] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [17257.669986] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000010 [17257.678931] R13: ffffffffa121ba64 R14: ffffffffffffff10 R15: ffff8803162f6000 [17257.687875] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [17257.697902] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17257.705304] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [17257.714269] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17257.723226] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [17257.732176] Stack: [17257.735386] ffff880419dc3a00 ffff8800b2a3746c 0000000000000000 00000000ffffffff [17257.744660] 0000000000000001 ffff880703603df0 ffffffffa121cff9 ffffffffa121b990 [17257.753962] ffffffffa1217a20 ffff8800b2a36000 00000000810d354f ffff8803162f6000 [17257.763253] Call Trace: [17257.766980] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [17257.776147] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [17257.785309] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [17257.794180] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [17257.802863] [] ? wake_up_state+0x20/0x20 [17257.810076] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [17257.819428] [] kthread+0xcf/0xe0 [17257.825855] [] ? kthread_create_on_node+0x140/0x140 [17257.834138] [] ret_from_fork+0x58/0x90 [17257.841155] [] ? kthread_create_on_node+0x140/0x140 [17257.849428] Code: 10 00 00 02 00 00 00 48 89 df e8 fc 69 47 e0 b8 05 00 00 00 48 8b 75 d0 65 48 33 34 25 28 00 00 00 0f 85 4c 02 00 00 48 83 c4 70 <5b> 41 5c 41 5d 41 5e 41 5f 5d c3 48 8b 05 be 58 9c ff 48 85 c0 [17285.444617] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [17285.454290] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [17285.586174] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [17285.599467] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [17285.615727] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [17285.625129] RIP: 0010:[] [] osd_scrub_next+0x15/0x3c0 [osd_ldiskfs] [17285.636786] RSP: 0018:ffff880703603cf0 EFLAGS: 00000287 [17285.643725] RAX: ffffffffa1217a20 RBX: 000000009f8f2001 RCX: ffff880703603d98 [17285.652700] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [17285.661663] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [17285.670620] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffff880703603c78 [17285.679569] R13: ffff880703603d08 R14: 0000000000000010 R15: ffffffffa121ba87 [17285.688525] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [17285.698557] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17285.705975] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [17285.714945] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17285.723911] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [17285.732859] Stack: [17285.736081] 0000000000000000 00000000ffffffff 0000000000000001 ffff880703603df0 [17285.745386] ffffffffa121cff9 ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 [17285.754702] 00000000810d354f ffff8803162f6000 ffff8800b2a37468 0000000020000000 [17285.764021] Call Trace: [17285.767748] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [17285.776911] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [17285.786081] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [17285.794954] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [17285.803621] [] ? wake_up_state+0x20/0x20 [17285.810842] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [17285.820201] [] kthread+0xcf/0xe0 [17285.826636] [] ? kthread_create_on_node+0x140/0x140 [17285.834937] [] ret_from_fork+0x58/0x90 [17285.841957] [] ? kthread_create_on_node+0x140/0x140 [17285.850238] Code: ff 48 83 c4 08 5b 41 5c 41 5d 41 5e 5d c3 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 89 e5 41 57 49 89 f7 41 56 41 55 49 89 fd <41> 54 49 89 d4 53 48 89 cb 48 83 ec 70 65 48 8b 04 25 28 00 00 [17313.445599] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [17313.455326] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [17313.587069] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [17313.600360] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [17313.616614] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [17313.626001] RIP: 0010:[] [] osd_scrub_exec+0x30/0x5b0 [osd_ldiskfs] [17313.637656] RSP: 0018:ffff880703603c78 EFLAGS: 00000296 [17313.644599] RAX: 00000000fb6db880 RBX: ffff880703603c37 RCX: 0000000000000000 [17313.653579] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [17313.662552] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [17313.671508] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [17313.680459] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [17313.689418] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [17313.699458] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17313.706858] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [17313.715825] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17313.724782] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [17313.733726] Stack: [17313.736936] ffff880703603d08 0000000000000001 ffffea00027e3c80 000000009f8f2001 [17313.746207] ffff880703603d57 ffff880703603d57 ffffffffa121b990 0000000000000000 [17313.755511] ffff880703603d78 ffff8800b2a36000 ffff8803162f6000 ffffffffffffff10 [17313.764803] Call Trace: [17313.768525] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [17313.777691] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [17313.786845] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [17313.795989] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [17313.804842] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [17313.813503] [] ? wake_up_state+0x20/0x20 [17313.820703] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [17313.830040] [] kthread+0xcf/0xe0 [17313.836458] [] ? kthread_create_on_node+0x140/0x140 [17313.844728] [] ret_from_fork+0x58/0x90 [17313.851722] [] ? kthread_create_on_node+0x140/0x140 [17313.859980] Code: 55 48 89 e5 41 57 49 89 f7 41 56 41 55 41 54 45 31 e4 53 48 83 ec 68 48 8b 9e 08 10 00 00 4c 89 45 90 65 48 8b 04 25 28 00 00 00 <48> 89 45 d0 31 c0 48 8d 43 08 48 85 db 4c 0f 45 e0 41 83 f9 04 [17341.446567] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [17341.456253] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [17341.587740] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [17341.600997] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [17341.617187] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [17341.626545] RIP: 0010:[] [] osd_inode_iteration+0x2be/0xcc0 [osd_ldiskfs] [17341.638749] RSP: 0018:ffff880703603d18 EFLAGS: 00000297 [17341.645671] RAX: 0000000000000004 RBX: 0000000000000001 RCX: ffff880703603d98 [17341.654629] RDX: ffff880703603d78 RSI: 0000000000000000 RDI: ffff8803162f6000 [17341.663568] RBP: ffff880703603df0 R08: 0000000000000001 R09: 0000000000000004 [17341.672512] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [17341.681448] R13: ffffea00027e3c80 R14: 000000009f8f2001 R15: ffff880703603d08 [17341.690395] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [17341.700416] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17341.707811] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [17341.716748] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17341.725693] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [17341.734629] Stack: [17341.737828] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [17341.747131] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [17341.756415] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [17341.765690] Call Trace: [17341.769396] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [17341.778552] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [17341.787400] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [17341.796058] [] ? wake_up_state+0x20/0x20 [17341.803260] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [17341.812615] [] kthread+0xcf/0xe0 [17341.819042] [] ? kthread_create_on_node+0x140/0x140 [17341.827320] [] ret_from_fork+0x58/0x90 [17341.834328] [] ? kthread_create_on_node+0x140/0x140 [17341.842602] Code: 8b b5 38 ff ff ff 48 8b bd 48 ff ff ff 48 8b 85 30 ff ff ff ff d0 83 f8 03 41 89 c5 0f 84 4b 02 00 00 0f 8e 15 01 00 00 83 f8 05 <66> 90 0f 84 8a 02 00 00 83 f8 06 0f 85 a1 01 00 00 48 8b 7d 90 [17369.447545] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [17369.457240] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [17369.589161] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [17369.602474] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [17369.618763] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [17369.628167] RIP: 0010:[] [] osd_scrub_exec+0x5/0x5b0 [osd_ldiskfs] [17369.639725] RSP: 0018:ffff880703603d10 EFLAGS: 00000293 [17369.646683] RAX: ffffffffa121b990 RBX: 0000000000000001 RCX: 0000000000000000 [17369.655666] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [17369.664640] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [17369.673598] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [17369.682544] R13: ffffea00027e3c80 R14: 000000009f8f2001 R15: 0000000000000018 [17369.691489] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [17369.701512] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17369.708926] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [17369.717891] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17369.726852] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [17369.735797] Stack: [17369.739007] ffffffffa121d1e9 ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 [17369.748290] 00000000810d354f ffff8803162f6000 ffff8800b2a37468 0000000020000000 [17369.757577] 010000000000000c 0000000000000000 0000000000000000 ffff8800b2a36000 [17369.766867] Call Trace: [17369.770589] [] ? osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [17369.779964] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [17369.789128] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [17369.798005] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [17369.806679] [] ? wake_up_state+0x20/0x20 [17369.813895] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [17369.823256] [] kthread+0xcf/0xe0 [17369.829688] [] ? kthread_create_on_node+0x140/0x140 [17369.837968] [] ret_from_fork+0x58/0x90 [17369.844979] [] ? kthread_create_on_node+0x140/0x140 [17369.853247] Code: 05 85 a5 02 00 00 00 02 00 89 54 24 08 48 8b 40 40 ba 07 00 00 00 48 89 04 24 31 c0 e8 a5 03 98 ff e9 c7 fd ff ff 66 66 66 66 90 <55> 48 89 e5 41 57 49 89 f7 41 56 41 55 41 54 45 31 e4 53 48 83 [17384.116055] INFO: rcu_sched self-detected stall on CPU { 24} (t=9960186 jiffies g=907112 c=907111 q=36007586) [17384.128510] Task dump for CPU 24: [17384.133240] OI_scrub R running task 0 5551 2 0x00000088 [17384.142170] ffff88083fde6dd0 00000000fb6db880 ffff88082da03db0 ffffffff810c47e8 [17384.151510] 0000000000000018 ffffffff81a1e7c0 ffff88082da03dc8 ffffffff810c8079 [17384.160813] 0000000000000009 ffff88082da03df8 ffffffff811378e0 ffff88082da101c0 [17384.170100] Call Trace: [17384.173791] [] sched_show_task+0xa8/0x110 [17384.181765] [] dump_cpu_task+0x39/0x70 [17384.188728] [] rcu_dump_cpu_stacks+0x90/0xd0 [17384.196249] [] rcu_check_callbacks+0x442/0x720 [17384.203938] [] ? tick_sched_handle.isra.13+0x60/0x60 [17384.212200] [] update_process_times+0x47/0x80 [17384.219758] [] tick_sched_handle.isra.13+0x25/0x60 [17384.227774] [] tick_sched_timer+0x41/0x70 [17384.234899] [] __hrtimer_run_queues+0xd2/0x260 [17384.242491] [] hrtimer_interrupt+0xb0/0x1e0 [17384.249759] [] ? call_softirq+0x1c/0x30 [17384.256644] [] local_apic_timer_interrupt+0x37/0x60 [17384.264672] [] smp_apic_timer_interrupt+0x3f/0x60 [17384.272501] [] apic_timer_interrupt+0x6d/0x80 [17384.279927] [] ? osd_inode_iteration+0x255/0xcc0 [osd_ldiskfs] [17384.289758] [] ? osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [17384.298859] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [17384.307771] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [17384.316389] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [17384.324803] [] ? wake_up_state+0x20/0x20 [17384.331757] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [17384.340846] [] kthread+0xcf/0xe0 [17384.347022] [] ? kthread_create_on_node+0x140/0x140 [17384.355058] [] ret_from_fork+0x58/0x90 [17384.361818] [] ? kthread_create_on_node+0x140/0x140 [17409.448937] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [17409.458413] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [17409.588105] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [17409.601286] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [17409.617354] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [17409.626652] RIP: 0010:[] [] osd_scrub_exec+0xd4/0x5b0 [osd_ldiskfs] [17409.638225] RSP: 0018:ffff880703603c78 EFLAGS: 00000286 [17409.645086] RAX: ffff8803162f1008 RBX: ffff880703603c37 RCX: 0000000000000000 [17409.653998] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [17409.662903] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [17409.671801] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [17409.680702] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [17409.689601] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [17409.699577] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17409.706929] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [17409.715842] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17409.724750] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [17409.733649] Stack: [17409.736818] ffffea00027e3c80 000000009f8f2001 ffff880703603df0 ffff880703603d57 [17409.746072] ffff880703603d57 000000009f8f2001 0000000000000004 ffff880703603d57 [17409.755323] ffff88042bd78800 ffff88042bd78800 00000000000072cf ffff880419dc3a00 [17409.764563] Call Trace: [17409.768244] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [17409.777376] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [17409.786502] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [17409.795337] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [17409.803987] [] ? wake_up_state+0x20/0x20 [17409.811171] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [17409.820487] [] kthread+0xcf/0xe0 [17409.826869] [] ? kthread_create_on_node+0x140/0x140 [17409.835107] [] ret_from_fork+0x58/0x90 [17409.842079] [] ? kthread_create_on_node+0x140/0x140 [17409.850321] Code: 4d 88 4c 89 ef e8 1d fb 46 e0 41 83 87 68 14 00 00 01 49 83 87 88 11 00 00 01 4c 89 ef e8 f5 9f e9 df 44 8b 4d 88 48 85 db 74 23 83 18 0a 00 00 08 74 1a 4d 85 e4 74 15 41 8b 87 6c 14 00 00 [17437.449913] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [17437.459488] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [17437.591103] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [17437.604392] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [17437.620630] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [17437.630012] RIP: 0010:[] [] osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [17437.641760] RSP: 0018:ffff880703603d10 EFLAGS: 00000287 [17437.648700] RAX: ffffffffa1217a20 RBX: ffff880703603d98 RCX: ffff880703603d98 [17437.657669] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [17437.666626] RBP: ffff880703603df0 R08: 0000000000000001 R09: 0000000000000004 [17437.675572] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000001 [17437.684510] R13: 0000000000000004 R14: 000000009f8f2001 R15: 0000000000000018 [17437.693448] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [17437.703452] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17437.710843] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [17437.719800] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17437.728751] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [17437.737678] Stack: [17437.740870] ffffffffa121cff9 ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 [17437.750142] 00000000810d354f ffff8803162f6000 ffff8800b2a37468 0000000020000000 [17437.759436] 010000000000000c 0000000000000000 0000000000000000 ffff8800b2a36000 [17437.768723] Call Trace: [17437.772419] [] ? osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [17437.781771] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [17437.790926] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [17437.799792] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [17437.808453] [] ? wake_up_state+0x20/0x20 [17437.815670] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [17437.825015] [] kthread+0xcf/0xe0 [17437.831443] [] ? kthread_create_on_node+0x140/0x140 [17437.839717] [] ret_from_fork+0x58/0x90 [17437.846728] [] ? kthread_create_on_node+0x140/0x140 [17437.854994] Code: f7 48 8d 04 80 48 c1 e0 03 48 8d 4c 02 10 48 01 c2 e8 25 fc ff ff 48 83 c4 08 5b 41 5c 41 5d 41 5e 5d c3 0f 1f 84 00 00 00 00 00 <66> 66 66 66 90 55 48 89 e5 41 57 49 89 f7 41 56 41 55 49 89 fd [17465.450888] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [17465.460565] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [17465.592238] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [17465.605522] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [17465.621773] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [17465.631157] RIP: 0010:[] [] osd_scrub_exec+0x95/0x5b0 [osd_ldiskfs] [17465.642800] RSP: 0018:ffff880703603d08 EFLAGS: 00000292 [17465.649736] RAX: 0000000000000000 RBX: ffff880703603d57 RCX: 0000000000000000 [17465.658707] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [17465.667668] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [17465.676619] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [17465.685568] R13: ffffea00027e3c80 R14: ffff880703603c78 R15: ffff880703603d08 [17465.694519] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [17465.704552] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17465.711957] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [17465.720924] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17465.729884] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [17465.738832] Stack: [17465.742046] ffff880703603df0 ffffffffa121d1e9 ffffffffa121b990 ffffffffa1217a20 [17465.751348] ffff8800b2a36000 00000000810d354f ffff8803162f6000 ffff8800b2a37468 [17465.760658] 0000000020000000 010000000000000c 0000000000000000 0000000000000000 [17465.769963] Call Trace: [17465.773694] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [17465.782872] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [17465.792038] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [17465.800911] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [17465.809586] [] ? wake_up_state+0x20/0x20 [17465.816791] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [17465.826136] [] kthread+0xcf/0xe0 [17465.832572] [] ? kthread_create_on_node+0x140/0x140 [17465.840857] [] ret_from_fork+0x58/0x90 [17465.847857] [] ? kthread_create_on_node+0x140/0x140 [17465.856125] Code: 01 00 00 41 80 a7 74 14 00 00 fe 48 8b 4d d0 65 48 33 0c 25 28 00 00 00 0f 85 b1 03 00 00 48 83 c4 68 5b 41 5c 41 5d 41 5e 41 5f <5d> c3 66 0f 1f 84 00 00 00 00 00 4c 8d ae f0 10 00 00 44 89 4d [17493.451864] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [17493.461515] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [17493.593257] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [17493.606540] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [17493.622784] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [17493.632165] RIP: 0010:[] [] osd_scrub_exec+0xf9/0x5b0 [osd_ldiskfs] [17493.643811] RSP: 0018:ffff880703603c78 EFLAGS: 00000246 [17493.650757] RAX: 0000000000000000 RBX: ffff880703603c37 RCX: 0000000000000000 [17493.659730] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [17493.668691] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [17493.677640] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [17493.686587] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [17493.695534] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [17493.705562] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17493.712967] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [17493.721932] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17493.730885] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [17493.739827] Stack: [17493.743035] ffff880703603d08 0000000000000001 ffffea00027e3c80 000000009f8f2001 [17493.752332] ffff880703603d57 ffff880703603d57 00000000000072ce ffff88042bd78800 [17493.761636] 00000000000072ce ffff880419dc3a00 ffff88042bd79800 ffffffffffffff10 [17493.770939] Call Trace: [17493.774665] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [17493.783830] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [17493.792990] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [17493.801875] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [17493.810551] [] ? wake_up_state+0x20/0x20 [17493.817765] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [17493.827121] [] kthread+0xcf/0xe0 [17493.833559] [] ? kthread_create_on_node+0x140/0x140 [17493.841842] [] ret_from_fork+0x58/0x90 [17493.848852] [] ? kthread_create_on_node+0x140/0x140 [17493.857125] Code: 88 48 85 db 74 23 f6 83 18 0a 00 00 08 74 1a 4d 85 e4 74 15 41 8b 87 6c 14 00 00 41 39 84 24 0c 0a 00 00 0f 82 d9 00 00 00 31 c0 <41> f6 87 74 14 00 00 04 0f 85 6e ff ff ff 41 83 f9 02 0f 84 64 [17521.452838] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [17521.462554] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [17521.594259] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [17521.607548] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [17521.623805] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [17521.633190] RIP: 0010:[] [] osd_scrub_next+0x160/0x3c0 [osd_ldiskfs] [17521.644933] RSP: 0018:ffff880703603ce0 EFLAGS: 00000282 [17521.651874] RAX: 0000000000000004 RBX: ffff880703603c70 RCX: ffff880703603d98 [17521.660854] RDX: ffff880703603d78 RSI: 0000000000000000 RDI: ffff8803162f6000 [17521.669819] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [17521.678769] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000010 [17521.687714] R13: ffffffffa1217b6d R14: ffffffffffffff10 R15: ffff8803162f6000 [17521.696669] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [17521.706705] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17521.714110] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [17521.723074] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17521.732030] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [17521.740974] Stack: [17521.744214] ffff880419dc3a00 ffff8800b2a3746c 0000000000000000 00000000ffffffff [17521.753517] 0000000000000001 ffff880703603df0 ffffffffa121cff9 ffffffffa121b990 [17521.762820] ffffffffa1217a20 ffff8800b2a36000 00000000810d354f ffff8803162f6000 [17521.772127] Call Trace: [17521.775847] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [17521.785015] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [17521.794178] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [17521.803043] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [17521.811720] [] ? wake_up_state+0x20/0x20 [17521.818931] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [17521.828288] [] kthread+0xcf/0xe0 [17521.834717] [] ? kthread_create_on_node+0x140/0x140 [17521.842993] [] ret_from_fork+0x58/0x90 [17521.850002] [] ? kthread_create_on_node+0x140/0x140 [17521.858271] Code: 10 00 00 02 00 00 00 48 89 df e8 fc 69 47 e0 b8 05 00 00 00 48 8b 75 d0 65 48 33 34 25 28 00 00 00 0f 85 4c 02 00 00 48 83 c4 70 <5b> 41 5c 41 5d 41 5e 41 5f 5d c3 48 8b 05 be 58 9c ff 48 85 c0 [17549.453813] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [17549.463444] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [17549.595297] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [17549.608586] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [17549.624848] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [17549.634232] RIP: 0010:[] [] osd_scrub_exec+0xf7/0x5b0 [osd_ldiskfs] [17549.645879] RSP: 0018:ffff880703603c78 EFLAGS: 00000246 [17549.652819] RAX: 0000000000280001 RBX: ffff880703603c37 RCX: 0000000000000000 [17549.661797] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [17549.670760] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [17549.679713] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [17549.688660] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [17549.697615] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [17549.707637] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17549.715038] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [17549.724006] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17549.732967] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [17549.741912] Stack: [17549.745162] 0000000000000004 0000000000000001 0000000000000004 ffff880703603d98 [17549.754443] ffff880703603d57 0000000000000000 ffff8803162f6000 ffffffffffffff10 [17549.763734] ffffffffa1217b80 0000000000000010 0000000000000282 ffff880703603ce0 [17549.773025] Call Trace: [17549.776745] [] ? osd_scrub_next+0x160/0x3c0 [osd_ldiskfs] [17549.785618] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [17549.794778] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [17549.803931] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [17549.812787] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [17549.821451] [] ? wake_up_state+0x20/0x20 [17549.828658] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [17549.838009] [] kthread+0xcf/0xe0 [17549.844436] [] ? kthread_create_on_node+0x140/0x140 [17549.852710] [] ret_from_fork+0x58/0x90 [17549.859723] [] ? kthread_create_on_node+0x140/0x140 [17549.867991] Code: 8b 4d 88 48 85 db 74 23 f6 83 18 0a 00 00 08 74 1a 4d 85 e4 74 15 41 8b 87 6c 14 00 00 41 39 84 24 0c 0a 00 00 0f 82 d9 00 00 00 <31> c0 41 f6 87 74 14 00 00 04 0f 85 6e ff ff ff 41 83 f9 02 0f [17564.125324] INFO: rcu_sched self-detected stall on CPU { 24} (t=10140189 jiffies g=907112 c=907111 q=37172155) [17564.137885] Task dump for CPU 24: [17564.142603] OI_scrub R running task 0 5551 2 0x00000088 [17564.151530] ffff88083fde6dd0 00000000fb6db880 ffff88082da03db0 ffffffff810c47e8 [17564.160831] 0000000000000018 ffffffff81a1e7c0 ffff88082da03dc8 ffffffff810c8079 [17564.170119] 0000000000000009 ffff88082da03df8 ffffffff811378e0 ffff88082da101c0 [17564.179390] Call Trace: [17564.183065] [] sched_show_task+0xa8/0x110 [17564.191026] [] dump_cpu_task+0x39/0x70 [17564.197963] [] rcu_dump_cpu_stacks+0x90/0xd0 [17564.205457] [] rcu_check_callbacks+0x442/0x720 [17564.213128] [] ? tick_sched_handle.isra.13+0x60/0x60 [17564.221357] [] update_process_times+0x47/0x80 [17564.228897] [] tick_sched_handle.isra.13+0x25/0x60 [17564.236891] [] tick_sched_timer+0x41/0x70 [17564.243989] [] __hrtimer_run_queues+0xd2/0x260 [17564.251552] [] hrtimer_interrupt+0xb0/0x1e0 [17564.258822] [] ? call_softirq+0x1c/0x30 [17564.265682] [] local_apic_timer_interrupt+0x37/0x60 [17564.273701] [] smp_apic_timer_interrupt+0x3f/0x60 [17564.281515] [] apic_timer_interrupt+0x6d/0x80 [17564.288943] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [17564.298273] [] ? osd_scrub_next+0x12/0x3c0 [osd_ldiskfs] [17564.306786] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [17564.315689] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [17564.324580] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [17564.333187] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [17564.341580] [] ? wake_up_state+0x20/0x20 [17564.348522] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [17564.357613] [] kthread+0xcf/0xe0 [17564.363780] [] ? kthread_create_on_node+0x140/0x140 [17564.371785] [] ret_from_fork+0x58/0x90 [17564.378527] [] ? kthread_create_on_node+0x140/0x140 [17589.455207] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [17589.464618] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [17589.594209] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [17589.607379] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [17589.623404] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [17589.632678] RIP: 0010:[] [] osd_inode_iteration+0x282/0xcc0 [osd_ldiskfs] [17589.644822] RSP: 0018:ffff880703603d18 EFLAGS: 00000287 [17589.651677] RAX: ffff88042bd78800 RBX: 0000000000000001 RCX: ffff88042bd78800 [17589.660571] RDX: 00000000000072cf RSI: ffff880419dc3a00 RDI: ffff88042bd79800 [17589.669465] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [17589.678361] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [17589.687251] R13: ffffea00027e3c80 R14: 000000009f8f2001 R15: ffff880703603d08 [17589.696140] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [17589.706102] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17589.713445] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [17589.722355] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17589.731267] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [17589.740163] Stack: [17589.743331] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [17589.752585] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [17589.761840] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [17589.771096] Call Trace: [17589.774783] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [17589.783919] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [17589.792766] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [17589.801424] [] ? wake_up_state+0x20/0x20 [17589.808610] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [17589.817942] [] kthread+0xcf/0xe0 [17589.824343] [] ? kthread_create_on_node+0x140/0x140 [17589.832598] [] ret_from_fork+0x58/0x90 [17589.839588] [] ? kthread_create_on_node+0x140/0x140 [17589.847836] Code: 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 e8 77 5b f4 ff 42 8d 14 38 48 8b 45 88 48 8b 80 a8 03 00 00 48 3b 50 20 0f 83 57 01 00 00 <44> 0f b6 85 67 ff ff ff 48 8d 4d a8 48 8d 55 88 48 8b b5 38 ff [17617.456183] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [17617.465790] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [17617.597696] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [17617.611013] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [17617.627317] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [17617.636736] RIP: 0010:[] [] osd_scrub_next+0x12/0x3c0 [osd_ldiskfs] [17617.648425] RSP: 0018:ffff880703603cf0 EFLAGS: 00000287 [17617.655392] RAX: ffffffffa1217a20 RBX: 000000009f8f2001 RCX: ffff880703603d98 [17617.664392] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [17617.673382] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [17617.682345] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffff880703603c78 [17617.691305] R13: 0000000000000282 R14: 0000000000000010 R15: ffffffffa121ba79 [17617.700266] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [17617.710295] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17617.717711] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [17617.726676] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17617.735637] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [17617.744582] Stack: [17617.747799] 0000000000000000 00000000ffffffff 0000000000000001 ffff880703603df0 [17617.757077] ffffffffa121cff9 ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 [17617.766353] 00000000810d354f ffff8803162f6000 ffff8800b2a37468 0000000020000000 [17617.775641] Call Trace: [17617.779364] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [17617.788535] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [17617.797700] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [17617.806582] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [17617.815274] [] ? wake_up_state+0x20/0x20 [17617.822486] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [17617.831839] [] kthread+0xcf/0xe0 [17617.838270] [] ? kthread_create_on_node+0x140/0x140 [17617.846558] [] ret_from_fork+0x58/0x90 [17617.853568] [] ? kthread_create_on_node+0x140/0x140 [17617.861844] Code: 25 fc ff ff 48 83 c4 08 5b 41 5c 41 5d 41 5e 5d c3 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 89 e5 41 57 49 89 f7 41 56 41 55 <49> 89 fd 41 54 49 89 d4 53 48 89 cb 48 83 ec 70 65 48 8b 04 25 [17645.457159] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [17645.466823] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [17645.598490] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [17645.611778] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [17645.628032] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [17645.637423] RIP: 0010:[] [] osd_scrub_exec+0xe9/0x5b0 [osd_ldiskfs] [17645.649071] RSP: 0018:ffff880703603c78 EFLAGS: 00000282 [17645.656010] RAX: 0000000000280001 RBX: ffff880703603c37 RCX: 0000000000000000 [17645.664987] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [17645.673953] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [17645.682913] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [17645.691863] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [17645.700815] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [17645.710845] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17645.718250] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [17645.727218] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17645.736176] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [17645.745122] Stack: [17645.748336] ffffea00027e3c80 000000009f8f2001 0000000000000004 0000000000000001 [17645.757611] ffff880703603d57 ffff880703603d98 ffff880703603d78 ffff8800b2a36000 [17645.766899] ffff8803162f6000 ffffffffffffff10 ffffffffa1217a32 0000000000000010 [17645.776195] Call Trace: [17645.779931] [] ? osd_scrub_next+0x12/0x3c0 [osd_ldiskfs] [17645.788714] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [17645.797880] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [17645.807037] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [17645.815898] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [17645.824569] [] ? wake_up_state+0x20/0x20 [17645.831780] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [17645.841129] [] kthread+0xcf/0xe0 [17645.847561] [] ? kthread_create_on_node+0x140/0x140 [17645.855839] [] ret_from_fork+0x58/0x90 [17645.862852] [] ? kthread_create_on_node+0x140/0x140 [17645.871131] Code: 88 11 00 00 01 4c 89 ef e8 f5 9f e9 df 44 8b 4d 88 48 85 db 74 23 f6 83 18 0a 00 00 08 74 1a 4d 85 e4 74 15 41 8b 87 6c 14 00 00 <41> 39 84 24 0c 0a 00 00 0f 82 d9 00 00 00 31 c0 41 f6 87 74 14 [17673.458133] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [17673.467822] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [17673.599502] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [17673.612774] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [17673.629009] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [17673.638378] RIP: 0010:[] [] osd_scrub_next+0x15/0x3c0 [osd_ldiskfs] [17673.650017] RSP: 0018:ffff880703603cf0 EFLAGS: 00000287 [17673.656954] RAX: ffffffffa1217a20 RBX: 000000009f8f2001 RCX: ffff880703603d98 [17673.665919] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [17673.674876] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [17673.683829] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffff880703603c78 [17673.692790] R13: 0000000000000286 R14: 0000000000000010 R15: ffffffffa121ba64 [17673.701743] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [17673.711770] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17673.719177] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [17673.728151] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17673.737101] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [17673.746049] Stack: [17673.749259] 0000000000000000 00000000ffffffff 0000000000000001 ffff880703603df0 [17673.758566] ffffffffa121cff9 ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 [17673.767851] 00000000810d354f ffff8803162f6000 ffff8800b2a37468 0000000020000000 [17673.777142] Call Trace: [17673.780869] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [17673.790033] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [17673.799187] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [17673.808053] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [17673.816722] [] ? wake_up_state+0x20/0x20 [17673.823929] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [17673.833280] [] kthread+0xcf/0xe0 [17673.839708] [] ? kthread_create_on_node+0x140/0x140 [17673.847990] [] ret_from_fork+0x58/0x90 [17673.855000] [] ? kthread_create_on_node+0x140/0x140 [17673.863267] Code: ff 48 83 c4 08 5b 41 5c 41 5d 41 5e 5d c3 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 89 e5 41 57 49 89 f7 41 56 41 55 49 89 fd <41> 54 49 89 d4 53 48 89 cb 48 83 ec 70 65 48 8b 04 25 28 00 00 [17701.459109] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [17701.468808] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [17701.600530] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [17701.613817] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [17701.630071] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [17701.639455] RIP: 0010:[] [] osd_inode_iteration+0x4a6/0xcc0 [osd_ldiskfs] [17701.651687] RSP: 0018:ffff880703603d18 EFLAGS: 00000246 [17701.658629] RAX: 0000000020000000 RBX: 000000009f8f2001 RCX: 0000000000000000 [17701.667607] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [17701.676575] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [17701.685537] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000001 [17701.694485] R13: ffff880703603d08 R14: 000000009f8f2001 R15: ffffea00027e3c80 [17701.703436] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [17701.713465] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17701.720871] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [17701.729838] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17701.738797] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [17701.747743] Stack: [17701.750956] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [17701.760245] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [17701.769552] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [17701.778847] Call Trace: [17701.782571] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [17701.791737] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [17701.800600] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [17701.809274] [] ? wake_up_state+0x20/0x20 [17701.816482] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [17701.825831] [] kthread+0xcf/0xe0 [17701.832265] [] ? kthread_create_on_node+0x140/0x140 [17701.840537] [] ret_from_fork+0x58/0x90 [17701.847552] [] ? kthread_create_on_node+0x140/0x140 [17701.855830] Code: 8b 4d a8 48 8d 55 88 48 8b b5 38 ff ff ff 48 8b bd 48 ff ff ff 48 8b 85 28 ff ff ff ff d0 85 c0 41 89 c5 75 1c 8b 85 58 ff ff ff <41> 3b 04 24 72 10 48 8b 85 50 ff ff ff 44 3b 30 0f 87 95 fd ff [17724.700936] Lustre: 4594:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1494371069/real 1494371069] req@ffff8804d4400900 x1566943532435152/t0(0) o104->soaked-MDT0000@192.168.1.120@o2ib100:15/16 lens 296/224 e 0 to 1 dl 1494371076 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [17729.460085] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [17729.469802] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [17729.601731] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [17729.615039] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [17729.631338] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [17729.640742] RIP: 0010:[] [] osd_inode_iteration+0x2a0/0xcc0 [osd_ldiskfs] [17729.653020] RSP: 0018:ffff880703603d18 EFLAGS: 00000287 [17729.659998] RAX: ffff88042bd78800 RBX: 000000009f8f2001 RCX: ffff880703603d98 [17729.669010] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [17729.678018] RBP: ffff880703603df0 R08: 0000000000000001 R09: 0000000000000004 [17729.687022] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000001 [17729.696029] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [17729.705037] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [17729.715120] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17729.722574] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [17729.731580] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17729.740583] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [17729.749579] Stack: [17729.752852] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [17729.762211] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [17729.771544] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [17729.780879] Call Trace: [17729.784648] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [17729.793859] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [17729.802775] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [17729.811495] [] ? wake_up_state+0x20/0x20 [17729.818755] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [17729.828153] [] kthread+0xcf/0xe0 [17729.834633] [] ? kthread_create_on_node+0x140/0x140 [17729.842958] [] ret_from_fork+0x58/0x90 [17729.850025] [] ? kthread_create_on_node+0x140/0x140 [17729.858347] Code: 03 00 00 48 3b 50 20 0f 83 57 01 00 00 44 0f b6 85 67 ff ff ff 48 8d 4d a8 48 8d 55 88 48 8b b5 38 ff ff ff 48 8b bd 48 ff ff ff <48> 8b 85 30 ff ff ff ff d0 83 f8 03 41 89 c5 0f 84 4b 02 00 00 [17744.134595] INFO: rcu_sched self-detected stall on CPU { 24} (t=10320192 jiffies g=907112 c=907111 q=38226674) [17744.147187] Task dump for CPU 24: [17744.151935] OI_scrub R running task 0 5551 2 0x00000088 [17744.160874] ffff88083fde6dd0 00000000fb6db880 ffff88082da03db0 ffffffff810c47e8 [17744.170209] 0000000000000018 ffffffff81a1e7c0 ffff88082da03dc8 ffffffff810c8079 [17744.179508] 0000000000000009 ffff88082da03df8 ffffffff811378e0 ffff88082da101c0 [17744.188823] Call Trace: [17744.192538] [] sched_show_task+0xa8/0x110 [17744.200530] [] dump_cpu_task+0x39/0x70 [17744.207501] [] rcu_dump_cpu_stacks+0x90/0xd0 [17744.215041] [] rcu_check_callbacks+0x442/0x720 [17744.222761] [] ? tick_sched_handle.isra.13+0x60/0x60 [17744.231029] [] update_process_times+0x47/0x80 [17744.238603] [] tick_sched_handle.isra.13+0x25/0x60 [17744.246634] [] tick_sched_timer+0x41/0x70 [17744.253769] [] __hrtimer_run_queues+0xd2/0x260 [17744.261368] [] hrtimer_interrupt+0xb0/0x1e0 [17744.268661] [] ? call_softirq+0x1c/0x30 [17744.275540] [] local_apic_timer_interrupt+0x37/0x60 [17744.283575] [] smp_apic_timer_interrupt+0x3f/0x60 [17744.291402] [] apic_timer_interrupt+0x6d/0x80 [17744.298835] [] ? osd_inode_iteration+0x49e/0xcc0 [osd_ldiskfs] [17744.308655] [] ? osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [17744.317743] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [17744.326650] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [17744.335267] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [17744.343684] [] ? wake_up_state+0x20/0x20 [17744.350639] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [17744.359736] [] kthread+0xcf/0xe0 [17744.365913] [] ? kthread_create_on_node+0x140/0x140 [17744.373938] [] ret_from_fork+0x58/0x90 [17744.380699] [] ? kthread_create_on_node+0x140/0x140 [17769.461478] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [17769.470909] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [17769.600770] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [17769.613945] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [17769.630034] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [17769.639376] RIP: 0010:[] [] osd_scrub_next+0x2f/0x3c0 [osd_ldiskfs] [17769.650959] RSP: 0018:ffff880703603c70 EFLAGS: 00000282 [17769.657820] RAX: 00000000fb6db880 RBX: ffff880703603c37 RCX: ffff880703603d98 [17769.666738] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [17769.675636] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [17769.684536] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [17769.693437] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [17769.702332] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [17769.712318] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17769.719672] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [17769.728608] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17769.737518] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [17769.746418] Stack: [17769.749586] 000000009f8f2001 ffff880703603d08 000000009f8f2001 ffff880703603df0 [17769.758846] 000000009f8f2001 ffff880703603d57 000000009f8f2001 0000000000000004 [17769.768098] 0000000000000001 0000000000000004 ffff880703603d98 ffff880703603d78 [17769.777331] Call Trace: [17769.781017] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [17769.790151] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [17769.799290] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [17769.808205] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [17769.816850] [] ? wake_up_state+0x20/0x20 [17769.824031] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [17769.833347] [] kthread+0xcf/0xe0 [17769.839748] [] ? kthread_create_on_node+0x140/0x140 [17769.848000] [] ret_from_fork+0x58/0x90 [17769.855059] [] ? kthread_create_on_node+0x140/0x140 [17769.863354] Code: 90 55 48 89 e5 41 57 49 89 f7 41 56 41 55 49 89 fd 41 54 49 89 d4 53 48 89 cb 48 83 ec 70 65 48 8b 04 25 28 00 00 00 48 89 45 d0 <31> c0 48 8b 05 f8 59 9c ff 48 85 c0 75 7b 41 f6 87 58 10 00 00 [17797.462453] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [17797.472097] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [17797.603810] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [17797.617091] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [17797.633327] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [17797.642696] RIP: 0010:[] [] osd_inode_iteration+0x2ac/0xcc0 [osd_ldiskfs] [17797.654919] RSP: 0018:ffff880703603d18 EFLAGS: 00000202 [17797.661846] RAX: 0000000000000004 RBX: 0000000000000001 RCX: ffff880703603d98 [17797.670817] RDX: ffff880703603d78 RSI: 0000000000000000 RDI: ffff8803162f6000 [17797.679783] RBP: ffff880703603df0 R08: 0000000000000001 R09: 0000000000000004 [17797.688727] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [17797.697665] R13: ffffea00027e3c80 R14: 000000009f8f2001 R15: 0000000000000018 [17797.706604] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [17797.716624] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17797.724020] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [17797.732964] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17797.741906] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [17797.750835] Stack: [17797.754031] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [17797.763319] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [17797.772648] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [17797.781921] Call Trace: [17797.785633] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [17797.794842] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [17797.803699] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [17797.812364] [] ? wake_up_state+0x20/0x20 [17797.819566] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [17797.828924] [] kthread+0xcf/0xe0 [17797.835349] [] ? kthread_create_on_node+0x140/0x140 [17797.843623] [] ret_from_fork+0x58/0x90 [17797.850633] [] ? kthread_create_on_node+0x140/0x140 [17797.858899] Code: 00 44 0f b6 85 67 ff ff ff 48 8d 4d a8 48 8d 55 88 48 8b b5 38 ff ff ff 48 8b bd 48 ff ff ff 48 8b 85 30 ff ff ff ff d0 83 f8 03 <41> 89 c5 0f 84 4b 02 00 00 0f 8e 15 01 00 00 83 f8 05 66 90 0f [17825.463429] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [17825.473162] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [17825.605007] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [17825.618309] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [17825.634598] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [17825.643996] RIP: 0010:[] [] osd_inode_iteration+0x251/0xcc0 [osd_ldiskfs] [17825.656244] RSP: 0018:ffff880703603d18 EFLAGS: 00000286 [17825.663210] RAX: ffff8800b2a37468 RBX: 000000009f8f2001 RCX: 0000000000000000 [17825.672196] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [17825.681171] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [17825.690130] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [17825.699078] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [17825.708023] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [17825.718053] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17825.725460] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [17825.734434] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17825.743396] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [17825.752345] Stack: [17825.755560] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [17825.764843] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [17825.774134] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [17825.783419] Call Trace: [17825.787152] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [17825.796319] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [17825.805193] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [17825.813875] [] ? wake_up_state+0x20/0x20 [17825.821091] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [17825.830453] [] kthread+0xcf/0xe0 [17825.836884] [] ? kthread_create_on_node+0x140/0x140 [17825.845163] [] ret_from_fork+0x58/0x90 [17825.852181] [] ? kthread_create_on_node+0x140/0x140 [17825.860455] Code: b0 00 00 00 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 00 8b 75 98 48 8b 7d 88 e8 dc 2d f5 ff 48 85 c0 48 89 45 90 0f 84 b3 08 00 00 <44> 8b 7d a0 48 8b 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 e8 77 [17853.464404] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [17853.474113] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [17853.606034] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [17853.619341] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [17853.635633] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [17853.645039] RIP: 0010:[] [] osd_inode_iteration+0x28a/0xcc0 [osd_ldiskfs] [17853.657288] RSP: 0018:ffff880703603d18 EFLAGS: 00000287 [17853.664251] RAX: ffff88042bd78800 RBX: ffff880703603d98 RCX: ffff88042bd78800 [17853.673242] RDX: 00000000000072cf RSI: ffff880419dc3a00 RDI: ffff88042bd79800 [17853.682228] RBP: ffff880703603df0 R08: 0000000000000001 R09: 0000000000000004 [17853.691195] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000001 [17853.700152] R13: 0000000000000004 R14: 000000009f8f2001 R15: ffffea00027e3c80 [17853.709109] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [17853.719133] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17853.726537] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [17853.735503] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17853.744465] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [17853.753418] Stack: [17853.756629] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [17853.765927] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [17853.775194] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [17853.784505] Call Trace: [17853.788227] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [17853.797390] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [17853.806262] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [17853.814946] [] ? wake_up_state+0x20/0x20 [17853.822160] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [17853.831516] [] kthread+0xcf/0xe0 [17853.837964] [] ? kthread_create_on_node+0x140/0x140 [17853.846241] [] ret_from_fork+0x58/0x90 [17853.853254] [] ? kthread_create_on_node+0x140/0x140 [17853.861526] Code: a8 00 00 00 00 e8 77 5b f4 ff 42 8d 14 38 48 8b 45 88 48 8b 80 a8 03 00 00 48 3b 50 20 0f 83 57 01 00 00 44 0f b6 85 67 ff ff ff <48> 8d 4d a8 48 8d 55 88 48 8b b5 38 ff ff ff 48 8b bd 48 ff ff [17881.465386] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [17881.475110] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [17881.606878] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [17881.620185] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [17881.636499] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [17881.645906] RIP: 0010:[] [] osd_scrub_next+0x169/0x3c0 [osd_ldiskfs] [17881.657677] RSP: 0018:ffff880703603d08 EFLAGS: 00000282 [17881.664638] RAX: 0000000000000004 RBX: 0000000000000001 RCX: ffff880703603d98 [17881.673623] RDX: ffff880703603d78 RSI: 0000000000000000 RDI: ffff8803162f6000 [17881.682596] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [17881.691545] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [17881.700490] R13: 0000000000000018 R14: ffff880703603c70 R15: 0000000000000282 [17881.709430] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [17881.719443] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17881.726831] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [17881.735786] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17881.744734] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [17881.753668] Stack: [17881.756867] ffff880703603df0 ffffffffa121cff9 ffffffffa121b990 ffffffffa1217a20 [17881.766143] ffff8800b2a36000 00000000810d354f ffff8803162f6000 ffff8800b2a37468 [17881.775422] 0000000020000000 010000000000000c 0000000000000000 0000000000000000 [17881.784697] Call Trace: [17881.788415] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [17881.797571] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [17881.806720] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [17881.815587] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [17881.824254] [] ? wake_up_state+0x20/0x20 [17881.831454] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [17881.840789] [] kthread+0xcf/0xe0 [17881.847205] [] ? kthread_create_on_node+0x140/0x140 [17881.855472] [] ret_from_fork+0x58/0x90 [17881.862474] [] ? kthread_create_on_node+0x140/0x140 [17881.870737] Code: df e8 fc 69 47 e0 b8 05 00 00 00 48 8b 75 d0 65 48 33 34 25 28 00 00 00 0f 85 4c 02 00 00 48 83 c4 70 5b 41 5c 41 5d 41 5e 41 5f <5d> c3 48 8b 05 be 58 9c ff 48 85 c0 0f 84 c2 fe ff ff 0f 1f 44 [17909.466356] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [17909.475997] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [17909.607665] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [17909.620951] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [17909.637183] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [17909.646554] RIP: 0010:[] [] osd_inode_iteration+0x490/0xcc0 [osd_ldiskfs] [17909.658773] RSP: 0018:ffff880703603d18 EFLAGS: 00000293 [17909.665710] RAX: 0000000000000004 RBX: 0000000000000001 RCX: 0000000000000000 [17909.674668] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [17909.683684] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [17909.692629] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [17909.701569] R13: ffffea00027e3c80 R14: 000000009f8f2001 R15: ffff880703603d08 [17909.710514] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [17909.720539] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17909.727944] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [17909.736910] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17909.745872] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [17909.754827] Stack: [17909.758043] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [17909.767333] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [17909.776635] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [17909.785931] Call Trace: [17909.789652] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [17909.798826] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [17909.807701] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [17909.816377] [] ? wake_up_state+0x20/0x20 [17909.823587] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [17909.832942] [] kthread+0xcf/0xe0 [17909.839377] [] ? kthread_create_on_node+0x140/0x140 [17909.847661] [] ret_from_fork+0x58/0x90 [17909.854693] [] ? kthread_create_on_node+0x140/0x140 [17909.862969] Code: 0f fc ff ff 0f 1f 80 00 00 00 00 45 89 e9 4c 8d 85 67 ff ff ff 48 8b 4d a8 48 8d 55 88 48 8b b5 38 ff ff ff 48 8b bd 48 ff ff ff <48> 8b 85 28 ff ff ff ff d0 85 c0 41 89 c5 75 1c 8b 85 58 ff ff [17924.143866] INFO: rcu_sched self-detected stall on CPU { 24} (t=10500195 jiffies g=907112 c=907111 q=39976003) [17924.156427] Task dump for CPU 24: [17924.161200] OI_scrub R running task 0 5551 2 0x00000088 [17924.170141] ffff88083fde6dd0 00000000fb6db880 ffff88082da03db0 ffffffff810c47e8 [17924.179479] 0000000000000018 ffffffff81a1e7c0 ffff88082da03dc8 ffffffff810c8079 [17924.188798] 0000000000000009 ffff88082da03df8 ffffffff811378e0 ffff88082da101c0 [17924.198112] Call Trace: [17924.201809] [] sched_show_task+0xa8/0x110 [17924.209816] [] dump_cpu_task+0x39/0x70 [17924.216792] [] rcu_dump_cpu_stacks+0x90/0xd0 [17924.224339] [] rcu_check_callbacks+0x442/0x720 [17924.232049] [] ? tick_sched_handle.isra.13+0x60/0x60 [17924.240325] [] update_process_times+0x47/0x80 [17924.247898] [] tick_sched_handle.isra.13+0x25/0x60 [17924.255944] [] tick_sched_timer+0x41/0x70 [17924.263085] [] __hrtimer_run_queues+0xd2/0x260 [17924.270688] [] hrtimer_interrupt+0xb0/0x1e0 [17924.277984] [] ? call_softirq+0x1c/0x30 [17924.284874] [] local_apic_timer_interrupt+0x37/0x60 [17924.292919] [] smp_apic_timer_interrupt+0x3f/0x60 [17924.300765] [] apic_timer_interrupt+0x6d/0x80 [17924.308197] [] ? osd_inode_iteration+0x2ac/0xcc0 [osd_ldiskfs] [17924.318022] [] ? osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [17924.327122] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [17924.336029] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [17924.344652] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [17924.353067] [] ? wake_up_state+0x20/0x20 [17924.360023] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [17924.369112] [] kthread+0xcf/0xe0 [17924.375280] [] ? kthread_create_on_node+0x140/0x140 [17924.383306] [] ret_from_fork+0x58/0x90 [17924.390076] [] ? kthread_create_on_node+0x140/0x140 [17942.540728] Lustre: soaked-MDT0000: Client 6f6036ce-2e3a-7e95-1c07-11e5d5a627fe (at 192.168.1.124@o2ib100) reconnecting [17942.554759] Lustre: soaked-MDT0000: Connection restored to 5a0e6555-4c2d-39c9-90c4-a332f62c8353 (at 192.168.1.124@o2ib100) [17949.467749] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [17949.477263] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [17949.607241] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [17949.620437] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [17949.636528] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [17949.645840] RIP: 0010:[] [] osd_inode_iteration+0x282/0xcc0 [osd_ldiskfs] [17949.658010] RSP: 0018:ffff880703603d18 EFLAGS: 00000287 [17949.664901] RAX: ffff88042bd78800 RBX: ffff880703603d57 RCX: ffff88042bd78800 [17949.673842] RDX: 00000000000072cf RSI: ffff880419dc3a00 RDI: ffff88042bd79800 [17949.682786] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [17949.691722] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [17949.700651] R13: ffffea00027e3c80 R14: 000000009f8f2001 R15: 0000000000000018 [17949.709571] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [17949.719575] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17949.726962] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [17949.735907] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17949.744851] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [17949.753787] Stack: [17949.757005] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [17949.766304] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [17949.775604] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [17949.784906] Call Trace: [17949.788632] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [17949.797805] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [17949.806697] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [17949.815376] [] ? wake_up_state+0x20/0x20 [17949.822592] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [17949.831947] [] kthread+0xcf/0xe0 [17949.838383] [] ? kthread_create_on_node+0x140/0x140 [17949.846675] [] ret_from_fork+0x58/0x90 [17949.853692] [] ? kthread_create_on_node+0x140/0x140 [17949.861977] Code: 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 e8 77 5b f4 ff 42 8d 14 38 48 8b 45 88 48 8b 80 a8 03 00 00 48 3b 50 20 0f 83 57 01 00 00 <44> 0f b6 85 67 ff ff ff 48 8d 4d a8 48 8d 55 88 48 8b b5 38 ff [17961.148283] FS-Cache: Loaded [17961.189726] FS-Cache: Netfs 'nfs' registered for caching [17961.207315] Key type dns_resolver registered [17961.254191] NFS: Registering the id_resolver key type [17961.261565] Key type id_resolver registered [17961.268153] Key type id_legacy registered [17977.468724] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [17977.478311] Modules linked in: rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [17977.613590] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [17977.626780] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [17977.642864] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [17977.652172] RIP: 0010:[] [] osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [17977.664149] RSP: 0018:ffff880703603d10 EFLAGS: 00000293 [17977.671050] RAX: ffffffffa121b990 RBX: 000000009f8f2001 RCX: 0000000000000000 [17977.679996] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [17977.688950] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [17977.697884] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000001 [17977.706821] R13: ffff880703603df0 R14: 000000009f8f2001 R15: 0000000000000018 [17977.715762] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [17977.725765] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17977.733161] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [17977.742114] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [17977.751057] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [17977.759987] Stack: [17977.763186] ffffffffa121d1e9 ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 [17977.772456] 00000000810d354f ffff8803162f6000 ffff8800b2a37468 0000000020000000 [17977.781708] 010000000000000c 0000000000000000 0000000000000000 ffff8800b2a36000 [17977.790970] Call Trace: [17977.794669] [] ? osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [17977.804009] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [17977.813166] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [17977.822019] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [17977.830672] [] ? wake_up_state+0x20/0x20 [17977.837872] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [17977.847209] [] kthread+0xcf/0xe0 [17977.853628] [] ? kthread_create_on_node+0x140/0x140 [17977.861883] [] ret_from_fork+0x58/0x90 [17977.868862] [] ? kthread_create_on_node+0x140/0x140 [17977.877095] Code: b5 07 00 00 c7 05 85 a5 02 00 00 00 02 00 89 54 24 08 48 8b 40 40 ba 07 00 00 00 48 89 04 24 31 c0 e8 a5 03 98 ff e9 c7 fd ff ff <66> 66 66 66 90 55 48 89 e5 41 57 49 89 f7 41 56 41 55 41 54 45 [18005.469700] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [18005.479308] Modules linked in: rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [18005.615655] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [18005.628948] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [18005.645206] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [18005.654593] RIP: 0010:[] [] osd_scrub_exec+0x12/0x5b0 [osd_ldiskfs] [18005.666251] RSP: 0018:ffff880703603cf0 EFLAGS: 00000293 [18005.673192] RAX: ffffffffa121b990 RBX: 000000009f8f2001 RCX: 0000000000000000 [18005.682171] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [18005.691154] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [18005.700113] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffff880703603c78 [18005.709067] R13: 0000000000000296 R14: 0000000000000010 R15: ffffffffa121b9ac [18005.718026] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [18005.728058] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18005.735463] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [18005.744453] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [18005.753410] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [18005.762353] Stack: [18005.765571] 0000000000000004 00000000ffffffff 0000000000000001 ffff880703603df0 [18005.774866] ffffffffa121d1e9 ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 [18005.784174] 00000000810d354f ffff8803162f6000 ffff8800b2a37468 0000000020000000 [18005.793477] Call Trace: [18005.797203] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [18005.806383] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [18005.815558] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [18005.824428] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [18005.833095] [] ? wake_up_state+0x20/0x20 [18005.840303] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [18005.849649] [] kthread+0xcf/0xe0 [18005.856088] [] ? kthread_create_on_node+0x140/0x140 [18005.864374] [] ret_from_fork+0x58/0x90 [18005.871387] [] ? kthread_create_on_node+0x140/0x140 [18005.879662] Code: 48 8b 40 40 ba 07 00 00 00 48 89 04 24 31 c0 e8 a5 03 98 ff e9 c7 fd ff ff 66 66 66 66 90 55 48 89 e5 41 57 49 89 f7 41 56 41 55 <41> 54 45 31 e4 53 48 83 ec 68 48 8b 9e 08 10 00 00 4c 89 45 90 [18033.470682] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [18033.480387] Modules linked in: rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [18033.616770] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [18033.630066] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [18033.646339] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [18033.655725] RIP: 0010:[] [] osd_scrub_exec+0xf9/0x5b0 [osd_ldiskfs] [18033.667392] RSP: 0018:ffff880703603c78 EFLAGS: 00000246 [18033.674332] RAX: 0000000000000000 RBX: ffff880703603c37 RCX: 0000000000000000 [18033.683308] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [18033.692273] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [18033.701226] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [18033.710182] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [18033.719132] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [18033.729160] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18033.736580] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [18033.745547] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [18033.754516] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [18033.763462] Stack: [18033.766672] ffff880703603df0 000000009f8f2001 ffff880703603df0 000000009f8f2001 [18033.775976] ffff880703603d57 000000009f8f2001 0000000000000004 ffff880703603d57 [18033.785280] ffffffffa121b990 0000000000000000 ffff880703603d78 ffff8800b2a36000 [18033.794573] Call Trace: [18033.798291] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [18033.807444] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [18033.816600] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [18033.825752] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [18033.834602] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [18033.843269] [] ? wake_up_state+0x20/0x20 [18033.850468] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [18033.859809] [] kthread+0xcf/0xe0 [18033.866241] [] ? kthread_create_on_node+0x140/0x140 [18033.874507] [] ret_from_fork+0x58/0x90 [18033.881507] [] ? kthread_create_on_node+0x140/0x140 [18033.889779] Code: 88 48 85 db 74 23 f6 83 18 0a 00 00 08 74 1a 4d 85 e4 74 15 41 8b 87 6c 14 00 00 41 39 84 24 0c 0a 00 00 0f 82 d9 00 00 00 31 c0 <41> f6 87 74 14 00 00 04 0f 85 6e ff ff ff 41 83 f9 02 0f 84 64 [18061.471650] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [18061.481372] Modules linked in: rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [18061.617488] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [18061.630748] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [18061.646944] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [18061.656301] RIP: 0010:[] [] osd_scrub_next+0x1b/0x3c0 [osd_ldiskfs] [18061.667933] RSP: 0018:ffff880703603ce0 EFLAGS: 00000287 [18061.674862] RAX: ffffffffa1217a20 RBX: ffff880703603c78 RCX: ffff880703603d98 [18061.683817] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [18061.692753] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [18061.701692] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000010 [18061.710627] R13: ffffffffa121ba89 R14: ffffffffffffff10 R15: ffff8803162f6000 [18061.719570] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [18061.729586] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18061.736983] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [18061.745938] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [18061.754868] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [18061.763795] Stack: [18061.767002] ffff880419dc3a00 ffff8800b2a3746c 0000000000000000 00000000ffffffff [18061.776306] 0000000000000001 ffff880703603df0 ffffffffa121cff9 ffffffffa121b990 [18061.785591] ffffffffa1217a20 ffff8800b2a36000 00000000810d354f ffff8803162f6000 [18061.794874] Call Trace: [18061.798586] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [18061.807748] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [18061.816903] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [18061.825759] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [18061.834429] [] ? wake_up_state+0x20/0x20 [18061.841637] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [18061.850985] [] kthread+0xcf/0xe0 [18061.857422] [] ? kthread_create_on_node+0x140/0x140 [18061.865700] [] ret_from_fork+0x58/0x90 [18061.872711] [] ? kthread_create_on_node+0x140/0x140 [18061.880979] Code: 41 5c 41 5d 41 5e 5d c3 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 89 e5 41 57 49 89 f7 41 56 41 55 49 89 fd 41 54 49 89 d4 53 <48> 89 cb 48 83 ec 70 65 48 8b 04 25 28 00 00 00 48 89 45 d0 31 [18089.472625] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [18089.482275] Modules linked in: rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [18089.618545] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [18089.631834] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [18089.648087] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [18089.657473] RIP: 0010:[] [] osd_scrub_exec+0xf7/0x5b0 [osd_ldiskfs] [18089.669122] RSP: 0018:ffff880703603c78 EFLAGS: 00000246 [18089.676073] RAX: 0000000000280001 RBX: ffff880703603c37 RCX: 0000000000000000 [18089.685050] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [18089.694024] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [18089.702986] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [18089.711937] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [18089.720891] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [18089.730922] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18089.738336] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [18089.747305] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [18089.756272] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [18089.765217] Stack: [18089.768442] 0000000000000004 0000000000000001 ffff880703603df0 ffff880703603d98 [18089.777745] ffff880703603d57 000000009f8f2001 0000000000000004 ffff880703603d57 [18089.787055] ffff8800b2a37468 0000000000000000 ffff880703603d78 ffff8800b2a36000 [18089.796342] Call Trace: [18089.800061] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [18089.809225] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [18089.818380] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [18089.827241] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [18089.835898] [] ? wake_up_state+0x20/0x20 [18089.843102] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [18089.852447] [] kthread+0xcf/0xe0 [18089.858878] [] ? kthread_create_on_node+0x140/0x140 [18089.867150] [] ret_from_fork+0x58/0x90 [18089.874155] [] ? kthread_create_on_node+0x140/0x140 [18089.882419] Code: 8b 4d 88 48 85 db 74 23 f6 83 18 0a 00 00 08 74 1a 4d 85 e4 74 15 41 8b 87 6c 14 00 00 41 39 84 24 0c 0a 00 00 0f 82 d9 00 00 00 <31> c0 41 f6 87 74 14 00 00 04 0f 85 6e ff ff ff 41 83 f9 02 0f [18104.153135] INFO: rcu_sched self-detected stall on CPU { 24} (t=10680198 jiffies g=907112 c=907111 q=41641355) [18104.165705] Task dump for CPU 24: [18104.170436] OI_scrub R running task 0 5551 2 0x00000088 [18104.179354] ffff88083fde6dd0 00000000fb6db880 ffff88082da03db0 ffffffff810c47e8 [18104.188664] 0000000000000018 ffffffff81a1e7c0 ffff88082da03dc8 ffffffff810c8079 [18104.197946] 0000000000000009 ffff88082da03df8 ffffffff811378e0 ffff88082da101c0 [18104.207233] Call Trace: [18104.210913] [] sched_show_task+0xa8/0x110 [18104.218884] [] dump_cpu_task+0x39/0x70 [18104.225845] [] rcu_dump_cpu_stacks+0x90/0xd0 [18104.233358] [] rcu_check_callbacks+0x442/0x720 [18104.241049] [] ? tick_sched_handle.isra.13+0x60/0x60 [18104.249299] [] update_process_times+0x47/0x80 [18104.256844] [] tick_sched_handle.isra.13+0x25/0x60 [18104.264854] [] tick_sched_timer+0x41/0x70 [18104.271968] [] __hrtimer_run_queues+0xd2/0x260 [18104.279547] [] hrtimer_interrupt+0xb0/0x1e0 [18104.286830] [] ? call_softirq+0x1c/0x30 [18104.293702] [] local_apic_timer_interrupt+0x37/0x60 [18104.301742] [] smp_apic_timer_interrupt+0x3f/0x60 [18104.309567] [] apic_timer_interrupt+0x6d/0x80 [18104.316984] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [18104.326595] [] ? osd_scrub_exec+0x1c/0x5b0 [osd_ldiskfs] [18104.335108] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [18104.344016] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [18104.352910] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [18104.361507] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [18104.369900] [] ? wake_up_state+0x20/0x20 [18104.376846] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [18104.385924] [] kthread+0xcf/0xe0 [18104.392097] [] ? kthread_create_on_node+0x140/0x140 [18104.400105] [] ret_from_fork+0x58/0x90 [18104.406854] [] ? kthread_create_on_node+0x140/0x140 [18121.929925] INFO: task mount.nfs:8923 blocked for more than 120 seconds. [18121.938469] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [18121.948031] mount.nfs D ffff8803ec731c00 0 8923 8922 0x00000080 [18121.956707] ffff88016f4a39a0 0000000000000082 ffff880186b7bec0 ffff88016f4a3fd8 [18121.965824] ffff88016f4a3fd8 ffff88016f4a3fd8 ffff880186b7bec0 ffff88016f4a3ae0 [18121.974919] ffff88016f4a3ae8 7fffffffffffffff ffff880186b7bec0 ffff8803ec731c00 [18121.984000] Call Trace: [18121.987521] [] schedule+0x29/0x70 [18121.993827] [] schedule_timeout+0x239/0x2c0 [18122.001090] [] ? __schedule+0x3b0/0x990 [18122.007963] [] wait_for_completion+0x116/0x170 [18122.015530] [] ? wake_up_state+0x20/0x20 [18122.022525] [] ? dyntick_save_progress_counter+0x30/0x30 [18122.031035] [] __stop_cpus+0x5f/0x90 [18122.037565] [] ? dyntick_save_progress_counter+0x30/0x30 [18122.046060] [] try_stop_cpus+0x3a/0x60 [18122.052779] [] synchronize_sched_expedited+0xd8/0x1d0 [18122.060960] [] synchronize_rcu_expedited+0xe/0x10 [18122.068735] [] bdi_unregister+0x143/0x1b0 [18122.075741] [] nfs_put_super+0x19/0x20 [nfs] [18122.083016] [] generic_shutdown_super+0x72/0xf0 [18122.090571] [] kill_anon_super+0x12/0x20 [18122.097456] [] nfs_kill_super+0x1b/0x30 [nfs] [18122.104812] [] deactivate_locked_super+0x49/0x60 [18122.112462] [] deactivate_super+0x46/0x60 [18122.119438] [] mntput_no_expire+0xc5/0x120 [18122.126494] [] mntput+0x24/0x40 [18122.132468] [] namespace_unlock+0x118/0x130 [18122.139631] [] put_mnt_ns+0x4b/0x60 [18122.146023] [] mount_subtree+0x58/0xb0 [18122.152718] [] nfs_follow_remote_path+0x115/0x1b0 [nfsv4] [18122.161242] [] nfs4_try_mount+0x56/0xc0 [nfsv4] [18122.168831] [] ? get_nfs_version+0x27/0x90 [nfs] [18122.176490] [] nfs_fs_mount+0x4c5/0xd90 [nfs] [18122.183862] [] ? nfs_clone_super+0x140/0x140 [nfs] [18122.191704] [] ? param_set_portnr+0x70/0x70 [nfs] [18122.199441] [] mount_fs+0x39/0x1b0 [18122.205721] [] ? __alloc_percpu+0x15/0x20 [18122.212671] [] vfs_kern_mount+0x5f/0xf0 [18122.219424] [] do_mount+0x24e/0xaa0 [18122.225791] [] SyS_mount+0x96/0xf0 [18122.232099] [] system_call_fastpath+0x16/0x1b [18129.474019] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [18129.483336] Modules linked in: rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [18129.616370] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [18129.629409] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [18129.645245] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [18129.654442] RIP: 0010:[] [] osd_scrub_exec+0xf9/0x5b0 [osd_ldiskfs] [18129.665934] RSP: 0018:ffff880703603c78 EFLAGS: 00000246 [18129.672732] RAX: 0000000000000000 RBX: ffff880703603c37 RCX: 0000000000000000 [18129.681570] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [18129.690414] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [18129.699251] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [18129.708119] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [18129.716990] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [18129.726925] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18129.734231] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [18129.743112] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [18129.751977] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [18129.760852] Stack: [18129.763973] 0000000000000004 0000000000000001 ffff880703603df0 ffff880703603d98 [18129.773170] ffff880703603d57 000000009f8f2001 0000000000000004 ffff880703603d57 [18129.782376] ffff8800b2a37468 0000000000000000 ffff880703603d78 ffff8800b2a36000 [18129.791583] Call Trace: [18129.795254] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [18129.804347] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [18129.813456] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [18129.822276] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [18129.830881] [] ? wake_up_state+0x20/0x20 [18129.838027] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [18129.847325] [] kthread+0xcf/0xe0 [18129.853709] [] ? kthread_create_on_node+0x140/0x140 [18129.861937] [] ret_from_fork+0x58/0x90 [18129.868895] [] ? kthread_create_on_node+0x140/0x140 [18129.877131] Code: 88 48 85 db 74 23 f6 83 18 0a 00 00 08 74 1a 4d 85 e4 74 15 41 8b 87 6c 14 00 00 41 39 84 24 0c 0a 00 00 0f 82 d9 00 00 00 31 c0 <41> f6 87 74 14 00 00 04 0f 85 6e ff ff ff 41 83 f9 02 0f 84 64 [18157.474993] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [18157.484611] Modules linked in: rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [18157.621615] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [18157.634911] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [18157.651172] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [18157.660649] RIP: 0010:[] [] osd_inode_iteration+0x299/0xcc0 [osd_ldiskfs] [18157.672939] RSP: 0018:ffff880703603d18 EFLAGS: 00000287 [18157.679888] RAX: ffff88042bd78800 RBX: 000000009f8f2001 RCX: ffff880703603d98 [18157.688870] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff88042bd79800 [18157.697827] RBP: ffff880703603df0 R08: 0000000000000001 R09: 0000000000000004 [18157.706805] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [18157.715756] R13: ffff880703603d08 R14: 000000009f8f2001 R15: ffff880703603d08 [18157.724720] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [18157.734751] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18157.742166] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [18157.751149] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [18157.760112] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [18157.769056] Stack: [18157.772267] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [18157.781558] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [18157.790859] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [18157.800156] Call Trace: [18157.803896] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [18157.813072] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [18157.821947] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [18157.830632] [] ? wake_up_state+0x20/0x20 [18157.837842] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [18157.847195] [] kthread+0xcf/0xe0 [18157.853633] [] ? kthread_create_on_node+0x140/0x140 [18157.861916] [] ret_from_fork+0x58/0x90 [18157.868934] [] ? kthread_create_on_node+0x140/0x140 [18157.877209] Code: 8b 45 88 48 8b 80 a8 03 00 00 48 3b 50 20 0f 83 57 01 00 00 44 0f b6 85 67 ff ff ff 48 8d 4d a8 48 8d 55 88 48 8b b5 38 ff ff ff <48> 8b bd 48 ff ff ff 48 8b 85 30 ff ff ff ff d0 83 f8 03 41 89 [18185.475976] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [18185.485699] Modules linked in: rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [18185.622997] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [18185.636311] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [18185.652617] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [18185.662030] RIP: 0010:[] [] osd_inode_iteration+0x251/0xcc0 [osd_ldiskfs] [18185.674287] RSP: 0018:ffff880703603d18 EFLAGS: 00000286 [18185.681262] RAX: ffff8800b2a37468 RBX: 0000000000000000 RCX: 0000000000000000 [18185.690258] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [18185.699243] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [18185.708212] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffff880703603d57 [18185.717171] R13: 0000000000000004 R14: 000000009f8f2001 R15: ffffea00027e3c80 [18185.726124] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [18185.736151] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18185.743558] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [18185.752525] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [18185.761487] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [18185.770433] Stack: [18185.773643] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [18185.782931] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [18185.792223] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [18185.801523] Call Trace: [18185.805255] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [18185.814420] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [18185.823290] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [18185.831964] [] ? wake_up_state+0x20/0x20 [18185.839181] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [18185.848526] [] kthread+0xcf/0xe0 [18185.854954] [] ? kthread_create_on_node+0x140/0x140 [18185.863239] [] ret_from_fork+0x58/0x90 [18185.870248] [] ? kthread_create_on_node+0x140/0x140 [18185.878518] Code: b0 00 00 00 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 00 8b 75 98 48 8b 7d 88 e8 dc 2d f5 ff 48 85 c0 48 89 45 90 0f 84 b3 08 00 00 <44> 8b 7d a0 48 8b 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 e8 77 Console [soak-8] joined by on pts/4 at 05-09 23:12. CentOS Linux 7 (Core) Kernel 3.10.0-514.16.1.el7_lustre.x86_64 on an x86_64 soak-8 login: root Password: Last login: Tue May 9 23:07:26 from soak-35.spirit.hpdd.intel.com [root@soak-8 ~]# echo t > /proc/systrq-trigger [18213.476945] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [18213.486566] Modules linked in: rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm; ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [18213.623160] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [18213.636504] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [18213.652800] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [18213.662216] RIP: 0010:[] [] osd_scrub_exec+0xf7/0x5b0 [osd_ldiskfs] [18213.673899] RSP: 0018:ffff880703603c78 EFLAGS: 00000246 [18213.680859] RAX: 0000000000280001 RBX: ffff880703603c37 RCX: 0000000000000000 [18213.689849] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [18213.698840] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [18213.707812] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [18213.716763] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [18213.725713] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [18213.735749] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18213.743156] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [18213.752132] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [18213.761671] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [18213.770617] Stack: [18213.773815] ffffea00027e3c80 000000009f8f2001 ffff880703603df0 ffff880703603d57 [18213.783107] ffff880703603d57 000000009f8f2001 0000000000000004 ffff880703603d57 [18213.792389] ffff8800b2a37468 0000000000000000 ffff880703603d78 ffff8800b2a36000 [18213.801688] Call Trace: [18213.805422] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [18213.814585] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [18213.823752] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [18213.832624] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [18213.841286] [] ? wake_up_state+0x20/0x20 [18213.848484] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [18213.857824] [] kthread+0xcf/0xe0 [18213.864245] [] ? kthread_create_on_node+0x140/0x140 [18213.872513] [] ret_from_fork+0x58/0x90 [18213.879504] [] ? kthread_create_on_node+0x140/0x140 [18213.887765] Code: 8b 4d 88 48 85 db 74 23 f6 83 18 0a 00 00 08 74 1a 4d 85 e4 74 15 41 8b 87 6c 14 00 00 41 39 84 24 0c 0a 00 00 0f 82 d9 00 00 00 <31> c0 41 f6 87 74 14 00 00 04 0f 85 6e ff ff ff 41 83 f9 02 0f Message from syslogd@soak-8 at May 9 23:12:45 ... kernel:NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] ;echo c > /proc/sysrq-trigger -bash: syntax error near unexpected token `;;' [root@soak-8 ~]# echo t > /proc/sysrq-trigger ;;echo c > /proc/sysrq-trigger  [18236.263970] SysRq : Show State [18236.268673] task PC stack pid father [18236.276321] systemd S 0000000000000000 0 1 0 0x00000000 [18236.285228] ffff88017a1a3db8 0000000000000082 ffff8804ba508000 ffff88017a1a3fd8 [18236.294541] ffff88017a1a3fd8 ffff88017a1a3fd8 ffff8804ba508000 0000000000000000 [18236.303836] 0000000000000000 ffff88082cb85260 ffff8804ba508000 0000000000000000 [18236.313113] Call Trace: [18236.316781] [] schedule+0x29/0x70 [18236.323241] [] schedule_hrtimeout_range_clock+0x12d/0x150 [18236.332031] [] ? ep_scan_ready_list.isra.7+0x1b9/0x1f0 [18236.340499] [] schedule_hrtimeout_range+0x13/0x20 [18236.348939] [] ep_poll+0x23e/0x360 [18236.355860] [] ? do_timerfd_settime+0x2ed/0x3a0 [18236.363635] [] ? wake_up_state+0x20/0x20 [18236.370644] [] SyS_epoll_wait+0xed/0x120 [18236.377630] [] system_call_fastpath+0x16/0x1b [18236.385087] kthreadd S ffff8804ba508fb0 0 2 0 0x00000000 [18236.393724] ffff88017a1a7e70 0000000000000046 ffff8804ba508fb0 ffff88017a1a7fd8 [18236.402768] ffff88017a1a7fd8 ffff88017a1a7fd8 ffff8804ba508fb0 ffffffff819fdd40 [18236.411793] ffff88061efabd38 ffff8804ba508fb0 ffff8804ba508fb0 ffff8804ba508fb0 [18236.420811] Call Trace: [18236.424240] [] schedule+0x29/0x70 [18236.430571] [] kthreadd+0x2e6/0x2f0 [18236.437019] [] ? kthread_create_on_cpu+0x60/0x60 [18236.444802] [] ret_from_fork+0x58/0x90 [18236.451525] [] ? kthread_create_on_cpu+0x60/0x60 [18236.459213] ksoftirqd/0 S 0000000000000000 0 3 2 0x00000000 [18236.467812] ffff88017a1b7e80 0000000000000046 ffff88017a248000 ffff88017a1b7fd8 [18236.476825] ffff88017a1b7fd8 ffff88017a1b7fd8 ffff88017a248000 ffff88017a248000 [18236.485816] ffff88017fc7c260 ffffffff819f1100 0000000000000000 0000000000000000 [18236.494834] Call Trace: [18236.498266] [] schedule+0x29/0x70 [18236.504495] [] smpboot_thread_fn+0xdd/0x180 [18236.511695] [] ? lg_double_unlock+0x90/0x90 [18236.518897] [] kthread+0xcf/0xe0 [18236.525028] [] ? kthread_create_on_node+0x140/0x140 [18236.533003] [] ret_from_fork+0x58/0x90 [18236.539699] [] ? kthread_create_on_node+0x140/0x140 [18236.547660] kworker/0:0H S ffff88042e016840 0 5 2 0x00000000 [18236.556257] ffff88017a257e50 0000000000000046 ffff88017a249f60 ffff88017a257fd8 [18236.565247] ffff88017a257fd8 ffff88017a257fd8 ffff88017a249f60 ffff88042e016860 [18236.574234] ffff8804bfc40eb0 ffff88017a249f60 ffff8804bfc40e80 ffff88042e016840 [18236.583216] Call Trace: [18236.586612] [] schedule+0x29/0x70 [18236.592829] [] worker_thread+0x1d9/0x410 [18236.599719] [] ? rescuer_thread+0x460/0x460 [18236.606908] [] kthread+0xcf/0xe0 [18236.612985] [] ? kthread_create_on_node+0x140/0x140 [18236.620909] [] ret_from_fork+0x58/0x90 [18236.627564] [] ? kthread_create_on_node+0x140/0x140 [18236.635479] migration/0 S 0000000000000000 0 8 2 0x00000000 [18236.644016] ffff88017a263e80 0000000000000046 ffff88017a24ce70 ffff88017a263fd8 [18236.652962] ffff88017a263fd8 ffff88017a263fd8 ffff88017a24ce70 ffff88017a24ce70 [18236.661907] ffff88017fc7c280 ffffffff81a08f40 0000000000000000 0000000000000000 [18236.670848] Call Trace: [18236.674206] [] schedule+0x29/0x70 [18236.680384] [] smpboot_thread_fn+0xdd/0x180 [18236.687529] [] ? lg_double_unlock+0x90/0x90 [18236.694673] [] kthread+0xcf/0xe0 [18236.700743] [] ? kthread_create_on_node+0x140/0x140 [18236.708656] [] ret_from_fork+0x58/0x90 [18236.715324] [] ? kthread_create_on_node+0x140/0x140 [18236.723262] rcu_bh S 0000000000000000 0 9 2 0x00000000 [18236.731818] ffff88017a267e10 0000000000000046 ffff88017a24de20 ffff88017a267fd8 [18236.740779] ffff88017a267fd8 ffff88017a267fd8 ffff88017a24de20 ffffffff81a09d80 [18236.749716] ffff88017a24de20 ffffffff81a1e3e0 0000000000000000 0000000000000000 [18236.758657] Call Trace: [18236.762028] [] schedule+0x29/0x70 [18236.768218] [] rcu_gp_kthread+0xb2/0x7e0 [18236.775081] [] ? __schedule+0x3b0/0x990 [18236.782199] [] ? wake_up_atomic_t+0x30/0x30 [18236.789639] [] ? rcu_gp_fqs+0xb0/0xb0 [18236.796201] [] kthread+0xcf/0xe0 [18236.802287] [] ? kthread_create_on_node+0x140/0x140 [18236.810207] [] ret_from_fork+0x58/0x90 [18236.816860] [] ? kthread_create_on_node+0x140/0x140 [18236.824785] rcu_sched S 000000000000f888 0 10 2 0x00000000 [18236.833318] ffff88017a26bd60 0000000000000046 ffff88017a24edd0 ffff88017a26bfd8 [18236.842290] ffff88017a26bfd8 ffff88017a26bfd8 ffff88017a24edd0 ffff8801795fc000 [18236.851237] ffff88017a26bd90 000000010111ad8b ffff8801795fc000 000000000000f888 [18236.860190] Call Trace: [18236.863549] [] schedule+0x29/0x70 [18236.869735] [] schedule_timeout+0x174/0x2c0 [18236.877280] [] ? internal_add_timer+0x70/0x70 [18236.884624] [] ? prepare_to_wait+0x56/0x90 [18236.891718] [] rcu_gp_kthread+0x349/0x7e0 [18236.898866] [] ? wake_up_atomic_t+0x30/0x30 [18236.906018] [] ? rcu_gp_fqs+0xb0/0xb0 [18236.912567] [] kthread+0xcf/0xe0 [18236.918629] [] ? kthread_create_on_node+0x140/0x140 [18236.926532] [] ret_from_fork+0x58/0x90 [18236.933172] [] ? kthread_create_on_node+0x140/0x140 [18236.941080] watchdog/0 S 0000000000000000 0 11 2 0x00000000 [18236.949603] ffff88017a26fe80 0000000000000046 ffff88017a270000 ffff88017a26ffd8 [18236.958544] ffff88017a26ffd8 ffff88017a26ffd8 ffff88017a270000 ffff88017a270000 [18236.967472] ffff88017fc7c780 ffffffff81a09820 0000000000000000 0000000000000000 [18236.976430] Call Trace: [18236.979796] [] schedule+0x29/0x70 [18236.985981] [] smpboot_thread_fn+0xdd/0x180 [18236.993112] [] ? lg_double_unlock+0x90/0x90 [18237.000336] [] kthread+0xcf/0xe0 [18237.006413] [] ? kthread_create_on_node+0x140/0x140 [18237.014317] [] ret_from_fork+0x58/0x90 [18237.020954] [] ? kthread_create_on_node+0x140/0x140 [18237.028866] watchdog/1 S 0000000000000000 0 12 2 0x00000000 [18237.037388] ffff880179d2be80 0000000000000046 ffff880179d20000 ffff880179d2bfd8 [18237.046345] ffff880179d2bfd8 ffff880179d2bfd8 ffff880179d20000 ffff880179d20000 [18237.055304] ffff88017fc7c790 ffffffff81a09820 0000000000000000 0000000000000000 [18237.064264] Call Trace: [18237.067624] [] schedule+0x29/0x70 [18237.073792] [] smpboot_thread_fn+0xdd/0x180 [18237.080923] [] ? lg_double_unlock+0x90/0x90 [18237.088057] [] kthread+0xcf/0xe0 [18237.094114] [] ? kthread_create_on_node+0x140/0x140 [18237.102027] [] ret_from_fork+0x58/0x90 [18237.108651] [] ? kthread_create_on_node+0x140/0x140 [18237.116551] migration/1 S 0000000000000000 0 13 2 0x00000000 [18237.125077] ffff880179d2fe80 0000000000000046 ffff880179d20fb0 ffff880179d2ffd8 [18237.134035] ffff880179d2ffd8 ffff880179d2ffd8 ffff880179d20fb0 ffff880179d20fb0 [18237.142973] ffff88017fc7c7a0 ffffffff81a08f40 0000000000000000 0000000000000000 [18237.151927] Call Trace: [18237.155292] [] schedule+0x29/0x70 [18237.161466] [] smpboot_thread_fn+0xdd/0x180 [18237.168611] [] ? lg_double_unlock+0x90/0x90 [18237.175738] [] kthread+0xcf/0xe0 [18237.181800] [] ? kthread_create_on_node+0x140/0x140 [18237.189712] [] ret_from_fork+0x58/0x90 [18237.196368] [] ? kthread_create_on_node+0x140/0x140 [18237.204284] ksoftirqd/1 S 0000000000000000 0 14 2 0x00000000 [18237.212811] ffff880179d33e80 0000000000000046 ffff880179d21f60 ffff880179d33fd8 [18237.221760] ffff880179d33fd8 ffff880179d33fd8 ffff880179d21f60 ffff880179d21f60 [18237.231067] ffff88017fc7c7b0 ffffffff819f1100 0000000000000000 0000000000000000 [18237.240064] Call Trace: [18237.243416] [] schedule+0x29/0x70 [18237.249576] [] smpboot_thread_fn+0xdd/0x180 [18237.256726] [] ? lg_double_unlock+0x90/0x90 [18237.263882] [] kthread+0xcf/0xe0 [18237.269980] [] ? kthread_create_on_node+0x140/0x140 [18237.277926] [] ret_from_fork+0x58/0x90 [18237.284588] [] ? kthread_create_on_node+0x140/0x140 [18237.292510] kworker/1:0H S ffff88042e056840 0 16 2 0x00000000 [18237.301065] ffff880179d4be50 0000000000000046 ffff880179d23ec0 ffff880179d4bfd8 [18237.310024] ffff880179d4bfd8 ffff880179d4bfd8 ffff880179d23ec0 ffff88042e056860 [18237.318985] ffff8804bfc41630 ffff880179d23ec0 ffff8804bfc41600 ffff88042e056840 [18237.327939] Call Trace: [18237.331300] [] schedule+0x29/0x70 [18237.337487] [] worker_thread+0x1d9/0x410 [18237.344347] [] ? rescuer_thread+0x460/0x460 [18237.351498] [] kthread+0xcf/0xe0 [18237.357575] [] ? kthread_create_on_node+0x140/0x140 [18237.365520] [] ret_from_fork+0x58/0x90 [18237.372210] [] ? kthread_create_on_node+0x140/0x140 [18237.380182] watchdog/2 S 0000000000000000 0 17 2 0x00000000 [18237.388713] ffff8801795dbe80 0000000000000046 ffff880179d24e70 ffff8801795dbfd8 [18237.397673] ffff8801795dbfd8 ffff8801795dbfd8 ffff880179d24e70 ffff880179d24e70 [18237.406618] ffff88017fc7c7c0 ffffffff81a09820 0000000000000000 0000000000000000 [18237.415575] Call Trace: [18237.418962] [] schedule+0x29/0x70 [18237.425169] [] smpboot_thread_fn+0xdd/0x180 [18237.432334] [] ? lg_double_unlock+0x90/0x90 [18237.439489] [] kthread+0xcf/0xe0 [18237.445571] [] ? kthread_create_on_node+0x140/0x140 [18237.453503] [] ret_from_fork+0x58/0x90 [18237.460176] [] ? kthread_create_on_node+0x140/0x140 [18237.468103] migration/2 S 0000000000000000 0 18 2 0x00000000 [18237.476636] ffff8801795dfe80 0000000000000046 ffff880179d25e20 ffff8801795dffd8 [18237.485592] ffff8801795dffd8 ffff8801795dffd8 ffff880179d25e20 ffff880179d25e20 [18237.494547] ffff88017fc7c7d0 ffffffff81a08f40 0000000000000000 0000000000000000 [18237.503501] Call Trace: [18237.506863] [] schedule+0x29/0x70 [18237.513052] [] smpboot_thread_fn+0xdd/0x180 [18237.520217] [] ? lg_double_unlock+0x90/0x90 [18237.527373] [] kthread+0xcf/0xe0 [18237.533459] [] ? kthread_create_on_node+0x140/0x140 [18237.541388] [] ret_from_fork+0x58/0x90 [18237.548062] [] ? kthread_create_on_node+0x140/0x140 [18237.555996] ksoftirqd/2 S 0000000000000000 0 19 2 0x00000000 [18237.564534] ffff8801795ebe80 0000000000000046 ffff880179d26dd0 ffff8801795ebfd8 [18237.573502] ffff8801795ebfd8 ffff8801795ebfd8 ffff880179d26dd0 ffff880179d26dd0 [18237.582459] ffff88017fc7c7e0 ffffffff819f1100 0000000000000000 0000000000000000 [18237.591411] Call Trace: [18237.594767] [] schedule+0x29/0x70 [18237.600962] [] smpboot_thread_fn+0xdd/0x180 [18237.608129] [] ? lg_double_unlock+0x90/0x90 [18237.615286] [] kthread+0xcf/0xe0 [18237.621372] [] ? kthread_create_on_node+0x140/0x140 [18237.629307] [] ret_from_fork+0x58/0x90 [18237.635984] [] ? kthread_create_on_node+0x140/0x140 [18237.643919] kworker/2:0H S ffff88042e096840 0 21 2 0x00000000 [18237.652456] ffff8801795fbe50 0000000000000046 ffff8801795f0fb0 ffff8801795fbfd8 [18237.661416] ffff8801795fbfd8 ffff8801795fbfd8 ffff8801795f0fb0 ffff88042e096860 [18237.670744] ffff8804b9daf3b0 ffff8801795f0fb0 ffff8804b9daf380 ffff88042e096840 [18237.679679] Call Trace: [18237.683057] [] schedule+0x29/0x70 [18237.689258] [] worker_thread+0x1d9/0x410 [18237.696128] [] ? rescuer_thread+0x460/0x460 [18237.703288] [] kthread+0xcf/0xe0 [18237.709370] [] ? kthread_create_on_node+0x140/0x140 [18237.717308] [] ret_from_fork+0x58/0x90 [18237.723971] [] ? kthread_create_on_node+0x140/0x140 [18237.731915] watchdog/3 S 0000000000000000 0 22 2 0x00000000 [18237.740453] ffff880179633e80 0000000000000046 ffff8801795f1f60 ffff880179633fd8 [18237.749400] ffff880179633fd8 ffff880179633fd8 ffff8801795f1f60 ffff8801795f1f60 [18237.758365] ffff88017fc7c7f0 ffffffff81a09820 0000000000000000 0000000000000000 [18237.767317] Call Trace: [18237.770674] [] schedule+0x29/0x70 [18237.776862] [] smpboot_thread_fn+0xdd/0x180 [18237.784026] [] ? lg_double_unlock+0x90/0x90 [18237.791194] [] kthread+0xcf/0xe0 [18237.797285] [] ? kthread_create_on_node+0x140/0x140 [18237.805221] [] ret_from_fork+0x58/0x90 [18237.811887] [] ? kthread_create_on_node+0x140/0x140 [18237.819814] migration/3 S 0000000000000000 0 23 2 0x00000000 [18237.828364] ffff880179637e80 0000000000000046 ffff8801795f2f10 ffff880179637fd8 [18237.837481] ffff880179637fd8 ffff880179637fd8 ffff8801795f2f10 ffff8801795f2f10 [18237.846440] ffff88017fc7c800 ffffffff81a08f40 0000000000000000 0000000000000000 [18237.855407] Call Trace: [18237.858772] [] schedule+0x29/0x70 [18237.864964] [] smpboot_thread_fn+0xdd/0x180 [18237.872118] [] ? lg_double_unlock+0x90/0x90 [18237.879289] [] kthread+0xcf/0xe0 [18237.885381] [] ? kthread_create_on_node+0x140/0x140 [18237.893318] [] ret_from_fork+0x58/0x90 [18237.899992] [] ? kthread_create_on_node+0x140/0x140 [18237.907916] ksoftirqd/3 S 0000000000000000 0 24 2 0x00000000 [18237.916455] ffff88017963be80 0000000000000046 ffff8801795f3ec0 ffff88017963bfd8 [18237.925424] ffff88017963bfd8 ffff88017963bfd8 ffff8801795f3ec0 ffff8801795f3ec0 [18237.934365] ffff88017fc7c810 ffffffff819f1100 0000000000000000 0000000000000000 [18237.943326] Call Trace: [18237.946690] [] schedule+0x29/0x70 [18237.952878] [] smpboot_thread_fn+0xdd/0x180 [18237.960041] [] ? lg_double_unlock+0x90/0x90 [18237.967193] [] kthread+0xcf/0xe0 [18237.973273] [] ? kthread_create_on_node+0x140/0x140 [18237.981207] [] ret_from_fork+0x58/0x90 [18237.987885] [] ? kthread_create_on_node+0x140/0x140 [18237.995807] kworker/3:0H S ffff88042e0d6840 0 26 2 0x00000000 [18238.004359] ffff880179643e50 0000000000000046 ffff8801795f5e20 ffff880179643fd8 [18238.013315] ffff880179643fd8 ffff880179643fd8 ffff8801795f5e20 ffff88042e0d6860 [18238.022280] ffff8804b9daf030 ffff8801795f5e20 ffff8804b9daf000 ffff88042e0d6840 [18238.031219] Call Trace: [18238.034577] [] schedule+0x29/0x70 [18238.040758] [] worker_thread+0x1d9/0x410 [18238.047618] [] ? rescuer_thread+0x460/0x460 [18238.054770] [] kthread+0xcf/0xe0 [18238.060863] [] ? kthread_create_on_node+0x140/0x140 [18238.068798] [] ret_from_fork+0x58/0x90 [18238.075461] [] ? kthread_create_on_node+0x140/0x140 [18238.083386] watchdog/4 S 0000000000000000 0 27 2 0x00000000 [18238.092295] ffff88017965be80 0000000000000046 ffff8801795f6dd0 ffff88017965bfd8 [18238.101309] ffff88017965bfd8 ffff88017965bfd8 ffff8801795f6dd0 ffff8801795f6dd0 [18238.110277] ffff88017fc7c820 ffffffff81a09820 0000000000000000 0000000000000000 [18238.119222] Call Trace: [18238.122587] [] schedule+0x29/0x70 [18238.128769] [] smpboot_thread_fn+0xdd/0x180 [18238.135940] [] ? lg_double_unlock+0x90/0x90 [18238.143105] [] kthread+0xcf/0xe0 [18238.149197] [] ? kthread_create_on_node+0x140/0x140 [18238.157126] [] ret_from_fork+0x58/0x90 [18238.163802] [] ? kthread_create_on_node+0x140/0x140 [18238.171727] migration/4 S 0000000000000000 0 28 2 0x00000000 [18238.180280] ffff88017965fe80 0000000000000046 ffff880179660000 ffff88017965ffd8 [18238.189234] ffff88017965ffd8 ffff88017965ffd8 ffff880179660000 ffff880179660000 [18238.198194] ffff88017fc7c830 ffffffff81a08f40 0000000000000000 0000000000000000 [18238.207140] Call Trace: [18238.210498] [] schedule+0x29/0x70 [18238.216684] [] smpboot_thread_fn+0xdd/0x180 [18238.223846] [] ? lg_double_unlock+0x90/0x90 [18238.231000] [] kthread+0xcf/0xe0 [18238.237081] [] ? kthread_create_on_node+0x140/0x140 [18238.245126] [] ret_from_fork+0x58/0x90 [18238.251905] [] ? kthread_create_on_node+0x140/0x140 [18238.259847] ksoftirqd/4 S 0000000000000000 0 29 2 0x00000000 [18238.268399] ffff88017966be80 0000000000000046 ffff880179660fb0 ffff88017966bfd8 [18238.277373] ffff88017966bfd8 ffff88017966bfd8 ffff880179660fb0 ffff880179660fb0 [18238.286332] ffff88017fc7c840 ffffffff819f1100 0000000000000000 0000000000000000 [18238.295284] Call Trace: [18238.298646] [] schedule+0x29/0x70 [18238.304834] [] smpboot_thread_fn+0xdd/0x180 [18238.311990] [] ? lg_double_unlock+0x90/0x90 [18238.319157] [] kthread+0xcf/0xe0 [18238.325259] [] ? kthread_create_on_node+0x140/0x140 [18238.333198] [] ret_from_fork+0x58/0x90 [18238.339868] [] ? kthread_create_on_node+0x140/0x140 [18238.347799] kworker/4:0 S ffff88042e116480 0 30 2 0x00000000 [18238.356353] ffff88017966fe50 0000000000000046 ffff880179661f60 ffff88017966ffd8 [18238.365318] ffff88017966ffd8 ffff88017966ffd8 ffff880179661f60 ffff88042e1164a0 [18238.374288] ffff8804b9daedb0 ffff880179661f60 ffff8804b9daed80 ffff88042e116480 [18238.383231] Call Trace: [18238.386586] [] schedule+0x29/0x70 [18238.392762] [] worker_thread+0x1d9/0x410 [18238.399621] [] ? rescuer_thread+0x460/0x460 [18238.406769] [] kthread+0xcf/0xe0 [18238.412857] [] ? kthread_create_on_node+0x140/0x140 [18238.420779] [] ret_from_fork+0x58/0x90 [18238.427448] [] ? kthread_create_on_node+0x140/0x140 [18238.435381] kworker/4:0H S ffff88042e116840 0 31 2 0x00000000 [18238.443931] ffff880179673e50 0000000000000046 ffff880179662f10 ffff880179673fd8 [18238.452881] ffff880179673fd8 ffff880179673fd8 ffff880179662f10 ffff88042e116860 [18238.461835] ffff8804b9daecb0 ffff880179662f10 ffff8804b9daec80 ffff88042e116840 [18238.470766] Call Trace: [18238.474126] [] schedule+0x29/0x70 [18238.480317] [] worker_thread+0x1d9/0x410 [18238.487181] [] ? rescuer_thread+0x460/0x460 [18238.494328] [] kthread+0xcf/0xe0 [18238.500408] [] ? kthread_create_on_node+0x140/0x140 [18238.508330] [] ret_from_fork+0x58/0x90 [18238.514997] [] ? kthread_create_on_node+0x140/0x140 [18238.522921] watchdog/5 S 0000000000000000 0 32 2 0x00000000 [18238.531452] ffff88017968fe80 0000000000000046 ffff880179663ec0 ffff88017968ffd8 [18238.540780] ffff88017968ffd8 ffff88017968ffd8 ffff880179663ec0 ffff880179663ec0 [18238.549735] ffff88017fc7c850 ffffffff81a09820 0000000000000000 0000000000000000 [18238.558697] Call Trace: [18238.562066] [] schedule+0x29/0x70 [18238.568262] [] smpboot_thread_fn+0xdd/0x180 [18238.575410] [] ? lg_double_unlock+0x90/0x90 [18238.582557] [] kthread+0xcf/0xe0 [18238.588635] [] ? kthread_create_on_node+0x140/0x140 [18238.596558] [] ret_from_fork+0x58/0x90 [18238.603218] [] ? kthread_create_on_node+0x140/0x140 [18238.611139] migration/5 S 0000000000000000 0 33 2 0x00000000 [18238.619671] ffff880179693e80 0000000000000046 ffff880179664e70 ffff880179693fd8 [18238.628639] ffff880179693fd8 ffff880179693fd8 ffff880179664e70 ffff880179664e70 [18238.637578] ffff88017fc7c860 ffffffff81a08f40 0000000000000000 0000000000000000 [18238.646523] Call Trace: [18238.649890] [] schedule+0x29/0x70 [18238.656078] [] smpboot_thread_fn+0xdd/0x180 [18238.663228] [] ? lg_double_unlock+0x90/0x90 [18238.670385] [] kthread+0xcf/0xe0 [18238.676468] [] ? kthread_create_on_node+0x140/0x140 [18238.684408] [] ret_from_fork+0x58/0x90 [18238.691069] [] ? kthread_create_on_node+0x140/0x140 [18238.698989] ksoftirqd/5 S 0000000000000000 0 34 2 0x00000000 [18238.707517] ffff880179697e80 0000000000000046 ffff880179665e20 ffff880179697fd8 [18238.716464] ffff880179697fd8 ffff880179697fd8 ffff880179665e20 ffff880179665e20 [18238.725413] ffff88017fc7c870 ffffffff819f1100 0000000000000000 0000000000000000 [18238.734371] Call Trace: [18238.737730] [] schedule+0x29/0x70 [18238.743931] [] smpboot_thread_fn+0xdd/0x180 [18238.751076] [] ? lg_double_unlock+0x90/0x90 [18238.758229] [] kthread+0xcf/0xe0 [18238.764322] [] ? kthread_create_on_node+0x140/0x140 [18238.772257] [] ret_from_fork+0x58/0x90 [18238.778920] [] ? kthread_create_on_node+0x140/0x140 [18238.786843] kworker/5:0H S ffff88042e156840 0 36 2 0x00000000 [18238.795387] ffff8801796a7e50 0000000000000046 ffff8801796b0000 ffff8801796a7fd8 [18238.804356] ffff8801796a7fd8 ffff8801796a7fd8 ffff8801796b0000 ffff88042e156860 [18238.813313] ffff880179d5ae30 ffff8801796b0000 ffff880179d5ae00 ffff88042e156840 [18238.822268] Call Trace: [18238.825624] [] schedule+0x29/0x70 [18238.831803] [] worker_thread+0x1d9/0x410 [18238.838659] [] ? rescuer_thread+0x460/0x460 [18238.845808] [] kthread+0xcf/0xe0 [18238.851896] [] ? kthread_create_on_node+0x140/0x140 [18238.859819] [] ret_from_fork+0x58/0x90 [18238.866480] [] ? kthread_create_on_node+0x140/0x140 [18238.874408] watchdog/6 S 0000000000000000 0 37 2 0x00000000 [18238.882963] ffff8801796bfe80 0000000000000046 ffff8801796b0fb0 ffff8801796bffd8 [18238.891920] ffff8801796bffd8 ffff8801796bffd8 ffff8801796b0fb0 ffff8801796b0fb0 [18238.900885] ffff88017fc7c880 ffffffff81a09820 0000000000000000 0000000000000000 [18238.909823] Call Trace: [18238.913196] [] schedule+0x29/0x70 [18238.919383] [] smpboot_thread_fn+0xdd/0x180 [18238.926545] [] ? lg_double_unlock+0x90/0x90 [18238.933692] [] kthread+0xcf/0xe0 [18238.939772] [] ? kthread_create_on_node+0x140/0x140 [18238.947702] [] ret_from_fork+0x58/0x90 [18238.954370] [] ? kthread_create_on_node+0x140/0x140 [18238.962657] migration/6 S 0000000000000000 0 38 2 0x00000000 [18238.971223] ffff8801796d3e80 0000000000000046 ffff8801796b1f60 ffff8801796d3fd8 [18238.980208] ffff8801796d3fd8 ffff8801796d3fd8 ffff8801796b1f60 ffff8801796b1f60 [18238.989161] ffff88017fc7c890 ffffffff81a08f40 0000000000000000 0000000000000000 [18238.998118] Call Trace: [18239.001478] [] schedule+0x29/0x70 [18239.007666] [] smpboot_thread_fn+0xdd/0x180 [18239.014816] [] ? lg_double_unlock+0x90/0x90 [18239.022087] [] kthread+0xcf/0xe0 [18239.028387] [] ? kthread_create_on_node+0x140/0x140 [18239.036349] [] ret_from_fork+0x58/0x90 [18239.043026] [] ? kthread_create_on_node+0x140/0x140 [18239.050980] ksoftirqd/6 S 0000000000000000 0 39 2 0x00000000 [18239.059528] ffff8801796d7e80 0000000000000046 ffff8801796b2f10 ffff8801796d7fd8 [18239.068501] ffff8801796d7fd8 ffff8801796d7fd8 ffff8801796b2f10 ffff8801796b2f10 [18239.077447] ffff88017fc7c8a0 ffffffff819f1100 0000000000000000 0000000000000000 [18239.086391] Call Trace: [18239.089749] [] schedule+0x29/0x70 [18239.095931] [] smpboot_thread_fn+0xdd/0x180 [18239.103098] [] ? lg_double_unlock+0x90/0x90 [18239.110251] [] kthread+0xcf/0xe0 [18239.116332] [] ? kthread_create_on_node+0x140/0x140 [18239.124263] [] ret_from_fork+0x58/0x90 [18239.130936] [] ? kthread_create_on_node+0x140/0x140 [18239.138875] kworker/6:0H S ffff88042e196840 0 41 2 0x00000000 [18239.147404] ffff8801796dfe50 0000000000000046 ffff8801796b4e70 ffff8801796dffd8 [18239.156373] ffff8801796dffd8 ffff8801796dffd8 ffff8801796b4e70 ffff88042e196860 [18239.165331] ffff880179d5b1b0 ffff8801796b4e70 ffff880179d5b180 ffff88042e196840 [18239.174290] Call Trace: [18239.177646] [] schedule+0x29/0x70 [18239.183824] [] worker_thread+0x1d9/0x410 [18239.190679] [] ? rescuer_thread+0x460/0x460 [18239.197829] [] kthread+0xcf/0xe0 [18239.203918] [] ? kthread_create_on_node+0x140/0x140 [18239.211845] [] ret_from_fork+0x58/0x90 [18239.218506] [] ? kthread_create_on_node+0x140/0x140 [18239.226437] watchdog/7 S 0000000000000000 0 42 2 0x00000000 [18239.234979] ffff8801796fbe80 0000000000000046 ffff8801796b5e20 ffff8801796fbfd8 [18239.243947] ffff8801796fbfd8 ffff8801796fbfd8 ffff8801796b5e20 ffff8801796b5e20 [18239.252899] ffff88017fc7c8b0 ffffffff81a09820 0000000000000000 0000000000000000 [18239.261852] Call Trace: [18239.265221] [] schedule+0x29/0x70 [18239.271404] [] smpboot_thread_fn+0xdd/0x180 [18239.278561] [] ? lg_double_unlock+0x90/0x90 [18239.285731] [] kthread+0xcf/0xe0 [18239.291809] [] ? kthread_create_on_node+0x140/0x140 [18239.299735] [] ret_from_fork+0x58/0x90 [18239.306406] [] ? kthread_create_on_node+0x140/0x140 [18239.314328] migration/7 S 0000000000000000 0 43 2 0x00000000 [18239.322879] ffff8801796ffe80 0000000000000046 ffff8801796b6dd0 ffff8801796fffd8 [18239.331822] ffff8801796fffd8 ffff8801796fffd8 ffff8801796b6dd0 ffff8801796b6dd0 [18239.340768] ffff88017fc7c8c0 ffffffff81a08f40 0000000000000000 0000000000000000 [18239.349734] Call Trace: [18239.353102] [] schedule+0x29/0x70 [18239.359299] [] smpboot_thread_fn+0xdd/0x180 [18239.366451] [] ? lg_double_unlock+0x90/0x90 [18239.373605] [] kthread+0xcf/0xe0 [18239.379675] [] ? kthread_create_on_node+0x140/0x140 [18239.387611] [] ret_from_fork+0x58/0x90 [18239.394283] [] ? kthread_create_on_node+0x140/0x140 [18239.402219] ksoftirqd/7 S 0000000000000000 0 44 2 0x00000000 [18239.410751] ffff880179713e80 0000000000000046 ffff880179708000 ffff880179713fd8 [18239.419714] ffff880179713fd8 ffff880179713fd8 ffff880179708000 ffff880179708000 [18239.428653] ffff88017fc7c8d0 ffffffff819f1100 0000000000000000 0000000000000000 [18239.437942] Call Trace: [18239.441658] [] schedule+0x29/0x70 [18239.447839] [] smpboot_thread_fn+0xdd/0x180 [18239.454995] [] ? lg_double_unlock+0x90/0x90 [18239.462153] [] kthread+0xcf/0xe0 [18239.468252] [] ? kthread_create_on_node+0x140/0x140 [18239.476185] [] ret_from_fork+0x58/0x90 [18239.482844] [] ? kthread_create_on_node+0x140/0x140 [18239.490761] kworker/7:0H S ffff88042e1d6840 0 46 2 0x00000000 [18239.499509] ffff88017971be50 0000000000000046 ffff880179709f60 ffff88017971bfd8 [18239.508632] ffff88017971bfd8 ffff88017971bfd8 ffff880179709f60 ffff88042e1d6860 [18239.517592] ffff8801796e0330 ffff880179709f60 ffff8801796e0300 ffff88042e1d6840 [18239.526526] Call Trace: [18239.529891] [] schedule+0x29/0x70 [18239.536072] [] worker_thread+0x1d9/0x410 [18239.542938] [] ? rescuer_thread+0x460/0x460 [18239.550092] [] kthread+0xcf/0xe0 [18239.556170] [] ? kthread_create_on_node+0x140/0x140 [18239.564097] [] ret_from_fork+0x58/0x90 [18239.570756] [] ? kthread_create_on_node+0x140/0x140 [18239.578675] watchdog/8 S 0000000000000000 0 47 2 0x00000000 [18239.587209] ffff8804b962be80 0000000000000046 ffff8804b9da0fb0 ffff8804b962bfd8 [18239.596166] ffff8804b962bfd8 ffff8804b962bfd8 ffff8804b9da0fb0 ffff8804b9da0fb0 [18239.605116] ffff8804ba74e000 ffffffff81a09820 0000000000000000 0000000000000000 [18239.614068] Call Trace: [18239.617426] [] schedule+0x29/0x70 [18239.623604] [] smpboot_thread_fn+0xdd/0x180 [18239.630752] [] ? lg_double_unlock+0x90/0x90 [18239.637917] [] kthread+0xcf/0xe0 [18239.644004] [] ? kthread_create_on_node+0x140/0x140 [18239.651930] [] ret_from_fork+0x58/0x90 [18239.658593] [] ? kthread_create_on_node+0x140/0x140 [18239.666513] migration/8 S 0000000000000000 0 48 2 0x00000000 [18239.675055] ffff8804b962fe80 0000000000000046 ffff8804b9da1f60 ffff8804b962ffd8 [18239.684004] ffff8804b962ffd8 ffff8804b962ffd8 ffff8804b9da1f60 ffff8804b9da1f60 [18239.692968] ffff8804ba74e010 ffffffff81a08f40 0000000000000000 0000000000000000 [18239.701911] Call Trace: [18239.705285] [] schedule+0x29/0x70 [18239.711470] [] smpboot_thread_fn+0xdd/0x180 [18239.718619] [] ? lg_double_unlock+0x90/0x90 [18239.725769] [] kthread+0xcf/0xe0 [18239.731845] [] ? kthread_create_on_node+0x140/0x140 [18239.739769] [] ret_from_fork+0x58/0x90 [18239.746434] [] ? kthread_create_on_node+0x140/0x140 [18239.754360] ksoftirqd/8 S 0000000000000000 0 49 2 0x00000000 [18239.762912] ffff8804b9633e80 0000000000000046 ffff8804b9da2f10 ffff8804b9633fd8 [18239.771853] ffff8804b9633fd8 ffff8804b9633fd8 ffff8804b9da2f10 ffff8804b9da2f10 [18239.780798] ffff8804ba74e020 ffffffff819f1100 0000000000000000 0000000000000000 [18239.789742] Call Trace: [18239.793115] [] schedule+0x29/0x70 [18239.799303] [] smpboot_thread_fn+0xdd/0x180 [18239.806458] [] ? lg_double_unlock+0x90/0x90 [18239.813609] [] kthread+0xcf/0xe0 [18239.819687] [] ? kthread_create_on_node+0x140/0x140 [18239.827613] [] ret_from_fork+0x58/0x90 [18239.834276] [] ? kthread_create_on_node+0x140/0x140 [18239.842211] kworker/8:0H S ffff88082d816840 0 51 2 0x00000000 [18239.850741] ffff8804b963be50 0000000000000046 ffff8804b9da4e70 ffff8804b963bfd8 [18239.859698] ffff8804b963bfd8 ffff8804b963bfd8 ffff8804b9da4e70 ffff88082d816860 [18239.868650] ffff8801796e06b0 ffff8804b9da4e70 ffff8801796e0680 ffff88082d816840 [18239.877597] Call Trace: [18239.880967] [] schedule+0x29/0x70 [18239.887177] [] worker_thread+0x1d9/0x410 [18239.894056] [] ? rescuer_thread+0x460/0x460 [18239.901209] [] kthread+0xcf/0xe0 [18239.907298] [] ? kthread_create_on_node+0x140/0x140 [18239.915240] [] ret_from_fork+0x58/0x90 [18239.921916] [] ? kthread_create_on_node+0x140/0x140 [18239.929841] watchdog/9 S 0000000000000000 0 53 2 0x00000000 [18239.938379] ffff8804b9667e80 0000000000000046 ffff8804b9668000 ffff8804b9667fd8 [18239.947345] ffff8804b9667fd8 ffff8804b9667fd8 ffff8804b9668000 ffff8804b9668000 [18239.956339] ffff8804ba74e030 ffffffff81a09820 0000000000000000 0000000000000000 [18239.965344] Call Trace: [18239.968703] [] schedule+0x29/0x70 [18239.974889] [] smpboot_thread_fn+0xdd/0x180 [18239.982042] [] ? lg_double_unlock+0x90/0x90 [18239.989202] [] kthread+0xcf/0xe0 [18239.995288] [] ? kthread_create_on_node+0x140/0x140 [18240.003215] [] ret_from_fork+0x58/0x90 [18240.009883] [] ? kthread_create_on_node+0x140/0x140 [18240.017798] migration/9 S 0000000000000000 0 54 2 0x00000000 [18240.026330] ffff8804b9673e80 0000000000000046 ffff8804b9668fb0 ffff8804b9673fd8 [18240.035296] ffff8804b9673fd8 ffff8804b9673fd8 ffff8804b9668fb0 ffff8804b9668fb0 [18240.044252] ffff8804ba74e040 ffffffff81a08f40 0000000000000000 0000000000000000 [18240.053210] Call Trace: [18240.056566] [] schedule+0x29/0x70 [18240.062747] [] smpboot_thread_fn+0xdd/0x180 [18240.069905] [] ? lg_double_unlock+0x90/0x90 [18240.077056] [] kthread+0xcf/0xe0 [18240.083144] [] ? kthread_create_on_node+0x140/0x140 [18240.091084] [] ret_from_fork+0x58/0x90 [18240.097742] [] ? kthread_create_on_node+0x140/0x140 [18240.105660] ksoftirqd/9 S 0000000000000000 0 55 2 0x00000000 [18240.114192] ffff8804b9677e80 0000000000000046 ffff8804b9669f60 ffff8804b9677fd8 [18240.123158] ffff8804b9677fd8 ffff8804b9677fd8 ffff8804b9669f60 ffff8804b9669f60 [18240.132118] ffff8804ba74e050 ffffffff819f1100 0000000000000000 0000000000000000 [18240.141077] Call Trace: [18240.144449] [] schedule+0x29/0x70 [18240.150626] [] smpboot_thread_fn+0xdd/0x180 [18240.157780] [] ? lg_double_unlock+0x90/0x90 [18240.164941] [] kthread+0xcf/0xe0 [18240.171029] [] ? kthread_create_on_node+0x140/0x140 [18240.178963] [] ret_from_fork+0x58/0x90 [18240.185623] [] ? kthread_create_on_node+0x140/0x140 [18240.193541] kworker/9:0 S ffff88082d856480 0 56 2 0x00000000 [18240.202076] ffff8804b9683e50 0000000000000046 ffff8804b966af10 ffff8804b9683fd8 [18240.211038] ffff8804b9683fd8 ffff8804b9683fd8 ffff8804b966af10 ffff88082d8564a0 [18240.220013] ffff8801796e0a30 ffff8804b966af10 ffff8801796e0a00 ffff88082d856480 [18240.228956] Call Trace: [18240.232320] [] schedule+0x29/0x70 [18240.238507] [] worker_thread+0x1d9/0x410 [18240.245377] [] ? rescuer_thread+0x460/0x460 [18240.252524] [] kthread+0xcf/0xe0 [18240.258599] [] ? kthread_create_on_node+0x140/0x140 [18240.266528] [] ret_from_fork+0x58/0x90 [18240.273188] [] ? kthread_create_on_node+0x140/0x140 [18240.281108] kworker/9:0H S ffff88082d856840 0 57 2 0x00000000 [18240.289650] ffff8804b9687e50 0000000000000046 ffff8804b966bec0 ffff8804b9687fd8 [18240.298602] ffff8804b9687fd8 ffff8804b9687fd8 ffff8804b966bec0 ffff88082d856860 [18240.307566] ffff8801796e0b30 ffff8804b966bec0 ffff8801796e0b00 ffff88082d856840 [18240.316505] Call Trace: [18240.319866] [] schedule+0x29/0x70 [18240.326042] [] worker_thread+0x1d9/0x410 [18240.332897] [] ? rescuer_thread+0x460/0x460 [18240.340056] [] kthread+0xcf/0xe0 [18240.346141] [] ? kthread_create_on_node+0x140/0x140 [18240.354074] [] ret_from_fork+0x58/0x90 [18240.360729] [] ? kthread_create_on_node+0x140/0x140 [18240.368651] watchdog/10 S 0000000000000000 0 58 2 0x00000000 [18240.377184] ffff8804b968fe80 0000000000000046 ffff8804b966ce70 ffff8804b968ffd8 [18240.386143] ffff8804b968ffd8 ffff8804b968ffd8 ffff8804b966ce70 ffff8804b966ce70 [18240.395083] ffff8804ba74e060 ffffffff81a09820 0000000000000000 0000000000000000 [18240.404032] Call Trace: [18240.407395] [] schedule+0x29/0x70 [18240.413569] [] smpboot_thread_fn+0xdd/0x180 [18240.420714] [] ? lg_double_unlock+0x90/0x90 [18240.427866] [] kthread+0xcf/0xe0 [18240.433944] [] ? kthread_create_on_node+0x140/0x140 [18240.441865] [] ret_from_fork+0x58/0x90 [18240.448526] [] ? kthread_create_on_node+0x140/0x140 [18240.456449] migration/10 S 0000000000000000 0 59 2 0x00000000 [18240.465000] ffff8804b96a3e80 0000000000000046 ffff8804b966de20 ffff8804b96a3fd8 [18240.473941] ffff8804b96a3fd8 ffff8804b96a3fd8 ffff8804b966de20 ffff8804b966de20 [18240.482908] ffff8804ba74e070 ffffffff81a08f40 0000000000000000 0000000000000000 [18240.491870] Call Trace: [18240.495238] [] schedule+0x29/0x70 [18240.501426] [] smpboot_thread_fn+0xdd/0x180 [18240.508574] [] ? lg_double_unlock+0x90/0x90 [18240.515722] [] kthread+0xcf/0xe0 [18240.521803] [] ? kthread_create_on_node+0x140/0x140 [18240.529727] [] ret_from_fork+0x58/0x90 [18240.536387] [] ? kthread_create_on_node+0x140/0x140 [18240.544308] ksoftirqd/10 S 0000000000000000 0 60 2 0x00000000 [18240.552824] ffff8804b96a7e80 0000000000000046 ffff8804b966edd0 ffff8804b96a7fd8 [18240.561770] ffff8804b96a7fd8 ffff8804b96a7fd8 ffff8804b966edd0 ffff8804b966edd0 [18240.570723] ffff8804ba74e080 ffffffff819f1100 0000000000000000 0000000000000000 [18240.579664] Call Trace: [18240.583043] [] schedule+0x29/0x70 [18240.589230] [] smpboot_thread_fn+0xdd/0x180 [18240.596382] [] ? lg_double_unlock+0x90/0x90 [18240.603541] [] kthread+0xcf/0xe0 [18240.609620] [] ? kthread_create_on_node+0x140/0x140 [18240.617552] [] ret_from_fork+0x58/0x90 [18240.624223] [] ? kthread_create_on_node+0x140/0x140 [18240.632153] kworker/10:0 S ffff88082d896480 0 61 2 0x00000000 [18240.640692] ffff8804b96b3e50 0000000000000046 ffff8804b96a8000 ffff8804b96b3fd8 [18240.649643] ffff8804b96b3fd8 ffff8804b96b3fd8 ffff8804b96a8000 ffff88082d8964a0 [18240.658612] ffff8801796e0db0 ffff8804b96a8000 ffff8801796e0d80 ffff88082d896480 [18240.667561] Call Trace: [18240.670929] [] schedule+0x29/0x70 [18240.677120] [] worker_thread+0x1d9/0x410 [18240.683987] [] ? rescuer_thread+0x460/0x460 [18240.691150] [] kthread+0xcf/0xe0 [18240.697233] [] ? kthread_create_on_node+0x140/0x140 [18240.705167] [] ret_from_fork+0x58/0x90 [18240.711827] [] ? kthread_create_on_node+0x140/0x140 [18240.719755] kworker/10:0H S ffff88082d896840 0 62 2 0x00000000 [18240.728305] ffff8804b96b7e50 0000000000000046 ffff8804b96a8fb0 ffff8804b96b7fd8 [18240.737279] ffff8804b96b7fd8 ffff8804b96b7fd8 ffff8804b96a8fb0 ffff88082d896860 [18240.746234] ffff8801796e0eb0 ffff8804b96a8fb0 ffff8801796e0e80 ffff88082d896840 [18240.755190] Call Trace: [18240.758546] [] schedule+0x29/0x70 [18240.764722] [] worker_thread+0x1d9/0x410 [18240.771575] [] ? rescuer_thread+0x460/0x460 [18240.778722] [] kthread+0xcf/0xe0 [18240.784798] [] ? kthread_create_on_node+0x140/0x140 [18240.792717] [] ret_from_fork+0x58/0x90 [18240.799368] [] ? kthread_create_on_node+0x140/0x140 [18240.807293] watchdog/11 S 0000000000000000 0 63 2 0x00000000 [18240.815822] ffff8804b96d3e80 0000000000000046 ffff8804b96a9f60 ffff8804b96d3fd8 [18240.824775] ffff8804b96d3fd8 ffff8804b96d3fd8 ffff8804b96a9f60 ffff8804b96a9f60 [18240.833725] ffff8804ba74e090 ffffffff81a09820 0000000000000000 0000000000000000 [18240.842663] Call Trace: [18240.846036] [] schedule+0x29/0x70 [18240.852219] [] smpboot_thread_fn+0xdd/0x180 [18240.859374] [] ? lg_double_unlock+0x90/0x90 [18240.866520] [] kthread+0xcf/0xe0 [18240.872597] [] ? kthread_create_on_node+0x140/0x140 [18240.880525] [] ret_from_fork+0x58/0x90 [18240.887199] [] ? kthread_create_on_node+0x140/0x140 [18240.895130] migration/11 S 0000000000000000 0 64 2 0x00000000 [18240.903662] ffff8804b96d7e80 0000000000000046 ffff8804b96aaf10 ffff8804b96d7fd8 [18240.912614] ffff8804b96d7fd8 ffff8804b96d7fd8 ffff8804b96aaf10 ffff8804b96aaf10 [18240.921555] ffff8804ba74e0a0 ffffffff81a08f40 0000000000000000 0000000000000000 [18240.930514] Call Trace: [18240.933868] [] schedule+0x29/0x70 [18240.940059] [] smpboot_thread_fn+0xdd/0x180 [18240.947213] [] ? lg_double_unlock+0x90/0x90 [18240.954385] [] kthread+0xcf/0xe0 [18240.960471] [] ? kthread_create_on_node+0x140/0x140 [18240.968410] [] ret_from_fork+0x58/0x90 [18240.975075] [] ? kthread_create_on_node+0x140/0x140 [18240.983007] ksoftirqd/11 S 0000000000000000 0 65 2 0x00000000 [18240.991532] ffff8804b96dbe80 0000000000000046 ffff8804b96abec0 ffff8804b96dbfd8 [18241.000548] ffff8804b96dbfd8 ffff8804b96dbfd8 ffff8804b96abec0 ffff8804b96abec0 [18241.009588] ffff8804ba74e0b0 ffffffff819f1100 0000000000000000 0000000000000000 [18241.018544] Call Trace: [18241.021908] [] schedule+0x29/0x70 [18241.028098] [] smpboot_thread_fn+0xdd/0x180 [18241.035252] [] ? lg_double_unlock+0x90/0x90 [18241.042416] [] kthread+0xcf/0xe0 [18241.048505] [] ? kthread_create_on_node+0x140/0x140 [18241.056434] [] ret_from_fork+0x58/0x90 [18241.063105] [] ? kthread_create_on_node+0x140/0x140 [18241.071047] kworker/11:0 S ffff88082d8d6480 0 66 2 0x00000000 [18241.079590] ffff8804b96dfe50 0000000000000046 ffff8804b96ace70 ffff8804b96dffd8 [18241.088542] ffff8804b96dffd8 ffff8804b96dffd8 ffff8804b96ace70 ffff88082d8d64a0 [18241.097506] ffff8801796e1130 ffff8804b96ace70 ffff8801796e1100 ffff88082d8d6480 [18241.106450] Call Trace: [18241.109808] [] schedule+0x29/0x70 [18241.116004] [] worker_thread+0x1d9/0x410 [18241.122861] [] ? rescuer_thread+0x460/0x460 [18241.130024] [] kthread+0xcf/0xe0 [18241.136120] [] ? kthread_create_on_node+0x140/0x140 [18241.144071] [] ret_from_fork+0x58/0x90 [18241.150739] [] ? kthread_create_on_node+0x140/0x140 [18241.158672] kworker/11:0H S ffff88082d8d6840 0 67 2 0x00000000 [18241.167219] ffff8804b96ebe50 0000000000000046 ffff8804b96ade20 ffff8804b96ebfd8 [18241.176184] ffff8804b96ebfd8 ffff8804b96ebfd8 ffff8804b96ade20 ffff88082d8d6860 [18241.185148] ffff8801796e1230 ffff8804b96ade20 ffff8801796e1200 ffff88082d8d6840 [18241.194112] Call Trace: [18241.197531] [] schedule+0x29/0x70 [18241.203709] [] worker_thread+0x1d9/0x410 [18241.210565] [] ? rescuer_thread+0x460/0x460 [18241.217721] [] kthread+0xcf/0xe0 [18241.223809] [] ? kthread_create_on_node+0x140/0x140 [18241.231733] [] ret_from_fork+0x58/0x90 [18241.238414] [] ? kthread_create_on_node+0x140/0x140 [18241.246341] watchdog/12 S 0000000000000000 0 68 2 0x00000000 [18241.254873] ffff8804b9727e80 0000000000000046 ffff8804b96aedd0 ffff8804b9727fd8 [18241.263834] ffff8804b9727fd8 ffff8804b9727fd8 ffff8804b96aedd0 ffff8804b96aedd0 [18241.272798] ffff8804ba74e0c0 ffffffff81a09820 0000000000000000 0000000000000000 [18241.281745] Call Trace: [18241.285120] [] schedule+0x29/0x70 [18241.291307] [] smpboot_thread_fn+0xdd/0x180 [18241.298476] [] ? lg_double_unlock+0x90/0x90 [18241.305622] [] kthread+0xcf/0xe0 [18241.311703] [] ? kthread_create_on_node+0x140/0x140 [18241.319627] [] ret_from_fork+0x58/0x90 [18241.326294] [] ? kthread_create_on_node+0x140/0x140 [18241.334221] migration/12 S 0000000000000000 0 69 2 0x00000000 [18241.342752] ffff8804b9733e80 0000000000000046 ffff8804b9728000 ffff8804b9733fd8 [18241.351704] ffff8804b9733fd8 ffff8804b9733fd8 ffff8804b9728000 ffff8804b9728000 [18241.360650] ffff8804ba74e0d0 ffffffff81a08f40 0000000000000000 0000000000000000 [18241.369577] Call Trace: [18241.372937] [] schedule+0x29/0x70 [18241.379121] [] smpboot_thread_fn+0xdd/0x180 [18241.386268] [] ? lg_double_unlock+0x90/0x90 [18241.393423] [] kthread+0xcf/0xe0 [18241.399511] [] ? kthread_create_on_node+0x140/0x140 [18241.407443] [] ret_from_fork+0x58/0x90 [18241.414114] [] ? kthread_create_on_node+0x140/0x140 [18241.422047] ksoftirqd/12 S 0000000000000000 0 70 2 0x00000000 [18241.430569] ffff8804b9737e80 0000000000000046 ffff8804b9728fb0 ffff8804b9737fd8 [18241.439530] ffff8804b9737fd8 ffff8804b9737fd8 ffff8804b9728fb0 ffff8804b9728fb0 [18241.448470] ffff8804ba74e0e0 ffffffff819f1100 0000000000000000 0000000000000000 [18241.457430] Call Trace: [18241.460790] [] schedule+0x29/0x70 [18241.466988] [] smpboot_thread_fn+0xdd/0x180 [18241.474141] [] ? lg_double_unlock+0x90/0x90 [18241.477925] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] Message from[18241.477961] Modules linked in: rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support syslogd@soak-8 at May 9 23:13:13 ... kernel mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes :NMI watchdog: BUG: soft lockup - CPU#24 stuck f[18241.477982] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 or 22s! [OI_scru[18241.477983] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 b:5551] [18241.477983] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [18241.478004] RIP: 0010:[] [] osd_scrub_next+0x3d/0x3c0 [osd_ldiskfs] [18241.478005] RSP: 0018:ffff880703603c70 EFLAGS: 00000246 [18241.478006] RAX: 0000000000000000 RBX: ffff880703603c37 RCX: ffff880703603d98 [18241.478006] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [18241.478007] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [18241.478008] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [18241.478008] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [18241.478009] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [18241.478010] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18241.478011] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [18241.478011] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [18241.478012] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [18241.478012] Stack: [18241.478013] 000000009f8f2001 ffff880703603d08 ffff880703603d57 ffff880703603df0 [18241.478014] 000000009f8f2001 ffff880703603d57 000000009f8f2001 0000000000000004 [18241.478015] 0000000000000001 0000000000000004 ffff880703603d98 ffff880703603d78 [18241.478015] Call Trace: [18241.478024] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [18241.478031] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [18241.478037] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [18241.478043] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [18241.478047] [] ? wake_up_state+0x20/0x20 [18241.478053] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [18241.478055] [] kthread+0xcf/0xe0 [18241.478056] [] ? kthread_create_on_node+0x140/0x140 [18241.478058] [] ret_from_fork+0x58/0x90 [18241.478059] [] ? kthread_create_on_node+0x140/0x140 [18241.478069] Code: 49 89 fd 41 54 49 89 d4 53 48 89 cb 48 83 ec 70 65 48 8b 04 25 28 00 00 00 48 89 45 d0 31 c0 48 8b 05 f8 59 9c ff 48 85 c0 75 7b <41> f6 87 58 10 00 00 08 0f 84 62 03 00 00 4d 8d b7 e0 10 00 00 [18241.909868] [] kthread+0xcf/0xe0 [18241.916280] [] ? kthread_create_on_node+0x140/0x140 [18241.924543] [] ret_from_fork+0x58/0x90 [18241.931533] [] ? kthread_create_on_node+0x140/0x140 [18241.939792] kworker/12:0H S ffff88082d916840 0 72 2 0x00000000 [18241.948664] ffff8804b973fe50 0000000000000046 ffff8804b972af10 ffff8804b973ffd8 [18241.957906] ffff8804b973ffd8 ffff8804b973ffd8 ffff8804b972af10 ffff88082d916860 [18241.967134] ffff8801796e15b0 ffff8804b972af10 ffff8801796e1580 ffff88082d916840 [18241.976338] Call Trace: [18241.979951] [] schedule+0x29/0x70 [18241.986346] [] worker_thread+0x1d9/0x410 [18241.993404] [] ? rescuer_thread+0x460/0x460 [18242.000738] [] kthread+0xcf/0xe0 [18242.006980] [] ? kthread_create_on_node+0x140/0x140 [18242.015039] [] ret_from_fork+0x58/0x90 [18242.021811] [] ? kthread_create_on_node+0x140/0x140 [18242.029830] watchdog/13 S 0000000000000000 0 73 2 0x00000000 [18242.038493] ffff8804b9747e80 0000000000000046 ffff8804b972bec0 ffff8804b9747fd8 [18242.047558] ffff8804b9747fd8 ffff8804b9747fd8 ffff8804b972bec0 ffff8804b972bec0 [18242.056585] ffff8804ba74e0f0 ffffffff81a09820 0000000000000000 0000000000000000 [18242.065589] Call Trace: [18242.069003] [] schedule+0x29/0x70 [18242.075238] [] smpboot_thread_fn+0xdd/0x180 [18242.082459] [] ? lg_double_unlock+0x90/0x90 [18242.089661] [] kthread+0xcf/0xe0 [18242.095789] [] ? kthread_create_on_node+0x140/0x140 [18242.103757] [] ret_from_fork+0x58/0x90 [18242.110456] [] ? kthread_create_on_node+0x140/0x140 [18242.118423] migration/13 S 0000000000000000 0 74 2 0x00000000 [18242.127005] ffff8804b975be80 0000000000000046 ffff8804b972ce70 ffff8804b975bfd8 [18242.136013] ffff8804b975bfd8 ffff8804b975bfd8 ffff8804b972ce70 ffff8804b972ce70 [18242.145018] ffff8804ba74e100 ffffffff81a08f40 0000000000000000 0000000000000000 [18242.154010] Call Trace: [18242.157419] [] schedule+0x29/0x70 [18242.163643] [] smpboot_thread_fn+0xdd/0x180 [18242.170830] [] ? lg_double_unlock+0x90/0x90 [18242.178022] [] kthread+0xcf/0xe0 [18242.184162] [] ? kthread_create_on_node+0x140/0x140 [18242.192123] [] ret_from_fork+0x58/0x90 [18242.198818] [] ? kthread_create_on_node+0x140/0x140 [18242.206778] ksoftirqd/13 S 0000000000000000 0 75 2 0x00000000 [18242.215359] ffff8804b975fe80 0000000000000046 ffff8804b972de20 ffff8804b975ffd8 [18242.224339] ffff8804b975ffd8 ffff8804b975ffd8 ffff8804b972de20 ffff8804b972de20 [18242.233339] ffff8804ba74e110 ffffffff819f1100 0000000000000000 0000000000000000 [18242.242325] Call Trace: [18242.244110] INFO: task mount.nfs:8923 blocked for more than 120 seconds. [18242.244111] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [18242.244113] mount.nfs D ffff8803ec731c00 0 8923 8922 0x00000080 [18242.244115] ffff88016f4a39a0 0000000000000082 ffff880186b7bec0 ffff88016f4a3fd8 [18242.244116] ffff88016f4a3fd8 ffff88016f4a3fd8 ffff880186b7bec0 ffff88016f4a3ae0 [18242.244118] ffff88016f4a3ae8 7fffffffffffffff ffff880186b7bec0 ffff8803ec731c00 [18242.244118] Call Trace: [18242.244123] [] schedule+0x29/0x70 [18242.244125] [] schedule_timeout+0x239/0x2c0 [18242.244127] [] ? __schedule+0x3b0/0x990 [18242.244130] [] wait_for_completion+0x116/0x170 [18242.244134] [] ? wake_up_state+0x20/0x20 [18242.244138] [] ? dyntick_save_progress_counter+0x30/0x30 [18242.244140] [] __stop_cpus+0x5f/0x90 [18242.244142] [] ? dyntick_save_progress_counter+0x30/0x30 [18242.244143] [] try_stop_cpus+0x3a/0x60 [18242.244146] [] synchronize_sched_expedited+0xd8/0x1d0 [18242.244147] [] synchronize_rcu_expedited+0xe/0x10 [18242.244152] [] bdi_unregister+0x143/0x1b0 [18242.244168] [] nfs_put_super+0x19/0x20 [nfs] [18242.244172] [] generic_shutdown_super+0x72/0xf0 [18242.244174] [] kill_anon_super+0x12/0x20 [18242.244179] [] nfs_kill_super+0x1b/0x30 [nfs] [18242.244180] [] deactivate_locked_super+0x49/0x60 [18242.244182] [] deactivate_super+0x46/0x60 [18242.244185] [] mntput_no_expire+0xc5/0x120 [18242.244187] [] mntput+0x24/0x40 [18242.244189] [] namespace_unlock+0x118/0x130 [18242.244191] [] put_mnt_ns+0x4b/0x60 [18242.244193] [] mount_subtree+0x58/0xb0 [18242.244209] [] nfs_follow_remote_path+0x115/0x1b0 [nfsv4] [18242.244215] [] nfs4_try_mount+0x56/0xc0 [nfsv4] [18242.244219] [] ? get_nfs_version+0x27/0x90 [nfs] [18242.244225] [] nfs_fs_mount+0x4c5/0xd90 [nfs] [18242.244230] [] ? nfs_clone_super+0x140/0x140 [nfs] [18242.244234] [] ? param_set_portnr+0x70/0x70 [nfs] [18242.244236] [] mount_fs+0x39/0x1b0 [18242.244238] [] ? __alloc_percpu+0x15/0x20 [18242.244240] [] vfs_kern_mount+0x5f/0xf0 [18242.244242] [] do_mount+0x24e/0xaa0 [18242.244244] [] SyS_mount+0x96/0xf0 [18242.244245] [] system_call_fastpath+0x16/0x1b [18242.551724] [] schedule+0x29/0x70 [18242.557835] [] smpboot_thread_fn+0xdd/0x180 [18242.564915] [] ? lg_double_unlock+0x90/0x90 [18242.572009] [] kthread+0xcf/0xe0 [18242.578044] [] ? kthread_create_on_node+0x140/0x140 [18242.585882] [] ret_from_fork+0x58/0x90 [18242.592455] [] ? kthread_create_on_node+0x140/0x140 [18242.600275] kworker/13:0H S ffff88082d956840 0 77 2 0x00000000 [18242.608709] ffff8804b9767e50 0000000000000046 ffff8804b9768000 ffff8804b9767fd8 [18242.617550] ffff8804b9767fd8 ffff8804b9767fd8 ffff8804b9768000 ffff88082d956860 [18242.626393] ffff8801796e1930 ffff8804b9768000 ffff8801796e1900 ffff88082d956840 [18242.635250] Call Trace: [18242.638512] [] schedule+0x29/0x70 [18242.644595] [] worker_thread+0x1d9/0x410 [18242.651359] [] ? rescuer_thread+0x460/0x460 [18242.658426] [] kthread+0xcf/0xe0 [18242.664423] [] ? kthread_create_on_node+0x140/0x140 [18242.672275] [] ret_from_fork+0x58/0x90 [18242.678836] [] ? kthread_create_on_node+0x140/0x140 [18242.686669] watchdog/14 S 0000000000000000 0 78 2 0x00000000 [18242.695129] ffff8804b9793e80 0000000000000046 ffff8804b9768fb0 ffff8804b9793fd8 [18242.704009] ffff8804b9793fd8 ffff8804b9793fd8 ffff8804b9768fb0 ffff8804b9768fb0 [18242.712872] ffff8804ba74e120 ffffffff81a09820 0000000000000000 0000000000000000 [18242.721743] Call Trace: [18242.725039] [] schedule+0x29/0x70 [18242.731154] [] smpboot_thread_fn+0xdd/0x180 [18242.738232] [] ? lg_double_unlock+0x90/0x90 [18242.745326] [] kthread+0xcf/0xe0 [18242.751332] [] ? kthread_create_on_node+0x140/0x140 [18242.759198] [] ret_from_fork+0x58/0x90 [18242.765793] [] ? kthread_create_on_node+0x140/0x140 [18242.773657] migration/14 S 0000000000000000 0 79 2 0x00000000 [18242.782140] ffff8804b9797e80 0000000000000046 ffff8804b9769f60 ffff8804b9797fd8 [18242.791052] ffff8804b9797fd8 ffff8804b9797fd8 ffff8804b9769f60 ffff8804b9769f60 [18242.799953] ffff8804ba74e130 ffffffff81a08f40 0000000000000000 0000000000000000 [18242.808862] Call Trace: [18242.812194] [] schedule+0x29/0x70 [18242.818346] [] smpboot_thread_fn+0xdd/0x180 [18242.825469] [] ? lg_double_unlock+0x90/0x90 [18242.832579] [] kthread+0xcf/0xe0 [18242.838623] [] ? kthread_create_on_node+0x140/0x140 [18242.846513] [] ret_from_fork+0x58/0x90 [18242.853150] [] ? kthread_create_on_node+0x140/0x140 [18242.861053] ksoftirqd/14 S 0000000000000000 0 80 2 0x00000000 [18242.869570] ffff8804b979be80 0000000000000046 ffff8804b976af10 ffff8804b979bfd8 [18242.878524] ffff8804b979bfd8 ffff8804b979bfd8 ffff8804b976af10 ffff8804b976af10 [18242.887479] ffff8804ba74e140 ffffffff819f1100 0000000000000000 0000000000000000 [18242.896436] Call Trace: [18242.899798] [] schedule+0x29/0x70 [18242.905983] [] smpboot_thread_fn+0xdd/0x180 [18242.913134] [] ? lg_double_unlock+0x90/0x90 [18242.920285] [] kthread+0xcf/0xe0 [18242.926370] [] ? kthread_create_on_node+0x140/0x140 [18242.934313] [] ret_from_fork+0x58/0x90 [18242.940980] [] ? kthread_create_on_node+0x140/0x140 [18242.948897] kworker/14:0H S ffff88082d996840 0 82 2 0x00000000 [18242.957436] ffff8804b97a3e50 0000000000000046 ffff8804b976ce70 ffff8804b97a3fd8 [18242.966397] ffff8804b97a3fd8 ffff8804b97a3fd8 ffff8804b976ce70 ffff88082d996860 [18242.975361] ffff8801796e1cb0 ffff8804b976ce70 ffff8801796e1c80 ffff88082d996840 [18242.984314] Call Trace: [18242.987676] [] schedule+0x29/0x70 [18242.993853] [] worker_thread+0x1d9/0x410 [18243.000708] [] ? rescuer_thread+0x460/0x460 [18243.007864] [] kthread+0xcf/0xe0 [18243.013943] [] ? kthread_create_on_node+0x140/0x140 [18243.021858] [] ret_from_fork+0x58/0x90 [18243.028538] [] ? kthread_create_on_node+0x140/0x140 [18243.036466] watchdog/15 S 0000000000000000 0 83 2 0x00000000 [18243.044995] ffff8804b97bbe80 0000000000000046 ffff8804b976de20 ffff8804b97bbfd8 [18243.054039] ffff8804b97bbfd8 ffff8804b97bbfd8 ffff8804b976de20 ffff8804b976de20 [18243.063032] ffff8804ba74e150 ffffffff81a09820 0000000000000000 0000000000000000 [18243.071995] Call Trace: [18243.075369] [] schedule+0x29/0x70 [18243.081572] [] smpboot_thread_fn+0xdd/0x180 [18243.088740] [] ? lg_double_unlock+0x90/0x90 [18243.095983] [] kthread+0xcf/0xe0 [18243.102099] [] ? kthread_create_on_node+0x140/0x140 [18243.110103] [] ret_from_fork+0x58/0x90 [18243.116785] [] ? kthread_create_on_node+0x140/0x140 [18243.124717] migration/15 S 0000000000000000 0 84 2 0x00000000 [18243.133274] ffff8804b97bfe80 0000000000000046 ffff8804b976edd0 ffff8804b97bffd8 [18243.142247] ffff8804b97bffd8 ffff8804b97bffd8 ffff8804b976edd0 ffff8804b976edd0 [18243.151210] ffff8804ba74e160 ffffffff81a08f40 0000000000000000 0000000000000000 [18243.160168] Call Trace: [18243.163527] [] schedule+0x29/0x70 [18243.169712] [] smpboot_thread_fn+0xdd/0x180 [18243.176855] [] ? lg_double_unlock+0x90/0x90 [18243.184015] [] kthread+0xcf/0xe0 [18243.190103] [] ? kthread_create_on_node+0x140/0x140 [18243.198048] [] ret_from_fork+0x58/0x90 [18243.204710] [] ? kthread_create_on_node+0x140/0x140 [18243.212630] ksoftirqd/15 S 0000000000000000 0 85 2 0x00000000 [18243.221165] ffff8804b97cbe80 0000000000000046 ffff8804b97c0000 ffff8804b97cbfd8 [18243.230119] ffff8804b97cbfd8 ffff8804b97cbfd8 ffff8804b97c0000 ffff8804b97c0000 [18243.239077] ffff8804ba74e170 ffffffff819f1100 0000000000000000 0000000000000000 [18243.248024] Call Trace: [18243.251382] [] schedule+0x29/0x70 [18243.257574] [] smpboot_thread_fn+0xdd/0x180 [18243.264724] [] ? lg_double_unlock+0x90/0x90 [18243.271872] [] kthread+0xcf/0xe0 [18243.277947] [] ? kthread_create_on_node+0x140/0x140 [18243.285870] [] ret_from_fork+0x58/0x90 [18243.292545] [] ? kthread_create_on_node+0x140/0x140 [18243.300478] kworker/15:0H S ffff88082d9d6840 0 87 2 0x00000000 [18243.309035] ffff8804b97d7e50 0000000000000046 ffff8804b97c1f60 ffff8804b97d7fd8 [18243.317991] ffff8804b97d7fd8 ffff8804b97d7fd8 ffff8804b97c1f60 ffff88082d9d6860 [18243.326942] ffff8804b97d0030 ffff8804b97c1f60 ffff8804b97d0000 ffff88082d9d6840 [18243.335879] Call Trace: [18243.339244] [] schedule+0x29/0x70 [18243.345438] [] worker_thread+0x1d9/0x410 [18243.352309] [] ? rescuer_thread+0x460/0x460 [18243.359464] [] kthread+0xcf/0xe0 [18243.365555] [] ? kthread_create_on_node+0x140/0x140 [18243.373486] [] ret_from_fork+0x58/0x90 [18243.380152] [] ? kthread_create_on_node+0x140/0x140 [18243.388068] watchdog/16 S 0000000000000000 0 88 2 0x00000000 [18243.396602] ffff880179757e80 0000000000000046 ffff88017970edd0 ffff880179757fd8 [18243.405554] ffff880179757fd8 ffff880179757fd8 ffff88017970edd0 ffff88017970edd0 [18243.414501] ffff88017fc7cff0 ffffffff81a09820 0000000000000000 0000000000000000 [18243.423457] Call Trace: [18243.426813] [] schedule+0x29/0x70 [18243.433003] [] smpboot_thread_fn+0xdd/0x180 [18243.440164] [] ? lg_double_unlock+0x90/0x90 [18243.447329] [] kthread+0xcf/0xe0 [18243.453413] [] ? kthread_create_on_node+0x140/0x140 [18243.461354] [] ret_from_fork+0x58/0x90 [18243.468017] [] ? kthread_create_on_node+0x140/0x140 [18243.475931] migration/16 S 0000000000000000 0 89 2 0x00000000 [18243.484489] ffff880179783e80 0000000000000046 ffff88017970de20 ffff880179783fd8 [18243.493449] ffff880179783fd8 ffff880179783fd8 ffff88017970de20 ffff88017970de20 [18243.502390] ffff88017fc7cfe0 ffffffff81a08f40 0000000000000000 0000000000000000 [18243.511336] Call Trace: [18243.514690] [] schedule+0x29/0x70 [18243.520864] [] smpboot_thread_fn+0xdd/0x180 [18243.528026] [] ? lg_double_unlock+0x90/0x90 [18243.535177] [] kthread+0xcf/0xe0 [18243.541262] [] ? kthread_create_on_node+0x140/0x140 [18243.549189] [] ret_from_fork+0x58/0x90 [18243.555844] [] ? kthread_create_on_node+0x140/0x140 [18243.563760] ksoftirqd/16 S 0000000000000000 0 90 2 0x00000000 [18243.572286] ffff880179787e80 0000000000000046 ffff88017970ce70 ffff880179787fd8 [18243.581236] ffff880179787fd8 ffff880179787fd8 ffff88017970ce70 ffff88017970ce70 [18243.590182] ffff88017fc7cfd0 ffffffff819f1100 0000000000000000 0000000000000000 [18243.599139] Call Trace: [18243.602500] [] schedule+0x29/0x70 [18243.608675] [] smpboot_thread_fn+0xdd/0x180 [18243.615821] [] ? lg_double_unlock+0x90/0x90 [18243.622976] [] kthread+0xcf/0xe0 [18243.629054] [] ? kthread_create_on_node+0x140/0x140 [18243.636978] [] ret_from_fork+0x58/0x90 [18243.643638] [] ? kthread_create_on_node+0x140/0x140 [18243.651560] kworker/16:0 S ffff88042e216480 0 91 2 0x00000000 [18243.660124] ffff88017978be50 0000000000000046 ffff88017970bec0 ffff88017978bfd8 [18243.669075] ffff88017978bfd8 ffff88017978bfd8 ffff88017970bec0 ffff88042e2164a0 [18243.678034] ffff8804b97d02b0 ffff88017970bec0 ffff8804b97d0280 ffff88042e216480 [18243.686970] Call Trace: [18243.690347] [] schedule+0x29/0x70 [18243.696536] [] worker_thread+0x1d9/0x410 [18243.703398] [] ? rescuer_thread+0x460/0x460 [18243.710560] [] kthread+0xcf/0xe0 [18243.716646] [] ? kthread_create_on_node+0x140/0x140 [18243.724582] [] ret_from_fork+0x58/0x90 [18243.731246] [] ? kthread_create_on_node+0x140/0x140 [18243.739171] kworker/16:0H S ffff88042e216840 0 92 2 0x00000000 [18243.747702] ffff88017978fe50 0000000000000046 ffff88017970af10 ffff88017978ffd8 [18243.756651] ffff88017978ffd8 ffff88017978ffd8 ffff88017970af10 ffff88042e216860 [18243.765604] ffff8804b97d03b0 ffff88017970af10 ffff8804b97d0380 ffff88042e216840 [18243.774556] Call Trace: [18243.777909] [] schedule+0x29/0x70 [18243.784094] [] worker_thread+0x1d9/0x410 [18243.790937] [] ? rescuer_thread+0x460/0x460 [18243.798084] [] kthread+0xcf/0xe0 [18243.804160] [] ? kthread_create_on_node+0x140/0x140 [18243.812093] [] ret_from_fork+0x58/0x90 [18243.818750] [] ? kthread_create_on_node+0x140/0x140 [18243.826674] watchdog/17 S 0000000000000000 0 93 2 0x00000000 [18243.835213] ffff8801797b3e80 0000000000000046 ffff8801797a8000 ffff8801797b3fd8 [18243.844160] ffff8801797b3fd8 ffff8801797b3fd8 ffff8801797a8000 ffff8801797a8000 [18243.853113] ffff88017fc7cfc0 ffffffff81a09820 0000000000000000 0000000000000000 [18243.862055] Call Trace: [18243.865416] [] schedule+0x29/0x70 [18243.871601] [] smpboot_thread_fn+0xdd/0x180 [18243.878745] [] ? lg_double_unlock+0x90/0x90 [18243.885890] [] kthread+0xcf/0xe0 [18243.891955] [] ? kthread_create_on_node+0x140/0x140 [18243.899876] [] ret_from_fork+0x58/0x90 [18243.906527] [] ? kthread_create_on_node+0x140/0x140 [18243.914446] migration/17 S 0000000000000000 0 94 2 0x00000000 [18243.922971] ffff8801797b7e80 0000000000000046 ffff8801797a8fb0 ffff8801797b7fd8 [18243.931923] ffff8801797b7fd8 ffff8801797b7fd8 ffff8801797a8fb0 ffff8801797a8fb0 [18243.940855] ffff88017fc7cfb0 ffffffff81a08f40 0000000000000000 0000000000000000 [18243.949790] Call Trace: [18243.953148] [] schedule+0x29/0x70 [18243.959326] [] smpboot_thread_fn+0xdd/0x180 [18243.966497] [] ? lg_double_unlock+0x90/0x90 [18243.973642] [] kthread+0xcf/0xe0 [18243.979716] [] ? kthread_create_on_node+0x140/0x140 [18243.987646] [] ret_from_fork+0x58/0x90 [18243.994302] [] ? kthread_create_on_node+0x140/0x140 [18244.002227] ksoftirqd/17 S 0000000000000000 0 95 2 0x00000000 [18244.010748] ffff8801797c3e80 0000000000000046 ffff8801797a9f60 ffff8801797c3fd8 [18244.019694] ffff8801797c3fd8 ffff8801797c3fd8 ffff8801797a9f60 ffff8801797a9f60 [18244.028636] ffff88017fc7cfa0 ffffffff819f1100 0000000000000000 0000000000000000 [18244.037581] Call Trace: [18244.040937] [] schedule+0x29/0x70 [18244.047123] [] smpboot_thread_fn+0xdd/0x180 [18244.054274] [] ? lg_double_unlock+0x90/0x90 [18244.061440] [] kthread+0xcf/0xe0 [18244.067535] [] ? kthread_create_on_node+0x140/0x140 [18244.075478] [] ret_from_fork+0x58/0x90 [18244.082143] [] ? kthread_create_on_node+0x140/0x140 [18244.090064] kworker/17:0H S ffff88042e256840 0 97 2 0x00000000 [18244.098624] ffff8801797cbe50 0000000000000046 ffff8801797abec0 ffff8801797cbfd8 [18244.107586] ffff8801797cbfd8 ffff8801797cbfd8 ffff8801797abec0 ffff88042e256860 [18244.116545] ffff8804b97d0730 ffff8801797abec0 ffff8804b97d0700 ffff88042e256840 [18244.125498] Call Trace: [18244.128852] [] schedule+0x29/0x70 [18244.135034] [] worker_thread+0x1d9/0x410 [18244.141892] [] ? rescuer_thread+0x460/0x460 [18244.149060] [] kthread+0xcf/0xe0 [18244.155154] [] ? kthread_create_on_node+0x140/0x140 [18244.163080] [] ret_from_fork+0x58/0x90 [18244.169738] [] ? kthread_create_on_node+0x140/0x140 [18244.177661] watchdog/18 S 0000000000000000 0 98 2 0x00000000 [18244.186200] ffff8801797e3e80 0000000000000046 ffff8801797ace70 ffff8801797e3fd8 [18244.195160] ffff8801797e3fd8 ffff8801797e3fd8 ffff8801797ace70 ffff8801797ace70 [18244.204111] ffff88017fc7cf90 ffffffff81a09820 0000000000000000 0000000000000000 [18244.213079] Call Trace: [18244.216452] [] schedule+0x29/0x70 [18244.222642] [] smpboot_thread_fn+0xdd/0x180 [18244.229791] [] ? lg_double_unlock+0x90/0x90 [18244.236960] [] kthread+0xcf/0xe0 [18244.243055] [] ? kthread_create_on_node+0x140/0x140 [18244.250980] [] ret_from_fork+0x58/0x90 [18244.257642] [] ? kthread_create_on_node+0x140/0x140 [18244.265563] migration/18 S 0000000000000000 0 99 2 0x00000000 [18244.274089] ffff8801797e7e80 0000000000000046 ffff8801797ade20 ffff8801797e7fd8 [18244.283039] ffff8801797e7fd8 ffff8801797e7fd8 ffff8801797ade20 ffff8801797ade20 [18244.291983] ffff88017fc7cf80 ffffffff81a08f40 0000000000000000 0000000000000000 [18244.300919] Call Trace: [18244.304277] [] schedule+0x29/0x70 [18244.310473] [] smpboot_thread_fn+0xdd/0x180 [18244.317625] [] ? lg_double_unlock+0x90/0x90 [18244.324776] [] kthread+0xcf/0xe0 [18244.330854] [] ? kthread_create_on_node+0x140/0x140 [18244.338787] [] ret_from_fork+0x58/0x90 [18244.345458] [] ? kthread_create_on_node+0x140/0x140 [18244.353382] ksoftirqd/18 S 0000000000000000 0 100 2 0x00000000 [18244.361914] ffff8801797f3e80 0000000000000046 ffff8801797aedd0 ffff8801797f3fd8 [18244.370873] ffff8801797f3fd8 ffff8801797f3fd8 ffff8801797aedd0 ffff8801797aedd0 [18244.379809] ffff88017fc7cf70 ffffffff819f1100 0000000000000000 0000000000000000 [18244.388743] Call Trace: [18244.392100] [] schedule+0x29/0x70 [18244.398281] [] smpboot_thread_fn+0xdd/0x180 [18244.405441] [] ? lg_double_unlock+0x90/0x90 [18244.412593] [] kthread+0xcf/0xe0 [18244.418669] [] ? kthread_create_on_node+0x140/0x140 [18244.426751] [] ret_from_fork+0x58/0x90 [18244.433431] [] ? kthread_create_on_node+0x140/0x140 [18244.441382] kworker/18:0H S ffff88042e296840 0 102 2 0x00000000 [18244.449926] ffff880179003e50 0000000000000046 ffff8801797f8fb0 ffff880179003fd8 [18244.458887] ffff880179003fd8 ffff880179003fd8 ffff8801797f8fb0 ffff88042e296860 [18244.467855] ffff8804b97d0ab0 ffff8801797f8fb0 ffff8804b97d0a80 ffff88042e296840 [18244.476794] Call Trace: [18244.480165] [] schedule+0x29/0x70 [18244.486357] [] worker_thread+0x1d9/0x410 [18244.493217] [] ? rescuer_thread+0x460/0x460 [18244.500375] [] kthread+0xcf/0xe0 [18244.506463] [] ? kthread_create_on_node+0x140/0x140 [18244.514398] [] ret_from_fork+0x58/0x90 [18244.521059] [] ? kthread_create_on_node+0x140/0x140 [18244.528979] watchdog/19 S 0000000000000000 0 103 2 0x00000000 [18244.537508] ffff88017901fe80 0000000000000046 ffff8801797f9f60 ffff88017901ffd8 [18244.546452] ffff88017901ffd8 ffff88017901ffd8 ffff8801797f9f60 ffff8801797f9f60 [18244.555409] ffff88017fc7cf60 ffffffff81a09820 0000000000000000 0000000000000000 [18244.564362] Call Trace: [18244.567719] [] schedule+0x29/0x70 [18244.573898] [] smpboot_thread_fn+0xdd/0x180 [18244.581049] [] ? lg_double_unlock+0x90/0x90 [18244.588208] [] kthread+0xcf/0xe0 [18244.594295] [] ? kthread_create_on_node+0x140/0x140 [18244.602229] [] ret_from_fork+0x58/0x90 [18244.608887] [] ? kthread_create_on_node+0x140/0x140 [18244.616807] migration/19 S 0000000000000000 0 104 2 0x00000000 [18244.625337] ffff880179023e80 0000000000000046 ffff8801797faf10 ffff880179023fd8 [18244.634293] ffff880179023fd8 ffff880179023fd8 ffff8801797faf10 ffff8801797faf10 [18244.643239] ffff88017fc7cf50 ffffffff81a08f40 0000000000000000 0000000000000000 [18244.652191] Call Trace: [18244.655550] [] schedule+0x29/0x70 [18244.661725] [] smpboot_thread_fn+0xdd/0x180 [18244.668875] [] ? lg_double_unlock+0x90/0x90 [18244.676023] [] kthread+0xcf/0xe0 [18244.682110] [] ? kthread_create_on_node+0x140/0x140 [18244.690036] [] ret_from_fork+0x58/0x90 [18244.696694] [] ? kthread_create_on_node+0x140/0x140 [18244.704621] ksoftirqd/19 S 0000000000000000 0 105 2 0x00000000 [18244.713151] ffff880179027e80 0000000000000046 ffff8801797fbec0 ffff880179027fd8 [18244.722109] ffff880179027fd8 ffff880179027fd8 ffff8801797fbec0 ffff8801797fbec0 [18244.731042] ffff88017fc7cf40 ffffffff819f1100 0000000000000000 0000000000000000 [18244.739976] Call Trace: [18244.743332] [] schedule+0x29/0x70 [18244.749510] [] smpboot_thread_fn+0xdd/0x180 [18244.756667] [] ? lg_double_unlock+0x90/0x90 [18244.763815] [] kthread+0xcf/0xe0 [18244.769891] [] ? kthread_create_on_node+0x140/0x140 [18244.777813] [] ret_from_fork+0x58/0x90 [18244.784483] [] ? kthread_create_on_node+0x140/0x140 [18244.792411] kworker/19:0H S ffff88042e2d6840 0 107 2 0x00000000 [18244.800946] ffff88017902fe50 0000000000000046 ffff8801797fde20 ffff88017902ffd8 [18244.809891] ffff88017902ffd8 ffff88017902ffd8 ffff8801797fde20 ffff88042e2d6860 [18244.818842] ffff8804b97d0e30 ffff8801797fde20 ffff8804b97d0e00 ffff88042e2d6840 [18244.827778] Call Trace: [18244.831135] [] schedule+0x29/0x70 [18244.837313] [] worker_thread+0x1d9/0x410 [18244.844179] [] ? rescuer_thread+0x460/0x460 [18244.851329] [] kthread+0xcf/0xe0 [18244.857408] [] ? kthread_create_on_node+0x140/0x140 [18244.865333] [] ret_from_fork+0x58/0x90 [18244.871990] [] ? kthread_create_on_node+0x140/0x140 [18244.879909] watchdog/20 S 0000000000000000 0 108 2 0x00000000 [18244.888458] ffff880179037e80 0000000000000046 ffff8801797fedd0 ffff880179037fd8 [18244.897420] ffff880179037fd8 ffff880179037fd8 ffff8801797fedd0 ffff8801797fedd0 [18244.906377] ffff88017fc7cf30 ffffffff81a09820 0000000000000000 0000000000000000 [18244.915333] Call Trace: [18244.918694] [] schedule+0x29/0x70 [18244.924869] [] smpboot_thread_fn+0xdd/0x180 [18244.932025] [] ? lg_double_unlock+0x90/0x90 [18244.939181] [] kthread+0xcf/0xe0 [18244.945273] [] ? kthread_create_on_node+0x140/0x140 [18244.953209] [] ret_from_fork+0x58/0x90 [18244.959867] [] ? kthread_create_on_node+0x140/0x140 [18244.967793] migration/20 S 0000000000000000 0 109 2 0x00000000 [18244.976351] ffff88017905be80 0000000000000046 ffff880179050000 ffff88017905bfd8 [18244.985308] ffff88017905bfd8 ffff88017905bfd8 ffff880179050000 ffff880179050000 [18244.994254] ffff88017fc7cf20 ffffffff81a08f40 0000000000000000 0000000000000000 [18245.003207] Call Trace: [18245.006582] [] schedule+0x29/0x70 [18245.012762] [] smpboot_thread_fn+0xdd/0x180 [18245.019909] [] ? lg_double_unlock+0x90/0x90 [18245.027072] [] kthread+0xcf/0xe0 [18245.033170] [] ? kthread_create_on_node+0x140/0x140 [18245.041121] [] ret_from_fork+0x58/0x90 [18245.047778] [] ? kthread_create_on_node+0x140/0x140 [18245.055695] ksoftirqd/20 S 0000000000000000 0 110 2 0x00000000 [18245.064242] ffff88017905fe80 0000000000000046 ffff880179050fb0 ffff88017905ffd8 [18245.073218] ffff88017905ffd8 ffff88017905ffd8 ffff880179050fb0 ffff880179050fb0 [18245.082183] ffff88017fc7cf10 ffffffff819f1100 0000000000000000 0000000000000000 [18245.091137] Call Trace: [18245.094502] [] schedule+0x29/0x70 [18245.100678] [] smpboot_thread_fn+0xdd/0x180 [18245.107823] [] ? lg_double_unlock+0x90/0x90 [18245.114969] [] kthread+0xcf/0xe0 [18245.121042] [] ? kthread_create_on_node+0x140/0x140 [18245.128964] [] ret_from_fork+0x58/0x90 [18245.135632] [] ? kthread_create_on_node+0x140/0x140 [18245.143568] kworker/20:0 S ffff88042e316480 0 111 2 0x00000000 [18245.152118] ffff880179063e50 0000000000000046 ffff880179051f60 ffff880179063fd8 [18245.161077] ffff880179063fd8 ffff880179063fd8 ffff880179051f60 ffff88042e3164a0 [18245.170017] ffff8804b97d10b0 ffff880179051f60 ffff8804b97d1080 ffff88042e316480 [18245.178968] Call Trace: [18245.182328] [] schedule+0x29/0x70 [18245.188506] [] worker_thread+0x1d9/0x410 [18245.195368] [] ? rescuer_thread+0x460/0x460 [18245.202523] [] kthread+0xcf/0xe0 [18245.208616] [] ? kthread_create_on_node+0x140/0x140 [18245.216544] [] ret_from_fork+0x58/0x90 [18245.223224] [] ? kthread_create_on_node+0x140/0x140 [18245.231151] kworker/20:0H S ffff88042e316840 0 112 2 0x00000000 [18245.239701] ffff880179067e50 0000000000000046 ffff880179052f10 ffff880179067fd8 [18245.248663] ffff880179067fd8 ffff880179067fd8 ffff880179052f10 ffff88042e316860 [18245.257638] ffff8804b97d11b0 ffff880179052f10 ffff8804b97d1180 ffff88042e316840 [18245.266578] Call Trace: [18245.269931] [] schedule+0x29/0x70 [18245.276116] [] worker_thread+0x1d9/0x410 [18245.282975] [] ? rescuer_thread+0x460/0x460 [18245.290140] [] kthread+0xcf/0xe0 [18245.296213] [] ? kthread_create_on_node+0x140/0x140 [18245.304144] [] ret_from_fork+0x58/0x90 [18245.310800] [] ? kthread_create_on_node+0x140/0x140 [18245.318713] watchdog/21 S 0000000000000000 0 113 2 0x00000000 [18245.327255] ffff88017908fe80 0000000000000046 ffff880179053ec0 ffff88017908ffd8 [18245.336211] ffff88017908ffd8 ffff88017908ffd8 ffff880179053ec0 ffff880179053ec0 [18245.345181] ffff88017fc7cf00 ffffffff81a09820 0000000000000000 0000000000000000 [18245.354123] Call Trace: [18245.357492] [] schedule+0x29/0x70 [18245.363678] [] smpboot_thread_fn+0xdd/0x180 [18245.370831] [] ? lg_double_unlock+0x90/0x90 [18245.377982] [] kthread+0xcf/0xe0 [18245.384051] [] ? kthread_create_on_node+0x140/0x140 [18245.391971] [] ret_from_fork+0x58/0x90 [18245.398634] [] ? kthread_create_on_node+0x140/0x140 [18245.406557] migration/21 S 0000000000000000 0 114 2 0x00000000 [18245.415103] ffff8801790abe80 0000000000000046 ffff880179054e70 ffff8801790abfd8 [18245.424065] ffff8801790abfd8 ffff8801790abfd8 ffff880179054e70 ffff880179054e70 [18245.433013] ffff88017fc7cef0 ffffffff81a08f40 0000000000000000 0000000000000000 [18245.441955] Call Trace: [18245.445326] [] schedule+0x29/0x70 [18245.451500] [] smpboot_thread_fn+0xdd/0x180 [18245.458658] [] ? lg_double_unlock+0x90/0x90 [18245.465802] [] kthread+0xcf/0xe0 [18245.471876] [] ? kthread_create_on_node+0x140/0x140 [18245.479810] [] ret_from_fork+0x58/0x90 [18245.486472] [] ? kthread_create_on_node+0x140/0x140 [18245.494393] ksoftirqd/21 S 0000000000000000 0 115 2 0x00000000 [18245.502930] ffff8801790afe80 0000000000000046 ffff880179055e20 ffff8801790affd8 [18245.511880] ffff8801790affd8 ffff8801790affd8 ffff880179055e20 ffff880179055e20 [18245.520833] ffff88017fc7cee0 ffffffff819f1100 0000000000000000 0000000000000000 [18245.529775] Call Trace: [18245.533138] [] schedule+0x29/0x70 [18245.539336] [] smpboot_thread_fn+0xdd/0x180 [18245.546489] [] ? lg_double_unlock+0x90/0x90 [18245.553649] [] kthread+0xcf/0xe0 [18245.559724] [] ? kthread_create_on_node+0x140/0x140 [18245.567652] [] ret_from_fork+0x58/0x90 [18245.574321] [] ? kthread_create_on_node+0x140/0x140 [18245.582243] kworker/21:0H S ffff88042e356840 0 117 2 0x00000000 [18245.590787] ffff8801790b7e50 0000000000000046 ffff8801790b8000 ffff8801790b7fd8 [18245.599719] ffff8801790b7fd8 ffff8801790b7fd8 ffff8801790b8000 ffff88042e356860 [18245.608674] ffff8804b97d1530 ffff8801790b8000 ffff8804b97d1500 ffff88042e356840 [18245.617638] Call Trace: [18245.620997] [] schedule+0x29/0x70 [18245.627186] [] worker_thread+0x1d9/0x410 [18245.634040] [] ? rescuer_thread+0x460/0x460 [18245.641186] [] kthread+0xcf/0xe0 [18245.647275] [] ? kthread_create_on_node+0x140/0x140 [18245.655208] [] ret_from_fork+0x58/0x90 [18245.661866] [] ? kthread_create_on_node+0x140/0x140 [18245.669785] watchdog/22 S 0000000000000000 0 118 2 0x00000000 [18245.678327] ffff8801790dbe80 0000000000000046 ffff8801790b8fb0 ffff8801790dbfd8 [18245.687294] ffff8801790dbfd8 ffff8801790dbfd8 ffff8801790b8fb0 ffff8801790b8fb0 [18245.696256] ffff88017fc7ced0 ffffffff81a09820 0000000000000000 0000000000000000 [18245.705200] Call Trace: [18245.708559] [] schedule+0x29/0x70 [18245.714735] [] smpboot_thread_fn+0xdd/0x180 [18245.721882] [] ? lg_double_unlock+0x90/0x90 [18245.729036] [] kthread+0xcf/0xe0 [18245.735118] [] ? kthread_create_on_node+0x140/0x140 [18245.743056] [] ret_from_fork+0x58/0x90 [18245.749720] [] ? kthread_create_on_node+0x140/0x140 [18245.757656] migration/22 S 0000000000000000 0 119 2 0x00000000 [18245.766199] ffff8801790dfe80 0000000000000046 ffff8801790b9f60 ffff8801790dffd8 [18245.775149] ffff8801790dffd8 ffff8801790dffd8 ffff8801790b9f60 ffff8801790b9f60 [18245.784096] ffff88017fc7cec0 ffffffff81a08f40 0000000000000000 0000000000000000 [18245.793035] Call Trace: [18245.796406] [] schedule+0x29/0x70 [18245.802599] [] smpboot_thread_fn+0xdd/0x180 [18245.809744] [] ? lg_double_unlock+0x90/0x90 [18245.816893] [] kthread+0xcf/0xe0 [18245.822976] [] ? kthread_create_on_node+0x140/0x140 [18245.830898] [] ret_from_fork+0x58/0x90 [18245.837572] [] ? kthread_create_on_node+0x140/0x140 [18245.845504] ksoftirqd/22 S 0000000000000000 0 120 2 0x00000000 [18245.854032] ffff8801790e3e80 0000000000000046 ffff8801790baf10 ffff8801790e3fd8 [18245.862985] ffff8801790e3fd8 ffff8801790e3fd8 ffff8801790baf10 ffff8801790baf10 [18245.871930] ffff88017fc7ceb0 ffffffff819f1100 0000000000000000 0000000000000000 [18245.880871] Call Trace: [18245.884241] [] schedule+0x29/0x70 [18245.890417] [] smpboot_thread_fn+0xdd/0x180 [18245.897562] [] ? lg_double_unlock+0x90/0x90 [18245.904716] [] kthread+0xcf/0xe0 [18245.910790] [] ? kthread_create_on_node+0x140/0x140 [18245.918718] [] ret_from_fork+0x58/0x90 [18245.925391] [] ? kthread_create_on_node+0x140/0x140 [18245.933316] kworker/22:0H S ffff88042e396840 0 122 2 0x00000000 [18245.941857] ffff8801790ebe50 0000000000000046 ffff8801790bce70 ffff8801790ebfd8 [18245.950816] ffff8801790ebfd8 ffff8801790ebfd8 ffff8801790bce70 ffff88042e396860 [18245.959774] ffff8804b97d18b0 ffff8801790bce70 ffff8804b97d1880 ffff88042e396840 [18245.968703] Call Trace: [18245.972060] [] schedule+0x29/0x70 [18245.978252] [] worker_thread+0x1d9/0x410 [18245.985128] [] ? rescuer_thread+0x460/0x460 [18245.992283] [] kthread+0xcf/0xe0 [18245.998366] [] ? kthread_create_on_node+0x140/0x140 [18246.006298] [] ret_from_fork+0x58/0x90 [18246.012957] [] ? kthread_create_on_node+0x140/0x140 [18246.020889] watchdog/23 S 0000000000000000 0 123 2 0x00000000 [18246.029423] ffff880179103e80 0000000000000046 ffff8801790bde20 ffff880179103fd8 [18246.038397] ffff880179103fd8 ffff880179103fd8 ffff8801790bde20 ffff8801790bde20 [18246.047363] ffff88017fc7cea0 ffffffff81a09820 0000000000000000 0000000000000000 [18246.056312] Call Trace: [18246.059684] [] schedule+0x29/0x70 [18246.065861] [] smpboot_thread_fn+0xdd/0x180 [18246.073009] [] ? lg_double_unlock+0x90/0x90 [18246.080172] [] kthread+0xcf/0xe0 [18246.086257] [] ? kthread_create_on_node+0x140/0x140 [18246.094188] [] ret_from_fork+0x58/0x90 [18246.100853] [] ? kthread_create_on_node+0x140/0x140 [18246.108780] migration/23 S 0000000000000000 0 124 2 0x00000000 [18246.117335] ffff880179113e80 0000000000000046 ffff8801790bedd0 ffff880179113fd8 [18246.126310] ffff880179113fd8 ffff880179113fd8 ffff8801790bedd0 ffff8801790bedd0 [18246.135277] ffff88017fc7ce90 ffffffff81a08f40 0000000000000000 0000000000000000 [18246.144245] Call Trace: [18246.147613] [] schedule+0x29/0x70 [18246.153795] [] smpboot_thread_fn+0xdd/0x180 [18246.160944] [] ? lg_double_unlock+0x90/0x90 [18246.168105] [] kthread+0xcf/0xe0 [18246.174186] [] ? kthread_create_on_node+0x140/0x140 [18246.182125] [] ret_from_fork+0x58/0x90 [18246.188785] [] ? kthread_create_on_node+0x140/0x140 [18246.196720] ksoftirqd/23 S 0000000000000000 0 125 2 0x00000000 [18246.205269] ffff880179117e80 0000000000000046 ffff880179118000 ffff880179117fd8 [18246.214227] ffff880179117fd8 ffff880179117fd8 ffff880179118000 ffff880179118000 [18246.223183] ffff88017fc7ce80 ffffffff819f1100 0000000000000000 0000000000000000 [18246.232125] Call Trace: [18246.235488] [] schedule+0x29/0x70 [18246.241669] [] smpboot_thread_fn+0xdd/0x180 [18246.248816] [] ? lg_double_unlock+0x90/0x90 [18246.255963] [] kthread+0xcf/0xe0 [18246.262039] [] ? kthread_create_on_node+0x140/0x140 [18246.269963] [] ret_from_fork+0x58/0x90 [18246.276637] [] ? kthread_create_on_node+0x140/0x140 [18246.284577] kworker/23:0H S ffff88042e3d6840 0 127 2 0x00000000 [18246.293126] ffff880179127e50 0000000000000046 ffff880179119f60 ffff880179127fd8 [18246.302080] ffff880179127fd8 ffff880179127fd8 ffff880179119f60 ffff88042e3d6860 [18246.311050] ffff8804b97d1c30 ffff880179119f60 ffff8804b97d1c00 ffff88042e3d6840 [18246.320001] Call Trace: [18246.323369] [] schedule+0x29/0x70 [18246.329544] [] worker_thread+0x1d9/0x410 [18246.336423] [] ? rescuer_thread+0x460/0x460 [18246.343577] [] kthread+0xcf/0xe0 [18246.349667] [] ? kthread_create_on_node+0x140/0x140 [18246.357596] [] ret_from_fork+0x58/0x90 [18246.364264] [] ? kthread_create_on_node+0x140/0x140 [18246.372187] watchdog/24 R running task 0 128 2 0x00000000 [18246.380734] ffff8804b97dfe80 0000000000000046 ffff8804b97c6dd0 ffff8804b97dffd8 [18246.389699] ffff8804b97dffd8 ffff8804b97dffd8 ffff8804b97c6dd0 ffff8804b97c6dd0 [18246.398661] ffff8804ba74eff0 ffffffff81a09820 0000000000000000 0000000000000000 [18246.407615] Call Trace: [18246.410973] [] schedule+0x29/0x70 [18246.417159] [] smpboot_thread_fn+0xdd/0x180 [18246.424325] [] ? lg_double_unlock+0x90/0x90 [18246.431484] [] kthread+0xcf/0xe0 [18246.437566] [] ? kthread_create_on_node+0x140/0x140 [18246.445499] [] ret_from_fork+0x58/0x90 [18246.452164] [] ? kthread_create_on_node+0x140/0x140 [18246.460105] migration/24 R running task 0 129 2 0x00000000 [18246.468668] ffff8804b9003e80 0000000000000046 ffff8804b97c5e20 ffff8804b9003fd8 [18246.477634] ffff8804b9003fd8 ffff8804b9003fd8 ffff8804b97c5e20 ffff8804b97c5e20 [18246.486577] ffff8804ba74efe0 ffffffff81a08f40 0000000000000000 0000000000000000 [18246.495542] Call Trace: [18246.498908] [] schedule+0x29/0x70 [18246.505092] [] smpboot_thread_fn+0xdd/0x180 [18246.512271] [] ? lg_double_unlock+0x90/0x90 [18246.519455] [] kthread+0xcf/0xe0 [18246.525545] [] ? kthread_create_on_node+0x140/0x140 [18246.533482] [] ret_from_fork+0x58/0x90 [18246.540160] [] ? kthread_create_on_node+0x140/0x140 [18246.548075] ksoftirqd/24 R running task 0 130 2 0x00000000 [18246.556627] ffff8804b9007e80 0000000000000046 ffff8804b97c4e70 ffff8804b9007fd8 [18246.565595] ffff8804b9007fd8 ffff8804b9007fd8 ffff8804b97c4e70 ffff8804b97c4e70 [18246.574554] ffff8804ba74efd0 ffffffff819f1100 0000000000000000 0000000000000000 [18246.583521] Call Trace: [18246.586895] [] schedule+0x29/0x70 [18246.593088] [] smpboot_thread_fn+0xdd/0x180 [18246.600254] [] ? lg_double_unlock+0x90/0x90 [18246.607421] [] kthread+0xcf/0xe0 [18246.613522] [] ? kthread_create_on_node+0x140/0x140 [18246.621473] [] ret_from_fork+0x58/0x90 [18246.628151] [] ? kthread_create_on_node+0x140/0x140 [18246.636093] kworker/24:0H S ffff88082da16840 0 132 2 0x00000000 [18246.644647] ffff8804b900fe50 0000000000000046 ffff8804b97c2f10 ffff8804b900ffd8 [18246.653618] ffff8804b900ffd8 ffff8804b900ffd8 ffff8804b97c2f10 ffff88082da16860 [18246.662589] ffff8804b97d1fb0 ffff8804b97c2f10 ffff8804b97d1f80 ffff88082da16840 [18246.671549] Call Trace: [18246.674916] [] schedule+0x29/0x70 [18246.681111] [] worker_thread+0x1d9/0x410 [18246.687990] [] ? rescuer_thread+0x460/0x460 [18246.695158] [] kthread+0xcf/0xe0 [18246.701258] [] ? kthread_create_on_node+0x140/0x140 [18246.709206] [] ret_from_fork+0x58/0x90 [18246.715882] [] ? kthread_create_on_node+0x140/0x140 [18246.723825] watchdog/25 S 0000000000000000 0 133 2 0x00000000 [18246.732364] ffff8804b9037e80 0000000000000046 ffff8804b9048000 ffff8804b9037fd8 [18246.741324] ffff8804b9037fd8 ffff8804b9037fd8 ffff8804b9048000 ffff8804b9048000 [18246.750286] ffff8804ba74efc0 ffffffff81a09820 0000000000000000 0000000000000000 [18246.759243] Call Trace: [18246.762602] [] schedule+0x29/0x70 [18246.768781] [] smpboot_thread_fn+0xdd/0x180 [18246.775939] [] ? lg_double_unlock+0x90/0x90 [18246.783097] [] kthread+0xcf/0xe0 [18246.789186] [] ? kthread_create_on_node+0x140/0x140 [18246.797112] [] ret_from_fork+0x58/0x90 [18246.803773] [] ? kthread_create_on_node+0x140/0x140 [18246.811695] migration/25 S 0000000000000000 0 134 2 0x00000000 [18246.820223] ffff8804b9053e80 0000000000000046 ffff8804b9048fb0 ffff8804b9053fd8 [18246.829184] ffff8804b9053fd8 ffff8804b9053fd8 ffff8804b9048fb0 ffff8804b9048fb0 [18246.838130] ffff8804ba74efb0 ffffffff81a08f40 0000000000000000 0000000000000000 [18246.847074] Call Trace: [18246.850430] [] schedule+0x29/0x70 [18246.856617] [] smpboot_thread_fn+0xdd/0x180 [18246.863785] [] ? lg_double_unlock+0x90/0x90 [18246.870930] [] kthread+0xcf/0xe0 [18246.877011] [] ? kthread_create_on_node+0x140/0x140 [18246.885303] [] ret_from_fork+0x58/0x90 [18246.892323] [] ? kthread_create_on_node+0x140/0x140 [18246.900256] ksoftirqd/25 S 0000000000000000 0 135 2 0x00000000 [18246.908805] ffff8804b9057e80 0000000000000046 ffff8804b9049f60 ffff8804b9057fd8 [18246.917768] ffff8804b9057fd8 ffff8804b9057fd8 ffff8804b9049f60 ffff8804b9049f60 [18246.926724] ffff8804ba74efa0 ffffffff819f1100 0000000000000000 0000000000000000 [18246.935665] Call Trace: [18246.939020] [] schedule+0x29/0x70 [18246.945194] [] smpboot_thread_fn+0xdd/0x180 [18246.952348] [] ? lg_double_unlock+0x90/0x90 [18246.959501] [] kthread+0xcf/0xe0 [18246.965583] [] ? kthread_create_on_node+0x140/0x140 [18246.973526] [] ret_from_fork+0x58/0x90 [18246.980190] [] ? kthread_create_on_node+0x140/0x140 [18246.988120] kworker/25:0H S ffff88082da56840 0 137 2 0x00000000 [18246.996652] ffff8804b9067e50 0000000000000046 ffff8804b904bec0 ffff8804b9067fd8 [18247.005615] ffff8804b9067fd8 ffff8804b9067fd8 ffff8804b904bec0 ffff88082da56860 [18247.014569] ffff880179134330 ffff8804b904bec0 ffff880179134300 ffff88082da56840 [18247.023511] Call Trace: [18247.026863] [] schedule+0x29/0x70 [18247.033047] [] worker_thread+0x1d9/0x410 [18247.039902] [] ? rescuer_thread+0x460/0x460 [18247.047047] [] kthread+0xcf/0xe0 [18247.053122] [] ? kthread_create_on_node+0x140/0x140 [18247.061045] [] ret_from_fork+0x58/0x90 [18247.067713] [] ? kthread_create_on_node+0x140/0x140 [18247.075631] watchdog/26 S 0000000000000000 0 138 2 0x00000000 [18247.084180] ffff8804b9083e80 0000000000000046 ffff8804b904ce70 ffff8804b9083fd8 [18247.093137] ffff8804b9083fd8 ffff8804b9083fd8 ffff8804b904ce70 ffff8804b904ce70 [18247.102077] ffff8804ba74ef90 ffffffff81a09820 0000000000000000 0000000000000000 [18247.111016] Call Trace: [18247.114379] [] schedule+0x29/0x70 [18247.120569] [] smpboot_thread_fn+0xdd/0x180 [18247.127733] [] ? lg_double_unlock+0x90/0x90 [18247.134883] [] kthread+0xcf/0xe0 [18247.140958] [] ? kthread_create_on_node+0x140/0x140 [18247.148882] [] ret_from_fork+0x58/0x90 [18247.155550] [] ? kthread_create_on_node+0x140/0x140 [18247.163479] migration/26 S 0000000000000000 0 139 2 0x00000000 [18247.172016] ffff8804b9087e80 0000000000000046 ffff8804b904de20 ffff8804b9087fd8 [18247.180968] ffff8804b9087fd8 ffff8804b9087fd8 ffff8804b904de20 ffff8804b904de20 [18247.189913] ffff8804ba74ef80 ffffffff81a08f40 0000000000000000 0000000000000000 [18247.198853] Call Trace: [18247.202210] [] schedule+0x29/0x70 [18247.208396] [] smpboot_thread_fn+0xdd/0x180 [18247.215554] [] ? lg_double_unlock+0x90/0x90 [18247.222711] [] kthread+0xcf/0xe0 [18247.228788] [] ? kthread_create_on_node+0x140/0x140 [18247.236720] [] ret_from_fork+0x58/0x90 [18247.243389] [] ? kthread_create_on_node+0x140/0x140 [18247.251314] ksoftirqd/26 S 0000000000000000 0 140 2 0x00000000 [18247.259847] ffff8804b908be80 0000000000000046 ffff8804b904edd0 ffff8804b908bfd8 [18247.268789] ffff8804b908bfd8 ffff8804b908bfd8 ffff8804b904edd0 ffff8804b904edd0 [18247.277740] ffff8804ba74ef70 ffffffff819f1100 0000000000000000 0000000000000000 [18247.286688] Call Trace: [18247.290048] [] schedule+0x29/0x70 [18247.296225] [] smpboot_thread_fn+0xdd/0x180 [18247.303375] [] ? lg_double_unlock+0x90/0x90 [18247.310523] [] kthread+0xcf/0xe0 [18247.316615] [] ? kthread_create_on_node+0x140/0x140 [18247.324896] [] ret_from_fork+0x58/0x90 [18247.331934] [] ? kthread_create_on_node+0x140/0x140 [18247.339833] kworker/26:0H S ffff88082da96840 0 142 2 0x00000000 [18247.348387] ffff8804b90a3e50 0000000000000046 ffff8804b9090fb0 ffff8804b90a3fd8 [18247.357354] ffff8804b90a3fd8 ffff8804b90a3fd8 ffff8804b9090fb0 ffff88082da96860 [18247.366321] ffff8801791346b0 ffff8804b9090fb0 ffff880179134680 ffff88082da96840 [18247.375270] Call Trace: [18247.378633] [] schedule+0x29/0x70 [18247.384812] [] worker_thread+0x1d9/0x410 [18247.391668] [] ? rescuer_thread+0x460/0x460 [18247.398822] [] kthread+0xcf/0xe0 [18247.404906] [] ? kthread_create_on_node+0x140/0x140 [18247.412828] [] ret_from_fork+0x58/0x90 [18247.419495] [] ? kthread_create_on_node+0x140/0x140 [18247.427427] watchdog/27 S 0000000000000000 0 143 2 0x00000000 [18247.435955] ffff8804b90bfe80 0000000000000046 ffff8804b9091f60 ffff8804b90bffd8 [18247.444897] ffff8804b90bffd8 ffff8804b90bffd8 ffff8804b9091f60 ffff8804b9091f60 [18247.453829] ffff8804ba74ef60 ffffffff81a09820 0000000000000000 0000000000000000 [18247.462758] Call Trace: [18247.466111] [] schedule+0x29/0x70 [18247.472290] [] smpboot_thread_fn+0xdd/0x180 [18247.479453] [] ? lg_double_unlock+0x90/0x90 [18247.486611] [] kthread+0xcf/0xe0 [18247.492696] [] ? kthread_create_on_node+0x140/0x140 [18247.500630] [] ret_from_fork+0x58/0x90 [18247.507296] [] ? kthread_create_on_node+0x140/0x140 [18247.515218] migration/27 S 0000000000000000 0 144 2 0x00000000 [18247.523748] ffff8804b90c3e80 0000000000000046 ffff8804b9092f10 ffff8804b90c3fd8 [18247.532702] ffff8804b90c3fd8 ffff8804b90c3fd8 ffff8804b9092f10 ffff8804b9092f10 [18247.541647] ffff8804ba74ef50 ffffffff81a08f40 0000000000000000 0000000000000000 [18247.550602] Call Trace: [18247.553964] [] schedule+0x29/0x70 [18247.560154] [] smpboot_thread_fn+0xdd/0x180 [18247.567319] [] ? lg_double_unlock+0x90/0x90 [18247.574472] [] kthread+0xcf/0xe0 [18247.580559] [] ? kthread_create_on_node+0x140/0x140 [18247.588499] [] ret_from_fork+0x58/0x90 [18247.595159] [] ? kthread_create_on_node+0x140/0x140 [18247.603065] ksoftirqd/27 S 0000000000000000 0 145 2 0x00000000 [18247.611595] ffff8804b90c7e80 0000000000000046 ffff8804b9093ec0 ffff8804b90c7fd8 [18247.620546] ffff8804b90c7fd8 ffff8804b90c7fd8 ffff8804b9093ec0 ffff8804b9093ec0 [18247.629506] ffff8804ba74ef40 ffffffff819f1100 0000000000000000 0000000000000000 [18247.638447] Call Trace: [18247.641805] [] schedule+0x29/0x70 [18247.647982] [] smpboot_thread_fn+0xdd/0x180 [18247.655129] [] ? lg_double_unlock+0x90/0x90 [18247.662304] [] kthread+0xcf/0xe0 [18247.668395] [] ? kthread_create_on_node+0x140/0x140 [18247.676333] [] ret_from_fork+0x58/0x90 [18247.682989] [] ? kthread_create_on_node+0x140/0x140 [18247.690911] kworker/27:0 S ffff88082dad6480 0 146 2 0x00000000 [18247.699447] ffff8804b90cbe50 0000000000000046 ffff8804b9094e70 ffff8804b90cbfd8 [18247.708411] ffff8804b90cbfd8 ffff8804b90cbfd8 ffff8804b9094e70 ffff88082dad64a0 [18247.717364] ffff880179134930 ffff8804b9094e70 ffff880179134900 ffff88082dad6480 [18247.726318] Call Trace: [18247.729672] [] schedule+0x29/0x70 [18247.735839] [] worker_thread+0x1d9/0x410 [18247.742703] [] ? rescuer_thread+0x460/0x460 [18247.749851] [] kthread+0xcf/0xe0 [18247.755926] [] ? kthread_create_on_node+0x140/0x140 [18247.763853] [] ret_from_fork+0x58/0x90 [18247.770890] [] ? kthread_create_on_node+0x140/0x140 [18247.779178] kworker/27:0H S ffff88082dad6840 0 147 2 0x00000000 [18247.787750] ffff8804b90cfe50 0000000000000046 ffff8804b9095e20 ffff8804b90cffd8 [18247.796725] ffff8804b90cffd8 ffff8804b90cffd8 ffff8804b9095e20 ffff88082dad6860 [18247.805676] ffff880179134a30 ffff8804b9095e20 ffff880179134a00 ffff88082dad6840 [18247.814619] Call Trace: [18247.817973] [] schedule+0x29/0x70 [18247.824136] [] worker_thread+0x1d9/0x410 [18247.831005] [] ? rescuer_thread+0x460/0x460 [18247.838152] [] kthread+0xcf/0xe0 [18247.844226] [] ? kthread_create_on_node+0x140/0x140 [18247.852146] [] ret_from_fork+0x58/0x90 [18247.858792] [] ? kthread_create_on_node+0x140/0x140 [18247.866716] watchdog/28 R running task 0 148 2 0x00000000 [18247.875249] ffff8804b90d7e80 0000000000000046 ffff8804b9096dd0 ffff8804b90d7fd8 [18247.884213] ffff8804b90d7fd8 ffff8804b90d7fd8 ffff8804b9096dd0 ffff8804b9096dd0 [18247.893159] ffff8804ba74ef30 ffffffff81a09820 0000000000000000 0000000000000000 [18247.902099] Call Trace: [18247.905454] [] schedule+0x29/0x70 [18247.911635] [] smpboot_thread_fn+0xdd/0x180 [18247.918797] [] ? lg_double_unlock+0x90/0x90 [18247.925938] [] kthread+0xcf/0xe0 [18247.932013] [] ? kthread_create_on_node+0x140/0x140 [18247.939936] [] ret_from_fork+0x58/0x90 [18247.946597] [] ? kthread_create_on_node+0x140/0x140 [18247.954533] migration/28 R running task 0 149 2 0x00000000 [18247.963059] ffff8804b90f3e80 0000000000000046 ffff8804b90e8000 ffff8804b90f3fd8 [18247.972014] ffff8804b90f3fd8 ffff8804b90f3fd8 ffff8804b90e8000 ffff8804b90e8000 [18247.980976] ffff8804ba74ef20 ffffffff81a08f40 0000000000000000 0000000000000000 [18247.989918] Call Trace: [18247.993296] [] schedule+0x29/0x70 [18247.999485] [] smpboot_thread_fn+0xdd/0x180 [18248.006651] [] ? lg_double_unlock+0x90/0x90 [18248.013805] [] kthread+0xcf/0xe0 [18248.019886] [] ? kthread_create_on_node+0x140/0x140 [18248.027814] [] ret_from_fork+0x58/0x90 [18248.034484] [] ? kthread_create_on_node+0x140/0x140 [18248.042415] ksoftirqd/28 R running task 0 150 2 0x00000000 [18248.050937] ffff8804b90f7e80 0000000000000046 ffff8804b90e8fb0 ffff8804b90f7fd8 [18248.059895] ffff8804b90f7fd8 ffff8804b90f7fd8 ffff8804b90e8fb0 ffff8804b90e8fb0 [18248.068851] ffff8804ba74ef10 ffffffff819f1100 0000000000000000 0000000000000000 [18248.077810] Call Trace: [18248.081175] [] schedule+0x29/0x70 [18248.087374] [] smpboot_thread_fn+0xdd/0x180 [18248.094541] [] ? lg_double_unlock+0x90/0x90 [18248.101702] [] kthread+0xcf/0xe0 [18248.107798] [] ? kthread_create_on_node+0x140/0x140 [18248.115741] [] ret_from_fork+0x58/0x90 [18248.122425] [] ? kthread_create_on_node+0x140/0x140 [18248.130353] kworker/28:0 S ffff88082db16480 0 151 2 0x00000000 [18248.138890] ffff8804b90fbe50 0000000000000046 ffff8804b90e9f60 ffff8804b90fbfd8 [18248.147857] ffff8804b90fbfd8 ffff8804b90fbfd8 ffff8804b90e9f60 ffff88082db164a0 [18248.156806] ffff880179134cb0 ffff8804b90e9f60 ffff880179134c80 ffff88082db16480 [18248.165775] Call Trace: [18248.169140] [] schedule+0x29/0x70 [18248.175343] [] worker_thread+0x1d9/0x410 [18248.182216] [] ? rescuer_thread+0x460/0x460 [18248.189737] [] kthread+0xcf/0xe0 [18248.196192] [] ? kthread_create_on_node+0x140/0x140 [18248.204107] [] ret_from_fork+0x58/0x90 [18248.210804] [] ? kthread_create_on_node+0x140/0x140 [18248.218741] kworker/28:0H S ffff88082db16840 0 152 2 0x00000000 [18248.227295] ffff8804b90ffe50 0000000000000046 ffff8804b90eaf10 ffff8804b90fffd8 [18248.236260] ffff8804b90fffd8 ffff8804b90fffd8 ffff8804b90eaf10 ffff88082db16860 [18248.245214] ffff880179134db0 ffff8804b90eaf10 ffff880179134d80 ffff88082db16840 [18248.254160] Call Trace: [18248.257536] [] schedule+0x29/0x70 [18248.263725] [] worker_thread+0x1d9/0x410 [18248.270595] [] ? rescuer_thread+0x460/0x460 [18248.277755] [] kthread+0xcf/0xe0 [18248.283837] [] ? kthread_create_on_node+0x140/0x140 [18248.291777] [] ret_from_fork+0x58/0x90 [18248.298450] [] ? kthread_create_on_node+0x140/0x140 [18248.306373] watchdog/29 S 0000000000000000 0 153 2 0x00000000 [18248.314900] ffff8804b911be80 0000000000000046 ffff8804b90ebec0 ffff8804b911bfd8 [18248.323852] ffff8804b911bfd8 ffff8804b911bfd8 ffff8804b90ebec0 ffff8804b90ebec0 [18248.332788] ffff8804ba74ef00 ffffffff81a09820 0000000000000000 0000000000000000 [18248.341733] Call Trace: [18248.345088] [] schedule+0x29/0x70 [18248.351323] [] smpboot_thread_fn+0xdd/0x180 [18248.358464] [] ? lg_double_unlock+0x90/0x90 [18248.365631] [] kthread+0xcf/0xe0 [18248.371717] [] ? kthread_create_on_node+0x140/0x140 [18248.379649] [] ret_from_fork+0x58/0x90 [18248.386320] [] ? kthread_create_on_node+0x140/0x140 [18248.394246] migration/29 S 0000000000000000 0 154 2 0x00000000 [18248.402779] ffff8804b911fe80 0000000000000046 ffff8804b90ece70 ffff8804b911ffd8 [18248.411737] ffff8804b911ffd8 ffff8804b911ffd8 ffff8804b90ece70 ffff8804b90ece70 [18248.420674] ffff8804ba74eef0 ffffffff81a08f40 0000000000000000 0000000000000000 [18248.429619] Call Trace: [18248.432970] [] schedule+0x29/0x70 [18248.439145] [] smpboot_thread_fn+0xdd/0x180 [18248.446294] [] ? lg_double_unlock+0x90/0x90 [18248.453442] [] kthread+0xcf/0xe0 [18248.459526] [] ? kthread_create_on_node+0x140/0x140 [18248.467464] [] ret_from_fork+0x58/0x90 [18248.474121] [] ? kthread_create_on_node+0x140/0x140 [18248.482037] ksoftirqd/29 S 0000000000000000 0 155 2 0x00000000 [18248.490568] ffff8804b9123e80 0000000000000046 ffff8804b90ede20 ffff8804b9123fd8 [18248.499524] ffff8804b9123fd8 ffff8804b9123fd8 ffff8804b90ede20 ffff8804b90ede20 [18248.508480] ffff8804ba74eee0 ffffffff819f1100 0000000000000000 0000000000000000 [18248.517426] Call Trace: [18248.520798] [] schedule+0x29/0x70 [18248.526982] [] smpboot_thread_fn+0xdd/0x180 [18248.534131] [] ? lg_double_unlock+0x90/0x90 [18248.541298] [] kthread+0xcf/0xe0 [18248.547386] [] ? kthread_create_on_node+0x140/0x140 [18248.555320] [] ret_from_fork+0x58/0x90 [18248.562350] [] ? kthread_create_on_node+0x140/0x140 [18248.570548] kworker/29:0H S ffff88082db56840 0 157 2 0x00000000 [18248.579079] ffff8804b9133e50 0000000000000046 ffff8804b9128000 ffff8804b9133fd8 [18248.588039] ffff8804b9133fd8 ffff8804b9133fd8 ffff8804b9128000 ffff88082db56860 [18248.596992] ffff880179135130 ffff8804b9128000 ffff880179135100 ffff88082db56840 [18248.606295] Call Trace: [18248.609658] [] schedule+0x29/0x70 [18248.615836] [] worker_thread+0x1d9/0x410 [18248.622705] [] ? rescuer_thread+0x460/0x460 [18248.630224] [] kthread+0xcf/0xe0 [18248.636324] [] ? kthread_create_on_node+0x140/0x140 [18248.644613] [] ret_from_fork+0x58/0x90 [18248.651293] [] ? kthread_create_on_node+0x140/0x140 [18248.659208] watchdog/30 S 0000000000000000 0 158 2 0x00000000 [18248.667773] ffff8804b916be80 0000000000000046 ffff8804b9128fb0 ffff8804b916bfd8 [18248.677076] ffff8804b916bfd8 ffff8804b916bfd8 ffff8804b9128fb0 ffff8804b9128fb0 [18248.686036] ffff8804ba74eed0 ffffffff81a09820 0000000000000000 0000000000000000 [18248.694986] Call Trace: [18248.698372] [] schedule+0x29/0x70 [18248.704576] [] smpboot_thread_fn+0xdd/0x180 [18248.711742] [] ? lg_double_unlock+0x90/0x90 [18248.719251] [] kthread+0xcf/0xe0 [18248.725345] [] ? kthread_create_on_node+0x140/0x140 [18248.733304] [] ret_from_fork+0x58/0x90 [18248.739964] [] ? kthread_create_on_node+0x140/0x140 [18248.747881] migration/30 S 0000000000000000 0 159 2 0x00000000 [18248.756756] ffff8804b916fe80 0000000000000046 ffff8804b9129f60 ffff8804b916ffd8 [18248.765740] ffff8804b916ffd8 ffff8804b916ffd8 ffff8804b9129f60 ffff8804b9129f60 [18248.774720] ffff8804ba74eec0 ffffffff81a08f40 0000000000000000 0000000000000000 [18248.783686] Call Trace: [18248.787413] [] schedule+0x29/0x70 [18248.793951] [] smpboot_thread_fn+0xdd/0x180 [18248.801082] [] ? lg_double_unlock+0x90/0x90 [18248.808243] [] kthread+0xcf/0xe0 [18248.814343] [] ? kthread_create_on_node+0x140/0x140 [18248.822276] [] ret_from_fork+0x58/0x90 [18248.829312] [] ? kthread_create_on_node+0x140/0x140 [18248.837255] ksoftirqd/30 S 0000000000000000 0 160 2 0x00000000 [18248.845801] ffff8804b9173e80 0000000000000046 ffff8804b912af10 ffff8804b9173fd8 [18248.854769] ffff8804b9173fd8 ffff8804b9173fd8 ffff8804b912af10 ffff8804b912af10 [18248.863724] ffff8804ba74eeb0 ffffffff819f1100 0000000000000000 0000000000000000 [18248.873009] Call Trace: [18248.876379] [] schedule+0x29/0x70 [18248.882561] [] smpboot_thread_fn+0xdd/0x180 [18248.889724] [] ? lg_double_unlock+0x90/0x90 [18248.896880] [] kthread+0xcf/0xe0 [18248.903039] [] ? kthread_create_on_node+0x140/0x140 [18248.911326] [] ret_from_fork+0x58/0x90 [18248.917971] [] ? kthread_create_on_node+0x140/0x140 [18248.925868] kworker/30:0 S ffff88082db96480 0 161 2 0x00000000 [18248.934418] ffff8804b9177e50 0000000000000046 ffff8804b912bec0 ffff8804b9177fd8 [18248.943394] ffff8804b9177fd8 ffff8804b9177fd8 ffff8804b912bec0 ffff88082db964a0 [18248.952714] ffff8801791353b0 ffff8804b912bec0 ffff880179135380 ffff88082db96480 [18248.961680] Call Trace: [18248.965042] [] schedule+0x29/0x70 [18248.971222] [] worker_thread+0x1d9/0x410 [18248.978078] [] ? rescuer_thread+0x460/0x460 [18248.985600] [] kthread+0xcf/0xe0 [18248.991863] [] ? kthread_create_on_node+0x140/0x140 [18248.999790] [] ret_from_fork+0x58/0x90 [18249.006469] [] ? kthread_create_on_node+0x140/0x140 [18249.014397] kworker/30:0H S ffff88082db96840 0 162 2 0x00000000 [18249.023288] ffff8804b917be50 0000000000000046 ffff8804b912ce70 ffff8804b917bfd8 [18249.032263] ffff8804b917bfd8 ffff8804b917bfd8 ffff8804b912ce70 ffff88082db96860 [18249.041228] ffff8801791354b0 ffff8804b912ce70 ffff880179135480 ffff88082db96840 [18249.050166] Call Trace: [18249.053528] [] schedule+0x29/0x70 [18249.060061] [] worker_thread+0x1d9/0x410 [18249.067255] [] ? rescuer_thread+0x460/0x460 [18249.074421] [] kthread+0xcf/0xe0 [18249.080500] [] ? kthread_create_on_node+0x140/0x140 [18249.088417] [] ret_from_fork+0x58/0x90 [18249.095077] [] ? kthread_create_on_node+0x140/0x140 [18249.103348] watchdog/31 S 0000000000000000 0 163 2 0x00000000 [18249.111879] ffff8804b9197e80 0000000000000046 ffff8804b912de20 ffff8804b9197fd8 [18249.120838] ffff8804b9197fd8 ffff8804b9197fd8 ffff8804b912de20 ffff8804b912de20 [18249.129800] ffff8804ba74eea0 ffffffff81a09820 0000000000000000 0000000000000000 [18249.139090] Call Trace: [18249.142790] [] schedule+0x29/0x70 [18249.148948] [] smpboot_thread_fn+0xdd/0x180 [18249.156100] [] ? lg_double_unlock+0x90/0x90 [18249.163249] [] kthread+0xcf/0xe0 [18249.169685] [] ? kthread_create_on_node+0x140/0x140 [18249.178016] [] ret_from_fork+0x58/0x90 [18249.185043] [] ? kthread_create_on_node+0x140/0x140 [18249.192970] migration/31 S 0000000000000000 0 164 2 0x00000000 [18249.201516] ffff8804b919be80 0000000000000046 ffff8804b912edd0 ffff8804b919bfd8 [18249.210561] ffff8804b919bfd8 ffff8804b919bfd8 ffff8804b912edd0 ffff8804b912edd0 [18249.219528] ffff8804ba74ee90 ffffffff81a08f40 0000000000000000 0000000000000000 [18249.228825] Call Trace: [18249.232183] [] schedule+0x29/0x70 [18249.238381] [] smpboot_thread_fn+0xdd/0x180 [18249.245541] [] ? lg_double_unlock+0x90/0x90 [18249.252698] [] kthread+0xcf/0xe0 [18249.259152] [] ? kthread_create_on_node+0x140/0x140 [18249.267434] [] ret_from_fork+0x58/0x90 [18249.274074] [] ? kthread_create_on_node+0x140/0x140 [18249.281992] ksoftirqd/31 S 0000000000000000 0 165 2 0x00000000 [18249.290534] ffff8804b919fe80 0000000000000046 ffff8804b91a0000 ffff8804b919ffd8 [18249.299496] ffff8804b919ffd8 ffff8804b919ffd8 ffff8804b91a0000 ffff8804b91a0000 [18249.308772] ffff8804ba74ee80 ffffffff819f1100 0000000000000000 0000000000000000 [18249.317731] Call Trace: [18249.321087] [] schedule+0x29/0x70 [18249.327278] [] smpboot_thread_fn+0xdd/0x180 [18249.334432] [] ? lg_double_unlock+0x90/0x90 [18249.341941] [] kthread+0xcf/0xe0 [18249.348248] [] ? kthread_create_on_node+0x140/0x140 [18249.356175] [] ret_from_fork+0x58/0x90 [18249.362835] [] ? kthread_create_on_node+0x140/0x140 [18249.370763] kworker/31:0H S ffff88082dbd6840 0 167 2 0x00000000 [18249.379480] ffff8804b91afe50 0000000000000046 ffff8804b91a1f60 ffff8804b91affd8 [18249.388828] ffff8804b91affd8 ffff8804b91affd8 ffff8804b91a1f60 ffff88082dbd6860 [18249.397807] ffff880179135830 ffff8804b91a1f60 ffff880179135800 ffff88082dbd6840 [18249.406747] Call Trace: [18249.410105] [] schedule+0x29/0x70 [18249.416298] [] worker_thread+0x1d9/0x410 [18249.423515] [] ? rescuer_thread+0x460/0x460 [18249.430689] [] kthread+0xcf/0xe0 [18249.436794] [] ? kthread_create_on_node+0x140/0x140 [18249.444737] [] ret_from_fork+0x58/0x90 [18249.451400] [] ? kthread_create_on_node+0x140/0x140 [18249.459322] kdevtmpfs S 0000000000000000 0 170 2 0x00000000 [18249.468186] ffff88042fde7e70 0000000000000046 ffff88017911ce70 ffff88042fde7fd8 [18249.477139] ffff88042fde7fd8 ffff88042fde7fd8 ffff88017911ce70 0000000000000000 [18249.486085] 0000000000000000 ffff88017911ce70 0000000000000000 0000000000000000 [18249.495033] Call Trace: [18249.498397] [] schedule+0x29/0x70 [18249.504938] [] devtmpfsd+0x168/0x180 [18249.511415] [] ? handle_create.isra.2+0x220/0x220 [18249.519151] [] kthread+0xcf/0xe0 [18249.525230] [] ? kthread_create_on_node+0x140/0x140 [18249.533164] [] ret_from_fork+0x58/0x90 [18249.539834] [] ? kthread_create_on_node+0x140/0x140 [18249.548110] netns S ffff88083fe98200 0 171 2 0x00000000 [18249.556674] ffff880179227e08 0000000000000046 ffff88017911bec0 ffff880179227fd8 [18249.565643] ffff880179227fd8 ffff880179227fd8 ffff88017911bec0 ffff88017911bec0 [18249.574601] ffff88083fe98200 ffff88082f5e8a80 ffff88083fe98230 ffff88083fe98200 [18249.583569] Call Trace: [18249.587239] [] schedule+0x29/0x70 [18249.593790] [] rescuer_thread+0x359/0x460 [18249.600764] [] ? __schedule+0x3b0/0x990 [18249.607545] [] ? manage_workers.isra.24+0x2d0/0x2d0 [18249.615491] [] kthread+0xcf/0xe0 [18249.621582] [] ? kthread_create_on_node+0x140/0x140 [18249.629519] [] ret_from_fork+0x58/0x90 [18249.636542] [] ? kthread_create_on_node+0x140/0x140 [18249.644469] khungtaskd S 000000000000f888 0 172 2 0x00000000 [18249.653012] ffff8801791fbd90 0000000000000046 ffff88017911af10 ffff8801791fbfd8 [18249.661969] ffff8801791fbfd8 ffff8801791fbfd8 ffff88017911af10 ffff8804b9740000 [18249.670919] ffff8801791fbdc0 0000000101139756 ffff8804b9740000 000000000000f888 [18249.680244] Call Trace: [18249.683963] [] schedule+0x29/0x70 [18249.690177] [] schedule_timeout+0x174/0x2c0 [18249.697360] [] ? internal_add_timer+0x70/0x70 [18249.704722] [] schedule_timeout_interruptible+0x1e/0x20 [18249.713412] [] watchdog+0x4c/0x2c0 [18249.719829] [] ? reset_hung_task_detector+0x20/0x20 [18249.728122] [] kthread+0xcf/0xe0 [18249.734527] [] ? kthread_create_on_node+0x140/0x140 [18249.742464] [] ret_from_fork+0x58/0x90 [18249.749130] [] ? kthread_create_on_node+0x140/0x140 [18249.757055] writeback S ffff88082f023200 0 173 2 0x00000000 [18249.765604] ffff88042fd67e08 0000000000000046 ffff88042fda8000 ffff88042fd67fd8 [18249.774947] ffff88042fd67fd8 ffff88042fd67fd8 ffff88042fda8000 ffff88042fda8000 [18249.783909] ffff88082f023200 ffff88082f5ee280 ffff88082f023230 ffff88082f023200 [18249.792863] Call Trace: [18249.796239] [] schedule+0x29/0x70 [18249.802449] [] rescuer_thread+0x359/0x460 [18249.809417] [] ? __schedule+0x3b0/0x990 [18249.816532] [] ? manage_workers.isra.24+0x2d0/0x2d0 [18249.824475] [] kthread+0xcf/0xe0 [18249.830570] [] ? kthread_create_on_node+0x140/0x140 [18249.838507] [] ret_from_fork+0x58/0x90 [18249.845164] [] ? kthread_create_on_node+0x140/0x140 [18249.853084] kintegrityd S ffff88082f023280 0 174 2 0x00000000 [18249.861998] ffff88042fd27e08 0000000000000046 ffff88042fda8fb0 ffff88042fd27fd8 [18249.870949] ffff88042fd27fd8 ffff88042fd27fd8 ffff88042fda8fb0 ffff88042fda8fb0 [18249.879933] ffff88082f023280 ffff88082f5ee480 ffff88082f0232b0 ffff88082f023280 [18249.888880] Call Trace: [18249.892258] [] schedule+0x29/0x70 [18249.898455] [] rescuer_thread+0x359/0x460 [18249.905772] [] ? __schedule+0x3b0/0x990 [18249.912543] [] ? manage_workers.isra.24+0x2d0/0x2d0 [18249.920482] [] kthread+0xcf/0xe0 [18249.926583] [] ? kthread_create_on_node+0x140/0x140 [18249.934500] [] ret_from_fork+0x58/0x90 [18249.941155] [] ? kthread_create_on_node+0x140/0x140 [18249.949470] bioset S ffff88082f023380 0 175 2 0x00000000 [18249.958381] ffff88042fce7e08 0000000000000046 ffff88042fda9f60 ffff88042fce7fd8 [18249.967731] ffff88042fce7fd8 ffff88042fce7fd8 ffff88042fda9f60 ffff88042fda9f60 [18249.977070] ffff88082f023380 ffff88082f5ee680 ffff88082f0233b0 ffff88082f023380 [18249.986409] Call Trace: [18249.990153] [] schedule+0x29/0x70 [18249.996717] [] rescuer_thread+0x359/0x460 [18250.003698] [] ? __schedule+0x3b0/0x990 [18250.010480] [] ? manage_workers.isra.24+0x2d0/0x2d0 [18250.018424] [] kthread+0xcf/0xe0 [18250.024511] [] ? kthread_create_on_node+0x140/0x140 [18250.032431] [] ret_from_fork+0x58/0x90 [18250.039441] [] ? kthread_create_on_node+0x140/0x140 [18250.047384] kblockd S ffff88082f023400 0 176 2 0x00000000 [18250.055936] ffff88042fca7e08 0000000000000046 ffff88042fdaaf10 ffff88042fca7fd8 [18250.064908] ffff88042fca7fd8 ffff88042fca7fd8 ffff88042fdaaf10 ffff88042fdaaf10 [18250.073857] ffff88082f023400 ffff88082f5eea80 ffff88082f023430 ffff88082f023400 [18250.082826] Call Trace: [18250.086190] [] schedule+0x29/0x70 [18250.092375] [] rescuer_thread+0x359/0x460 [18250.099342] [] ? __schedule+0x3b0/0x990 [18250.106103] [] ? manage_workers.isra.24+0x2d0/0x2d0 [18250.114024] [] kthread+0xcf/0xe0 [18250.120106] [] ? kthread_create_on_node+0x140/0x140 [18250.128023] [] ret_from_fork+0x58/0x90 [18250.135045] [] ? kthread_create_on_node+0x140/0x140 [18250.142943] md S ffff88082f534380 0 178 2 0x00000000 [18250.151495] ffff88042f9fbe08 0000000000000046 ffff88042fdace70 ffff88042f9fbfd8 [18250.160462] ffff88042f9fbfd8 ffff88042f9fbfd8 ffff88042fdace70 ffff88042fdace70 [18250.169747] ffff88082f534380 ffff88042f9f2480 ffff88082f5343b0 ffff88082f534380 [18250.178840] Call Trace: [18250.182189] [] schedule+0x29/0x70 [18250.188393] [] rescuer_thread+0x359/0x460 [18250.195375] [] ? __schedule+0x3b0/0x990 [18250.202136] [] ? manage_workers.isra.24+0x2d0/0x2d0 [18250.210443] [] kthread+0xcf/0xe0 [18250.216866] [] ? kthread_create_on_node+0x140/0x140 [18250.224797] [] ret_from_fork+0x58/0x90 [18250.231478] [] ? kthread_create_on_node+0x140/0x140 [18250.239407] kworker/19:1 S ffff88042e2d6480 0 180 2 0x00000000 [18250.247956] ffff88042ec3fe50 0000000000000046 ffff88042fdaedd0 ffff88042ec3ffd8 [18250.257273] ffff88042ec3ffd8 ffff88042ec3ffd8 ffff88042fdaedd0 ffff88042e2d64a0 [18250.266229] ffff88042f03a430 ffff88042fdaedd0 ffff88042f03a400 ffff88042e2d6480 [18250.275193] Call Trace: [18250.278921] [] schedule+0x29/0x70 [18250.285460] [] worker_thread+0x1d9/0x410 [18250.292338] [] ? rescuer_thread+0x460/0x460 [18250.299853] [] kthread+0xcf/0xe0 [18250.305907] [] ? kthread_create_on_node+0x140/0x140 [18250.313831] [] ret_from_fork+0x58/0x90 [18250.320500] [] ? kthread_create_on_node+0x140/0x140 [18250.328422] kswapd0 S 0000000000000000 0 185 2 0x00000000 [18250.337297] ffff88042c4cbe08 0000000000000046 ffff88042cb03ec0 ffff88042c4cbfd8 [18250.346263] ffff88042c4cbfd8 ffff88042c4cbfd8 ffff88042cb03ec0 ffff88043ffd9000 [18250.355208] ffff88043ffff4e8 0000000000000000 ffff88042c4cbe80 0000000000000000 [18250.364150] Call Trace: [18250.367513] [] schedule+0x29/0x70 [18250.374065] [] kswapd+0x394/0x450 [18250.380566] [] ? wake_up_atomic_t+0x30/0x30 [18250.387721] [] ? balance_pgdat+0x5e0/0x5e0 [18250.394781] [] kthread+0xcf/0xe0 [18250.400849] [] ? kthread_create_on_node+0x140/0x140 [18250.408763] [] ret_from_fork+0x58/0x90 [18250.415761] [] ? kthread_create_on_node+0x140/0x140 [18250.423694] kswapd1 S 0000000000000000 0 186 2 0x00000000 [18250.432216] ffff88042c4cfe08 0000000000000046 ffff88042cb04e70 ffff88042c4cffd8 [18250.441150] ffff88042c4cffd8 ffff88042c4cffd8 ffff88042cb04e70 ffff88083ffd6000 [18250.450471] ffff88083fffc4e8 0000000000000000 ffff88042c4cfe80 0000000000000000 [18250.459437] Call Trace: [18250.462809] [] schedule+0x29/0x70 [18250.468982] [] kswapd+0x394/0x450 [18250.475153] [] ? wake_up_atomic_t+0x30/0x30 [18250.482303] [] ? balance_pgdat+0x5e0/0x5e0 [18250.489713] [] kthread+0xcf/0xe0 [18250.496107] [] ? kthread_create_on_node+0x140/0x140 [18250.503990] [] ret_from_fork+0x58/0x90 [18250.510640] [] ? kthread_create_on_node+0x140/0x140 [18250.518556] ksmd S 0000000000000000 0 187 2 0x00000000 [18250.527393] ffff88042cb0be50 0000000000000046 ffff88042cb05e20 ffff88042cb0bfd8 [18250.536418] ffff88042cb0bfd8 ffff88042cb0bfd8 ffff88042cb05e20 ffff88042cb05e20 [18250.545358] ffff88042cb05e20 ffff88042cb0be88 ffff88042cb05e20 0000000000000000 [18250.554308] Call Trace: [18250.557655] [] schedule+0x29/0x70 [18250.563842] [] ksm_scan_thread+0x16a/0x240 [18250.571234] [] ? wake_up_atomic_t+0x30/0x30 [18250.578393] [] ? ksm_do_scan+0x11e0/0x11e0 [18250.585426] [] kthread+0xcf/0xe0 [18250.591496] [] ? kthread_create_on_node+0x140/0x140 [18250.599392] [] ret_from_fork+0x58/0x90 [18250.606364] [] ? kthread_create_on_node+0x140/0x140 [18250.614438] khugepaged S 000000000000f888 0 188 2 0x00000000 [18250.622919] ffff88042cb13d80 0000000000000046 ffff88042cb06dd0 ffff88042cb13fd8 [18250.631871] ffff88042cb13fd8 ffff88042cb13fd8 ffff88042cb06dd0 ffff8804b96b8000 [18250.640804] ffff88042cb13db0 000000010111f398 ffff8804b96b8000 000000000000f888 [18250.650089] Call Trace: [18250.653530] [] schedule+0x29/0x70 [18250.659698] [] schedule_timeout+0x174/0x2c0 [18250.666841] [] ? internal_add_timer+0x70/0x70 [18250.674170] [] ? prepare_to_wait+0x56/0x90 [18250.681198] [] khugepaged+0x1e3/0x480 [18250.688078] [] ? wake_up_atomic_t+0x30/0x30 [18250.695188] [] ? khugepaged_scan_mm_slot+0xc70/0xc70 [18250.703185] [] kthread+0xcf/0xe0 [18250.709283] [] ? kthread_create_on_node+0x140/0x140 [18250.717176] [] ret_from_fork+0x58/0x90 [18250.724149] [] ? kthread_create_on_node+0x140/0x140 [18250.732048] fsnotify_mark S ffff88042c52fe00 0 189 2 0x00000000 [18250.740579] ffff88042c52fe28 0000000000000046 ffff88042cb02f10 ffff88042c52ffd8 [18250.749534] ffff88042c52ffd8 ffff88042c52ffd8 ffff88042cb02f10 ffff88042c52fe58 [18250.758488] ffff88042c52fe58 ffff88042cb02f10 ffff88042bc90258 ffff88042c52fe00 [18250.767757] Call Trace: [18250.771082] [] schedule+0x29/0x70 [18250.777233] [] fsnotify_mark_destroy+0x141/0x180 [18250.784846] [] ? wake_up_atomic_t+0x30/0x30 [18250.791972] [] ? fsnotify_put_mark+0x40/0x40 [18250.799522] [] kthread+0xcf/0xe0 [18250.805907] [] ? kthread_create_on_node+0x140/0x140 [18250.813818] [] ret_from_fork+0x58/0x90 [18250.820470] [] ? kthread_create_on_node+0x140/0x140 [18250.828386] crypto S ffff88042c4d0180 0 190 2 0x00000000 [18250.836905] ffff88042c54be08 0000000000000046 ffff88042cb01f60 ffff88042c54bfd8 [18250.845851] ffff88042c54bfd8 ffff88042c54bfd8 ffff88042cb01f60 ffff88042cb01f60 [18250.854842] ffff88042c4d0180 ffff88042cae4680 ffff88042c4d01b0 ffff88042c4d0180 [18250.863788] Call Trace: [18250.867145] [] schedule+0x29/0x70 [18250.873335] [] rescuer_thread+0x359/0x460 [18250.880287] [] ? __schedule+0x3b0/0x990 [18250.887386] [] ? manage_workers.isra.24+0x2d0/0x2d0 [18250.895294] [] kthread+0xcf/0xe0 [18250.901365] [] ? kthread_create_on_node+0x140/0x140 [18250.909273] [] ret_from_fork+0x58/0x90 [18250.915920] [] ? kthread_create_on_node+0x140/0x140 [18250.924197] kthrotld S ffff88042c4d0280 0 198 2 0x00000000 [18250.932746] ffff88042c573e08 0000000000000046 ffff88042c55ce70 ffff88042c573fd8 [18250.941703] ffff88042c573fd8 ffff88042c573fd8 ffff88042c55ce70 ffff88042c55ce70 [18250.950647] ffff88042c4d0280 ffff88042cae4280 ffff88042c4d02b0 ffff88042c4d0280 [18250.959589] Call Trace: [18250.962987] [] schedule+0x29/0x70 [18250.969508] [] rescuer_thread+0x359/0x460 [18250.976464] [] ? __schedule+0x3b0/0x990 [18250.983209] [] ? manage_workers.isra.24+0x2d0/0x2d0 [18250.991122] [] kthread+0xcf/0xe0 [18250.997178] [] ? kthread_create_on_node+0x140/0x140 [18251.005438] [] ret_from_fork+0x58/0x90 [18251.012439] [] ? kthread_create_on_node+0x140/0x140 [18251.020367] kworker/8:1 S ffff88082d816480 0 199 2 0x00000000 [18251.028910] ffff88082c8a3e50 0000000000000046 ffff88082c898000 ffff88082c8a3fd8 [18251.037877] ffff88082c8a3fd8 ffff88082c8a3fd8 ffff88082c898000 ffff88082d8164a0 [18251.046834] ffff88082c886030 ffff88082c898000 ffff88082c886000 ffff88082d816480 [18251.056119] Call Trace: [18251.059487] [] schedule+0x29/0x70 [18251.065680] [] worker_thread+0x1d9/0x410 [18251.072541] [] ? rescuer_thread+0x460/0x460 [18251.079693] [] kthread+0xcf/0xe0 [18251.085767] [] ? kthread_create_on_node+0x140/0x140 [18251.094022] [] ret_from_fork+0x58/0x90 [18251.100690] [] ? kthread_create_on_node+0x140/0x140 [18251.108615] kmpath_rdacd S ffff88042bd40980 0 203 2 0x00000000 [18251.117141] ffff88042bd6fe08 0000000000000046 ffff88082c89de20 ffff88042bd6ffd8 [18251.126094] ffff88042bd6ffd8 ffff88042bd6ffd8 ffff88082c89de20 ffff88082c89de20 [18251.135398] ffff88042bd40980 ffff88042bd70480 ffff88042bd409b0 ffff88042bd40980 [18251.144349] Call Trace: [18251.147716] [] schedule+0x29/0x70 [18251.153895] [] rescuer_thread+0x359/0x460 [18251.160856] [] ? __schedule+0x3b0/0x990 [18251.167618] [] ? manage_workers.isra.24+0x2d0/0x2d0 [18251.175892] [] kthread+0xcf/0xe0 [18251.181971] [] ? kthread_create_on_node+0x140/0x140 [18251.189871] [] ret_from_fork+0x58/0x90 [18251.196527] [] ? kthread_create_on_node+0x140/0x140 [18251.204458] kworker/0:2 S ffff88042e016480 0 204 2 0x00000000 [18251.213345] ffff88042bd8be50 0000000000000046 ffff88042c558000 ffff88042bd8bfd8 [18251.222323] ffff88042bd8bfd8 ffff88042bd8bfd8 ffff88042c558000 ffff88042e0164a0 [18251.231251] ffff88042c4d16b0 ffff88042c558000 ffff88042c4d1680 ffff88042e016480 [18251.240191] Call Trace: [18251.243557] [] schedule+0x29/0x70 [18251.249746] [] worker_thread+0x1d9/0x410 [18251.256982] [] ? rescuer_thread+0x460/0x460 [18251.264467] [] kthread+0xcf/0xe0 [18251.270553] [] ? kthread_create_on_node+0x140/0x140 [18251.278484] [] ret_from_fork+0x58/0x90 [18251.285125] [] ? kthread_create_on_node+0x140/0x140 [18251.293038] kworker/15:1 S ffff88082d9d6480 0 205 2 0x00000000 [18251.301913] ffff88082cb1be50 0000000000000046 ffff88082c898fb0 ffff88082cb1bfd8 [18251.310876] ffff88082cb1bfd8 ffff88082cb1bfd8 ffff88082c898fb0 ffff88082d9d64a0 [18251.319822] ffff88082c9d8130 ffff88082c898fb0 ffff88082c9d8100 ffff88082d9d6480 [18251.328776] Call Trace: [18251.332130] [] schedule+0x29/0x70 [18251.338314] [] worker_thread+0x1d9/0x410 [18251.345444] [] ? rescuer_thread+0x460/0x460 [18251.352947] [] kthread+0xcf/0xe0 [18251.358996] [] ? kthread_create_on_node+0x140/0x140 [18251.366909] [] ret_from_fork+0x58/0x90 [18251.373570] [] ? kthread_create_on_node+0x140/0x140 [18251.381486] kpsmoused S ffff88082c9d8180 0 206 2 0x00000000 [18251.390368] ffff88042bd8fe08 0000000000000046 ffff88082c899f60 ffff88042bd8ffd8 [18251.399336] ffff88042bd8ffd8 ffff88042bd8ffd8 ffff88082c899f60 ffff88082c899f60 [18251.408280] ffff88082c9d8180 ffff88082cb10080 ffff88082c9d81b0 ffff88082c9d8180 [18251.417221] Call Trace: [18251.420591] [] schedule+0x29/0x70 [18251.426774] [] rescuer_thread+0x359/0x460 [18251.433737] [] ? __schedule+0x3b0/0x990 [18251.440494] [] ? manage_workers.isra.24+0x2d0/0x2d0 [18251.448787] [] kthread+0xcf/0xe0 [18251.454887] [] ? kthread_create_on_node+0x140/0x140 [18251.462821] [] ret_from_fork+0x58/0x90 [18251.469476] [] ? kthread_create_on_node+0x140/0x140 [18251.477389] ipv6_addrconf S ffff88042bde3900 0 207 2 0x00000000 [18251.486234] ffff88042be1be08 0000000000000046 ffff88082c89af10 ffff88042be1bfd8 [18251.495190] ffff88042be1bfd8 ffff88042be1bfd8 ffff88082c89af10 ffff88082c89af10 [18251.504131] ffff88042bde3900 ffff88082c8ace80 ffff88042bde3930 ffff88042bde3900 [18251.513068] Call Trace: [18251.516436] [] schedule+0x29/0x70 [18251.522629] [] rescuer_thread+0x359/0x460 [18251.529581] [] ? __schedule+0x3b0/0x990 [18251.536708] [] ? manage_workers.isra.24+0x2d0/0x2d0 [18251.544664] [] kthread+0xcf/0xe0 [18251.550760] [] ? kthread_create_on_node+0x140/0x140 [18251.558690] [] ret_from_fork+0x58/0x90 [18251.565340] [] ? kthread_create_on_node+0x140/0x140 [18251.573262] deferwq S ffff88042bde3b80 0 226 2 0x00000000 [18251.582180] ffff88042be5fe08 0000000000000046 ffff88042c558fb0 ffff88042be5ffd8 [18251.591111] ffff88042be5ffd8 ffff88042be5ffd8 ffff88042c558fb0 ffff88042c558fb0 [18251.600021] ffff88042bde3b80 ffff88082c8ae880 ffff88042bde3bb0 ffff88042bde3b80 [18251.608979] Call Trace: [18251.612359] [] schedule+0x29/0x70 [18251.618554] [] rescuer_thread+0x359/0x460 [18251.625893] [] ? __schedule+0x3b0/0x990 [18251.632859] [] ? manage_workers.isra.24+0x2d0/0x2d0 [18251.640797] [] kthread+0xcf/0xe0 [18251.646883] [] ? finish_task_switch+0x53/0x180 [18251.654323] [] ? kthread_create_on_node+0x140/0x140 [18251.662239] [] ret_from_fork+0x58/0x90 [18251.668897] [] ? kthread_create_on_node+0x140/0x140 [18251.677162] kworker/1:1 S ffff88042e056480 0 228 2 0x00000000 [18251.685718] ffff88042be77e50 0000000000000046 ffff88042be88000 ffff88042be77fd8 [18251.694688] ffff88042be77fd8 ffff88042be77fd8 ffff88042be88000 ffff88042e0564a0 [18251.703646] ffff880179222430 ffff88042be88000 ffff880179222400 ffff88042e056480 [18251.712602] Call Trace: [18251.715966] [] schedule+0x29/0x70 [18251.722522] [] worker_thread+0x1d9/0x410 [18251.729414] [] ? rescuer_thread+0x460/0x460 [18251.736573] [] kthread+0xcf/0xe0 [18251.742641] [] ? kthread_create_on_node+0x140/0x140 [18251.750564] [] ret_from_fork+0x58/0x90 [18251.757216] [] ? kthread_create_on_node+0x140/0x140 [18251.765128] kworker/2:1 S ffff88042e096480 0 229 2 0x00000000 [18251.774012] ffff88042beebe50 0000000000000046 ffff88042be88fb0 ffff88042beebfd8 [18251.782968] ffff88042beebfd8 ffff88042beebfd8 ffff88042be88fb0 ffff88042e0964a0 [18251.791927] ffff88042fce3530 ffff88042be88fb0 ffff88042fce3500 ffff88042e096480 [18251.800891] Call Trace: [18251.804249] [] schedule+0x29/0x70 [18251.810430] [] worker_thread+0x1d9/0x410 [18251.817642] [] ? rescuer_thread+0x460/0x460 [18251.824793] [] kthread+0xcf/0xe0 [18251.830872] [] ? kthread_create_on_node+0x140/0x140 [18251.838790] [] ret_from_fork+0x58/0x90 [18251.845441] [] ? kthread_create_on_node+0x140/0x140 [18251.853356] kworker/4:1 S ffff88042e116480 0 230 2 0x00000000 [18251.862243] ffff88042beefe50 0000000000000046 ffff88042be89f60 ffff88042beeffd8 [18251.871255] ffff88042beeffd8 ffff88042beeffd8 ffff88042be89f60 ffff88042e1164a0 [18251.880206] ffff88042c610030 ffff88042be89f60 ffff88042c610000 ffff88042e116480 [18251.889144] Call Trace: [18251.892508] [] schedule+0x29/0x70 [18251.898691] [] worker_thread+0x1d9/0x410 [18251.905916] [] ? rescuer_thread+0x460/0x460 [18251.913413] [] kthread+0xcf/0xe0 [18251.919554] [] ? kthread_create_on_node+0x140/0x140 [18251.927492] [] ret_from_fork+0x58/0x90 [18251.934131] [] ? kthread_create_on_node+0x140/0x140 [18251.942068] kworker/6:1 S ffff88042e196480 0 231 2 0x00000000 [18251.950601] ffff88042befbe50 0000000000000046 ffff88042be8af10 ffff88042befbfd8 [18251.959906] ffff88042befbfd8 ffff88042befbfd8 ffff88042be8af10 ffff88042e1964a0 [18251.968895] ffff88082c970030 ffff88042be8af10 ffff88082c970000 ffff88042e196480 [18251.977852] Call Trace: [18251.981207] [] schedule+0x29/0x70 [18251.987378] [] worker_thread+0x1d9/0x410 [18251.994225] [] ? rescuer_thread+0x460/0x460 [18252.001375] [] kthread+0xcf/0xe0 [18252.007818] [] ? kthread_create_on_node+0x140/0x140 [18252.015907] [] ret_from_fork+0x58/0x90 [18252.022670] [] ? kthread_create_on_node+0x140/0x140 [18252.030613] kworker/7:1 S ffff88042e1d6480 0 232 2 0x00000000 [18252.039144] ffff88042beffe50 0000000000000046 ffff88042be8bec0 ffff88042befffd8 [18252.048084] ffff88042befffd8 ffff88042befffd8 ffff88042be8bec0 ffff88042e1d64a0 [18252.057377] ffff88042c654030 ffff88042be8bec0 ffff88042c654000 ffff88042e1d6480 [18252.066448] Call Trace: [18252.069852] [] schedule+0x29/0x70 [18252.076030] [] worker_thread+0x1d9/0x410 [18252.082890] [] ? rescuer_thread+0x460/0x460 [18252.090046] [] kthread+0xcf/0xe0 [18252.096116] [] ? kthread_create_on_node+0x140/0x140 [18252.104378] [] ret_from_fork+0x58/0x90 [18252.111169] [] ? kthread_create_on_node+0x140/0x140 [18252.119083] kworker/13:1 S ffff88082d956480 0 234 2 0x00000000 [18252.127615] ffff88082c4ebe50 0000000000000046 ffff88082c4e0fb0 ffff88082c4ebfd8 [18252.136564] ffff88082c4ebfd8 ffff88082c4ebfd8 ffff88082c4e0fb0 ffff88082d9564a0 [18252.145511] ffff88042c748030 ffff88082c4e0fb0 ffff88042c748000 ffff88082d956480 [18252.154775] Call Trace: [18252.158487] [] schedule+0x29/0x70 [18252.164676] [] worker_thread+0x1d9/0x410 [18252.171528] [] ? rescuer_thread+0x460/0x460 [18252.178686] [] kthread+0xcf/0xe0 [18252.184764] [] ? kthread_create_on_node+0x140/0x140 [18252.192689] [] ret_from_fork+0x58/0x90 [18252.199344] [] ? kthread_create_on_node+0x140/0x140 [18252.207606] kworker/12:1 S ffff88082d916480 0 235 2 0x00000000 [18252.216188] ffff88082c4efe50 0000000000000046 ffff88082c4e1f60 ffff88082c4effd8 [18252.225138] ffff88082c4effd8 ffff88082c4effd8 ffff88082c4e1f60 ffff88082d9164a0 [18252.234083] ffff88042c720030 ffff88082c4e1f60 ffff88042c720000 ffff88082d916480 [18252.243019] Call Trace: [18252.246374] [] schedule+0x29/0x70 [18252.252918] [] worker_thread+0x1d9/0x410 [18252.260119] [] ? rescuer_thread+0x460/0x460 [18252.267271] [] kthread+0xcf/0xe0 [18252.273350] [] ? kthread_create_on_node+0x140/0x140 [18252.281265] [] ret_from_fork+0x58/0x90 [18252.287922] [] ? kthread_create_on_node+0x140/0x140 [18252.295840] kworker/14:1 S ffff88082d996480 0 236 2 0x00000000 [18252.304593] ffff88082c4f3e50 0000000000000046 ffff88082c4e2f10 ffff88082c4f3fd8 [18252.313867] ffff88082c4f3fd8 ffff88082c4f3fd8 ffff88082c4e2f10 ffff88082d9964a0 [18252.322825] ffff88082c9b0030 ffff88082c4e2f10 ffff88082c9b0000 ffff88082d996480 [18252.331758] Call Trace: [18252.335106] [] schedule+0x29/0x70 [18252.341285] [] worker_thread+0x1d9/0x410 [18252.348130] [] ? rescuer_thread+0x460/0x460 [18252.355276] [] kthread+0xcf/0xe0 [18252.361698] [] ? kthread_create_on_node+0x140/0x140 [18252.369636] [] ret_from_fork+0x58/0x90 [18252.376289] [] ? kthread_create_on_node+0x140/0x140 [18252.384196] kworker/17:1 S ffff88042e256480 0 237 2 0x00000000 [18252.392728] ffff88042bf03e50 0000000000000046 ffff88042be8edd0 ffff88042bf03fd8 [18252.401671] ffff88042bf03fd8 ffff88042bf03fd8 ffff88042be8edd0 ffff88042e2564a0 [18252.410974] ffff88082ca28030 ffff88042be8edd0 ffff88082ca28000 ffff88042e256480 [18252.419915] Call Trace: [18252.423251] [] schedule+0x29/0x70 [18252.429421] [] worker_thread+0x1d9/0x410 [18252.436269] [] ? rescuer_thread+0x460/0x460 [18252.443420] [] kthread+0xcf/0xe0 [18252.449507] [] ? kthread_create_on_node+0x140/0x140 [18252.457421] [] ret_from_fork+0x58/0x90 [18252.464421] [] ? kthread_create_on_node+0x140/0x140 [18252.472390] kworker/18:1 S ffff88042e296480 0 239 2 0x00000000 [18252.480919] ffff88042bf0be50 0000000000000046 ffff88042be8ce70 ffff88042bf0bfd8 [18252.489871] ffff88042bf0bfd8 ffff88042bf0bfd8 ffff88042be8ce70 ffff88042e2964a0 [18252.499165] ffff88082ca4a030 ffff88042be8ce70 ffff88082ca4a000 ffff88042e296480 [18252.508088] Call Trace: [18252.511438] [] schedule+0x29/0x70 [18252.517640] [] worker_thread+0x1d9/0x410 [18252.524487] [] ? rescuer_thread+0x460/0x460 [18252.531642] [] kthread+0xcf/0xe0 [18252.537720] [] ? kthread_create_on_node+0x140/0x140 [18252.545649] [] ret_from_fork+0x58/0x90 [18252.552298] [] ? kthread_create_on_node+0x140/0x140 [18252.560207] kworker/20:1 S ffff88042e316480 0 240 2 0x00000000 [18252.568749] ffff88042bf0fe50 0000000000000046 ffff88042bf10000 ffff88042bf0ffd8 [18252.578058] ffff88042bf0ffd8 ffff88042bf0ffd8 ffff88042bf10000 ffff88042e3164a0 [18252.587027] ffff88042c7b2030 ffff88042bf10000 ffff88042c7b2000 ffff88042e316480 [18252.595960] Call Trace: [18252.599306] [] schedule+0x29/0x70 [18252.605501] [] worker_thread+0x1d9/0x410 [18252.612599] [] ? rescuer_thread+0x460/0x460 [18252.619783] [] kthread+0xcf/0xe0 [18252.625862] [] ? kthread_create_on_node+0x140/0x140 [18252.633782] [] ret_from_fork+0x58/0x90 [18252.640466] [] ? kthread_create_on_node+0x140/0x140 [18252.648413] kworker/22:1 S ffff88042e396480 0 241 2 0x00000000 [18252.656940] ffff88042bf1be50 0000000000000046 ffff88042bf10fb0 ffff88042bf1bfd8 [18252.665889] ffff88042bf1bfd8 ffff88042bf1bfd8 ffff88042bf10fb0 ffff88042e3964a0 [18252.674843] ffff88042bc04030 ffff88042bf10fb0 ffff88042bc04000 ffff88042e396480 [18252.683775] Call Trace: [18252.687121] [] schedule+0x29/0x70 [18252.693278] [] worker_thread+0x1d9/0x410 [18252.700105] [] ? rescuer_thread+0x460/0x460 [18252.707244] [] kthread+0xcf/0xe0 [18252.713315] [] ? kthread_create_on_node+0x140/0x140 [18252.721227] [] ret_from_fork+0x58/0x90 [18252.727879] [] ? kthread_create_on_node+0x140/0x140 [18252.735790] kworker/23:1 S ffff88042e3d6480 0 242 2 0x00000000 [18252.744304] ffff88042bf1fe50 0000000000000046 ffff88042bf11f60 ffff88042bf1ffd8 [18252.753254] ffff88042bf1ffd8 ffff88042bf1ffd8 ffff88042bf11f60 ffff88042e3d64a0 [18252.762198] ffff88042bc2c030 ffff88042bf11f60 ffff88042bc2c000 ffff88042e3d6480 [18252.771141] Call Trace: [18252.774495] [] schedule+0x29/0x70 [18252.780663] [] worker_thread+0x1d9/0x410 [18252.787506] [] ? rescuer_thread+0x460/0x460 [18252.794660] [] kthread+0xcf/0xe0 [18252.801082] [] ? kthread_create_on_node+0x140/0x140 [18252.809021] [] ret_from_fork+0x58/0x90 [18252.815680] [] ? kthread_create_on_node+0x140/0x140 [18252.823598] kworker/21:1 S ffff88042e356480 0 243 2 0x00000000 [18252.832136] ffff88042bf23e50 0000000000000046 ffff88042bf12f10 ffff88042bf23fd8 [18252.841093] ffff88042bf23fd8 ffff88042bf23fd8 ffff88042bf12f10 ffff88042e3564a0 [18252.850048] ffff88042c7da030 ffff88042bf12f10 ffff88042c7da000 ffff88042e356480 [18252.859323] Call Trace: [18252.862973] [] schedule+0x29/0x70 [18252.869121] [] worker_thread+0x1d9/0x410 [18252.875967] [] ? rescuer_thread+0x460/0x460 [18252.883093] [] kthread+0xcf/0xe0 [18252.889156] [] ? kthread_create_on_node+0x140/0x140 [18252.897068] [] ret_from_fork+0x58/0x90 [18252.903715] [] ? kthread_create_on_node+0x140/0x140 [18252.911969] kworker/25:1 S ffff88082da56480 0 244 2 0x00000000 [18252.920557] ffff88082c4f7e50 0000000000000046 ffff88082c4e6dd0 ffff88082c4f7fd8 [18252.929516] ffff88082c4f7fd8 ffff88082c4f7fd8 ffff88082c4e6dd0 ffff88082da564a0 [18252.938472] ffff88082c8840b0 ffff88082c4e6dd0 ffff88082c884080 ffff88082da56480 [18252.947421] Call Trace: [18252.950769] [] schedule+0x29/0x70 [18252.956951] [] worker_thread+0x1d9/0x410 [18252.963804] [] ? rescuer_thread+0x460/0x460 [18252.971290] [] kthread+0xcf/0xe0 [18252.977685] [] ? kthread_create_on_node+0x140/0x140 [18252.985608] [] ret_from_fork+0x58/0x90 [18252.992254] [] ? kthread_create_on_node+0x140/0x140 [18253.000168] kworker/9:1 S ffff88082d856480 0 245 2 0x00000000 [18253.008689] ffff88082c4fbe50 0000000000000046 ffff88082c4e5e20 ffff88082c4fbfd8 [18253.017635] ffff88082c4fbfd8 ffff88082c4fbfd8 ffff88082c4e5e20 ffff88082d8564a0 [18253.026951] ffff88042c69c330 ffff88082c4e5e20 ffff88042c69c300 ffff88082d856480 [18253.036225] Call Trace: [18253.039604] [] schedule+0x29/0x70 [18253.045780] [] worker_thread+0x1d9/0x410 [18253.052625] [] ? rescuer_thread+0x460/0x460 [18253.059771] [] kthread+0xcf/0xe0 [18253.065851] [] ? kthread_create_on_node+0x140/0x140 [18253.073781] [] ret_from_fork+0x58/0x90 [18253.080429] [] ? kthread_create_on_node+0x140/0x140 [18253.088687] kworker/24:1 R running task 0 246 2 0x00000000 [18253.097201] ffff88082c4ffe50 0000000000000046 ffff88082c4e4e70 ffff88082c4fffd8 [18253.106146] ffff88082c4fffd8 ffff88082c4fffd8 ffff88082c4e4e70 ffff88082da164a0 [18253.115089] ffff88042bc56030 ffff88082c4e4e70 ffff88042bc56000 ffff88082da16480 [18253.124008] Call Trace: [18253.127359] [] schedule+0x29/0x70 [18253.133531] [] worker_thread+0x1d9/0x410 [18253.140770] [] ? rescuer_thread+0x460/0x460 [18253.148239] [] kthread+0xcf/0xe0 [18253.154309] [] ? kthread_create_on_node+0x140/0x140 [18253.162223] [] ret_from_fork+0x58/0x90 [18253.168873] [] ? kthread_create_on_node+0x140/0x140 [18253.176792] kworker/27:1 S ffff88082dad6480 0 247 2 0x00000000 [18253.185314] ffff88082c503e50 0000000000000046 ffff88082c4e3ec0 ffff88082c503fd8 [18253.194291] ffff88082c503fd8 ffff88082c503fd8 ffff88082c4e3ec0 ffff88082dad64a0 [18253.203600] ffff88042bcc6030 ffff88082c4e3ec0 ffff88042bcc6000 ffff88082dad6480 [18253.212559] Call Trace: [18253.215916] [] schedule+0x29/0x70 [18253.222090] [] worker_thread+0x1d9/0x410 [18253.228956] [] ? rescuer_thread+0x460/0x460 [18253.236108] [] kthread+0xcf/0xe0 [18253.242546] [] ? kthread_create_on_node+0x140/0x140 [18253.250485] [] ret_from_fork+0x58/0x90 [18253.257504] [] ? kthread_create_on_node+0x140/0x140 [18253.265438] kworker/31:1 S ffff88082dbd6480 0 248 2 0x00000000 [18253.273972] ffff88082c507e50 0000000000000046 ffff88082c508000 ffff88082c507fd8 [18253.282926] ffff88082c507fd8 ffff88082c507fd8 ffff88082c508000 ffff88082dbd64a0 [18253.291906] ffff88042bd40a30 ffff88082c508000 ffff88042bd40a00 ffff88082dbd6480 [18253.300849] Call Trace: [18253.304199] [] schedule+0x29/0x70 [18253.310378] [] worker_thread+0x1d9/0x410 [18253.317587] [] ? rescuer_thread+0x460/0x460 [18253.324742] [] kthread+0xcf/0xe0 [18253.330823] [] ? kthread_create_on_node+0x140/0x140 [18253.338749] [] ret_from_fork+0x58/0x90 [18253.345420] [] ? kthread_create_on_node+0x140/0x140 [18253.353326] kworker/28:1 R running task 0 249 2 0x00000000 [18253.361861] ffff88082c513e50 0000000000000046 ffff88082c508fb0 ffff88082c513fd8 [18253.371160] ffff88082c513fd8 ffff88082c513fd8 ffff88082c508fb0 ffff88082db164a0 [18253.380163] ffff88082ca6a030 ffff88082c508fb0 ffff88082ca6a000 ffff88082db16480 [18253.389096] Call Trace: [18253.392460] [] schedule+0x29/0x70 [18253.398652] [] worker_thread+0x1d9/0x410 [18253.405508] [] ? rescuer_thread+0x460/0x460 [18253.412666] [] kthread+0xcf/0xe0 [18253.418749] [] ? kthread_create_on_node+0x140/0x140 [18253.426673] [] ret_from_fork+0x58/0x90 [18253.433690] [] ? kthread_create_on_node+0x140/0x140 [18253.441603] kworker/29:1 S ffff88082db56480 0 250 2 0x00000000 [18253.450119] ffff88082c517e50 0000000000000046 ffff88082c509f60 ffff88082c517fd8 [18253.459058] ffff88082c517fd8 ffff88082c517fd8 ffff88082c509f60 ffff88082db564a0 [18253.468002] ffff88082ca8c030 ffff88082c509f60 ffff88082ca8c000 ffff88082db56480 [18253.476939] Call Trace: [18253.480282] [] schedule+0x29/0x70 [18253.486458] [] worker_thread+0x1d9/0x410 [18253.493653] [] ? rescuer_thread+0x460/0x460 [18253.500801] [] kthread+0xcf/0xe0 [18253.506884] [] ? kthread_create_on_node+0x140/0x140 [18253.514825] [] ret_from_fork+0x58/0x90 [18253.521476] [] ? kthread_create_on_node+0x140/0x140 [18253.529410] kworker/30:1 S ffff88082db96480 0 251 2 0x00000000 [18253.537937] ffff88082c51be50 0000000000000046 ffff88082c50af10 ffff88082c51bfd8 [18253.546870] ffff88082c51bfd8 ffff88082c51bfd8 ffff88082c50af10 ffff88082db964a0 [18253.556172] ffff88042bd16030 ffff88082c50af10 ffff88042bd16000 ffff88082db96480 [18253.565133] Call Trace: [18253.568494] [] schedule+0x29/0x70 [18253.574669] [] worker_thread+0x1d9/0x410 [18253.581515] [] ? rescuer_thread+0x460/0x460 [18253.588660] [] kthread+0xcf/0xe0 [18253.594725] [] ? kthread_create_on_node+0x140/0x140 [18253.602639] [] ret_from_fork+0x58/0x90 [18253.609367] [] ? kthread_create_on_node+0x140/0x140 [18253.617625] kworker/10:1 S ffff88082d896480 0 252 2 0x00000000 [18253.626116] ffff88082c51fe50 0000000000000046 ffff88082c50bec0 ffff88082c51ffd8 [18253.635062] ffff88082c51ffd8 ffff88082c51ffd8 ffff88082c50bec0 ffff88082d8964a0 [18253.644014] ffff88042c6c6030 ffff88082c50bec0 ffff88042c6c6000 ffff88082d896480 [18253.652941] Call Trace: [18253.656289] [] schedule+0x29/0x70 [18253.662471] [] worker_thread+0x1d9/0x410 [18253.669313] [] ? rescuer_thread+0x460/0x460 [18253.676801] [] kthread+0xcf/0xe0 [18253.683207] [] ? kthread_create_on_node+0x140/0x140 [18253.691100] [] ret_from_fork+0x58/0x90 [18253.697765] [] ? kthread_create_on_node+0x140/0x140 [18253.705672] kworker/26:1 S ffff88082da96480 0 253 2 0x00000000 [18253.714191] ffff88082c523e50 0000000000000046 ffff88082c50ce70 ffff88082c523fd8 [18253.723137] ffff88082c523fd8 ffff88082c523fd8 ffff88082c50ce70 ffff88082da964a0 [18253.732083] ffff88042bc9e330 ffff88082c50ce70 ffff88042bc9e300 ffff88082da96480 [18253.741392] Call Trace: [18253.745088] [] schedule+0x29/0x70 [18253.751592] [] worker_thread+0x1d9/0x410 [18253.758466] [] ? rescuer_thread+0x460/0x460 [18253.765622] [] kthread+0xcf/0xe0 [18253.771709] [] ? kthread_create_on_node+0x140/0x140 [18253.779637] [] ret_from_fork+0x58/0x90 [18253.786291] [] ? kthread_create_on_node+0x140/0x140 [18253.794201] kworker/0:3 S ffff88042e016480 0 254 2 0x00000000 [18253.802746] ffff88042bf4be50 0000000000000046 ffff88042bf16dd0 ffff88042bf4bfd8 [18253.811684] ffff88042bf4bfd8 ffff88042bf4bfd8 ffff88042bf16dd0 ffff88042e0164a0 [18253.820639] ffff88042c4d1730 ffff88042bf16dd0 ffff88042c4d1700 ffff88042e016480 [18253.829598] Call Trace: [18253.832955] [] schedule+0x29/0x70 [18253.839125] [] worker_thread+0x1d9/0x410 [18253.845984] [] ? rescuer_thread+0x460/0x460 [18253.853123] [] kthread+0xcf/0xe0 [18253.859191] [] ? kthread_create_on_node+0x140/0x140 [18253.867113] [] ret_from_fork+0x58/0x90 [18253.873773] [] ? kthread_create_on_node+0x140/0x140 [18253.881695] kworker/5:1 S ffff88042e156480 0 255 2 0x00000000 [18253.890218] ffff88042bf4fe50 0000000000000046 ffff88042bf58000 ffff88042bf4ffd8 [18253.899158] ffff88042bf4ffd8 ffff88042bf4ffd8 ffff88042bf58000 ffff88042e1564a0 [18253.908114] ffff88082c946db0 ffff88042bf58000 ffff88082c946d80 ffff88042e156480 [18253.917047] Call Trace: [18253.920748] [] schedule+0x29/0x70 [18253.927268] [] worker_thread+0x1d9/0x410 [18253.934122] [] ? rescuer_thread+0x460/0x460 [18253.941618] [] kthread+0xcf/0xe0 [18253.947934] [] ? kthread_create_on_node+0x140/0x140 [18253.955883] [] ret_from_fork+0x58/0x90 [18253.962557] [] ? kthread_create_on_node+0x140/0x140 [18253.970479] kauditd S 0000000000000000 0 367 2 0x00000000 [18253.978992] ffff88042c6bfe48 0000000000000046 ffff88042bf58fb0 ffff88042c6bffd8 [18253.987935] ffff88042c6bffd8 ffff88042c6bffd8 ffff88042bf58fb0 0000000000000000 [18253.996883] ffff88042bf58fb0 ffff88042bf58fb0 ffff88042bf58fb0 0000000000000000 [18254.006160] Call Trace: [18254.009847] [] schedule+0x29/0x70 [18254.016012] [] kauditd_thread+0x184/0x1e0 [18254.022966] [] ? wake_up_state+0x20/0x20 [18254.029824] [] ? audit_printk_skb+0x70/0x70 [18254.036969] [] kthread+0xcf/0xe0 [18254.043035] [] ? kthread_create_on_node+0x140/0x140 [18254.050973] [] ret_from_fork+0x58/0x90 [18254.057626] [] ? kthread_create_on_node+0x140/0x140 [18254.065869] ata_sff S ffff88042c69c780 0 452 2 0x00000000 [18254.074418] ffff88042a6d7e08 0000000000000046 ffff88042bf59f60 ffff88042a6d7fd8 [18254.083381] ffff88042a6d7fd8 ffff88042a6d7fd8 ffff88042bf59f60 ffff88042bf59f60 [18254.092320] ffff88042c69c780 ffff88082be84680 ffff88042c69c7b0 ffff88042c69c780 [18254.101247] Call Trace: [18254.104598] [] schedule+0x29/0x70 [18254.110782] [] rescuer_thread+0x359/0x460 [18254.117728] [] ? __schedule+0x3b0/0x990 [18254.124821] [] ? manage_workers.isra.24+0x2d0/0x2d0 [18254.133066] [] kthread+0xcf/0xe0 [18254.139124] [] ? kthread_create_on_node+0x140/0x140 [18254.147047] [] ret_from_fork+0x58/0x90 [18254.153688] [] ? kthread_create_on_node+0x140/0x140 [18254.161609] scsi_eh_0 S 0000000000000000 0 482 2 0x00000000 [18254.170126] ffff880036a9be18 0000000000000046 ffff88042a68af10 ffff880036a9bfd8 [18254.179055] ffff880036a9bfd8 ffff880036a9bfd8 ffff88042a68af10 ffff8801790d0000 [18254.188327] ffff88042a68af10 ffff88042a68b588 0000000000000000 0000000000000000 [18254.197265] Call Trace: [18254.200641] [] schedule+0x29/0x70 [18254.206837] [] scsi_error_handler+0xa3/0x8b0 [18254.214099] [] ? scsi_eh_get_sense+0x250/0x250 [18254.221558] [] kthread+0xcf/0xe0 [18254.227634] [] ? kthread_create_on_node+0x140/0x140 [18254.235582] [] ret_from_fork+0x58/0x90 [18254.242236] [] ? kthread_create_on_node+0x140/0x140 [18254.250506] scsi_tmf_0 S ffff88042c7db600 0 483 2 0x00000000 [18254.259094] ffff880036a9fe08 0000000000000046 ffff88042a68bec0 ffff880036a9ffd8 [18254.268075] ffff880036a9ffd8 ffff880036a9ffd8 ffff88042a68bec0 ffff88042a68bec0 [18254.277020] ffff88042c7db600 ffff88042bac4480 ffff88042c7db630 ffff88042c7db600 [18254.285969] Call Trace: [18254.289328] [] schedule+0x29/0x70 [18254.295510] [] rescuer_thread+0x359/0x460 [18254.302474] [] ? __schedule+0x3b0/0x990 [18254.309373] [] ? manage_workers.isra.24+0x2d0/0x2d0 [18254.317632] [] kthread+0xcf/0xe0 [18254.323719] [] ? kthread_create_on_node+0x140/0x140 [18254.331648] [] ret_from_fork+0x58/0x90 [18254.338309] [] ? kthread_create_on_node+0x140/0x140 [18254.346346] fw_event_mpt2sa S ffff88042c7db680 0 484 2 0x00000000 [18254.354911] ffff8800361a3e08 0000000000000046 ffff88042a68ce70 ffff8800361a3fd8 [18254.363934] ffff8800361a3fd8 ffff8800361a3fd8 ffff88042a68ce70 ffff88042a68ce70 [18254.373235] ffff88042c7db680 ffff88042bac4680 ffff88042c7db6b0 ffff88042c7db680 [18254.382390] Call Trace: [18254.385798] [] schedule+0x29/0x70 [18254.392007] [] rescuer_thread+0x359/0x460 [18254.398983] [] ? __schedule+0x3b0/0x990 [18254.405773] [] ? manage_workers.isra.24+0x2d0/0x2d0 [18254.413726] [] kthread+0xcf/0xe0 [18254.419839] [] ? kthread_create_on_node+0x140/0x140 [18254.427786] [] ret_from_fork+0x58/0x90 [18254.434815] [] ? kthread_create_on_node+0x140/0x140 [18254.443076] mlx4 S ffff88042c610300 0 490 2 0x00000000 [18254.451634] ffff880429e07e08 0000000000000046 ffff88042a68edd0 ffff880429e07fd8 [18254.460605] ffff880429e07fd8 ffff880429e07fd8 ffff88042a68edd0 ffff88042a68edd0 [18254.469581] ffff88042c610300 ffff88042a5fc480 ffff88042c610330 ffff88042c610300 [18254.478527] Call Trace: [18254.481912] [] schedule+0x29/0x70 [18254.488098] [] rescuer_thread+0x359/0x460 [18254.495067] [] ? __schedule+0x3b0/0x990 [18254.502216] [] ? manage_workers.isra.24+0x2d0/0x2d0 [18254.510520] [] kthread+0xcf/0xe0 [18254.516621] [] ? kthread_create_on_node+0x140/0x140 [18254.524549] [] ret_from_fork+0x58/0x90 [18254.531219] [] ? kthread_create_on_node+0x140/0x140 [18254.539143] mlx4_health S ffff88042c610380 0 491 2 0x00000000 [18254.547698] ffff880429e13e08 0000000000000046 ffff880429e08000 ffff880429e13fd8 [18254.556670] ffff880429e13fd8 ffff880429e13fd8 ffff880429e08000 ffff880429e08000 [18254.565646] ffff88042c610380 ffff88042a5fc280 ffff88042c6103b0 ffff88042c610380 [18254.574967] Call Trace: [18254.578681] [] schedule+0x29/0x70 [18254.584884] [] rescuer_thread+0x359/0x460 [18254.591864] [] ? __schedule+0x3b0/0x990 [18254.598645] [] ? manage_workers.isra.24+0x2d0/0x2d0 [18254.606580] [] kthread+0xcf/0xe0 [18254.612662] [] ? kthread_create_on_node+0x140/0x140 [18254.620609] [] ret_from_fork+0x58/0x90 [18254.627630] [] ? kthread_create_on_node+0x140/0x140 [18254.635565] scsi_eh_1 S 0000000000000000 0 496 2 0x00000000 [18254.644306] ffff880429a4be18 0000000000000046 ffff88042bf13ec0 ffff880429a4bfd8 [18254.653544] ffff880429a4bfd8 ffff880429a4bfd8 ffff88042bf13ec0 ffff8800361de000 [18254.662510] ffff88042bf13ec0 ffff88042bf14538 0000000000000000 0000000000000000 [18254.671465] Call Trace: [18254.674833] [] schedule+0x29/0x70 [18254.681027] [] scsi_error_handler+0xa3/0x8b0 [18254.688274] [] ? scsi_eh_get_sense+0x250/0x250 [18254.695722] [] kthread+0xcf/0xe0 [18254.701815] [] ? kthread_create_on_node+0x140/0x140 [18254.709790] [] ret_from_fork+0x58/0x90 [18254.716812] [] ? kthread_create_on_node+0x140/0x140 [18254.724768] scsi_tmf_1 S ffff88042c654a80 0 497 2 0x00000000 [18254.733304] ffff880429a4fe08 0000000000000046 ffff880429f00000 ffff880429a4ffd8 [18254.742253] ffff880429a4ffd8 ffff880429a4ffd8 ffff880429f00000 ffff880429f00000 [18254.751221] ffff88042c654a80 ffff88042a432280 ffff88042c654ab0 ffff88042c654a80 [18254.760171] Call Trace: [18254.763547] [] schedule+0x29/0x70 [18254.769733] [] rescuer_thread+0x359/0x460 [18254.776892] [] ? __schedule+0x3b0/0x990 [18254.784014] [] ? manage_workers.isra.24+0x2d0/0x2d0 [18254.791944] [] kthread+0xcf/0xe0 [18254.798028] [] ? kthread_create_on_node+0x140/0x140 [18254.805943] [] ret_from_fork+0x58/0x90 [18254.812614] [] ? kthread_create_on_node+0x140/0x140 [18254.820566] scsi_eh_2 S 0000000000000000 0 498 2 0x00000000 [18254.829103] ffff880429f0fe18 0000000000000046 ffff880429f00fb0 ffff880429f0ffd8 [18254.838051] ffff880429f0ffd8 ffff880429f0ffd8 ffff880429f00fb0 ffff8800361d9800 [18254.847356] ffff880429f00fb0 ffff880429f01628 0000000000000000 0000000000000000 [18254.856327] Call Trace: [18254.859701] [] schedule+0x29/0x70 [18254.865904] [] scsi_error_handler+0xa3/0x8b0 [18254.873169] [] ? scsi_eh_get_sense+0x250/0x250 [18254.880622] [] kthread+0xcf/0xe0 [18254.886707] [] ? kthread_create_on_node+0x140/0x140 [18254.894645] [] ret_from_fork+0x58/0x90 [18254.901296] [] ? kthread_create_on_node+0x140/0x140 [18254.909590] scsi_tmf_2 S ffff88042c654b80 0 499 2 0x00000000 [18254.918513] ffff880429f1be08 0000000000000046 ffff880429f01f60 ffff880429f1bfd8 [18254.927488] ffff880429f1bfd8 ffff880429f1bfd8 ffff880429f01f60 ffff880429f01f60 [18254.936451] ffff88042c654b80 ffff88042a432880 ffff88042c654bb0 ffff88042c654b80 [18254.945401] Call Trace: [18254.948783] [] schedule+0x29/0x70 [18254.954983] [] rescuer_thread+0x359/0x460 [18254.961958] [] ? __schedule+0x3b0/0x990 [18254.968723] [] ? manage_workers.isra.24+0x2d0/0x2d0 [18254.976663] [] kthread+0xcf/0xe0 [18254.983119] [] ? kthread_create_on_node+0x140/0x140 [18254.991364] [] ret_from_fork+0x58/0x90 [18254.998042] [] ? kthread_create_on_node+0x140/0x140 [18255.005959] scsi_eh_3 S 0000000000000000 0 500 2 0x00000000 [18255.014511] ffff880429f1fe18 0000000000000046 ffff880429f02f10 ffff880429f1ffd8 [18255.023464] ffff880429f1ffd8 ffff880429f1ffd8 ffff880429f02f10 ffff8800361de800 [18255.032417] ffff880429f02f10 ffff880429f03588 0000000000000000 0000000000000000 [18255.041360] Call Trace: [18255.044735] [] schedule+0x29/0x70 [18255.051285] [] scsi_error_handler+0xa3/0x8b0 [18255.058908] [] ? scsi_eh_get_sense+0x250/0x250 [18255.066360] [] kthread+0xcf/0xe0 [18255.072458] [] ? kthread_create_on_node+0x140/0x140 [18255.080380] [] ret_from_fork+0x58/0x90 [18255.087033] [] ? kthread_create_on_node+0x140/0x140 [18255.094963] scsi_tmf_3 S ffff88042c654c80 0 501 2 0x00000000 [18255.103520] ffff880429f23e08 0000000000000046 ffff880429f03ec0 ffff880429f23fd8 [18255.112478] ffff880429f23fd8 ffff880429f23fd8 ffff880429f03ec0 ffff880429f03ec0 [18255.121497] ffff88042c654c80 ffff88042a432e80 ffff88042c654cb0 ffff88042c654c80 [18255.130807] Call Trace: [18255.134169] [] schedule+0x29/0x70 [18255.140351] [] rescuer_thread+0x359/0x460 [18255.147306] [] ? __schedule+0x3b0/0x990 [18255.154067] [] ? manage_workers.isra.24+0x2d0/0x2d0 [18255.161994] [] kthread+0xcf/0xe0 [18255.168443] [] ? kthread_create_on_node+0x140/0x140 [18255.176367] [] ret_from_fork+0x58/0x90 [18255.183041] [] ? kthread_create_on_node+0x140/0x140 [18255.190980] scsi_eh_4 S 0000000000000000 0 502 2 0x00000000 [18255.199537] ffff880429f27e18 0000000000000046 ffff880429f04e70 ffff880429f27fd8 [18255.208521] ffff880429f27fd8 ffff880429f27fd8 ffff880429f04e70 ffff8800361d9000 [18255.217476] ffff880429f04e70 ffff880429f054e8 0000000000000000 0000000000000000 [18255.226429] Call Trace: [18255.229788] [] schedule+0x29/0x70 [18255.235977] [] scsi_error_handler+0xa3/0x8b0 [18255.243585] [] ? scsi_eh_get_sense+0x250/0x250 [18255.251038] [] kthread+0xcf/0xe0 [18255.257098] [] ? kthread_create_on_node+0x140/0x140 [18255.265032] [] ret_from_fork+0x58/0x90 [18255.271702] [] ? kthread_create_on_node+0x140/0x140 [18255.279979] scsi_tmf_4 S ffff88042c654d80 0 503 2 0x00000000 [18255.288557] ffff880429f33e08 0000000000000046 ffff880429f05e20 ffff880429f33fd8 [18255.297507] ffff880429f33fd8 ffff880429f33fd8 ffff880429f05e20 ffff880429f05e20 [18255.306464] ffff88042c654d80 ffff88042a433480 ffff88042c654db0 ffff88042c654d80 [18255.315788] Call Trace: [18255.319523] [] schedule+0x29/0x70 [18255.325716] [] rescuer_thread+0x359/0x460 [18255.333058] [] ? __schedule+0x3b0/0x990 [18255.339847] [] ? manage_workers.isra.24+0x2d0/0x2d0 [18255.347802] [] kthread+0xcf/0xe0 [18255.353889] [] ? kthread_create_on_node+0x140/0x140 [18255.362169] [] ret_from_fork+0x58/0x90 [18255.368840] [] ? kthread_create_on_node+0x140/0x140 [18255.376771] scsi_eh_5 S 0000000000000000 0 504 2 0x00000000 [18255.385308] ffff880429f37e18 0000000000000046 ffff880429f06dd0 ffff880429f37fd8 [18255.394606] ffff880429f37fd8 ffff880429f37fd8 ffff880429f06dd0 ffff8800361df000 [18255.403565] ffff880429f06dd0 ffff880429f07448 0000000000000000 0000000000000000 [18255.412529] Call Trace: [18255.415911] [] schedule+0x29/0x70 [18255.422095] [] scsi_error_handler+0xa3/0x8b0 [18255.429343] [] ? scsi_eh_get_sense+0x250/0x250 [18255.436789] [] kthread+0xcf/0xe0 [18255.442862] [] ? kthread_create_on_node+0x140/0x140 [18255.450800] [] ret_from_fork+0x58/0x90 [18255.457451] [] ? kthread_create_on_node+0x140/0x140 [18255.465359] scsi_tmf_5 S ffff88042c654e80 0 505 2 0x00000000 [18255.474276] ffff880429f43e08 0000000000000046 ffff880429f38000 ffff880429f43fd8 [18255.483246] ffff880429f43fd8 ffff880429f43fd8 ffff880429f38000 ffff880429f38000 [18255.492194] ffff88042c654e80 ffff88042a433a80 ffff88042c654eb0 ffff88042c654e80 [18255.501145] Call Trace: [18255.504525] [] schedule+0x29/0x70 [18255.511070] [] rescuer_thread+0x359/0x460 [18255.518077] [] ? __schedule+0x3b0/0x990 [18255.524842] [] ? manage_workers.isra.24+0x2d0/0x2d0 [18255.532776] [] kthread+0xcf/0xe0 [18255.538857] [] ? kthread_create_on_node+0x140/0x140 [18255.547148] [] ret_from_fork+0x58/0x90 [18255.553946] [] ? kthread_create_on_node+0x140/0x140 [18255.561885] scsi_eh_6 S 0000000000000000 0 506 2 0x00000000 [18255.570431] ffff880429f47e18 0000000000000046 ffff880429f38fb0 ffff880429f47fd8 [18255.579376] ffff880429f47fd8 ffff880429f47fd8 ffff880429f38fb0 ffff8800361d8800 [18255.588682] ffff880429f38fb0 ffff880429f39628 0000000000000000 0000000000000000 [18255.597649] Call Trace: [18255.601025] [] schedule+0x29/0x70 [18255.607214] [] scsi_error_handler+0xa3/0x8b0 [18255.614486] [] ? scsi_eh_get_sense+0x250/0x250 [18255.622305] [] kthread+0xcf/0xe0 [18255.628755] [] ? kthread_create_on_node+0x140/0x140 [18255.636690] [] ret_from_fork+0x58/0x90 [18255.643339] [] ? kthread_create_on_node+0x140/0x140 [18255.651271] scsi_tmf_6 S ffff88042c654f80 0 507 2 0x00000000 [18255.659925] ffff880429f53e08 0000000000000046 ffff880429f39f60 ffff880429f53fd8 [18255.669286] ffff880429f53fd8 ffff880429f53fd8 ffff880429f39f60 ffff880429f39f60 [18255.678298] ffff88042c654f80 ffff88042a434080 ffff88042c654fb0 ffff88042c654f80 [18255.687261] Call Trace: [18255.690637] [] schedule+0x29/0x70 [18255.696837] [] rescuer_thread+0x359/0x460 [18255.704124] [] ? __schedule+0x3b0/0x990 [18255.710899] [] ? manage_workers.isra.24+0x2d0/0x2d0 [18255.718837] [] kthread+0xcf/0xe0 [18255.724931] [] ? kthread_create_on_node+0x140/0x140 [18255.733216] [] ret_from_fork+0x58/0x90 [18255.740216] [] ? kthread_create_on_node+0x140/0x140 [18255.748136] scsi_eh_7 S 0000000000000000 0 509 2 0x00000000 [18255.756687] ffff880429f83e18 0000000000000046 ffff880429f3bec0 ffff880429f83fd8 [18255.765656] ffff880429f83fd8 ffff880429f83fd8 ffff880429f3bec0 ffff88042bfdf000 [18255.774941] ffff880429f3bec0 ffff880429f3c538 0000000000000000 0000000000000000 [18255.783906] Call Trace: [18255.787250] [] schedule+0x29/0x70 [18255.793443] [] scsi_error_handler+0xa3/0x8b0 [18255.800703] [] ? scsi_eh_get_sense+0x250/0x250 [18255.808137] [] kthread+0xcf/0xe0 [18255.814227] [] ? kthread_create_on_node+0x140/0x140 [18255.822148] [] ret_from_fork+0x58/0x90 [18255.828814] [] ? kthread_create_on_node+0x140/0x140 [18255.836749] scsi_tmf_7 S ffff88042b859900 0 511 2 0x00000000 [18255.845296] ffff880429f8be08 0000000000000046 ffff880429f3de20 ffff880429f8bfd8 [18255.854261] ffff880429f8bfd8 ffff880429f8bfd8 ffff880429f3de20 ffff880429f3de20 [18255.863224] ffff88042b859900 ffff88042cae0880 ffff88042b859930 ffff88042b859900 [18255.872173] Call Trace: [18255.875559] [] schedule+0x29/0x70 [18255.881749] [] rescuer_thread+0x359/0x460 [18255.888719] [] ? __schedule+0x3b0/0x990 [18255.895498] [] ? manage_workers.isra.24+0x2d0/0x2d0 [18255.903424] [] kthread+0xcf/0xe0 [18255.909526] [] ? kthread_create_on_node+0x140/0x140 [18255.917452] [] ret_from_fork+0x58/0x90 [18255.924102] [] ? kthread_create_on_node+0x140/0x140 [18255.932031] scsi_wq_7 S ffff88042b859a80 0 513 2 0x00000000 [18255.940591] ffff880036337e08 0000000000000046 ffff880036338000 ffff880036337fd8 [18255.949549] ffff880036337fd8 ffff880036337fd8 ffff880036338000 ffff880036338000 [18255.958497] ffff88042b859a80 ffff88042cae1480 ffff88042b859ab0 ffff88042b859a80 [18255.967455] Call Trace: [18255.970823] [] schedule+0x29/0x70 [18255.977011] [] rescuer_thread+0x359/0x460 [18255.983985] [] ? __schedule+0x3b0/0x990 [18255.990760] [] ? manage_workers.isra.24+0x2d0/0x2d0 [18255.998700] [] kthread+0xcf/0xe0 [18256.004779] [] ? kthread_create_on_node+0x140/0x140 [18256.012700] [] ret_from_fork+0x58/0x90 [18256.019350] [] ? kthread_create_on_node+0x140/0x140 [18256.027274] ttm_swap S ffff880428204180 0 524 2 0x00000000 [18256.035826] ffff88042822be08 0000000000000046 ffff88003633ce70 ffff88042822bfd8 [18256.044786] ffff88042822bfd8 ffff88042822bfd8 ffff88003633ce70 ffff88003633ce70 [18256.053745] ffff880428204180 ffff88042b9bae80 ffff8804282041b0 ffff880428204180 [18256.062716] Call Trace: [18256.066091] [] schedule+0x29/0x70 [18256.072277] [] rescuer_thread+0x359/0x460 [18256.079238] [] ? __schedule+0x3b0/0x990 [18256.086019] [] ? manage_workers.isra.24+0x2d0/0x2d0 [18256.093963] [] kthread+0xcf/0xe0 [18256.100059] [] ? kthread_create_on_node+0x140/0x140 [18256.107994] [] ret_from_fork+0x58/0x90 [18256.114667] [] ? kthread_create_on_node+0x140/0x140 [18256.122611] poll_mpt2sas0_s S ffff88042ba6c480 0 528 2 0x00000000 [18256.131156] ffff880429dffe08 0000000000000046 ffff88003633de20 ffff880429dfffd8 [18256.140121] ffff880429dfffd8 ffff880429dfffd8 ffff88003633de20 ffff88003633de20 [18256.149093] ffff88042ba6c480 ffff88042b9b8880 ffff88042ba6c4b0 ffff88042ba6c480 [18256.158065] Call Trace: [18256.161443] [] schedule+0x29/0x70 [18256.167649] [] rescuer_thread+0x359/0x460 [18256.174625] [] ? __schedule+0x3b0/0x990 [18256.181387] [] ? manage_workers.isra.24+0x2d0/0x2d0 [18256.189311] [] kthread+0xcf/0xe0 [18256.195388] [] ? kthread_create_on_node+0x140/0x140 [18256.203308] [] ret_from_fork+0x58/0x90 [18256.209979] [] ? kthread_create_on_node+0x140/0x140 [18256.217906] kworker/0:1H S ffff88042e016840 0 532 2 0x00000000 [18256.226459] ffff88042bcbfe50 0000000000000046 ffff88042bd00000 ffff88042bcbffd8 [18256.235423] ffff88042bcbffd8 ffff88042bcbffd8 ffff88042bd00000 ffff88042e016860 [18256.244395] ffff88042ba6c6b0 ffff88042bd00000 ffff88042ba6c680 ffff88042e016840 [18256.253339] Call Trace: [18256.256726] [] schedule+0x29/0x70 [18256.262932] [] worker_thread+0x1d9/0x410 [18256.269809] [] ? rescuer_thread+0x460/0x460 [18256.276970] [] kthread+0xcf/0xe0 [18256.283064] [] ? kthread_create_on_node+0x140/0x140 [18256.290994] [] ret_from_fork+0x58/0x90 [18256.297663] [] ? kthread_create_on_node+0x140/0x140 [18256.305597] mlx4_en S ffff88042ba6c880 0 541 2 0x00000000 [18256.314141] ffff88041b133e08 0000000000000046 ffff88042bd01f60 ffff88041b133fd8 [18256.323100] ffff88041b133fd8 ffff88041b133fd8 ffff88042bd01f60 ffff88042bd01f60 [18256.332077] ffff88042ba6c880 ffff880036b64280 ffff88042ba6c8b0 ffff88042ba6c880 [18256.341044] Call Trace: [18256.344422] [] schedule+0x29/0x70 [18256.350625] [] rescuer_thread+0x359/0x460 [18256.357604] [] ? __schedule+0x3b0/0x990 [18256.364370] [] ? manage_workers.isra.24+0x2d0/0x2d0 [18256.372289] [] kthread+0xcf/0xe0 [18256.378361] [] ? kthread_create_on_node+0x140/0x140 [18256.386278] [] ret_from_fork+0x58/0x90 [18256.392942] [] ? kthread_create_on_node+0x140/0x140 [18256.400877] kworker/u482:4 S ffff8804ba627400 0 547 2 0x00000000 [18256.409415] ffff88082c8c7e50 0000000000000046 ffff88082c8c9f60 ffff88082c8c7fd8 [18256.418364] ffff88082c8c7fd8 ffff88082c8c7fd8 ffff88082c8c9f60 ffff8804ba627420 [18256.427317] ffff88082c885bb0 ffff88082c8c9f60 ffff88082c885b80 ffff8804ba627400 [18256.436271] Call Trace: [18256.439649] [] schedule+0x29/0x70 [18256.445836] [] worker_thread+0x1d9/0x410 [18256.452700] [] ? rescuer_thread+0x460/0x460 [18256.459864] [] kthread+0xcf/0xe0 [18256.465953] [] ? kthread_create_on_node+0x140/0x140 [18256.473877] [] ret_from_fork+0x58/0x90 [18256.480542] [] ? kthread_create_on_node+0x140/0x140 [18256.488459] kworker/14:2 S ffff88082d996480 0 584 2 0x00000000 [18256.497010] ffff880829f3fe50 0000000000000046 ffff880829d3bec0 ffff880829f3ffd8 [18256.505973] ffff880829f3ffd8 ffff880829f3ffd8 ffff880829d3bec0 ffff88082d9964a0 [18256.514937] ffff88082c9b0db0 ffff880829d3bec0 ffff88082c9b0d80 ffff88082d996480 [18256.523882] Call Trace: [18256.527246] [] schedule+0x29/0x70 [18256.533424] [] worker_thread+0x1d9/0x410 [18256.540279] [] ? rescuer_thread+0x460/0x460 [18256.547426] [] kthread+0xcf/0xe0 [18256.553508] [] ? kthread_create_on_node+0x140/0x140 [18256.561431] [] ret_from_fork+0x58/0x90 [18256.568081] [] ? kthread_create_on_node+0x140/0x140 [18256.576001] kworker/8:2 S ffff88082d816480 0 585 2 0x00000000 [18256.584542] ffff880829f3be50 0000000000000046 ffff880829d3ce70 ffff880829f3bfd8 [18256.593509] ffff880829f3bfd8 ffff880829f3bfd8 ffff880829d3ce70 ffff88082d8164a0 [18256.602478] ffff88082c30dab0 ffff880829d3ce70 ffff88082c30da80 ffff88082d816480 [18256.611409] Call Trace: [18256.614774] [] schedule+0x29/0x70 [18256.620955] [] worker_thread+0x1d9/0x410 [18256.627814] [] ? rescuer_thread+0x460/0x460 [18256.634971] [] kthread+0xcf/0xe0 [18256.641048] [] ? kthread_create_on_node+0x140/0x140 [18256.648974] [] ret_from_fork+0x58/0x90 [18256.655636] [] ? kthread_create_on_node+0x140/0x140 [18256.663552] kworker/14:1H S ffff88082d996840 0 590 2 0x00000000 [18256.672091] ffff88082a7ffe50 0000000000000046 ffff880829d3af10 ffff88082a7fffd8 [18256.681060] ffff88082a7fffd8 ffff88082a7fffd8 ffff880829d3af10 ffff88082d996860 [18256.690023] ffff88082c9b08b0 ffff880829d3af10 ffff88082c9b0880 ffff88082d996840 [18256.698968] Call Trace: [18256.702327] [] schedule+0x29/0x70 [18256.708493] [] worker_thread+0x1d9/0x410 [18256.715346] [] ? rescuer_thread+0x460/0x460 [18256.722489] [] kthread+0xcf/0xe0 [18256.728565] [] ? kthread_create_on_node+0x140/0x140 [18256.736487] [] ret_from_fork+0x58/0x90 [18256.743144] [] ? kthread_create_on_node+0x140/0x140 [18256.751076] kworker/13:1H S ffff88082d956840 0 591 2 0x00000000 [18256.759612] ffff880829e7be50 0000000000000046 ffff880829d3de20 ffff880829e7bfd8 [18256.768556] ffff880829e7bfd8 ffff880829e7bfd8 ffff880829d3de20 ffff88082d956860 [18256.777509] ffff88042c749cb0 ffff880829d3de20 ffff88042c749c80 ffff88082d956840 [18256.786436] Call Trace: [18256.789793] [] schedule+0x29/0x70 [18256.795981] [] worker_thread+0x1d9/0x410 [18256.802833] [] ? rescuer_thread+0x460/0x460 [18256.809987] [] kthread+0xcf/0xe0 [18256.816073] [] ? kthread_create_on_node+0x140/0x140 [18256.824006] [] ret_from_fork+0x58/0x90 [18256.830665] [] ? kthread_create_on_node+0x140/0x140 [18256.838578] kworker/15:1H S ffff88082d9d6840 0 593 2 0x00000000 [18256.847100] ffff880829d53e50 0000000000000046 ffff880829d3edd0 ffff880829d53fd8 [18256.856041] ffff880829d53fd8 ffff880829d53fd8 ffff880829d3edd0 ffff88082d9d6860 [18256.864991] ffff88082c9d96b0 ffff880829d3edd0 ffff88082c9d9680 ffff88082d9d6840 [18256.873941] Call Trace: [18256.877293] [] schedule+0x29/0x70 [18256.883469] [] worker_thread+0x1d9/0x410 [18256.890318] [] ? rescuer_thread+0x460/0x460 [18256.897462] [] kthread+0xcf/0xe0 [18256.903535] [] ? kthread_create_on_node+0x140/0x140 [18256.911452] [] ret_from_fork+0x58/0x90 [18256.918105] [] ? kthread_create_on_node+0x140/0x140 [18256.926027] kworker/8:1H S ffff88082d816840 0 595 2 0x00000000 [18256.934566] ffff880829fd3e50 0000000000000046 ffff880829fc0000 ffff880829fd3fd8 [18256.943520] ffff880829fd3fd8 ffff880829fd3fd8 ffff880829fc0000 ffff88082d816860 [18256.952451] ffff88082c30df30 ffff880829fc0000 ffff88082c30df00 ffff88082d816840 [18256.961379] Call Trace: [18256.964737] [] schedule+0x29/0x70 [18256.970909] [] worker_thread+0x1d9/0x410 [18256.977764] [] ? rescuer_thread+0x460/0x460 [18256.984919] [] kthread+0xcf/0xe0 [18256.991002] [] ? kthread_create_on_node+0x140/0x140 [18256.998922] [] ret_from_fork+0x58/0x90 [18257.005587] [] ? kthread_create_on_node+0x140/0x140 [18257.013509] kworker/28:1H S ffff88082db16840 0 598 2 0x00000000 [18257.022039] ffff880829ffbe50 0000000000000046 ffff880829fc0fb0 ffff880829ffbfd8 [18257.031001] ffff880829ffbfd8 ffff880829ffbfd8 ffff880829fc0fb0 ffff88082db16860 [18257.039973] ffff88082ca6a7b0 ffff880829fc0fb0 ffff88082ca6a780 ffff88082db16840 [18257.048925] Call Trace: [18257.052271] [] schedule+0x29/0x70 [18257.058442] [] worker_thread+0x1d9/0x410 [18257.065292] [] ? rescuer_thread+0x460/0x460 [18257.072433] [] kthread+0xcf/0xe0 [18257.078510] [] ? kthread_create_on_node+0x140/0x140 [18257.086423] [] ret_from_fork+0x58/0x90 [18257.093081] [] ? kthread_create_on_node+0x140/0x140 [18257.100993] kworker/25:1H S ffff88082da56840 0 600 2 0x00000000 [18257.109518] ffff880829407e50 0000000000000046 ffff880829fc1f60 ffff880829407fd8 [18257.118474] ffff880829407fd8 ffff880829407fd8 ffff880829fc1f60 ffff88082da56860 [18257.127420] ffff88082c8851b0 ffff880829fc1f60 ffff88082c885180 ffff88082da56840 [18257.136358] Call Trace: [18257.139714] [] schedule+0x29/0x70 [18257.145893] [] worker_thread+0x1d9/0x410 [18257.152750] [] ? rescuer_thread+0x460/0x460 [18257.159892] [] kthread+0xcf/0xe0 [18257.165970] [] ? kthread_create_on_node+0x140/0x140 [18257.173896] [] ret_from_fork+0x58/0x90 [18257.180549] [] ? kthread_create_on_node+0x140/0x140 [18257.188454] kworker/26:1H S ffff88082da96840 0 607 2 0x00000000 [18257.196973] ffff88082a763e50 0000000000000046 ffff880829fc2f10 ffff88082a763fd8 [18257.205934] ffff88082a763fd8 ffff88082a763fd8 ffff880829fc2f10 ffff88082da96860 [18257.214897] ffff88042bc9f830 ffff880829fc2f10 ffff88042bc9f800 ffff88082da96840 [18257.223849] Call Trace: [18257.227199] [] schedule+0x29/0x70 [18257.233371] [] worker_thread+0x1d9/0x410 [18257.240220] [] ? rescuer_thread+0x460/0x460 [18257.247364] [] kthread+0xcf/0xe0 [18257.253434] [] ? kthread_create_on_node+0x140/0x140 [18257.261350] [] ret_from_fork+0x58/0x90 [18257.268012] [] ? kthread_create_on_node+0x140/0x140 [18257.275933] kworker/11:1H S ffff88082d8d6840 0 608 2 0x00000000 [18257.284451] ffff880829f0fe50 0000000000000046 ffff880829fc3ec0 ffff880829f0ffd8 [18257.293388] ffff880829f0ffd8 ffff880829f0ffd8 ffff880829fc3ec0 ffff88082d8d6860 [18257.302318] ffff88042c6ef4b0 ffff880829fc3ec0 ffff88042c6ef480 ffff88082d8d6840 [18257.311248] Call Trace: [18257.314615] [] schedule+0x29/0x70 [18257.320806] [] worker_thread+0x1d9/0x410 [18257.327660] [] ? rescuer_thread+0x460/0x460 [18257.334796] [] kthread+0xcf/0xe0 [18257.340868] [] ? kthread_create_on_node+0x140/0x140 [18257.348797] [] ret_from_fork+0x58/0x90 [18257.355452] [] ? kthread_create_on_node+0x140/0x140 [18257.363363] kworker/12:1H S ffff88082d916840 0 609 2 0x00000000 [18257.371905] ffff880829effe50 0000000000000046 ffff880829fc4e70 ffff880829efffd8 [18257.380852] ffff880829efffd8 ffff880829efffd8 ffff880829fc4e70 ffff88082d916860 [18257.389809] ffff88042c7212b0 ffff880829fc4e70 ffff88042c721280 ffff88082d916840 [18257.398768] Call Trace: [18257.402125] [] schedule+0x29/0x70 [18257.408300] [] worker_thread+0x1d9/0x410 [18257.415151] [] ? rescuer_thread+0x460/0x460 [18257.422297] [] kthread+0xcf/0xe0 [18257.428372] [] ? kthread_create_on_node+0x140/0x140 [18257.436293] [] ret_from_fork+0x58/0x90 [18257.442951] [] ? kthread_create_on_node+0x140/0x140 [18257.450866] kworker/11:2 S ffff88082d8d6480 0 610 2 0x00000000 [18257.459389] ffff88082a6f3e50 0000000000000046 ffff880829fc5e20 ffff88082a6f3fd8 [18257.468320] ffff88082a6f3fd8 ffff88082a6f3fd8 ffff880829fc5e20 ffff88082d8d64a0 [18257.477279] ffff88042c6ef5b0 ffff880829fc5e20 ffff88042c6ef580 ffff88082d8d6480 [18257.486233] Call Trace: [18257.489591] [] schedule+0x29/0x70 [18257.495757] [] worker_thread+0x1d9/0x410 [18257.502619] [] ? rescuer_thread+0x460/0x460 [18257.509766] [] kthread+0xcf/0xe0 [18257.515844] [] ? kthread_create_on_node+0x140/0x140 [18257.523777] [] ret_from_fork+0x58/0x90 [18257.530434] [] ? kthread_create_on_node+0x140/0x140 [18257.538352] jbd2/sda1-8 S ffff88082c7c0800 0 613 2 0x00000000 [18257.546893] ffff88041ac53e30 0000000000000046 ffff880829fc6dd0 ffff88041ac53fd8 [18257.555852] ffff88041ac53fd8 ffff88041ac53fd8 ffff880829fc6dd0 ffff88082c7c0828 [18257.564821] ffff88082c7c0be8 ffff88082c7c08a8 ffff88041ac53e80 ffff88082c7c0800 [18257.573776] Call Trace: [18257.577132] [] schedule+0x29/0x70 [18257.583331] [] kjournald2+0x203/0x260 [jbd2] [18257.590585] [] ? wake_up_atomic_t+0x30/0x30 [18257.597750] [] ? commit_timeout+0x10/0x10 [jbd2] [18257.605387] [] kthread+0xcf/0xe0 [18257.611467] [] ? kthread_create_on_node+0x140/0x140 [18257.619393] [] ret_from_fork+0x58/0x90 [18257.626064] [] ? kthread_create_on_node+0x140/0x140 [18257.633984] ext4-rsv-conver S ffff88042c749e80 0 614 2 0x00000000 [18257.642543] ffff88042a60be08 0000000000000046 ffff88041b388000 ffff88042a60bfd8 [18257.651519] ffff88042a60bfd8 ffff88042a60bfd8 ffff88041b388000 ffff88041b388000 [18257.660456] ffff88042c749e80 ffff88082c20c880 ffff88042c749eb0 ffff88042c749e80 [18257.669392] Call Trace: [18257.672769] [] schedule+0x29/0x70 [18257.678964] [] rescuer_thread+0x359/0x460 [18257.685936] [] ? __schedule+0x3b0/0x990 [18257.692703] [] ? manage_workers.isra.24+0x2d0/0x2d0 [18257.700636] [] kthread+0xcf/0xe0 [18257.706719] [] ? kthread_create_on_node+0x140/0x140 [18257.714657] [] ret_from_fork+0x58/0x90 [18257.721304] [] ? kthread_create_on_node+0x140/0x140 [18257.729227] kworker/13:2 S ffff88082d956480 0 616 2 0x00000000 [18257.737792] ffff880829f5be50 0000000000000046 ffff880829f50000 ffff880829f5bfd8 [18257.746771] ffff880829f5bfd8 ffff880829f5bfd8 ffff880829f50000 ffff88082d9564a0 [18257.755736] ffff88042c7492b0 ffff880829f50000 ffff88042c749280 ffff88082d956480 [18257.764693] Call Trace: [18257.768076] [] schedule+0x29/0x70 [18257.774270] [] worker_thread+0x1d9/0x410 [18257.781139] [] ? rescuer_thread+0x460/0x460 [18257.788290] [] kthread+0xcf/0xe0 [18257.794376] [] ? kthread_create_on_node+0x140/0x140 [18257.802306] [] ret_from_fork+0x58/0x90 [18257.808971] [] ? kthread_create_on_node+0x140/0x140 [18257.816903] kworker/27:1H S ffff88082dad6840 0 625 2 0x00000000 [18257.825446] ffff880829fefe50 0000000000000046 ffff880829f50fb0 ffff880829feffd8 [18257.834397] ffff880829feffd8 ffff880829feffd8 ffff880829f50fb0 ffff88082dad6860 [18257.843356] ffff88042bcc7d30 ffff880829f50fb0 ffff88042bcc7d00 ffff88082dad6840 [18257.852320] Call Trace: [18257.855706] [] schedule+0x29/0x70 [18257.861902] [] worker_thread+0x1d9/0x410 [18257.868768] [] ? rescuer_thread+0x460/0x460 [18257.875930] [] kthread+0xcf/0xe0 [18257.882004] [] ? kthread_create_on_node+0x140/0x140 [18257.889941] [] ret_from_fork+0x58/0x90 [18257.896595] [] ? kthread_create_on_node+0x140/0x140 [18257.904525] kworker/31:1H S ffff88082dbd6840 0 642 2 0x00000000 [18257.913091] ffff880829c17e50 0000000000000046 ffff880829f51f60 ffff880829c17fd8 [18257.922063] ffff880829c17fd8 ffff880829c17fd8 ffff880829f51f60 ffff88082dbd6860 [18257.931021] ffff88042bd41cb0 ffff880829f51f60 ffff88042bd41c80 ffff88082dbd6840 [18257.939982] Call Trace: [18257.943350] [] schedule+0x29/0x70 [18257.949521] [] worker_thread+0x1d9/0x410 [18257.956380] [] ? rescuer_thread+0x460/0x460 [18257.963541] [] kthread+0xcf/0xe0 [18257.969628] [] ? kthread_create_on_node+0x140/0x140 [18257.977560] [] ret_from_fork+0x58/0x90 [18257.984221] [] ? kthread_create_on_node+0x140/0x140 [18257.992121] kworker/31:2 S ffff88082dbd6480 0 644 2 0x00000000 [18258.000667] ffff88082a747e50 0000000000000046 ffff880829f52f10 ffff88082a747fd8 [18258.009628] ffff88082a747fd8 ffff88082a747fd8 ffff880829f52f10 ffff88082dbd64a0 [18258.018579] ffff88042bd41e30 ffff880829f52f10 ffff88042bd41e00 ffff88082dbd6480 [18258.027529] Call Trace: [18258.030886] [] schedule+0x29/0x70 [18258.037070] [] worker_thread+0x1d9/0x410 [18258.043934] [] ? rescuer_thread+0x460/0x460 [18258.051085] [] kthread+0xcf/0xe0 [18258.057158] [] ? kthread_create_on_node+0x140/0x140 [18258.065090] [] ret_from_fork+0x58/0x90 [18258.071748] [] ? kthread_create_on_node+0x140/0x140 [18258.079662] kworker/2:1H S ffff88042e096840 0 664 2 0x00000000 [18258.088192] ffff88041b1e3e50 0000000000000046 ffff88041b38edd0 ffff88041b1e3fd8 [18258.097149] ffff88041b1e3fd8 ffff88041b1e3fd8 ffff88041b38edd0 ffff88042e096860 [18258.106105] ffff88041b2bcfb0 ffff88041b38edd0 ffff88041b2bcf80 ffff88042e096840 [18258.115063] Call Trace: [18258.118428] [] schedule+0x29/0x70 [18258.124603] [] worker_thread+0x1d9/0x410 [18258.131453] [] ? rescuer_thread+0x460/0x460 [18258.138598] [] kthread+0xcf/0xe0 [18258.144670] [] ? kthread_create_on_node+0x140/0x140 [18258.152601] [] ret_from_fork+0x58/0x90 [18258.159257] [] ? kthread_create_on_node+0x140/0x140 [18258.167176] kworker/18:1H S ffff88042e296840 0 692 2 0x00000000 [18258.175730] ffff880036aebe50 0000000000000046 ffff88041b38de20 ffff880036aebfd8 [18258.184678] ffff880036aebfd8 ffff880036aebfd8 ffff88041b38de20 ffff88042e296860 [18258.193638] ffff88082ca4a330 ffff88041b38de20 ffff88082ca4a300 ffff88042e296840 [18258.202573] Call Trace: [18258.205940] [] schedule+0x29/0x70 [18258.212124] [] worker_thread+0x1d9/0x410 [18258.218974] [] ? rescuer_thread+0x460/0x460 [18258.226120] [] kthread+0xcf/0xe0 [18258.232187] [] ? kthread_create_on_node+0x140/0x140 [18258.240109] [] ret_from_fork+0x58/0x90 [18258.246768] [] ? kthread_create_on_node+0x140/0x140 [18258.254691] kworker/1:1H S ffff88042e056840 0 693 2 0x00000000 [18258.263234] ffff88041b18fe50 0000000000000046 ffff88041b38ce70 ffff88041b18ffd8 [18258.272192] ffff88041b18ffd8 ffff88041b18ffd8 ffff88041b38ce70 ffff88042e056860 [18258.281145] ffff880179222fb0 ffff88041b38ce70 ffff880179222f80 ffff88042e056840 [18258.290086] Call Trace: [18258.293448] [] schedule+0x29/0x70 [18258.299623] [] worker_thread+0x1d9/0x410 [18258.306468] [] ? rescuer_thread+0x460/0x460 [18258.313615] [] kthread+0xcf/0xe0 [18258.319691] [] ? kthread_create_on_node+0x140/0x140 [18258.327614] [] ret_from_fork+0x58/0x90 [18258.334280] [] ? kthread_create_on_node+0x140/0x140 [18258.342194] systemd-journal W 0000000000000000 0 700 1 0x00000000 [18258.350731] ffff88042ae4bdb8 0000000000000082 ffff8804287f4e70 ffff88042ae4bfd8 [18258.359685] ffff88042ae4bfd8 ffff88042ae4bfd8 ffff8804287f4e70 0000000000000000 [18258.368638] 0000000000000000 ffff88082a541020 ffff8804287f4e70 0000000000000000 [18258.377580] Call Trace: [18258.380931] [] schedule+0x29/0x70 [18258.387120] [] schedule_hrtimeout_range_clock+0x12d/0x150 [18258.395641] [] ? ep_scan_ready_list.isra.7+0x1b9/0x1f0 [18258.403869] [] schedule_hrtimeout_range+0x13/0x20 [18258.411595] [] ep_poll+0x23e/0x360 [18258.417875] [] ? do_truncate+0x88/0xc0 [18258.424546] [] ? __sb_start_write+0x58/0x110 [18258.431789] [] ? wake_up_state+0x20/0x20 [18258.438636] [] SyS_epoll_wait+0xed/0x120 [18258.445485] [] system_call_fastpath+0x16/0x1b [18258.452816] kworker/12:2 S ffff88082d916480 0 703 2 0x00000000 [18258.461345] ffff88082912fe50 0000000000000046 ffff880829f56dd0 ffff88082912ffd8 [18258.470294] ffff88082912ffd8 ffff88082912ffd8 ffff880829f56dd0 ffff88082d9164a0 [18258.479240] ffff88042c7213b0 ffff880829f56dd0 ffff88042c721380 ffff88082d916480 [18258.488195] Call Trace: [18258.491549] [] schedule+0x29/0x70 [18258.497736] [] worker_thread+0x1d9/0x410 [18258.504596] [] ? rescuer_thread+0x460/0x460 [18258.511737] [] kthread+0xcf/0xe0 [18258.517816] [] ? kthread_create_on_node+0x140/0x140 [18258.525722] [] ret_from_fork+0x58/0x90 [18258.532366] [] ? kthread_create_on_node+0x140/0x140 [18258.540277] rpciod S ffff88042c6efd80 0 714 2 0x00000000 [18258.548813] ffff88042c713e08 0000000000000046 ffff880829f55e20 ffff88042c713fd8 [18258.557776] ffff88042c713fd8 ffff88042c713fd8 ffff880829f55e20 ffff880829f55e20 [18258.566729] ffff88042c6efd80 ffff88082bda7280 ffff88042c6efdb0 ffff88042c6efd80 [18258.575695] Call Trace: [18258.579062] [] schedule+0x29/0x70 [18258.585239] [] rescuer_thread+0x359/0x460 [18258.592190] [] ? __schedule+0x3b0/0x990 [18258.598954] [] ? manage_workers.isra.24+0x2d0/0x2d0 [18258.606871] [] kthread+0xcf/0xe0 [18258.612943] [] ? kthread_create_on_node+0x140/0x140 [18258.620865] [] ret_from_fork+0x58/0x90 [18258.627507] [] ? kthread_create_on_node+0x140/0x140 [18258.635409] kworker/3:1H S ffff88042e0d6840 0 716 2 0x00000000 [18258.643967] ffff880036a63e50 0000000000000046 ffff88041b388fb0 ffff880036a63fd8 [18258.652937] ffff880036a63fd8 ffff880036a63fd8 ffff88041b388fb0 ffff88042e0d6860 [18258.661908] ffff88042f772d30 ffff88041b388fb0 ffff88042f772d00 ffff88042e0d6840 [18258.670846] Call Trace: [18258.674206] [] schedule+0x29/0x70 [18258.680385] [] worker_thread+0x1d9/0x410 [18258.687235] [] ? rescuer_thread+0x460/0x460 [18258.694370] [] kthread+0xcf/0xe0 [18258.700437] [] ? kthread_create_on_node+0x140/0x140 [18258.708352] [] ret_from_fork+0x58/0x90 [18258.715008] [] ? kthread_create_on_node+0x140/0x140 [18258.722930] kworker/26:2 S ffff88082da96480 0 718 2 0x00000000 [18258.731459] ffff88082e6f3e50 0000000000000046 ffff880829f53ec0 ffff88082e6f3fd8 [18258.740408] ffff88082e6f3fd8 ffff88082e6f3fd8 ffff880829f53ec0 ffff88082da964a0 [18258.749362] ffff88042bc9e0b0 ffff880829f53ec0 ffff88042bc9e080 ffff88082da96480 [18258.758314] Call Trace: [18258.761680] [] schedule+0x29/0x70 [18258.767864] [] worker_thread+0x1d9/0x410 [18258.774710] [] ? rescuer_thread+0x460/0x460 [18258.781856] [] kthread+0xcf/0xe0 [18258.787942] [] ? kthread_create_on_node+0x140/0x140 [18258.795865] [] ret_from_fork+0x58/0x90 [18258.802520] [] ? kthread_create_on_node+0x140/0x140 [18258.810433] kworker/1:2 S ffff88042e056480 0 723 2 0x00000000 [18258.818974] ffff880036a4fe50 0000000000000046 ffff88041b38bec0 ffff880036a4ffd8 [18258.827947] ffff880036a4ffd8 ffff880036a4ffd8 ffff88041b38bec0 ffff88042e0564a0 [18258.836889] ffff880179223630 ffff88041b38bec0 ffff880179223600 ffff88042e056480 [18258.845846] Call Trace: [18258.849209] [] schedule+0x29/0x70 [18258.855391] [] worker_thread+0x1d9/0x410 [18258.862255] [] ? rescuer_thread+0x460/0x460 [18258.869407] [] kthread+0xcf/0xe0 [18258.875489] [] ? kthread_create_on_node+0x140/0x140 [18258.883416] [] ret_from_fork+0x58/0x90 [18258.890082] [] ? kthread_create_on_node+0x140/0x140 [18258.898011] lvmetad S 0000000000000004 0 731 1 0x00000000 [18258.906558] ffff880429bdf8e8 0000000000000086 ffff88042ba03ec0 ffff880429bdffd8 [18258.915505] ffff880429bdffd8 ffff880429bdffd8 ffff88042ba03ec0 0000000000000000 [18258.924443] ffff880429bdfad0 00000000fffffffc 0000000000000004 0000000000000004 [18258.933380] Call Trace: [18258.936759] [] schedule+0x29/0x70 [18258.942942] [] schedule_hrtimeout_range_clock+0x12d/0x150 [18258.951457] [] ? add_wait_queue+0x3c/0x50 [18258.958420] [] ? __pollwait+0x7f/0xf0 [18258.964990] [] schedule_hrtimeout_range+0x13/0x20 [18258.972730] [] poll_schedule_timeout+0x55/0xb0 [18258.980175] [] do_select+0x6d1/0x7c0 [18258.986631] [] ? poll_select_copy_remaining+0x150/0x150 [18258.994947] [] ? radix_tree_insert+0x41/0xf0 [18259.002184] [] ? list_del+0xd/0x30 [18259.008452] [] ? __rmqueue+0x8a/0x460 [18259.015018] [] ? shmem_add_to_page_cache.isra.19+0xf0/0x170 [18259.023709] [] ? zone_statistics+0x89/0xa0 [18259.030763] [] ? list_del+0xd/0x30 [18259.037026] [] ? __rmqueue+0x8a/0x460 [18259.043590] [] ? zone_statistics+0x89/0xa0 [18259.050641] [] ? radix_tree_lookup_slot+0x22/0x50 [18259.058362] [] ? __find_get_page+0x1e/0xa0 [18259.065399] [] ? filemap_fault+0x215/0x410 [18259.072422] [] core_sys_select+0x1db/0x300 [18259.079442] [] ? handle_mm_fault+0x6b1/0xfe0 [18259.086656] [] ? enqueue_task+0x51/0x80 [18259.093396] [] ? __do_page_fault+0x171/0x450 [18259.100613] [] SyS_select+0xba/0x110 [18259.107050] [] system_call_fastpath+0x16/0x1b [18259.114354] systemd-udevd S 0000000000000000 0 733 1 0x00000000 [18259.122873] ffff88042a47fdb8 0000000000000086 ffff88042ba04e70 ffff88042a47ffd8 [18259.131812] ffff88042a47ffd8 ffff88042a47ffd8 ffff88042ba04e70 0000000000000000 [18259.140744] 0000000000000000 ffff8804b9721da0 ffff88042ba04e70 0000000000000000 [18259.149684] Call Trace: [18259.153023] [] schedule+0x29/0x70 [18259.159170] [] schedule_hrtimeout_range_clock+0x12d/0x150 [18259.167648] [] ? ep_scan_ready_list.isra.7+0x1b9/0x1f0 [18259.175844] [] schedule_hrtimeout_range+0x13/0x20 [18259.183547] [] ep_poll+0x23e/0x360 [18259.189795] [] ? do_unlinkat+0xf1/0x2b0 [18259.196515] [] ? wake_up_state+0x20/0x20 [18259.203337] [] SyS_epoll_wait+0xed/0x120 [18259.210154] [] system_call_fastpath+0x16/0x1b [18259.217459] kworker/29:1H S ffff88082db56840 0 734 2 0x00000000 [18259.225969] ffff880829e3be50 0000000000000046 ffff880829f54e70 ffff880829e3bfd8 [18259.234919] ffff880829e3bfd8 ffff880829e3bfd8 ffff880829f54e70 ffff88082db56860 [18259.243840] ffff88082ca8c7b0 ffff880829f54e70 ffff88082ca8c780 ffff88082db56840 [18259.252742] Call Trace: [18259.256069] [] schedule+0x29/0x70 [18259.262219] [] worker_thread+0x1d9/0x410 [18259.269044] [] ? rescuer_thread+0x460/0x460 [18259.276149] [] kthread+0xcf/0xe0 [18259.282182] [] ? kthread_create_on_node+0x140/0x140 [18259.290064] [] ret_from_fork+0x58/0x90 [18259.296683] [] ? kthread_create_on_node+0x140/0x140 [18259.304574] kmpathd S ffff88042bc9f780 0 745 2 0x00000000 [18259.313089] ffff88042b687e08 0000000000000046 ffff88041b389f60 ffff88042b687fd8 [18259.322032] ffff88042b687fd8 ffff88042b687fd8 ffff88041b389f60 ffff88041b389f60 [18259.330969] ffff88042bc9f780 ffff88083fa05c80 ffff88042bc9f7b0 ffff88042bc9f780 [18259.339908] Call Trace: [18259.343240] [] schedule+0x29/0x70 [18259.349387] [] rescuer_thread+0x359/0x460 [18259.356313] [] ? __schedule+0x3b0/0x990 [18259.363041] [] ? manage_workers.isra.24+0x2d0/0x2d0 [18259.370933] [] kthread+0xcf/0xe0 [18259.376982] [] ? kthread_create_on_node+0x140/0x140 [18259.384887] [] ret_from_fork+0x58/0x90 [18259.391517] [] ? kthread_create_on_node+0x140/0x140 [18259.399415] kmpath_handlerd S ffff88042bc9f000 0 746 2 0x00000000 [18259.407940] ffff88041aaf7e08 0000000000000046 ffff88041b38af10 ffff88041aaf7fd8 [18259.416891] ffff88041aaf7fd8 ffff88041aaf7fd8 ffff88041b38af10 ffff88041b38af10 [18259.425831] ffff88042bc9f000 ffff88083fa05e80 ffff88042bc9f030 ffff88042bc9f000 [18259.434781] Call Trace: [18259.438144] [] schedule+0x29/0x70 [18259.444313] [] rescuer_thread+0x359/0x460 [18259.451265] [] ? __schedule+0x3b0/0x990 [18259.458026] [] ? manage_workers.isra.24+0x2d0/0x2d0 [18259.465940] [] kthread+0xcf/0xe0 [18259.472022] [] ? kthread_create_on_node+0x140/0x140 [18259.479943] [] ret_from_fork+0x58/0x90 [18259.486602] [] ? kthread_create_on_node+0x140/0x140 [18259.494525] kworker/15:2 S ffff88082d9d6480 0 751 2 0x00000000 [18259.503065] ffff88082d69fe50 0000000000000046 ffff88082a400000 ffff88082d69ffd8 [18259.512026] ffff88082d69ffd8 ffff88082d69ffd8 ffff88082a400000 ffff88082d9d64a0 [18259.520985] ffff88082c9d9830 ffff88082a400000 ffff88082c9d9800 ffff88082d9d6480 [18259.529941] Call Trace: [18259.533294] [] schedule+0x29/0x70 [18259.539469] [] worker_thread+0x1d9/0x410 [18259.546329] [] ? rescuer_thread+0x460/0x460 [18259.553478] [] kthread+0xcf/0xe0 [18259.559556] [] ? kthread_create_on_node+0x140/0x140 [18259.567468] [] ret_from_fork+0x58/0x90 [18259.574130] [] ? kthread_create_on_node+0x140/0x140 [18259.582058] multipathd S ffffc900035a7d80 0 767 1 0x00000000 [18259.590610] ffff8800b6ddbcd0 0000000000000086 ffff88042b921f60 ffff8800b6ddbfd8 [18259.599542] ffff8800b6ddbfd8 ffff8800b6ddbfd8 ffff88042b921f60 ffff88042b921f60 [18259.608491] ffff88042b921f60 0000000000000000 ffff8800b6ddbde0 ffffc900035a7d80 [18259.617460] Call Trace: [18259.620836] [] schedule+0x29/0x70 [18259.627031] [] futex_wait_queue_me+0xc4/0x120 [18259.634386] [] futex_wait+0x179/0x280 [18259.640963] [] ? __generic_file_aio_write+0x1e2/0x400 [18259.649096] [] do_futex+0xfe/0x5b0 [18259.655373] [] ? mntput+0x24/0x40 [18259.661539] [] ? __fput+0x183/0x260 [18259.667909] [] SyS_futex+0x80/0x180 [18259.674269] [] ? do_notify_resume+0x92/0xb0 [18259.681395] [] system_call_fastpath+0x16/0x1b [18259.688723] multipathd S ffffc90003729d40 0 770 1 0x00000000 [18259.697240] ffff88042a547cd0 0000000000000086 ffff88042e97af10 ffff88042a547fd8 [18259.706189] ffff88042a547fd8 ffff88042a547fd8 ffff88042e97af10 ffff88042e97af10 [18259.715147] ffff88042e97af10 0000000000000000 ffff88042a547de0 ffffc90003729d40 [18259.724117] Call Trace: [18259.727484] [] schedule+0x29/0x70 [18259.733658] [] futex_wait_queue_me+0xc4/0x120 [18259.741001] [] futex_wait+0x179/0x280 [18259.747546] [] ? __sb_start_write+0x58/0x110 [18259.754778] [] ? __mark_inode_dirty+0xa0/0x290 [18259.762194] [] do_futex+0xfe/0x5b0 [18259.768447] [] SyS_futex+0x80/0x180 [18259.774791] [] ? fd_install+0x25/0x30 [18259.781314] [] system_call_fastpath+0x16/0x1b [18259.788634] multipathd S ffff88042c2ffb70 0 832 1 0x00000000 [18259.797144] ffff88042c2ffa00 0000000000000086 ffff88042ba00fb0 ffff88042c2fffd8 [18259.806096] ffff88042c2fffd8 ffff88042c2fffd8 ffff88042ba00fb0 ffff88082bf9f090 [18259.815039] ffff88082bf9f090 ffff88042c2ffb74 ffff88082bf9f000 ffff88042c2ffb70 [18259.823972] Call Trace: [18259.827307] [] schedule+0x29/0x70 [18259.833462] [] schedule_timeout+0x239/0x2c0 [18259.840586] [] ? __slab_free+0x81/0x2f0 [18259.847312] [] ? __slab_free+0x81/0x2f0 [18259.854045] [] __skb_recv_datagram+0x3ca/0x460 [18259.861447] [] ? datagram_poll+0x100/0x100 [18259.868444] [] skb_recv_datagram+0x41/0x60 [18259.875450] [] netlink_recvmsg+0x62/0x450 [18259.882359] [] sock_recvmsg+0xbf/0x100 [18259.888972] [] ? enqueue_pushable_task+0x6d/0x90 [18259.896561] [] ___sys_recvmsg+0x11e/0x2b0 [18259.903465] [] ? do_futex+0x172/0x5b0 [18259.909983] [] ? _raw_spin_unlock_bh+0x1b/0x40 [18259.917373] [] ? release_sock+0x118/0x170 [18259.924270] [] __sys_recvmsg+0x51/0x90 [18259.930867] [] SyS_recvmsg+0x12/0x20 [18259.937273] [] system_call_fastpath+0x16/0x1b [18259.944545] multipathd S ffff88042a393b64 0 833 1 0x00000000 [18259.953005] ffff88042a3939f0 0000000000000086 ffff88042ba00000 ffff88042a393fd8 [18259.961911] ffff88042a393fd8 ffff88042a393fd8 ffff88042ba00000 ffff88042a393b48 [18259.970812] 0000000000000000 0000000000000000 0000000000000000 ffff88042a393b64 [18259.979704] Call Trace: [18259.983019] [] schedule+0x29/0x70 [18259.989136] [] schedule_hrtimeout_range_clock+0xb2/0x150 [18259.997497] [] ? hrtimer_get_res+0x50/0x50 [18260.004504] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [18260.013082] [] schedule_hrtimeout_range+0x13/0x20 [18260.020785] [] poll_schedule_timeout+0x55/0xb0 [18260.028197] [] do_sys_poll+0x4cd/0x580 [18260.034820] [] ? zone_statistics+0x89/0xa0 [18260.041830] [] ? zone_statistics+0x89/0xa0 [18260.048834] [] ? get_page_from_freelist+0x51a/0x9f0 [18260.056713] [] ? poll_select_copy_remaining+0x150/0x150 [18260.064971] [] ? __alloc_pages_nodemask+0x176/0x420 [18260.072845] [] ? __mem_cgroup_commit_charge+0xe7/0x370 [18260.081013] [] ? add_page_to_unevictable_list+0xcb/0x160 [18260.089367] [] ? __kmalloc+0x55/0x240 [18260.095884] [] ? ktime_get_ts64+0x76/0xf0 [18260.102797] [] ? recalc_sigpending+0x1b/0x50 [18260.110005] [] ? __set_task_blocked+0x41/0xa0 [18260.117296] [] SyS_ppoll+0xce/0x1d0 [18260.123627] [] system_call_fastpath+0x16/0x1b [18260.130920] multipathd S 00007f4308cd0670 0 839 1 0x00000000 [18260.139399] ffff88042bb9fe68 0000000000000086 ffff88042ba01f60 ffff88042bb9ffd8 [18260.148297] ffff88042bb9ffd8 ffff88042bb9ffd8 ffff88042ba01f60 ffff88042ba01f60 [18260.157207] ffff88042bb9feb8 ffff88042ba01f60 ffff88042bb9ff58 00007f4308cd0670 [18260.166110] Call Trace: [18260.169416] [] schedule+0x29/0x70 [18260.175540] [] do_nanosleep+0x86/0xf0 [18260.182062] [] hrtimer_nanosleep+0xbb/0x180 [18260.189164] [] ? hrtimer_get_res+0x50/0x50 [18260.196178] [] ? do_nanosleep+0x72/0xf0 [18260.202891] [] SyS_nanosleep+0x7e/0x90 [18260.209503] [] system_call_fastpath+0x16/0x1b [18260.216807] multipathd S ffffc9000394d440 0 840 1 0x00000000 [18260.225290] ffff88042bb7fcd0 0000000000000086 ffff88042ba02f10 ffff88042bb7ffd8 [18260.234216] ffff88042bb7ffd8 ffff88042bb7ffd8 ffff88042ba02f10 ffff88042ba02f10 [18260.243150] ffff88042ba02f10 0000000000000000 ffff88042bb7fde0 ffffc9000394d440 [18260.252086] Call Trace: [18260.255421] [] schedule+0x29/0x70 [18260.261577] [] futex_wait_queue_me+0xc4/0x120 [18260.268900] [] futex_wait+0x179/0x280 [18260.275440] [] do_futex+0xfe/0x5b0 [18260.281697] [] ? dm_ctl_ioctl+0x13/0x20 [dm_mod] [18260.289316] [] ? do_vfs_ioctl+0x2d5/0x4b0 [18260.296244] [] SyS_futex+0x80/0x180 [18260.302571] [] ? SyS_ioctl+0x8b/0xc0 [18260.309000] [] system_call_fastpath+0x16/0x1b [18260.316308] multipathd S 0000000000004000 0 919 1 0x00000000 [18260.324806] ffff88042b69fc50 0000000000000086 ffff88042e97de20 ffff88042b69ffd8 [18260.333729] ffff88042b69ffd8 ffff88042b69ffd8 ffff88042e97de20 ffff88042a624800 [18260.342655] 0000000000000000 ffff88042a624a10 ffff88042e97de20 0000000000004000 [18260.351553] Call Trace: [18260.354886] [] schedule+0x29/0x70 [18260.361036] [] dm_wait_event+0xa1/0xf0 [dm_mod] [18260.368535] [] ? strlcpy+0x42/0x60 [18260.374763] [] ? wake_up_atomic_t+0x30/0x30 [18260.381888] [] dev_wait+0x4d/0xd0 [dm_mod] [18260.388917] [] ? security_capable+0x18/0x20 [18260.396046] [] ? table_deps+0x1a0/0x1a0 [dm_mod] [18260.403672] [] ctl_ioctl+0x1e5/0x500 [dm_mod] [18260.410999] [] dm_ctl_ioctl+0x13/0x20 [dm_mod] [18260.418404] [] do_vfs_ioctl+0x2d5/0x4b0 [18260.425133] [] ? __set_task_blocked+0x41/0xa0 [18260.432438] [] SyS_ioctl+0xa1/0xc0 [18260.438679] [] system_call_fastpath+0x16/0x1b [18260.445991] multipathd S 0000000000004000 0 941 1 0x00000000 [18260.454494] ffff88042c37fc50 0000000000000086 ffff88042e97ce70 ffff88042c37ffd8 [18260.463420] ffff88042c37ffd8 ffff88042c37ffd8 ffff88042e97ce70 ffff88082bd28000 [18260.472343] 0000000000000000 ffff88082bd28210 ffff88042e97ce70 0000000000004000 [18260.481251] Call Trace: [18260.484582] [] schedule+0x29/0x70 [18260.490737] [] dm_wait_event+0xa1/0xf0 [dm_mod] [18260.498249] [] ? strlcpy+0x42/0x60 [18260.504489] [] ? wake_up_atomic_t+0x30/0x30 [18260.511605] [] dev_wait+0x4d/0xd0 [dm_mod] [18260.518633] [] ? security_capable+0x18/0x20 [18260.525763] [] ? table_deps+0x1a0/0x1a0 [dm_mod] [18260.533373] [] ctl_ioctl+0x1e5/0x500 [dm_mod] [18260.540695] [] dm_ctl_ioctl+0x13/0x20 [dm_mod] [18260.548111] [] do_vfs_ioctl+0x2d5/0x4b0 [18260.554842] [] ? __set_task_blocked+0x41/0xa0 [18260.562157] [] SyS_ioctl+0xa1/0xc0 [18260.568385] [] system_call_fastpath+0x16/0x1b [18260.575688] multipathd S 0000000000004000 0 950 1 0x00000000 [18260.584196] ffff880429d57c50 0000000000000086 ffff88042e978fb0 ffff880429d57fd8 [18260.593124] ffff880429d57fd8 ffff880429d57fd8 ffff88042e978fb0 ffff8800b4779000 [18260.602052] 0000000000000000 ffff8800b4779210 ffff88042e978fb0 0000000000004000 [18260.610968] Call Trace: [18260.614296] [] schedule+0x29/0x70 [18260.620451] [] dm_wait_event+0xa1/0xf0 [dm_mod] [18260.627971] [] ? strlcpy+0x42/0x60 [18260.634224] [] ? wake_up_atomic_t+0x30/0x30 [18260.641347] [] dev_wait+0x4d/0xd0 [dm_mod] [18260.648378] [] ? security_capable+0x18/0x20 [18260.655508] [] ? table_deps+0x1a0/0x1a0 [dm_mod] [18260.663119] [] ctl_ioctl+0x1e5/0x500 [dm_mod] [18260.670435] [] dm_ctl_ioctl+0x13/0x20 [dm_mod] [18260.677855] [] do_vfs_ioctl+0x2d5/0x4b0 [18260.684577] [] ? __set_task_blocked+0x41/0xa0 [18260.691881] [] SyS_ioctl+0xa1/0xc0 [18260.698120] [] system_call_fastpath+0x16/0x1b [18260.705418] multipathd S 0000000000004000 0 960 1 0x00000000 [18260.713928] ffff880036a93c50 0000000000000086 ffff88042e979f60 ffff880036a93fd8 [18260.722865] ffff880036a93fd8 ffff880036a93fd8 ffff88042e979f60 ffff8800b477e000 [18260.731800] 0000000000000000 ffff8800b477e210 ffff88042e979f60 0000000000004000 [18260.740729] Call Trace: [18260.744074] [] schedule+0x29/0x70 [18260.750227] [] dm_wait_event+0xa1/0xf0 [dm_mod] [18260.757739] [] ? strlcpy+0x42/0x60 [18260.764002] [] ? wake_up_atomic_t+0x30/0x30 [18260.771149] [] dev_wait+0x4d/0xd0 [dm_mod] [18260.778180] [] ? security_capable+0x18/0x20 [18260.785310] [] ? table_deps+0x1a0/0x1a0 [dm_mod] [18260.792930] [] ctl_ioctl+0x1e5/0x500 [dm_mod] [18260.800243] [] dm_ctl_ioctl+0x13/0x20 [dm_mod] [18260.807653] [] do_vfs_ioctl+0x2d5/0x4b0 [18260.814385] [] ? __set_task_blocked+0x41/0xa0 [18260.821696] [] SyS_ioctl+0xa1/0xc0 [18260.827935] [] system_call_fastpath+0x16/0x1b [18260.835232] multipathd S 0000000000004000 0 969 1 0x00000000 [18260.843738] ffff880429d2fc50 0000000000000086 ffff88042e978000 ffff880429d2ffd8 [18260.852645] ffff880429d2ffd8 ffff880429d2ffd8 ffff88042e978000 ffff880036a14800 [18260.861551] 0000000000000000 ffff880036a14a10 ffff88042e978000 0000000000004000 [18260.870468] Call Trace: [18260.873807] [] schedule+0x29/0x70 [18260.879969] [] dm_wait_event+0xa1/0xf0 [dm_mod] [18260.887480] [] ? strlcpy+0x42/0x60 [18260.893741] [] ? wake_up_atomic_t+0x30/0x30 [18260.900880] [] dev_wait+0x4d/0xd0 [dm_mod] [18260.907910] [] ? security_capable+0x18/0x20 [18260.915050] [] ? table_deps+0x1a0/0x1a0 [dm_mod] [18260.922685] [] ctl_ioctl+0x1e5/0x500 [dm_mod] [18260.929997] [] dm_ctl_ioctl+0x13/0x20 [dm_mod] [18260.937406] [] do_vfs_ioctl+0x2d5/0x4b0 [18260.944138] [] ? __set_task_blocked+0x41/0xa0 [18260.951431] [] SyS_ioctl+0xa1/0xc0 [18260.957678] [] system_call_fastpath+0x16/0x1b [18260.964987] kworker/9:1H S ffff88082d856840 0 830 2 0x00000000 [18260.973488] ffff880829727e50 0000000000000046 ffff88082a400fb0 ffff880829727fd8 [18260.982416] ffff880829727fd8 ffff880829727fd8 ffff88082a400fb0 ffff88082d856860 [18260.991351] ffff88042c69dab0 ffff88082a400fb0 ffff88042c69da80 ffff88082d856840 [18261.000271] Call Trace: [18261.003601] [] schedule+0x29/0x70 [18261.009746] [] worker_thread+0x1d9/0x410 [18261.016574] [] ? rescuer_thread+0x460/0x460 [18261.023696] [] kthread+0xcf/0xe0 [18261.029756] [] ? kthread_create_on_node+0x140/0x140 [18261.037675] [] ret_from_fork+0x58/0x90 [18261.044314] [] ? kthread_create_on_node+0x140/0x140 [18261.052217] kworker/24:1H S ffff88082da16840 0 831 2 0x00000000 [18261.060743] ffff88082906fe50 0000000000000046 ffff88082a401f60 ffff88082906ffd8 [18261.069678] ffff88082906ffd8 ffff88082906ffd8 ffff88082a401f60 ffff88082da16860 [18261.078598] ffff88042bc57830 ffff88082a401f60 ffff88042bc57800 ffff88082da16840 [18261.087530] Call Trace: [18261.090872] [] schedule+0x29/0x70 [18261.097026] [] worker_thread+0x1d9/0x410 [18261.103860] [] ? rescuer_thread+0x460/0x460 [18261.110991] [] kthread+0xcf/0xe0 [18261.117051] [] ? kthread_create_on_node+0x140/0x140 [18261.124947] [] ret_from_fork+0x58/0x90 [18261.131584] [] ? kthread_create_on_node+0x140/0x140 [18261.139483] kworker/10:1H S ffff88082d896840 0 881 2 0x00000000 [18261.148023] ffff88082c6bbe50 0000000000000046 ffff88082d610000 ffff88082c6bbfd8 [18261.156963] ffff88082c6bbfd8 ffff88082c6bbfd8 ffff88082d610000 ffff88082d896860 [18261.165913] ffff88042c6c6bb0 ffff88082d610000 ffff88042c6c6b80 ffff88082d896840 [18261.174846] Call Trace: [18261.178214] [] schedule+0x29/0x70 [18261.184378] [] worker_thread+0x1d9/0x410 [18261.191233] [] ? rescuer_thread+0x460/0x460 [18261.198368] [] kthread+0xcf/0xe0 [18261.204429] [] ? kthread_create_on_node+0x140/0x140 [18261.212331] [] ret_from_fork+0x58/0x90 [18261.218970] [] ? kthread_create_on_node+0x140/0x140 [18261.226874] kipmi0 S ffff880036b654c8 0 882 2 0x00000000 [18261.235379] ffff88042c72fe48 0000000000000046 ffff88042bd05e20 ffff88042c72ffd8 [18261.244321] ffff88042c72ffd8 ffff88042c72ffd8 ffff88042bd05e20 ffff880036b65400 [18261.253263] ffff880036b65424 0000000000000004 0000000000000246 ffff880036b654c8 [18261.262218] Call Trace: [18261.265582] [] schedule+0x29/0x70 [18261.271756] [] ipmi_thread+0x118/0x220 [ipmi_si] [18261.279397] [] ? poll+0x70/0x70 [ipmi_si] [18261.286354] [] kthread+0xcf/0xe0 [18261.292434] [] ? kthread_create_on_node+0x140/0x140 [18261.300358] [] ret_from_fork+0x58/0x90 [18261.307025] [] ? kthread_create_on_node+0x140/0x140 [18261.314950] edac-poller S ffff88042c6c6480 0 883 2 0x00000000 [18261.323486] ffff88042c733e08 0000000000000046 ffff88042bd04e70 ffff88042c733fd8 [18261.332453] ffff88042c733fd8 ffff88042c733fd8 ffff88042bd04e70 ffff88042bd04e70 [18261.341428] ffff88042c6c6480 ffff88082be38e80 ffff88042c6c64b0 ffff88042c6c6480 [18261.350378] Call Trace: [18261.353750] [] schedule+0x29/0x70 [18261.359934] [] rescuer_thread+0x359/0x460 [18261.366903] [] ? __schedule+0x3b0/0x990 [18261.373667] [] ? manage_workers.isra.24+0x2d0/0x2d0 [18261.381598] [] kthread+0xcf/0xe0 [18261.387681] [] ? kthread_create_on_node+0x140/0x140 [18261.395622] [] ret_from_fork+0x58/0x90 [18261.402284] [] ? kthread_create_on_node+0x140/0x140 [18261.410220] kworker/5:2 S ffff88042e156480 0 894 2 0x00000000 [18261.418777] ffff880429dbfe50 0000000000000046 ffff88042bd02f10 ffff880429dbffd8 [18261.427749] ffff880429dbffd8 ffff880429dbffd8 ffff88042bd02f10 ffff88042e1564a0 [18261.436721] ffff88042b6176b0 ffff88042bd02f10 ffff88042b617680 ffff88042e156480 [18261.445677] Call Trace: [18261.449066] [] schedule+0x29/0x70 [18261.455266] [] worker_thread+0x1d9/0x410 [18261.462140] [] ? rescuer_thread+0x460/0x460 [18261.469297] [] kthread+0xcf/0xe0 [18261.475381] [] ? kthread_create_on_node+0x140/0x140 [18261.483316] [] ret_from_fork+0x58/0x90 [18261.489985] [] ? kthread_create_on_node+0x140/0x140 [18261.497915] kdmflush S ffff88042c7d2c80 0 912 2 0x00000000 [18261.506448] ffff88042bb77e08 0000000000000046 ffff88042bd06dd0 ffff88042bb77fd8 [18261.515415] ffff88042bb77fd8 ffff88042bb77fd8 ffff88042bd06dd0 ffff88042bd06dd0 [18261.524375] ffff88042c7d2c80 ffff880036b66e80 ffff88042c7d2cb0 ffff88042c7d2c80 [18261.533332] Call Trace: [18261.536703] [] schedule+0x29/0x70 [18261.542892] [] rescuer_thread+0x359/0x460 [18261.549859] [] ? __schedule+0x3b0/0x990 [18261.556629] [] ? manage_workers.isra.24+0x2d0/0x2d0 [18261.564558] [] kthread+0xcf/0xe0 [18261.570642] [] ? kthread_create_on_node+0x140/0x140 [18261.578573] [] ret_from_fork+0x58/0x90 [18261.585236] [] ? kthread_create_on_node+0x140/0x140 [18261.593177] bioset S ffff88042c7d2b00 0 917 2 0x00000000 [18261.601740] ffff8804296f7e08 0000000000000046 ffff88042bacbec0 ffff8804296f7fd8 [18261.610696] ffff8804296f7fd8 ffff8804296f7fd8 ffff88042bacbec0 ffff88042bacbec0 [18261.619641] ffff88042c7d2b00 ffff880036b66080 ffff88042c7d2b30 ffff88042c7d2b00 [18261.628586] Call Trace: [18261.631964] [] schedule+0x29/0x70 [18261.638159] [] rescuer_thread+0x359/0x460 [18261.645119] [] ? __schedule+0x3b0/0x990 [18261.651889] [] ? manage_workers.isra.24+0x2d0/0x2d0 [18261.659832] [] kthread+0xcf/0xe0 [18261.665916] [] ? kthread_create_on_node+0x140/0x140 [18261.673839] [] ret_from_fork+0x58/0x90 [18261.680497] [] ? kthread_create_on_node+0x140/0x140 [18261.688415] kdmwork-253:0 S ffff88042bacedd0 0 918 2 0x00000000 [18261.696954] ffff88042a2a7e68 0000000000000046 ffff88042bacedd0 ffff88042a2a7fd8 [18261.705912] ffff88042a2a7fd8 ffff88042a2a7fd8 ffff88042bacedd0 ffff88042a624bc0 [18261.714872] ffff88042bacedd0 ffff88042bacedd0 0000000000000000 ffff88042bacedd0 [18261.723839] Call Trace: [18261.727226] [] schedule+0x29/0x70 [18261.733412] [] kthread_worker_fn+0x113/0x1d0 [18261.740670] [] ? kthread_stop+0xe0/0xe0 [18261.747429] [] kthread+0xcf/0xe0 [18261.753504] [] ? kthread_create_on_node+0x140/0x140 [18261.761426] [] ret_from_fork+0x58/0x90 [18261.768094] [] ? kthread_create_on_node+0x140/0x140 [18261.776022] kworker/17:2 S ffff88042e256480 0 933 2 0x00000000 [18261.784575] ffff88042a35be50 0000000000000046 ffff88042bf5bec0 ffff88042a35bfd8 [18261.793534] ffff88042a35bfd8 ffff88042a35bfd8 ffff88042bf5bec0 ffff88042e2564a0 [18261.802485] ffff88082ca283b0 ffff88042bf5bec0 ffff88082ca28380 ffff88042e256480 [18261.811447] Call Trace: [18261.814819] [] schedule+0x29/0x70 [18261.821017] [] worker_thread+0x1d9/0x410 [18261.827893] [] ? rescuer_thread+0x460/0x460 [18261.835049] [] kthread+0xcf/0xe0 [18261.841139] [] ? kthread_create_on_node+0x140/0x140 [18261.849070] [] ret_from_fork+0x58/0x90 [18261.855723] [] ? kthread_create_on_node+0x140/0x140 [18261.863651] kdmflush S ffff88042c7d2280 0 937 2 0x00000000 [18261.872217] ffff88041ab1fe08 0000000000000046 ffff88042bf5af10 ffff88041ab1ffd8 [18261.881184] ffff88041ab1ffd8 ffff88041ab1ffd8 ffff88042bf5af10 ffff88042bf5af10 [18261.890155] ffff88042c7d2280 ffff880036b63080 ffff88042c7d22b0 ffff88042c7d2280 [18261.899108] Call Trace: [18261.902482] [] schedule+0x29/0x70 [18261.908666] [] rescuer_thread+0x359/0x460 [18261.915623] [] ? __schedule+0x3b0/0x990 [18261.922389] [] ? manage_workers.isra.24+0x2d0/0x2d0 [18261.930313] [] kthread+0xcf/0xe0 [18261.936398] [] ? kthread_create_on_node+0x140/0x140 [18261.944322] [] ret_from_fork+0x58/0x90 [18261.950991] [] ? kthread_create_on_node+0x140/0x140 [18261.958910] bioset S ffff88042c7d2400 0 939 2 0x00000000 [18261.967462] ffff88041b353e08 0000000000000046 ffff88042bf5ce70 ffff88041b353fd8 [18261.976415] ffff88041b353fd8 ffff88041b353fd8 ffff88042bf5ce70 ffff88042bf5ce70 [18261.985364] ffff88042c7d2400 ffff880036b63480 ffff88042c7d2430 ffff88042c7d2400 [18261.994329] Call Trace: [18261.997694] [] schedule+0x29/0x70 [18262.003879] [] rescuer_thread+0x359/0x460 [18262.010851] [] ? __schedule+0x3b0/0x990 [18262.017618] [] ? manage_workers.isra.24+0x2d0/0x2d0 [18262.025550] [] kthread+0xcf/0xe0 [18262.031629] [] ? kthread_create_on_node+0x140/0x140 [18262.039547] [] ret_from_fork+0x58/0x90 [18262.046216] [] ? kthread_create_on_node+0x140/0x140 [18262.054142] kdmwork-253:1 S ffff88042bf5de20 0 940 2 0x00000000 [18262.062696] ffff880429767e68 0000000000000046 ffff88042bf5de20 ffff880429767fd8 [18262.071658] 00000000efd9ed30 ffff880429767fd8 ffff88042bf5de20 ffff88082bd283c0 [18262.080607] ffff88042bf5de20 ffff88042bf5de20 0000000000000000 ffff88042bf5de20 [18262.089557] Call Trace: [18262.092931] [] schedule+0x29/0x70 [18262.099141] [] kthread_worker_fn+0x113/0x1d0 [18262.106397] [] ? kthread_stop+0xe0/0xe0 [18262.113176] [] kthread+0xcf/0xe0 [18262.119283] [] ? kthread_create_on_node+0x140/0x140 [18262.127204] [] ret_from_fork+0x58/0x90 [18262.133875] [] ? kthread_create_on_node+0x140/0x140 [18262.141809] kworker/22:2 S ffff88042e396480 0 943 2 0x00000000 [18262.150357] ffff88041831be50 0000000000000046 ffff88042bf5edd0 ffff88041831bfd8 [18262.159324] ffff88041831bfd8 ffff88041831bfd8 ffff88042bf5edd0 ffff88042e3964a0 [18262.168297] ffff88042bc057b0 ffff88042bf5edd0 ffff88042bc05780 ffff88042e396480 [18262.177254] Call Trace: [18262.180630] [] schedule+0x29/0x70 [18262.186831] [] worker_thread+0x1d9/0x410 [18262.193700] [] ? rescuer_thread+0x460/0x460 [18262.200883] [] kthread+0xcf/0xe0 [18262.206983] [] ? kthread_create_on_node+0x140/0x140 [18262.214913] [] ret_from_fork+0x58/0x90 [18262.221565] [] ? kthread_create_on_node+0x140/0x140 [18262.229493] kdmflush S ffff88042c7d2100 0 944 2 0x00000000 [18262.238068] ffff88042b23be08 0000000000000046 ffff88042ae88000 ffff88042b23bfd8 [18262.247049] ffff88042b23bfd8 ffff88042b23bfd8 ffff88042ae88000 ffff88042ae88000 [18262.255998] ffff88042c7d2100 ffff880036b62a80 ffff88042c7d2130 ffff88042c7d2100 [18262.264973] Call Trace: [18262.268348] [] schedule+0x29/0x70 [18262.274539] [] rescuer_thread+0x359/0x460 [18262.281507] [] ? __schedule+0x3b0/0x990 [18262.288282] [] ? manage_workers.isra.24+0x2d0/0x2d0 [18262.296212] [] kthread+0xcf/0xe0 [18262.302297] [] ? kthread_create_on_node+0x140/0x140 [18262.310242] [] ret_from_fork+0x58/0x90 [18262.316915] [] ? kthread_create_on_node+0x140/0x140 [18262.324844] bioset S ffff88042c7d3b80 0 948 2 0x00000000 [18262.333393] ffff88042b2f7e08 0000000000000046 ffff88042ae88fb0 ffff88042b2f7fd8 [18262.342351] ffff88042b2f7fd8 ffff88042b2f7fd8 ffff88042ae88fb0 ffff88042ae88fb0 [18262.351320] ffff88042c7d3b80 ffff880036b65c80 ffff88042c7d3bb0 ffff88042c7d3b80 [18262.360284] Call Trace: [18262.363659] [] schedule+0x29/0x70 [18262.369858] [] rescuer_thread+0x359/0x460 [18262.376824] [] ? __schedule+0x3b0/0x990 [18262.383598] [] ? manage_workers.isra.24+0x2d0/0x2d0 [18262.391527] [] kthread+0xcf/0xe0 [18262.397606] [] ? kthread_create_on_node+0x140/0x140 [18262.405526] [] ret_from_fork+0x58/0x90 [18262.412189] [] ? kthread_create_on_node+0x140/0x140 [18262.420123] kdmwork-253:2 S ffff88042ae89f60 0 949 2 0x00000000 [18262.428667] ffff88042a46fe68 0000000000000046 ffff88042ae89f60 ffff88042a46ffd8 [18262.437619] ffff88042a46ffd8 ffff88042a46ffd8 ffff88042ae89f60 ffff8800b47793c0 [18262.446570] ffff88042ae89f60 ffff88042ae89f60 0000000000000000 ffff88042ae89f60 [18262.455512] Call Trace: [18262.458899] [] schedule+0x29/0x70 [18262.465101] [] kthread_worker_fn+0x113/0x1d0 [18262.472362] [] ? kthread_stop+0xe0/0xe0 [18262.479139] [] kthread+0xcf/0xe0 [18262.485223] [] ? kthread_create_on_node+0x140/0x140 [18262.493155] [] ret_from_fork+0x58/0x90 [18262.499818] [] ? kthread_create_on_node+0x140/0x140 [18262.507745] kworker/19:2 S ffff88042e2d6480 0 951 2 0x00000000 [18262.516286] ffff88042a477e50 0000000000000046 ffff88042ae8af10 ffff88042a477fd8 [18262.525250] ffff88042a477fd8 ffff88042a477fd8 ffff88042ae8af10 ffff88042e2d64a0 [18262.534197] ffff88042f03bbb0 ffff88042ae8af10 ffff88042f03bb80 ffff88042e2d6480 [18262.543174] Call Trace: [18262.546549] [] schedule+0x29/0x70 [18262.552737] [] worker_thread+0x1d9/0x410 [18262.559603] [] ? rescuer_thread+0x460/0x460 [18262.566757] [] kthread+0xcf/0xe0 [18262.572843] [] ? kthread_create_on_node+0x140/0x140 [18262.580778] [] ret_from_fork+0x58/0x90 [18262.587435] [] ? kthread_create_on_node+0x140/0x140 [18262.595365] kdmflush S ffff88042c7d3d00 0 956 2 0x00000000 [18262.603911] ffff880036a7be08 0000000000000046 ffff88042ae8bec0 ffff880036a7bfd8 [18262.612875] ffff880036a7bfd8 ffff880036a7bfd8 ffff88042ae8bec0 ffff88042ae8bec0 [18262.621833] ffff88042c7d3d00 ffff880036b60080 ffff88042c7d3d30 ffff88042c7d3d00 [18262.630795] Call Trace: [18262.634164] [] schedule+0x29/0x70 [18262.640348] [] rescuer_thread+0x359/0x460 [18262.647312] [] ? __schedule+0x3b0/0x990 [18262.654081] [] ? manage_workers.isra.24+0x2d0/0x2d0 [18262.662022] [] kthread+0xcf/0xe0 [18262.668111] [] ? kthread_create_on_node+0x140/0x140 [18262.676042] [] ret_from_fork+0x58/0x90 [18262.682699] [] ? kthread_create_on_node+0x140/0x140 [18262.690619] bioset S ffff88042c7d3080 0 958 2 0x00000000 [18262.699175] ffff880036103e08 0000000000000046 ffff88042ae8ce70 ffff880036103fd8 [18262.708146] ffff880036103fd8 ffff880036103fd8 ffff88042ae8ce70 ffff88042ae8ce70 [18262.717108] ffff88042c7d3080 ffff880036b67a80 ffff88042c7d30b0 ffff88042c7d3080 [18262.726071] Call Trace: [18262.729448] [] schedule+0x29/0x70 [18262.735635] [] rescuer_thread+0x359/0x460 [18262.742594] [] ? __schedule+0x3b0/0x990 [18262.749364] [] ? manage_workers.isra.24+0x2d0/0x2d0 [18262.757309] [] kthread+0xcf/0xe0 [18262.763392] [] ? kthread_create_on_node+0x140/0x140 [18262.771313] [] ret_from_fork+0x58/0x90 [18262.777967] [] ? kthread_create_on_node+0x140/0x140 [18262.785895] kdmwork-253:3 S ffff88042ae8de20 0 959 2 0x00000000 [18262.794432] ffff880036b4be68 0000000000000046 ffff88042ae8de20 ffff880036b4bfd8 [18262.803390] ffff880036b4bfd8 ffff880036b4bfd8 ffff88042ae8de20 ffff8800b477e3c0 [18262.812340] ffff88042ae8de20 ffff88042ae8de20 0000000000000000 ffff88042ae8de20 [18262.821291] Call Trace: [18262.824654] [] schedule+0x29/0x70 [18262.830845] [] kthread_worker_fn+0x113/0x1d0 [18262.838114] [] ? kthread_stop+0xe0/0xe0 [18262.844881] [] kthread+0xcf/0xe0 [18262.850974] [] ? kthread_create_on_node+0x140/0x140 [18262.858908] [] ret_from_fork+0x58/0x90 [18262.865557] [] ? kthread_create_on_node+0x140/0x140 [18262.873478] kdmflush S ffff88042c7d2700 0 964 2 0x00000000 [18262.882028] ffff8804182f3e08 0000000000000046 ffff88042ae8edd0 ffff8804182f3fd8 [18262.890999] ffff8804182f3fd8 ffff8804182f3fd8 ffff88042ae8edd0 ffff88042ae8edd0 [18262.899973] ffff88042c7d2700 ffff880036b67e80 ffff88042c7d2730 ffff88042c7d2700 [18262.908947] Call Trace: [18262.912317] [] schedule+0x29/0x70 [18262.918508] [] rescuer_thread+0x359/0x460 [18262.925473] [] ? __schedule+0x3b0/0x990 [18262.932257] [] ? manage_workers.isra.24+0x2d0/0x2d0 [18262.940198] [] kthread+0xcf/0xe0 [18262.946297] [] ? kthread_create_on_node+0x140/0x140 [18262.954227] [] ret_from_fork+0x58/0x90 [18262.960893] [] ? kthread_create_on_node+0x140/0x140 [18262.968821] bioset S ffff88042c7d2880 0 967 2 0x00000000 [18262.977363] ffff88042a49be08 0000000000000046 ffff880419670000 ffff88042a49bfd8 [18262.986319] ffff88042a49bfd8 ffff88042a49bfd8 ffff880419670000 ffff880419670000 [18262.995292] ffff88042c7d2880 ffff880036b67480 ffff88042c7d28b0 ffff88042c7d2880 [18263.004263] Call Trace: [18263.007640] [] schedule+0x29/0x70 [18263.013842] [] rescuer_thread+0x359/0x460 [18263.020812] [] ? __schedule+0x3b0/0x990 [18263.027574] [] ? manage_workers.isra.24+0x2d0/0x2d0 [18263.035503] [] kthread+0xcf/0xe0 [18263.041587] [] ? kthread_create_on_node+0x140/0x140 [18263.049510] [] ret_from_fork+0x58/0x90 [18263.056176] [] ? kthread_create_on_node+0x140/0x140 [18263.064101] kdmwork-253:4 S ffff880419670fb0 0 968 2 0x00000000 [18263.072634] ffff88042a4a3e68 0000000000000046 ffff880419670fb0 ffff88042a4a3fd8 [18263.081596] ffff88042a4a3fd8 ffff88042a4a3fd8 ffff880419670fb0 ffff880036a14bc0 [18263.090560] ffff880419670fb0 ffff880419670fb0 0000000000000000 ffff880419670fb0 [18263.099519] Call Trace: [18263.102906] [] schedule+0x29/0x70 [18263.109104] [] kthread_worker_fn+0x113/0x1d0 [18263.116368] [] ? kthread_stop+0xe0/0xe0 [18263.123154] [] kthread+0xcf/0xe0 [18263.129261] [] ? kthread_create_on_node+0x140/0x140 [18263.137203] [] ret_from_fork+0x58/0x90 [18263.143861] [] ? kthread_create_on_node+0x140/0x140 [18263.151789] kworker/18:2 S ffff88042e296480 0 975 2 0x00000000 [18263.160353] ffff8804182a3e50 0000000000000046 ffff880419671f60 ffff8804182a3fd8 [18263.169304] ffff8804182a3fd8 ffff8804182a3fd8 ffff880419671f60 ffff88042e2964a0 [18263.178270] ffff88082ca4a130 ffff880419671f60 ffff88082ca4a100 ffff88042e296480 [18263.187239] Call Trace: [18263.190613] [] schedule+0x29/0x70 [18263.196814] [] worker_thread+0x1d9/0x410 [18263.203684] [] ? rescuer_thread+0x460/0x460 [18263.210844] [] kthread+0xcf/0xe0 [18263.216933] [] ? kthread_create_on_node+0x140/0x140 [18263.224870] [] ret_from_fork+0x58/0x90 [18263.231526] [] ? kthread_create_on_node+0x140/0x140 [18263.239458] kworker/5:1H S ffff88042e156840 0 998 2 0x00000000 [18263.248024] ffff8800b3ce7e50 0000000000000046 ffff8800b3cf0000 ffff8800b3ce7fd8 [18263.256999] ffff8800b3ce7fd8 ffff8800b3ce7fd8 ffff8800b3cf0000 ffff88042e156860 [18263.265964] ffff88042b617d30 ffff8800b3cf0000 ffff88042b617d00 ffff88042e156840 [18263.274927] Call Trace: [18263.278310] [] schedule+0x29/0x70 [18263.284490] [] worker_thread+0x1d9/0x410 [18263.291347] [] ? rescuer_thread+0x460/0x460 [18263.298501] [] kthread+0xcf/0xe0 [18263.304584] [] ? kthread_create_on_node+0x140/0x140 [18263.312513] [] ret_from_fork+0x58/0x90 [18263.319189] [] ? kthread_create_on_node+0x140/0x140 [18263.327121] kworker/6:1H S ffff88042e196840 0 1010 2 0x00000000 [18263.335665] ffff88042a217e50 0000000000000046 ffff8804182f9f60 ffff88042a217fd8 [18263.344607] ffff88042a217fd8 ffff88042a217fd8 ffff8804182f9f60 ffff88042e196860 [18263.353545] ffff88042a6d8a30 ffff8804182f9f60 ffff88042a6d8a00 ffff88042e196840 [18263.362506] Call Trace: [18263.365876] [] schedule+0x29/0x70 [18263.372063] [] worker_thread+0x1d9/0x410 [18263.378934] [] ? rescuer_thread+0x460/0x460 [18263.386082] [] kthread+0xcf/0xe0 [18263.392170] [] ? kthread_create_on_node+0x140/0x140 [18263.400095] [] ret_from_fork+0x58/0x90 [18263.406763] [] ? kthread_create_on_node+0x140/0x140 [18263.414677] kworker/4:1H S ffff88042e116840 0 1011 2 0x00000000 [18263.423229] ffff880418247e50 0000000000000046 ffff88042b05bec0 ffff880418247fd8 [18263.432183] ffff880418247fd8 ffff880418247fd8 ffff88042b05bec0 ffff88042e116860 [18263.441151] ffff880417e57030 ffff88042b05bec0 ffff880417e57000 ffff88042e116840 [18263.450118] Call Trace: [18263.453477] [] schedule+0x29/0x70 [18263.459651] [] worker_thread+0x1d9/0x410 [18263.466503] [] ? rescuer_thread+0x460/0x460 [18263.473647] [] kthread+0xcf/0xe0 [18263.479718] [] ? kthread_create_on_node+0x140/0x140 [18263.487634] [] ret_from_fork+0x58/0x90 [18263.494296] [] ? kthread_create_on_node+0x140/0x140 [18263.502222] kworker/17:1H S ffff88042e256840 0 1014 2 0x00000000 [18263.510770] ffff880417f43e50 0000000000000046 ffff88042b05de20 ffff880417f43fd8 [18263.519735] ffff880417f43fd8 ffff880417f43fd8 ffff88042b05de20 ffff88042e256860 [18263.528677] ffff88082ca29330 ffff88042b05de20 ffff88082ca29300 ffff88042e256840 [18263.537609] Call Trace: [18263.540965] [] schedule+0x29/0x70 [18263.547152] [] worker_thread+0x1d9/0x410 [18263.554007] [] ? rescuer_thread+0x460/0x460 [18263.561162] [] kthread+0xcf/0xe0 [18263.567242] [] ? kthread_create_on_node+0x140/0x140 [18263.575173] [] ret_from_fork+0x58/0x90 [18263.581832] [] ? kthread_create_on_node+0x140/0x140 [18263.589757] kworker/7:1H S ffff88042e1d6840 0 1015 2 0x00000000 [18263.598305] ffff88042bf87e50 0000000000000046 ffff88042b05edd0 ffff88042bf87fd8 [18263.607266] ffff88042bf87fd8 ffff88042bf87fd8 ffff88042b05edd0 ffff88042e1d6860 [18263.616228] ffff88042c307fb0 ffff88042b05edd0 ffff88042c307f80 ffff88042e1d6840 [18263.625186] Call Trace: [18263.628544] [] schedule+0x29/0x70 [18263.634719] [] worker_thread+0x1d9/0x410 [18263.641572] [] ? rescuer_thread+0x460/0x460 [18263.648723] [] kthread+0xcf/0xe0 [18263.654799] [] ? kthread_create_on_node+0x140/0x140 [18263.662723] [] ret_from_fork+0x58/0x90 [18263.669387] [] ? kthread_create_on_node+0x140/0x140 [18263.677301] kdmflush S ffff88042bd41600 0 1035 2 0x00000000 [18263.685833] ffff88082d207e08 0000000000000046 ffff880829d66dd0 ffff88082d207fd8 [18263.694782] ffff88082d207fd8 ffff88082d207fd8 ffff880829d66dd0 ffff880829d66dd0 [18263.703740] ffff88042bd41600 ffff88042bd77680 ffff88042bd41630 ffff88042bd41600 [18263.712671] Call Trace: [18263.716035] [] schedule+0x29/0x70 [18263.722220] [] rescuer_thread+0x359/0x460 [18263.729182] [] ? __schedule+0x3b0/0x990 [18263.735944] [] ? manage_workers.isra.24+0x2d0/0x2d0 [18263.743876] [] kthread+0xcf/0xe0 [18263.749965] [] ? kthread_create_on_node+0x140/0x140 [18263.757891] [] ret_from_fork+0x58/0x90 [18263.764550] [] ? kthread_create_on_node+0x140/0x140 [18263.772468] kworker/16:1H S ffff88042e216840 0 1043 2 0x00000000 [18263.781008] ffff88041ac13e50 0000000000000046 ffff88042af60000 ffff88041ac13fd8 [18263.789977] ffff88041ac13fd8 ffff88041ac13fd8 ffff88042af60000 ffff88042e216860 [18263.798942] ffff88042c790530 ffff88042af60000 ffff88042c790500 ffff88042e216840 [18263.807894] Call Trace: [18263.811255] [] schedule+0x29/0x70 [18263.817436] [] worker_thread+0x1d9/0x410 [18263.824303] [] ? rescuer_thread+0x460/0x460 [18263.831460] [] kthread+0xcf/0xe0 [18263.837545] [] ? kthread_create_on_node+0x140/0x140 [18263.845478] [] ret_from_fork+0x58/0x90 [18263.852135] [] ? kthread_create_on_node+0x140/0x140 [18263.860057] kdmflush S ffff88042bc9f100 0 1050 2 0x00000000 [18263.868588] ffff88082d7cfe08 0000000000000046 ffff88042af60fb0 ffff88082d7cffd8 [18263.877537] ffff88082d7cffd8 ffff88082d7cffd8 ffff88042af60fb0 ffff88042af60fb0 [18263.886493] ffff88042bc9f100 ffff88083fa02280 ffff88042bc9f130 ffff88042bc9f100 [18263.895429] Call Trace: [18263.898804] [] schedule+0x29/0x70 [18263.904992] [] rescuer_thread+0x359/0x460 [18263.911952] [] ? __schedule+0x3b0/0x990 [18263.918727] [] ? manage_workers.isra.24+0x2d0/0x2d0 [18263.926649] [] kthread+0xcf/0xe0 [18263.932732] [] ? kthread_create_on_node+0x140/0x140 [18263.940652] [] ret_from_fork+0x58/0x90 [18263.947311] [] ? kthread_create_on_node+0x140/0x140 [18263.955230] bioset S ffff88042bc9e780 0 1052 2 0x00000000 [18263.963786] ffff88082d7dfe08 0000000000000046 ffff88042af61f60 ffff88082d7dffd8 [18263.972735] ffff88082d7dffd8 ffff88082d7dffd8 ffff88042af61f60 ffff88042af61f60 [18263.981669] ffff88042bc9e780 ffff88083fa02680 ffff88042bc9e7b0 ffff88042bc9e780 [18263.990619] Call Trace: [18263.993982] [] schedule+0x29/0x70 [18264.000170] [] rescuer_thread+0x359/0x460 [18264.007138] [] ? __schedule+0x3b0/0x990 [18264.013901] [] ? manage_workers.isra.24+0x2d0/0x2d0 [18264.021837] [] kthread+0xcf/0xe0 [18264.027917] [] ? kthread_create_on_node+0x140/0x140 [18264.035845] [] ret_from_fork+0x58/0x90 [18264.042508] [] ? kthread_create_on_node+0x140/0x140 [18264.050434] kdmflush S ffff88042bc9e700 0 1060 2 0x00000000 [18264.058992] ffff88082d7bbe08 0000000000000046 ffff88042af63ec0 ffff88082d7bbfd8 [18264.067958] ffff88082d7bbfd8 ffff88082d7bbfd8 ffff88042af63ec0 ffff88042af63ec0 [18264.076905] ffff88042bc9e700 ffff88083fa05280 ffff88042bc9e730 ffff88042bc9e700 [18264.085873] Call Trace: [18264.089249] [] schedule+0x29/0x70 [18264.095439] [] rescuer_thread+0x359/0x460 [18264.102397] [] ? __schedule+0x3b0/0x990 [18264.109166] [] ? manage_workers.isra.24+0x2d0/0x2d0 [18264.117115] [] kthread+0xcf/0xe0 [18264.123207] [] ? kthread_create_on_node+0x140/0x140 [18264.131142] [] ret_from_fork+0x58/0x90 [18264.137802] [] ? kthread_create_on_node+0x140/0x140 [18264.145724] bioset S ffff88042bc9f900 0 1062 2 0x00000000 [18264.154277] ffff880823efbe08 0000000000000046 ffff88042af64e70 ffff880823efbfd8 [18264.163263] ffff880823efbfd8 ffff880823efbfd8 ffff88042af64e70 ffff88042af64e70 [18264.172227] ffff88042bc9f900 ffff88083fa05080 ffff88042bc9f930 ffff88042bc9f900 [18264.181194] Call Trace: [18264.184560] [] schedule+0x29/0x70 [18264.190748] [] rescuer_thread+0x359/0x460 [18264.197709] [] ? __schedule+0x3b0/0x990 [18264.204472] [] ? manage_workers.isra.24+0x2d0/0x2d0 [18264.212398] [] kthread+0xcf/0xe0 [18264.218475] [] ? kthread_create_on_node+0x140/0x140 [18264.226406] [] ret_from_fork+0x58/0x90 [18264.233081] [] ? kthread_create_on_node+0x140/0x140 [18264.241008] kdmflush S ffff88042f773080 0 1071 2 0x00000000 [18264.249557] ffff88082a557e08 0000000000000046 ffff88042af66dd0 ffff88082a557fd8 [18264.258514] ffff88082a557fd8 ffff88082a557fd8 ffff88042af66dd0 ffff88042af66dd0 [18264.267483] ffff88042f773080 ffff88042f9f4080 ffff88042f7730b0 ffff88042f773080 [18264.276444] Call Trace: [18264.279818] [] schedule+0x29/0x70 [18264.286018] [] rescuer_thread+0x359/0x460 [18264.292988] [] ? __schedule+0x3b0/0x990 [18264.299768] [] ? manage_workers.isra.24+0x2d0/0x2d0 [18264.307690] [] kthread+0xcf/0xe0 [18264.313769] [] ? kthread_create_on_node+0x140/0x140 [18264.321687] [] ret_from_fork+0x58/0x90 [18264.328358] [] ? kthread_create_on_node+0x140/0x140 [18264.336288] bioset S ffff88042f772d80 0 1073 2 0x00000000 [18264.344836] ffff88082c6f7e08 0000000000000046 ffff88042af65e20 ffff88082c6f7fd8 [18264.353801] ffff88082c6f7fd8 ffff88082c6f7fd8 ffff88042af65e20 ffff88042af65e20 [18264.362776] ffff88042f772d80 ffff88042f9f3a80 ffff88042f772db0 ffff88042f772d80 [18264.371741] Call Trace: [18264.375117] [] schedule+0x29/0x70 [18264.381307] [] rescuer_thread+0x359/0x460 [18264.388283] [] ? __schedule+0x3b0/0x990 [18264.395053] [] ? manage_workers.isra.24+0x2d0/0x2d0 [18264.402986] [] kthread+0xcf/0xe0 [18264.409070] [] ? kthread_create_on_node+0x140/0x140 [18264.417004] [] ret_from_fork+0x58/0x90 [18264.423670] [] ? kthread_create_on_node+0x140/0x140 [18264.431585] kworker/u483:0 S ffff88082a749800 0 1114 2 0x00000000 [18264.440139] ffff88082a3a3e50 0000000000000046 ffff88082bcb0000 ffff88082a3a3fd8 [18264.449122] ffff88082a3a3fd8 ffff88082a3a3fd8 ffff88082bcb0000 ffff88082a749820 [18264.458087] ffff88082c884e30 ffff88082bcb0000 ffff88082c884e00 ffff88082a749800 [18264.467063] Call Trace: [18264.470441] [] schedule+0x29/0x70 [18264.476630] [] worker_thread+0x1d9/0x410 [18264.483494] [] ? rescuer_thread+0x460/0x460 [18264.490650] [] kthread+0xcf/0xe0 [18264.496729] [] ? kthread_create_on_node+0x140/0x140 [18264.504641] [] ret_from_fork+0x58/0x90 [18264.511295] [] ? kthread_create_on_node+0x140/0x140 [18264.519237] kworker/u484:0 S ffff88082a749000 0 1115 2 0x00000000 [18264.527779] ffff880036a4be50 0000000000000046 ffff8800b3cbedd0 ffff880036a4bfd8 [18264.536737] ffff880036a4bfd8 ffff880036a4bfd8 ffff8800b3cbedd0 ffff88082a749020 [18264.545690] ffff88082c8847b0 ffff8800b3cbedd0 ffff88082c884780 ffff88082a749000 [18264.554635] Call Trace: [18264.558005] [] schedule+0x29/0x70 [18264.564192] [] worker_thread+0x1d9/0x410 [18264.571064] [] ? rescuer_thread+0x460/0x460 [18264.578210] [] kthread+0xcf/0xe0 [18264.584292] [] ? kthread_create_on_node+0x140/0x140 [18264.592224] [] ret_from_fork+0x58/0x90 [18264.598876] [] ? kthread_create_on_node+0x140/0x140 [18264.606788] kworker/u485:0 S ffff88082a748800 0 1116 2 0x00000000 [18264.615350] ffff88083fb43e50 0000000000000046 ffff88082d612f10 ffff88083fb43fd8 [18264.624301] ffff88083fb43fd8 ffff88083fb43fd8 ffff88082d612f10 ffff88082a748820 [18264.633265] ffff88082c885030 ffff88082d612f10 ffff88082c885000 ffff88082a748800 [18264.642229] Call Trace: [18264.645592] [] schedule+0x29/0x70 [18264.651770] [] worker_thread+0x1d9/0x410 [18264.658631] [] ? rescuer_thread+0x460/0x460 [18264.665790] [] kthread+0xcf/0xe0 [18264.671879] [] ? kthread_create_on_node+0x140/0x140 [18264.679809] [] ret_from_fork+0x58/0x90 [18264.686463] [] ? kthread_create_on_node+0x140/0x140 [18264.694372] ib-comp-wq S ffff88082c884f80 0 1117 2 0x00000000 [18264.702910] ffff880824b77e08 0000000000000046 ffff88082d616dd0 ffff880824b77fd8 [18264.711867] ffff880824b77fd8 ffff880824b77fd8 ffff88082d616dd0 ffff88082d616dd0 [18264.720822] ffff88082c884f80 ffff88082d52ba80 ffff88082c884fb0 ffff88082c884f80 [18264.729767] Call Trace: [18264.733126] [] schedule+0x29/0x70 [18264.739320] [] rescuer_thread+0x359/0x460 [18264.746278] [] ? __schedule+0x3b0/0x990 [18264.753047] [] ? manage_workers.isra.24+0x2d0/0x2d0 [18264.760972] [] kthread+0xcf/0xe0 [18264.767051] [] ? kthread_create_on_node+0x140/0x140 [18264.774974] [] ret_from_fork+0x58/0x90 [18264.781629] [] ? kthread_create_on_node+0x140/0x140 [18264.789539] ib_addr S ffff88082c884800 0 1118 2 0x00000000 [18264.798091] ffff88082aaf7e08 0000000000000046 ffff88082d615e20 ffff88082aaf7fd8 [18264.807060] ffff88082aaf7fd8 ffff88082aaf7fd8 ffff88082d615e20 ffff88082d615e20 [18264.816001] ffff88082c884800 ffff88082d52c880 ffff88082c884830 ffff88082c884800 [18264.824959] Call Trace: [18264.828320] [] schedule+0x29/0x70 [18264.834502] [] rescuer_thread+0x359/0x460 [18264.841462] [] ? __schedule+0x3b0/0x990 [18264.848227] [] ? manage_workers.isra.24+0x2d0/0x2d0 [18264.856153] [] kthread+0xcf/0xe0 [18264.862244] [] ? kthread_create_on_node+0x140/0x140 [18264.870165] [] ret_from_fork+0x58/0x90 [18264.876827] [] ? kthread_create_on_node+0x140/0x140 [18264.884746] ib_mcast S ffff88082c884c00 0 1119 2 0x00000000 [18264.893283] ffff88081f707e08 0000000000000046 ffff88082d614e70 ffff88081f707fd8 [18264.902247] ffff88081f707fd8 ffff88081f707fd8 ffff88082d614e70 ffff88082d614e70 [18264.911214] ffff88082c884c00 ffff88082d52b680 ffff88082c884c30 ffff88082c884c00 [18264.920184] Call Trace: [18264.923557] [] schedule+0x29/0x70 [18264.929743] [] rescuer_thread+0x359/0x460 [18264.936706] [] ? __schedule+0x3b0/0x990 [18264.943472] [] ? manage_workers.isra.24+0x2d0/0x2d0 [18264.951399] [] kthread+0xcf/0xe0 [18264.957472] [] ? kthread_create_on_node+0x140/0x140 [18264.965394] [] ret_from_fork+0x58/0x90 [18264.972047] [] ? kthread_create_on_node+0x140/0x140 [18264.979981] ib_nl_sa_wq S ffff88082c885680 0 1120 2 0x00000000 [18264.988509] ffff8808297ebe08 0000000000000046 ffff88082d611f60 ffff8808297ebfd8 [18264.997465] ffff8808297ebfd8 ffff8808297ebfd8 ffff88082d611f60 ffff88082d611f60 [18265.006432] ffff88082c885680 ffff88082d52ca80 ffff88082c8856b0 ffff88082c885680 [18265.015393] Call Trace: [18265.018770] [] schedule+0x29/0x70 [18265.024963] [] rescuer_thread+0x359/0x460 [18265.031936] [] ? __schedule+0x3b0/0x990 [18265.038705] [] ? manage_workers.isra.24+0x2d0/0x2d0 [18265.046634] [] kthread+0xcf/0xe0 [18265.052710] [] ? kthread_create_on_node+0x140/0x140 [18265.060634] [] ret_from_fork+0x58/0x90 [18265.067302] [] ? kthread_create_on_node+0x140/0x140 [18265.075231] mlx4_ib S ffff88082c884900 0 1121 2 0x00000000 [18265.083762] ffff880824377e08 0000000000000046 ffff88082d613ec0 ffff880824377fd8 [18265.092723] ffff880824377fd8 ffff880824377fd8 ffff88082d613ec0 ffff88082d613ec0 [18265.101664] ffff88082c884900 ffff88082d52b480 ffff88082c884930 ffff88082c884900 [18265.110617] Call Trace: [18265.113993] [] schedule+0x29/0x70 [18265.120198] [] rescuer_thread+0x359/0x460 [18265.127167] [] ? __schedule+0x3b0/0x990 [18265.133935] [] ? manage_workers.isra.24+0x2d0/0x2d0 [18265.141880] [] kthread+0xcf/0xe0 [18265.147958] [] ? kthread_create_on_node+0x140/0x140 [18265.155882] [] ret_from_fork+0x58/0x90 [18265.162549] [] ? kthread_create_on_node+0x140/0x140 [18265.170471] mlx4_ib_mcg S ffff88082c885580 0 1122 2 0x00000000 [18265.179042] ffff880824b6fe08 0000000000000046 ffff88082d610fb0 ffff880824b6ffd8 [18265.188013] ffff880824b6ffd8 ffff880824b6ffd8 ffff88082d610fb0 ffff88082d610fb0 [18265.196975] ffff88082c885580 ffff88082d52ce80 ffff88082c8855b0 ffff88082c885580 [18265.205947] Call Trace: [18265.209333] [] schedule+0x29/0x70 [18265.215516] [] rescuer_thread+0x359/0x460 [18265.222480] [] ? __schedule+0x3b0/0x990 [18265.229250] [] ? manage_workers.isra.24+0x2d0/0x2d0 [18265.237178] [] kthread+0xcf/0xe0 [18265.243254] [] ? kthread_create_on_node+0x140/0x140 [18265.251177] [] ret_from_fork+0x58/0x90 [18265.257853] [] ? kthread_create_on_node+0x140/0x140 [18265.265782] ib_mad1 S ffff880179222080 0 1125 2 0x00000000 [18265.274337] ffff88082bfefe08 0000000000000046 ffff880829d63ec0 ffff88082bfeffd8 [18265.283320] ffff88082bfeffd8 ffff88082bfeffd8 ffff880829d63ec0 ffff880829d63ec0 [18265.292292] ffff880179222080 ffff8800368d1280 ffff8801792220b0 ffff880179222080 [18265.301273] Call Trace: [18265.304656] [] schedule+0x29/0x70 [18265.310848] [] rescuer_thread+0x359/0x460 [18265.317819] [] ? __schedule+0x3b0/0x990 [18265.324595] [] ? manage_workers.isra.24+0x2d0/0x2d0 [18265.332533] [] kthread+0xcf/0xe0 [18265.338602] [] ? kthread_create_on_node+0x140/0x140 [18265.346535] [] ret_from_fork+0x58/0x90 [18265.353211] [] ? kthread_create_on_node+0x140/0x140 [18265.361142] ib_mad2 S ffff880179223f80 0 1126 2 0x00000000 [18265.369672] ffff8804b935fe08 0000000000000046 ffff880829d60000 ffff8804b935ffd8 [18265.378624] ffff8804b935ffd8 ffff8804b935ffd8 ffff880829d60000 ffff880829d60000 [18265.387594] ffff880179223f80 ffff8800b38b5c80 ffff880179223fb0 ffff880179223f80 [18265.396560] Call Trace: [18265.399930] [] schedule+0x29/0x70 [18265.406126] [] rescuer_thread+0x359/0x460 [18265.413096] [] ? __schedule+0x3b0/0x990 [18265.419872] [] ? manage_workers.isra.24+0x2d0/0x2d0 [18265.427810] [] kthread+0xcf/0xe0 [18265.433896] [] ? kthread_create_on_node+0x140/0x140 [18265.441832] [] ret_from_fork+0x58/0x90 [18265.448492] [] ? kthread_create_on_node+0x140/0x140 [18265.456417] iw_cm_wq S ffff88082bdbc900 0 1152 2 0x00000000 [18265.464964] ffff88082e937e08 0000000000000046 ffff880829d61f60 ffff88082e937fd8 [18265.473931] ffff88082e937fd8 ffff88082e937fd8 ffff880829d61f60 ffff880829d61f60 [18265.482884] ffff88082bdbc900 ffff88082d52ae80 ffff88082bdbc930 ffff88082bdbc900 [18265.491856] Call Trace: [18265.495240] [] schedule+0x29/0x70 [18265.501430] [] rescuer_thread+0x359/0x460 [18265.508403] [] ? __schedule+0x3b0/0x990 [18265.515181] [] ? manage_workers.isra.24+0x2d0/0x2d0 [18265.523122] [] kthread+0xcf/0xe0 [18265.529205] [] ? kthread_create_on_node+0x140/0x140 [18265.537140] [] ret_from_fork+0x58/0x90 [18265.543816] [] ? kthread_create_on_node+0x140/0x140 [18265.551746] ib_cm S ffff88082bdbcb00 0 1157 2 0x00000000 [18265.560295] ffff88083fefbe08 0000000000000046 ffff880829d60fb0 ffff88083fefbfd8 [18265.569272] ffff88083fefbfd8 ffff88083fefbfd8 ffff880829d60fb0 ffff880829d60fb0 [18265.578232] ffff88082bdbcb00 ffff88082d52aa80 ffff88082bdbcb30 ffff88082bdbcb00 [18265.587201] Call Trace: [18265.590583] [] schedule+0x29/0x70 [18265.596770] [] rescuer_thread+0x359/0x460 [18265.603736] [] ? __schedule+0x3b0/0x990 [18265.610506] [] ? manage_workers.isra.24+0x2d0/0x2d0 [18265.618437] [] kthread+0xcf/0xe0 [18265.624514] [] ? kthread_create_on_node+0x140/0x140 [18265.632440] [] ret_from_fork+0x58/0x90 [18265.639114] [] ? kthread_create_on_node+0x140/0x140 [18265.647042] rdma_cm S ffff88082bdbcd00 0 1162 2 0x00000000 [18265.655577] ffff88082d22fe08 0000000000000046 ffff880829d62f10 ffff88082d22ffd8 [18265.664537] ffff88082d22ffd8 ffff88082d22ffd8 ffff880829d62f10 ffff880829d62f10 [18265.673491] ffff88082bdbcd00 ffff88082d52d880 ffff88082bdbcd30 ffff88082bdbcd00 [18265.682453] Call Trace: [18265.685832] [] schedule+0x29/0x70 [18265.692030] [] rescuer_thread+0x359/0x460 [18265.699002] [] ? __schedule+0x3b0/0x990 [18265.705767] [] ? manage_workers.isra.24+0x2d0/0x2d0 [18265.713685] [] kthread+0xcf/0xe0 [18265.719761] [] ? kthread_create_on_node+0x140/0x140 [18265.727679] [] ret_from_fork+0x58/0x90 [18265.734352] [] ? kthread_create_on_node+0x140/0x140 [18265.742273] ipoib_flush S ffff88082bdbd000 0 1183 2 0x00000000 [18265.750822] ffff88082ab3fe08 0000000000000046 ffff880829d65e20 ffff88082ab3ffd8 [18265.759782] ffff88082ab3ffd8 ffff88082ab3ffd8 ffff880829d65e20 ffff880829d65e20 [18265.768737] ffff88082bdbd000 ffff88082d52dc80 ffff88082bdbd030 ffff88082bdbd000 [18265.777686] Call Trace: [18265.781054] [] schedule+0x29/0x70 [18265.787254] [] rescuer_thread+0x359/0x460 [18265.794216] [] ? __schedule+0x3b0/0x990 [18265.800981] [] ? manage_workers.isra.24+0x2d0/0x2d0 [18265.808905] [] kthread+0xcf/0xe0 [18265.814988] [] ? kthread_create_on_node+0x140/0x140 [18265.822916] [] ret_from_fork+0x58/0x90 [18265.829579] [] ? kthread_create_on_node+0x140/0x140 [18265.837496] ipoib_wq S ffff88041b24cf80 0 1184 2 0x00000000 [18265.846051] ffff88082d003e08 0000000000000046 ffff88083fde0000 ffff88082d003fd8 [18265.855023] ffff88082d003fd8 ffff88082d003fd8 ffff88083fde0000 ffff88083fde0000 [18265.863989] ffff88041b24cf80 ffff8800b3930880 ffff88041b24cfb0 ffff88041b24cf80 [18265.872954] Call Trace: [18265.876336] [] schedule+0x29/0x70 [18265.882525] [] rescuer_thread+0x359/0x460 [18265.889487] [] ? __schedule+0x3b0/0x990 [18265.896255] [] ? manage_workers.isra.24+0x2d0/0x2d0 [18265.904184] [] kthread+0xcf/0xe0 [18265.910272] [] ? kthread_create_on_node+0x140/0x140 [18265.918211] [] ret_from_fork+0x58/0x90 [18265.924873] [] ? kthread_create_on_node+0x140/0x140 [18265.932794] ipoib_wq S ffff88041b24db80 0 1185 2 0x00000000 [18265.941339] ffff8808247fbe08 0000000000000046 ffff88083fde0fb0 ffff8808247fbfd8 [18265.950312] ffff8808247fbfd8 ffff8808247fbfd8 ffff88083fde0fb0 ffff88083fde0fb0 [18265.959274] ffff88041b24db80 ffff8800b3931a80 ffff88041b24dbb0 ffff88041b24db80 [18265.968249] Call Trace: [18265.971615] [] schedule+0x29/0x70 [18265.977811] [] rescuer_thread+0x359/0x460 [18265.984775] [] ? __schedule+0x3b0/0x990 [18265.991538] [] ? manage_workers.isra.24+0x2d0/0x2d0 [18265.999470] [] kthread+0xcf/0xe0 [18266.005551] [] ? kthread_create_on_node+0x140/0x140 [18266.013476] [] ret_from_fork+0x58/0x90 [18266.020161] [] ? kthread_create_on_node+0x140/0x140 [18266.028085] srp_remove S ffff88082bdbd580 0 1192 2 0x00000000 [18266.036622] ffff8808283c3e08 0000000000000046 ffff88083fde4e70 ffff8808283c3fd8 [18266.045571] ffff8808283c3fd8 ffff8808283c3fd8 ffff88083fde4e70 ffff88083fde4e70 [18266.054544] ffff88082bdbd580 ffff88082d529680 ffff88082bdbd5b0 ffff88082bdbd580 [18266.063505] Call Trace: [18266.066883] [] schedule+0x29/0x70 [18266.073085] [] rescuer_thread+0x359/0x460 [18266.080044] [] ? __schedule+0x3b0/0x990 [18266.086811] [] ? manage_workers.isra.24+0x2d0/0x2d0 [18266.094743] [] kthread+0xcf/0xe0 [18266.100825] [] ? kthread_create_on_node+0x140/0x140 [18266.108749] [] ret_from_fork+0x58/0x90 [18266.115425] [] ? kthread_create_on_node+0x140/0x140 [18266.123366] target_completi S ffff88082bdbcf00 0 1197 2 0x00000000 [18266.131930] ffff8804b931fe08 0000000000000046 ffff880823a68fb0 ffff8804b931ffd8 [18266.140914] ffff8804b931ffd8 ffff8804b931ffd8 ffff880823a68fb0 ffff880823a68fb0 [18266.149891] ffff88082bdbcf00 ffff88082d528e80 ffff88082bdbcf30 ffff88082bdbcf00 [18266.158856] Call Trace: [18266.162235] [] schedule+0x29/0x70 [18266.168437] [] rescuer_thread+0x359/0x460 [18266.175410] [] ? __schedule+0x3b0/0x990 [18266.182194] [] ? manage_workers.isra.24+0x2d0/0x2d0 [18266.190138] [] kthread+0xcf/0xe0 [18266.196229] [] ? kthread_create_on_node+0x140/0x140 [18266.204168] [] ret_from_fork+0x58/0x90 [18266.210848] [] ? kthread_create_on_node+0x140/0x140 [18266.218777] tmr-rd_mcp S ffff88082bdbd380 0 1198 2 0x00000000 [18266.227346] ffff880829ddfe08 0000000000000046 ffff880823a69f60 ffff880829ddffd8 [18266.236320] ffff880829ddffd8 ffff880829ddffd8 ffff880823a69f60 ffff880823a69f60 [18266.245291] ffff88082bdbd380 ffff88082d52f280 ffff88082bdbd3b0 ffff88082bdbd380 [18266.254265] Call Trace: [18266.257636] [] schedule+0x29/0x70 [18266.263830] [] rescuer_thread+0x359/0x460 [18266.270801] [] ? __schedule+0x3b0/0x990 [18266.277578] [] ? manage_workers.isra.24+0x2d0/0x2d0 [18266.285508] [] kthread+0xcf/0xe0 [18266.291585] [] ? kthread_create_on_node+0x140/0x140 [18266.299504] [] ret_from_fork+0x58/0x90 [18266.306174] [] ? kthread_create_on_node+0x140/0x140 [18266.314107] xcopy_wq S ffff88082bdbd300 0 1199 2 0x00000000 [18266.322655] ffff880824273e08 0000000000000046 ffff880823a6af10 ffff880824273fd8 [18266.331621] ffff880824273fd8 ffff880824273fd8 ffff880823a6af10 ffff880823a6af10 [18266.340596] ffff88082bdbd300 ffff88082d528c80 ffff88082bdbd330 ffff88082bdbd300 [18266.349560] Call Trace: [18266.352938] [] schedule+0x29/0x70 [18266.359129] [] rescuer_thread+0x359/0x460 [18266.366103] [] ? __schedule+0x3b0/0x990 [18266.372871] [] ? manage_workers.isra.24+0x2d0/0x2d0 [18266.380801] [] kthread+0xcf/0xe0 [18266.386873] [] ? kthread_create_on_node+0x140/0x140 [18266.394797] [] ret_from_fork+0x58/0x90 [18266.401462] [] ? kthread_create_on_node+0x140/0x140 [18266.409396] iscsi_eh S ffff88082bdbd800 0 1204 2 0x00000000 [18266.417952] ffff880823eb3e08 0000000000000046 ffff880823a6bec0 ffff880823eb3fd8 [18266.426917] ffff880823eb3fd8 ffff880823eb3fd8 ffff880823a6bec0 ffff880823a6bec0 [18266.435885] ffff88082bdbd800 ffff88082d529080 ffff88082bdbd830 ffff88082bdbd800 [18266.444838] Call Trace: [18266.448209] [] schedule+0x29/0x70 [18266.454411] [] rescuer_thread+0x359/0x460 [18266.461387] [] ? __schedule+0x3b0/0x990 [18266.468162] [] ? manage_workers.isra.24+0x2d0/0x2d0 [18266.476097] [] kthread+0xcf/0xe0 [18266.482187] [] ? kthread_create_on_node+0x140/0x140 [18266.490111] [] ret_from_fork+0x58/0x90 [18266.496772] [] ? kthread_create_on_node+0x140/0x140 [18266.504691] frwr_recovery S ffff88082bdbd480 0 1213 2 0x00000000 [18266.513243] ffff88082954be08 0000000000000046 ffff880823a6ce70 ffff88082954bfd8 [18266.522232] ffff88082954bfd8 ffff88082954bfd8 ffff880823a6ce70 ffff880823a6ce70 [18266.531199] ffff88082bdbd480 ffff88082d52e280 ffff88082bdbd4b0 ffff88082bdbd480 [18266.540167] Call Trace: [18266.543535] [] schedule+0x29/0x70 [18266.549721] [] rescuer_thread+0x359/0x460 [18266.556683] [] ? __schedule+0x3b0/0x990 [18266.563447] [] ? manage_workers.isra.24+0x2d0/0x2d0 [18266.571390] [] kthread+0xcf/0xe0 [18266.577462] [] ? kthread_create_on_node+0x140/0x140 [18266.585399] [] ret_from_fork+0x58/0x90 [18266.592057] [] ? kthread_create_on_node+0x140/0x140 [18266.599986] xprtrdma_receiv S ffff88082bdbd180 0 1214 2 0x00000000 [18266.608516] ffff88082954fe08 0000000000000046 ffff880823a6de20 ffff88082954ffd8 [18266.617475] ffff88082954ffd8 ffff88082954ffd8 ffff880823a6de20 ffff880823a6de20 [18266.626435] ffff88082bdbd180 ffff88082d52e080 ffff88082bdbd1b0 ffff88082bdbd180 [18266.635396] Call Trace: [18266.638759] [] schedule+0x29/0x70 [18266.644946] [] rescuer_thread+0x359/0x460 [18266.651917] [] ? __schedule+0x3b0/0x990 [18266.658680] [] ? manage_workers.isra.24+0x2d0/0x2d0 [18266.666607] [] kthread+0xcf/0xe0 [18266.672682] [] ? kthread_create_on_node+0x140/0x140 [18266.680597] [] ret_from_fork+0x58/0x90 [18266.687264] [] ? kthread_create_on_node+0x140/0x140 [18266.695184] auditd S ffff88042a577ea8 0 1241 1 0x00000000 [18266.703703] ffff88042a577db8 0000000000000082 ffff88042ba05e20 ffff88042a577fd8 [18266.712659] ffff88042a577fd8 ffff88042a577fd8 ffff88042ba05e20 ffff88042a577ea8 [18266.721627] 00000000038f9b17 0000000000000000 ffff88042ba05e20 ffff88042a577ea8 [18266.730586] Call Trace: [18266.733968] [] schedule+0x29/0x70 [18266.740162] [] schedule_hrtimeout_range_clock+0xb2/0x150 [18266.748585] [] ? hrtimer_get_res+0x50/0x50 [18266.755648] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [18266.764270] [] schedule_hrtimeout_range+0x13/0x20 [18266.772014] [] ep_poll+0x23e/0x360 [18266.778303] [] ? wake_up_state+0x20/0x20 [18266.785172] [] SyS_epoll_wait+0xed/0x120 [18266.792028] [] system_call_fastpath+0x16/0x1b [18266.799375] auditd S ffffc9000391b040 0 1244 1 0x00000000 [18266.807915] ffff88041613bcd0 0000000000000082 ffff88042b8a8000 ffff88041613bfd8 [18266.816884] ffff88041613bfd8 ffff88041613bfd8 ffff88042b8a8000 ffff88042b8a8000 [18266.825843] ffff88042b8a8000 0000000000000000 ffff88041613bde0 ffffc9000391b040 [18266.834794] Call Trace: [18266.838184] [] schedule+0x29/0x70 [18266.844381] [] futex_wait_queue_me+0xc4/0x120 [18266.851741] [] futex_wait+0x179/0x280 [18266.858314] [] ? set_next_entity+0x3c/0xe0 [18266.865376] [] ? __switch_to+0x15c/0x4c0 [18266.872241] [] do_futex+0xfe/0x5b0 [18266.878517] [] ? wake_up_atomic_t+0x30/0x30 [18266.885662] [] ? jbd2_complete_transaction+0x52/0xa0 [jbd2] [18266.894367] [] SyS_futex+0x80/0x180 [18266.900729] [] ? do_fsync+0x76/0xa0 [18266.907093] [] system_call_fastpath+0x16/0x1b [18266.914444] irqbalance S 00007ffcaadc6fb0 0 1263 1 0x00000080 [18266.922980] ffff8800b3323e68 0000000000000082 ffff88082c558fb0 ffff8800b3323fd8 [18266.931948] ffff8800b3323fd8 ffff8800b3323fd8 ffff88082c558fb0 ffff88082c558fb0 [18266.940898] ffff8800b3323eb8 ffff88082c558fb0 ffff8800b3323f58 00007ffcaadc6fb0 [18266.949855] Call Trace: [18266.953210] [] schedule+0x29/0x70 [18266.959388] [] do_nanosleep+0x86/0xf0 [18266.965950] [] hrtimer_nanosleep+0xbb/0x180 [18266.973095] [] ? hrtimer_get_res+0x50/0x50 [18266.980139] [] ? do_nanosleep+0x72/0xf0 [18266.986891] [] SyS_nanosleep+0x7e/0x90 [18266.993536] [] system_call_fastpath+0x16/0x1b [18267.000871] gssproxy S ffff88082d123ea8 0 1264 1 0x00000080 [18267.009415] ffff88082d123db8 0000000000000082 ffff88082bcb4e70 ffff88082d123fd8 [18267.018381] ffff88082d123fd8 ffff88082d123fd8 ffff88082bcb4e70 ffff88082d123ea8 [18267.027347] 0000000001c9c37f 0000000000000000 ffff88082bcb4e70 ffff88082d123ea8 [18267.036304] Call Trace: [18267.039667] [] schedule+0x29/0x70 [18267.045834] [] schedule_hrtimeout_range_clock+0xb2/0x150 [18267.054240] [] ? hrtimer_get_res+0x50/0x50 [18267.061277] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [18267.069879] [] schedule_hrtimeout_range+0x13/0x20 [18267.077606] [] ep_poll+0x23e/0x360 [18267.083874] [] ? wake_up_state+0x20/0x20 [18267.090727] [] SyS_epoll_wait+0xed/0x120 [18267.097575] [] system_call_fastpath+0x16/0x1b [18267.104900] gssproxy S ffffc900035bc2c0 0 1266 1 0x00000080 [18267.113426] ffff880429623cd0 0000000000000082 ffff88042d142f10 ffff880429623fd8 [18267.122379] ffff880429623fd8 ffff880429623fd8 ffff88042d142f10 ffff88042d142f10 [18267.131337] ffff88042d142f10 0000000000000000 ffff880429623de0 ffffc900035bc2c0 [18267.140289] Call Trace: [18267.143653] [] schedule+0x29/0x70 [18267.149825] [] futex_wait_queue_me+0xc4/0x120 [18267.157158] [] futex_wait+0x179/0x280 [18267.163711] [] ? __schedule+0x3b0/0x990 [18267.170454] [] do_futex+0xfe/0x5b0 [18267.176701] [] ? __do_page_fault+0x171/0x450 [18267.183913] [] SyS_futex+0x80/0x180 [18267.190265] [] ? do_page_fault+0x35/0x90 [18267.197105] [] system_call_fastpath+0x16/0x1b [18267.204424] gssproxy S ffffc900035f9f40 0 1267 1 0x00000080 [18267.212959] ffff88041ac3fcd0 0000000000000082 ffff88042d143ec0 ffff88041ac3ffd8 [18267.221901] ffff88041ac3ffd8 ffff88041ac3ffd8 ffff88042d143ec0 ffff88042d143ec0 [18267.230856] ffff88042d143ec0 0000000000000000 ffff88041ac3fde0 ffffc900035f9f40 [18267.239786] Call Trace: [18267.243140] [] schedule+0x29/0x70 [18267.249329] [] futex_wait_queue_me+0xc4/0x120 [18267.256664] [] futex_wait+0x179/0x280 [18267.263240] [] ? ext4_filemap_fault+0x41/0x50 [ext4] [18267.271251] [] ? unlock_page+0x2b/0x30 [18267.277911] [] do_futex+0xfe/0x5b0 [18267.284176] [] ? __do_page_fault+0x171/0x450 [18267.291415] [] SyS_futex+0x80/0x180 [18267.297761] [] ? do_page_fault+0x35/0x90 [18267.304596] [] system_call_fastpath+0x16/0x1b [18267.311925] gssproxy S ffffc900037a3dc0 0 1268 1 0x00000080 [18267.320455] ffff88042bc7bcd0 0000000000000082 ffff88042d141f60 ffff88042bc7bfd8 [18267.329401] ffff88042bc7bfd8 ffff88042bc7bfd8 ffff88042d141f60 ffff88042d141f60 [18267.338362] ffff88042d141f60 0000000000000000 ffff88042bc7bde0 ffffc900037a3dc0 [18267.347314] Call Trace: [18267.350674] [] schedule+0x29/0x70 [18267.356838] [] futex_wait_queue_me+0xc4/0x120 [18267.364172] [] futex_wait+0x179/0x280 [18267.370730] [] do_futex+0xfe/0x5b0 [18267.376994] [] SyS_futex+0x80/0x180 [18267.383340] [] system_call_fastpath+0x16/0x1b [18267.390657] gssproxy S ffffc9000360f3c0 0 1269 1 0x00000080 [18267.399177] ffff88042b907cd0 0000000000000082 ffff88042d140fb0 ffff88042b907fd8 [18267.408130] ffff88042b907fd8 ffff88042b907fd8 ffff88042d140fb0 ffff88042d140fb0 [18267.417077] ffff88042d140fb0 0000000000000000 ffff88042b907de0 ffffc9000360f3c0 [18267.426020] Call Trace: [18267.429365] [] schedule+0x29/0x70 [18267.435531] [] futex_wait_queue_me+0xc4/0x120 [18267.442871] [] futex_wait+0x179/0x280 [18267.449426] [] do_futex+0xfe/0x5b0 [18267.455665] [] SyS_futex+0x80/0x180 [18267.462015] [] system_call_fastpath+0x16/0x1b [18267.469327] gssproxy S ffffc9000386fc80 0 1270 1 0x00000080 [18267.477843] ffff88041ac23cd0 0000000000000082 ffff88042d144e70 ffff88041ac23fd8 [18267.486758] ffff88041ac23fd8 ffff88041ac23fd8 ffff88042d144e70 ffff88042d144e70 [18267.495685] ffff88042d144e70 0000000000000000 ffff88041ac23de0 ffffc9000386fc80 [18267.504607] Call Trace: [18267.507963] [] schedule+0x29/0x70 [18267.514137] [] futex_wait_queue_me+0xc4/0x120 [18267.521463] [] futex_wait+0x179/0x280 [18267.528013] [] ? __sb_end_write+0x31/0x60 [18267.534968] [] ? pipe_write+0x46a/0x5b0 [18267.541711] [] do_futex+0xfe/0x5b0 [18267.547969] [] SyS_futex+0x80/0x180 [18267.554317] [] system_call_fastpath+0x16/0x1b [18267.561625] chronyd S 0000000000000007 0 1273 1 0x00000080 [18267.570146] ffff8804161db8e8 0000000000000086 ffff88042b920fb0 ffff8804161dbfd8 [18267.579075] ffff8804161dbfd8 ffff8804161dbfd8 ffff88042b920fb0 ffff8804161dbac8 [18267.587992] 0000000005f5e100 0000000000000000 0000000000000007 0000000000000007 [18267.596916] Call Trace: [18267.600254] [] schedule+0x29/0x70 [18267.606426] [] schedule_hrtimeout_range_clock+0xb2/0x150 [18267.614815] [] ? hrtimer_get_res+0x50/0x50 [18267.621864] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [18267.630465] [] schedule_hrtimeout_range+0x13/0x20 [18267.638198] [] poll_schedule_timeout+0x55/0xb0 [18267.645631] [] do_select+0x6d1/0x7c0 [18267.652098] [] ? poll_select_copy_remaining+0x150/0x150 [18267.660403] [] ? poll_select_copy_remaining+0x150/0x150 [18267.668692] [] ? poll_select_copy_remaining+0x150/0x150 [18267.676977] [] ? poll_select_copy_remaining+0x150/0x150 [18267.685248] [] ? poll_select_copy_remaining+0x150/0x150 [18267.693503] [] ? unix_dgram_sendmsg+0x3a8/0x720 [18267.700990] [] ? sock_sendmsg+0xb0/0xf0 [18267.707683] [] core_sys_select+0x1db/0x300 [18267.714671] [] ? SYSC_sendto+0x121/0x1c0 [18267.721455] [] ? ktime_get_ts64+0x4c/0xf0 [18267.728363] [] SyS_select+0xba/0x110 [18267.734764] [] system_call_fastpath+0x16/0x1b [18267.742049] polkitd S ffff8804161abbac 0 1274 1 0x00000080 [18267.750522] ffff8804161aba30 0000000000000082 ffff88082c55af10 ffff8804161abfd8 [18267.759428] ffff8804161abfd8 ffff8804161abfd8 ffff88082c55af10 0000000000000000 [18267.768338] ffff8804161abc90 00000000fffffffc 0000000000000000 ffff8804161abbac [18267.777247] Call Trace: [18267.780558] [] schedule+0x29/0x70 [18267.786684] [] schedule_hrtimeout_range_clock+0x12d/0x150 [18267.795152] [] ? add_wait_queue+0x3c/0x50 [18267.802065] [] schedule_hrtimeout_range+0x13/0x20 [18267.809761] [] poll_schedule_timeout+0x55/0xb0 [18267.817175] [] do_sys_poll+0x4cd/0x580 [18267.823793] [] ? dequeue_task_fair+0x41e/0x660 [18267.831185] [] ? sched_clock_cpu+0x85/0xc0 [18267.838190] [] ? __switch_to+0x15c/0x4c0 [18267.844986] [] ? update_curr+0x104/0x190 [18267.851777] [] ? __enqueue_entity+0x78/0x80 [18267.858858] [] ? poll_select_copy_remaining+0x150/0x150 [18267.867109] [] ? poll_select_copy_remaining+0x150/0x150 [18267.875359] [] ? ttwu_do_wakeup+0x19/0xd0 [18267.882255] [] ? ttwu_do_activate.constprop.91+0x5d/0x70 [18267.890597] [] ? inotify_free_event+0xe/0x10 [18267.897775] [] ? kfree+0x103/0x140 [18267.904005] [] ? eventfd_ctx_read+0x67/0x210 [18267.911191] [] ? vfs_read+0xf7/0x170 [18267.917591] [] SyS_poll+0x74/0x110 [18267.923790] [] ? __audit_syscall_exit+0x1e6/0x280 [18267.931446] [] system_call_fastpath+0x16/0x1b [18267.938713] gmain S ffff88041622bbac 0 1275 1 0x00000080 [18267.947184] ffff88041622ba30 0000000000000082 ffff88042b923ec0 ffff88041622bfd8 [18267.956080] ffff88041622bfd8 ffff88041622bfd8 ffff88042b923ec0 0000000000000000 [18267.964961] ffff88041622bc90 00000000fffffffc 0000000000000000 ffff88041622bbac [18267.973852] Call Trace: [18267.977152] [] schedule+0x29/0x70 [18267.983270] [] schedule_hrtimeout_range_clock+0x12d/0x150 [18267.991713] [] ? add_wait_queue+0x3c/0x50 [18267.998604] [] schedule_hrtimeout_range+0x13/0x20 [18268.006274] [] poll_schedule_timeout+0x55/0xb0 [18268.013659] [] do_sys_poll+0x4cd/0x580 [18268.020277] [] ? zone_statistics+0x89/0xa0 [18268.027300] [] ? get_page_from_freelist+0x51a/0x9f0 [18268.035210] [] ? poll_select_copy_remaining+0x150/0x150 [18268.043498] [] ? poll_select_copy_remaining+0x150/0x150 [18268.051784] [] ? ext4_filemap_fault+0x41/0x50 [ext4] [18268.059772] [] ? unlock_page+0x2b/0x30 [18268.066419] [] ? do_read_fault.isra.42+0xe4/0x130 [18268.074132] [] ? eventfd_ctx_read+0x67/0x210 [18268.081352] [] ? vfs_read+0xf7/0x170 [18268.087785] [] SyS_poll+0x74/0x110 [18268.094019] [] ? __audit_syscall_exit+0x1e6/0x280 [18268.101704] [] system_call_fastpath+0x16/0x1b [18268.109004] gdbus S ffff88082a2e7bac 0 1289 1 0x00000080 [18268.117494] ffff88082a2e7a30 0000000000000082 ffff88082bc13ec0 ffff88082a2e7fd8 [18268.126425] ffff88082a2e7fd8 ffff88082a2e7fd8 ffff88082bc13ec0 0000000000000000 [18268.135343] ffff88082a2e7c90 00000000fffffffc 0000000000000000 ffff88082a2e7bac [18268.144258] Call Trace: [18268.147581] [] schedule+0x29/0x70 [18268.153716] [] schedule_hrtimeout_range_clock+0x12d/0x150 [18268.162199] [] ? add_wait_queue+0x3c/0x50 [18268.169128] [] ? add_wait_queue+0x3c/0x50 [18268.176042] [] schedule_hrtimeout_range+0x13/0x20 [18268.183735] [] poll_schedule_timeout+0x55/0xb0 [18268.191147] [] do_sys_poll+0x4cd/0x580 [18268.197776] [] ? unix_stream_recvmsg+0x54/0x70 [18268.205193] [] ? unix_state_double_unlock+0x50/0x50 [18268.213087] [] ? sock_recvmsg+0xbf/0x100 [18268.219929] [] ? __enqueue_entity+0x78/0x80 [18268.227052] [] ? poll_select_copy_remaining+0x150/0x150 [18268.235348] [] ? poll_select_copy_remaining+0x150/0x150 [18268.243628] [] ? ttwu_do_wakeup+0x19/0xd0 [18268.250542] [] ? ttwu_do_activate.constprop.91+0x5d/0x70 [18268.258932] [] ? try_to_wake_up+0x1c8/0x320 [18268.266055] [] ? default_wake_function+0x12/0x20 [18268.273653] [] ? pollwake+0x73/0x90 [18268.279999] [] ? wake_up_state+0x20/0x20 [18268.286817] [] ? eventfd_ctx_read+0x67/0x210 [18268.294022] [] ? vfs_read+0xf7/0x170 [18268.300448] [] SyS_poll+0x74/0x110 [18268.306670] [] ? __audit_syscall_exit+0x1e6/0x280 [18268.314347] [] system_call_fastpath+0x16/0x1b [18268.321629] JS GC Helper S ffffc90003683a80 0 1290 1 0x00000080 [18268.330112] ffff88082d287cd0 0000000000000082 ffff88082a63ce70 ffff88082d287fd8 [18268.339039] ffff88082d287fd8 ffff88082d287fd8 ffff88082a63ce70 ffff88082a63ce70 [18268.347936] ffff88082a63ce70 0000000000000000 ffff88082d287de0 ffffc90003683a80 [18268.356818] Call Trace: [18268.360136] [] schedule+0x29/0x70 [18268.366277] [] futex_wait_queue_me+0xc4/0x120 [18268.373575] [] futex_wait+0x179/0x280 [18268.380102] [] ? page_add_file_rmap+0x8c/0xc0 [18268.387406] [] ? unlock_page+0x2b/0x30 [18268.394020] [] do_futex+0xfe/0x5b0 [18268.400248] [] ? __do_page_fault+0x171/0x450 [18268.407448] [] ? strlcpy+0x42/0x60 [18268.413666] [] SyS_futex+0x80/0x180 [18268.419987] [] ? do_page_fault+0x35/0x90 [18268.426794] [] system_call_fastpath+0x16/0x1b [18268.434087] JS Sour~ Thread S ffffc90003674680 0 1291 1 0x00000080 [18268.442575] ffff880823f23cd0 0000000000000082 ffff88082a639f60 ffff880823f23fd8 [18268.451490] ffff880823f23fd8 ffff880823f23fd8 ffff88082a639f60 ffff88082a639f60 [18268.460393] ffff88082a639f60 0000000000000000 ffff880823f23de0 ffffc90003674680 [18268.469307] Call Trace: [18268.472620] [] schedule+0x29/0x70 [18268.478740] [] futex_wait_queue_me+0xc4/0x120 [18268.486025] [] futex_wait+0x179/0x280 [18268.492529] [] ? wake_up_state+0x10/0x20 [18268.499321] [] ? wake_futex+0x66/0x80 [18268.505815] [] ? get_futex_key+0x1c8/0x2c0 [18268.512793] [] do_futex+0xfe/0x5b0 [18268.519020] [] ? __do_page_fault+0x171/0x450 [18268.526201] [] ? strlcpy+0x42/0x60 [18268.532414] [] SyS_futex+0x80/0x180 [18268.538707] [] system_call_fastpath+0x16/0x1b [18268.545982] runaway-killer- S ffff88082a593ba4 0 1292 1 0x00000080 [18268.554451] ffff88082a593a30 0000000000000082 ffff88082a63bec0 ffff88082a593fd8 [18268.563365] ffff88082a593fd8 ffff88082a593fd8 ffff88082a63bec0 0000000000000000 [18268.572276] ffff88082a593c90 00000000fffffffc 0000000000000000 ffff88082a593ba4 [18268.581197] Call Trace: [18268.584517] [] schedule+0x29/0x70 [18268.590657] [] schedule_hrtimeout_range_clock+0x12d/0x150 [18268.599128] [] ? add_wait_queue+0x3c/0x50 [18268.606048] [] schedule_hrtimeout_range+0x13/0x20 [18268.613745] [] poll_schedule_timeout+0x55/0xb0 [18268.621151] [] do_sys_poll+0x4cd/0x580 [18268.627781] [] ? zone_statistics+0x89/0xa0 [18268.634803] [] ? zone_statistics+0x89/0xa0 [18268.641805] [] ? get_page_from_freelist+0x51a/0x9f0 [18268.649693] [] ? file_free+0x40/0x40 [18268.656131] [] ? __enqueue_entity+0x78/0x80 [18268.663242] [] ? poll_select_copy_remaining+0x150/0x150 [18268.671488] [] ? check_preempt_curr+0x75/0xa0 [18268.678785] [] ? ttwu_do_wakeup+0x19/0xd0 [18268.685689] [] ? ttwu_do_activate.constprop.91+0x5d/0x70 [18268.694053] [] ? try_to_wake_up+0x1c8/0x320 [18268.701145] [] ? wake_up_state+0x10/0x20 [18268.707944] [] ? wake_futex+0x66/0x80 [18268.714455] [] ? futex_wake+0x80/0x160 [18268.721072] [] ? eventfd_ctx_read+0x67/0x210 [18268.728253] [] ? vfs_read+0xf7/0x170 [18268.734645] [] SyS_poll+0x74/0x110 [18268.740853] [] ? __audit_syscall_exit+0x1e6/0x280 [18268.748519] [] system_call_fastpath+0x16/0x1b [18268.755794] dbus-daemon S 0000000000000000 0 1282 1 0x00000080 [18268.764286] ffff8804162afdb8 0000000000000086 ffff88082c55ce70 ffff8804162affd8 [18268.773182] ffff8804162affd8 ffff8804162affd8 ffff88082c55ce70 0000000000000000 [18268.782102] 0000000000000000 ffff88081e690c60 ffff88082c55ce70 0000000000000000 [18268.791005] Call Trace: [18268.794326] [] schedule+0x29/0x70 [18268.800449] [] schedule_hrtimeout_range_clock+0x12d/0x150 [18268.808904] [] ? ep_scan_ready_list.isra.7+0x1b9/0x1f0 [18268.817074] [] schedule_hrtimeout_range+0x13/0x20 [18268.824751] [] ep_poll+0x23e/0x360 [18268.830984] [] ? __fput+0x183/0x260 [18268.837304] [] ? wake_up_state+0x20/0x20 [18268.844124] [] SyS_epoll_wait+0xed/0x120 [18268.850928] [] system_call_fastpath+0x16/0x1b [18268.858224] NetworkManager S ffff88082955bbd4 0 1293 1 0x00000080 [18268.866703] ffff88082955ba30 0000000000000082 ffff88082a60edd0 ffff88082955bfd8 [18268.875626] ffff88082955bfd8 ffff88082955bfd8 ffff88082a60edd0 ffff88082955bb88 [18268.884540] 0000000005f5e100 0000000000000000 0000000000000000 ffff88082955bbd4 [18268.893481] Call Trace: [18268.896810] [] schedule+0x29/0x70 [18268.902958] [] schedule_hrtimeout_range_clock+0xb2/0x150 [18268.911335] [] ? hrtimer_get_res+0x50/0x50 [18268.918367] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [18268.926940] [] schedule_hrtimeout_range+0x13/0x20 [18268.934640] [] poll_schedule_timeout+0x55/0xb0 [18268.942040] [] do_sys_poll+0x4cd/0x580 [18268.948662] [] ? dequeue_task_fair+0x41e/0x660 [18268.956069] [] ? sched_clock_cpu+0x85/0xc0 [18268.963075] [] ? __switch_to+0x15c/0x4c0 [18268.969884] [] ? get_futex_key+0x1c8/0x2c0 [18268.976890] [] ? __enqueue_entity+0x78/0x80 [18268.984000] [] ? poll_select_copy_remaining+0x150/0x150 [18268.992278] [] ? poll_select_copy_remaining+0x150/0x150 [18269.000548] [] ? poll_select_copy_remaining+0x150/0x150 [18269.008806] [] ? poll_select_copy_remaining+0x150/0x150 [18269.017061] [] ? poll_select_copy_remaining+0x150/0x150 [18269.025305] [] ? poll_select_copy_remaining+0x150/0x150 [18269.033534] [] ? poll_select_copy_remaining+0x150/0x150 [18269.041747] [] ? ktime_get_ts64+0x4c/0xf0 [18269.048606] [] SyS_poll+0x74/0x110 [18269.054780] [] system_call_fastpath+0x16/0x1b [18269.062041] gmain S ffff880823e73bac 0 1302 1 0x00000080 [18269.070481] ffff880823e73a30 0000000000000082 ffff8800b3cbbec0 ffff880823e73fd8 [18269.079374] ffff880823e73fd8 ffff880823e73fd8 ffff8800b3cbbec0 0000000000000000 [18269.088268] ffff880823e73c90 00000000fffffffc 0000000000000000 ffff880823e73bac [18269.097148] Call Trace: [18269.100460] [] schedule+0x29/0x70 [18269.106565] [] schedule_hrtimeout_range_clock+0x12d/0x150 [18269.115023] [] ? add_wait_queue+0x3c/0x50 [18269.121922] [] schedule_hrtimeout_range+0x13/0x20 [18269.129589] [] poll_schedule_timeout+0x55/0xb0 [18269.136971] [] do_sys_poll+0x4cd/0x580 [18269.143565] [] ? zone_statistics+0x89/0xa0 [18269.150548] [] ? get_page_from_freelist+0x51a/0x9f0 [18269.158417] [] ? __mem_cgroup_uncharge_common+0xcd/0x2f0 [18269.166758] [] ? poll_select_copy_remaining+0x150/0x150 [18269.174998] [] ? poll_select_copy_remaining+0x150/0x150 [18269.183233] [] ? ext4_filemap_fault+0x41/0x50 [ext4] [18269.191189] [] ? unlock_page+0x2b/0x30 [18269.197776] [] ? do_read_fault.isra.42+0xe4/0x130 [18269.205439] [] ? eventfd_ctx_read+0x67/0x210 [18269.212617] [] ? vfs_read+0xf7/0x170 [18269.219008] [] SyS_poll+0x74/0x110 [18269.225194] [] ? __audit_syscall_exit+0x1e6/0x280 [18269.232840] [] system_call_fastpath+0x16/0x1b [18269.240107] gdbus S ffff88083f93fbb4 0 1307 1 0x00000080 [18269.248557] ffff88083f93fa30 0000000000000082 ffff88082a60de20 ffff88083f93ffd8 [18269.257436] ffff88083f93ffd8 ffff88083f93ffd8 ffff88082a60de20 0000000000000000 [18269.266319] ffff88083f93fc90 00000000fffffffc 0000000000000000 ffff88083f93fbb4 [18269.275190] Call Trace: [18269.278472] [] schedule+0x29/0x70 [18269.284573] [] schedule_hrtimeout_range_clock+0x12d/0x150 [18269.293012] [] ? add_wait_queue+0x3c/0x50 [18269.299893] [] ? __pollwait+0x7f/0xf0 [18269.306388] [] ? add_wait_queue+0x3c/0x50 [18269.313266] [] schedule_hrtimeout_range+0x13/0x20 [18269.320931] [] poll_schedule_timeout+0x55/0xb0 [18269.328311] [] do_sys_poll+0x4cd/0x580 [18269.334903] [] ? unix_stream_recvmsg+0x54/0x70 [18269.342274] [] ? unix_state_double_unlock+0x50/0x50 [18269.350146] [] ? sock_recvmsg+0xbf/0x100 [18269.356948] [] ? __enqueue_entity+0x78/0x80 [18269.364055] [] ? poll_select_copy_remaining+0x150/0x150 [18269.372346] [] ? poll_select_copy_remaining+0x150/0x150 [18269.380633] [] ? poll_select_copy_remaining+0x150/0x150 [18269.388909] [] ? try_to_wake_up+0x1c8/0x320 [18269.396036] [] ? default_wake_function+0x12/0x20 [18269.403645] [] ? pollwake+0x73/0x90 [18269.409990] [] ? wake_up_state+0x20/0x20 [18269.416805] [] ? eventfd_ctx_read+0x67/0x210 [18269.424009] [] ? vfs_read+0xf7/0x170 [18269.430437] [] SyS_poll+0x74/0x110 [18269.436655] [] ? __audit_syscall_exit+0x1e6/0x280 [18269.444328] [] system_call_fastpath+0x16/0x1b [18269.451606] rpc.gssd S 0000000000000000 0 1299 1 0x00000080 [18269.460105] ffff880823ec7db8 0000000000000082 ffff880829cf2f10 ffff880823ec7fd8 [18269.469007] ffff880823ec7fd8 ffff880823ec7fd8 ffff880829cf2f10 0000000000000000 [18269.477907] 0000000000000000[18269.478900] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] Message from[18269.478902] Modules linked in: syslogd@soak-8 rpcsec_gss_krb5at May 9 23:13: nfsv4 dns_resolver41 ... kernel nfs fscache:NMI watchdog: B osp(OE)UG: soft lockup mdd(OE) lod(OE)- CPU#24 stuck f mdt(OE) lfsck(OE)or 22s! [OI_scru mgs(OE)b:5551] mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [18269.478950] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [18269.478951] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [18269.478952] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [18269.478953] RIP: 0010:[] [] osd_scrub_next+0x167/0x3c0 [osd_ldiskfs] [18269.478982] RSP: 0018:ffff880703603d00 EFLAGS: 00000282 [18269.478983] RAX: 0000000000000004 RBX: 0000000000000001 RCX: ffff880703603d98 [18269.478983] RDX: ffff880703603d78 RSI: 0000000000000000 RDI: ffff8803162f6000 [18269.478984] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [18269.478985] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [18269.478985] R13: 0000000000000018 R14: ffff880703603c70 R15: 0000000000000246 [18269.478986] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [18269.478987] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18269.478988] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [18269.478988] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [18269.478989] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [18269.478989] Stack: [18269.478990] 0000000000000001 ffff880703603df0 ffffffffa121cff9 ffffffffa121b990 [18269.478991] ffffffffa1217a20 ffff8800b2a36000 00000000810d354f ffff8803162f6000 [18269.478992] ffff8800b2a37468 0000000020000000 010000000000000c 0000000000000000 [18269.478993] Call Trace: [18269.478994] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [18269.479002] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [18269.479008] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [18269.479015] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [18269.479021] [] ? wake_up_state+0x20/0x20 [18269.479023] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [18269.479029] [] kthread+0xcf/0xe0 [18269.479031] [] ? kthread_create_on_node+0x140/0x140 [18269.479033] [] ret_from_fork+0x58/0x90 [18269.479034] [] ? kthread_create_on_node+0x140/0x140 [18269.479036] Code: 48 89 df e8 fc 69 47 e0 b8 05 00 00 00 48 8b 75 d0 65 48 33 34 25 28 00 00 00 0f 85 4c 02 00 00 48 83 c4 70 5b 41 5c 41 5d 41 5e <41> 5f 5d c3 48 8b 05 be 58 9c ff 48 85 c0 0f 84 c2 fe ff ff 0f [18269.919883] ffff88081e6dd620 ffff880829cf2f10 0000000000000000 [18269.928676] Call Trace: [18269.932363] [] schedule+0x29/0x70 [18269.938860] [] schedule_hrtimeout_range_clock+0x12d/0x150 [18269.947669] [] ? ep_scan_ready_list.isra.7+0x1b9/0x1f0 [18269.956168] [] schedule_hrtimeout_range+0x13/0x20 [18269.964161] [] ep_poll+0x23e/0x360 [18269.970689] [] ? wake_up_state+0x20/0x20 [18269.977775] [] SyS_epoll_wait+0xed/0x120 [18269.984830] [] system_call_fastpath+0x16/0x1b [18269.992352] sssd S ffff8800b330bea8 0 1300 1 0x00000080 [18270.001059] ffff8800b330bdb8 0000000000000082 ffff88042d145e20 ffff8800b330bfd8 [18270.010167] ffff8800b330bfd8 ffff8800b330bfd8 ffff88042d145e20 ffff8800b330bea8 [18270.019234] 0000000001c9c37f 0000000000000000 ffff88042d145e20 ffff8800b330bea8 [18270.028296] Call Trace: [18270.031752] [] schedule+0x29/0x70 [18270.038004] [] schedule_hrtimeout_range_clock+0xb2/0x150 [18270.046492] [] ? hrtimer_get_res+0x50/0x50 [18270.053613] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [18270.062263] [] schedule_hrtimeout_range+0x13/0x20 [18270.070036] [] ep_poll+0x23e/0x360 [18270.076355] [] ? wake_up_state+0x20/0x20 [18270.083263] [] SyS_epoll_wait+0xed/0x120 [18270.090172] [] system_call_fastpath+0x16/0x1b [18270.097578] sssd_be S ffff88081e65fea8 0 1301 1300 0x00000080 [18270.106172] ffff88081e65fdb8 0000000000000082 ffff88042c55de20 ffff88081e65ffd8 [18270.115185] ffff88081e65ffd8 ffff88081e65ffd8 ffff88042c55de20 ffff88081e65fea8 [18270.124193] 000000000098967f 0000000000000000 ffff88042c55de20 ffff88081e65fea8 [18270.133195] Call Trace: [18270.136617] [] schedule+0x29/0x70 [18270.142850] [] schedule_hrtimeout_range_clock+0xb2/0x150 [18270.151316] [] ? hrtimer_get_res+0x50/0x50 [18270.158426] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [18270.167100] [] schedule_hrtimeout_range+0x13/0x20 [18270.174876] [] ep_poll+0x23e/0x360 [18270.181191] [] ? wake_up_state+0x20/0x20 [18270.188095] [] SyS_epoll_wait+0xed/0x120 [18270.194997] [] system_call_fastpath+0x16/0x1b [18270.202386] sssd_nss S ffff88081e71bea8 0 1304 1300 0x00000080 [18270.210978] ffff88081e71bdb8 0000000000000086 ffff88082bcb5e20 ffff88081e71bfd8 [18270.219980] ffff88081e71bfd8 ffff88081e71bfd8 ffff88082bcb5e20 ffff88081e71bea8 [18270.228957] 000000000098967f 0000000000000000 ffff88082bcb5e20 ffff88081e71bea8 [18270.237947] Call Trace: [18270.241350] [] schedule+0x29/0x70 [18270.247580] [] schedule_hrtimeout_range_clock+0xb2/0x150 [18270.256031] [] ? hrtimer_get_res+0x50/0x50 [18270.263111] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [18270.271723] [] schedule_hrtimeout_range+0x13/0x20 [18270.279465] [] ep_poll+0x23e/0x360 [18270.285743] [] ? wake_up_state+0x20/0x20 [18270.292602] [] SyS_epoll_wait+0xed/0x120 [18270.299465] [] system_call_fastpath+0x16/0x1b [18270.306801] sssd_pam S ffff88083f84fea8 0 1305 1300 0x00000080 [18270.315359] ffff88083f84fdb8 0000000000000086 ffff88082bcb6dd0 ffff88083f84ffd8 [18270.324334] ffff88083f84ffd8 ffff88083f84ffd8 ffff88082bcb6dd0 ffff88083f84fea8 [18270.333300] 000000000098967f 0000000000000000 ffff88082bcb6dd0 ffff88083f84fea8 [18270.342270] Call Trace: [18270.345635] [] schedule+0x29/0x70 [18270.351820] [] schedule_hrtimeout_range_clock+0xb2/0x150 [18270.360240] [] ? hrtimer_get_res+0x50/0x50 [18270.367301] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [18270.375913] [] schedule_hrtimeout_range+0x13/0x20 [18270.383652] [] ep_poll+0x23e/0x360 [18270.389930] [] ? wake_up_state+0x20/0x20 [18270.396787] [] SyS_epoll_wait+0xed/0x120 [18270.403648] [] system_call_fastpath+0x16/0x1b [18270.410992] sssd_ssh S ffff88083f8bbea8 0 1306 1300 0x00000080 [18270.419543] ffff88083f8bbdb8 0000000000000082 ffff88082bcb2f10 ffff88083f8bbfd8 [18270.428517] ffff88083f8bbfd8 ffff88083f8bbfd8 ffff88082bcb2f10 ffff88083f8bbea8 [18270.437479] 000000000098967f 0000000000000000 ffff88082bcb2f10 ffff88083f8bbea8 [18270.446447] Call Trace: [18270.449816] [] schedule+0x29/0x70 [18270.456014] [] schedule_hrtimeout_range_clock+0xb2/0x150 [18270.464430] [] ? hrtimer_get_res+0x50/0x50 [18270.471507] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [18270.480133] [] schedule_hrtimeout_range+0x13/0x20 [18270.487874] [] ep_poll+0x23e/0x360 [18270.494155] [] ? wake_up_state+0x20/0x20 [18270.501017] [] SyS_epoll_wait+0xed/0x120 [18270.507875] [] system_call_fastpath+0x16/0x1b [18270.515220] sssd_pac S ffff88083f917ea8 0 1308 1300 0x00000080 [18270.523777] ffff88083f917db8 0000000000000086 ffff88082bcb1f60 ffff88083f917fd8 [18270.532740] ffff88083f917fd8 ffff88083f917fd8 ffff88082bcb1f60 ffff88083f917ea8 [18270.541705] 000000000098967f 0000000000000000 ffff88082bcb1f60 ffff88083f917ea8 [18270.550676] Call Trace: [18270.554070] [] schedule+0x29/0x70 [18270.560256] [] schedule_hrtimeout_range_clock+0xb2/0x150 [18270.568670] [] ? hrtimer_get_res+0x50/0x50 [18270.575725] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [18270.584346] [] schedule_hrtimeout_range+0x13/0x20 [18270.592088] [] ep_poll+0x23e/0x360 [18270.598375] [] ? wake_up_state+0x20/0x20 [18270.605244] [] SyS_epoll_wait+0xed/0x120 [18270.612103] [] system_call_fastpath+0x16/0x1b [18270.619445] systemd-logind S 0000000000000000 0 1315 1 0x00000080 [18270.628016] ffff88083f7d7db8 0000000000000082 ffff88082a63edd0 ffff88083f7d7fd8 [18270.636983] ffff88083f7d7fd8 ffff88083f7d7fd8 ffff88082a63edd0 0000000000000000 [18270.645950] 0000000000000000 ffff8804b9033b60 ffff88082a63edd0 0000000000000000 [18270.654892] Call Trace: [18270.658271] [] schedule+0x29/0x70 [18270.664468] [] schedule_hrtimeout_range_clock+0x12d/0x150 [18270.672978] [] ? ep_scan_ready_list.isra.7+0x1b9/0x1f0 [18270.681215] [] schedule_hrtimeout_range+0x13/0x20 [18270.688972] [] ep_poll+0x23e/0x360 [18270.695254] [] ? wake_up_state+0x20/0x20 [18270.702124] [] SyS_epoll_wait+0xed/0x120 [18270.708984] [] system_call_fastpath+0x16/0x1b [18270.716332] crond S 0000000000000000 0 1320 1 0x00000080 [18270.724871] ffff88082a4bbe68 0000000000000086 ffff88082a638000 ffff88082a4bbfd8 [18270.733819] ffff88082a4bbfd8 ffff88082a4bbfd8 ffff88082a638000 ffff88082a638000 [18270.742769] ffff88082a4bbeb8 ffff88082a638000 ffff88082a4bbf58 0000000000000000 [18270.751720] Call Trace: [18270.755091] [] schedule+0x29/0x70 [18270.761280] [] do_nanosleep+0x86/0xf0 [18270.767844] [] hrtimer_nanosleep+0xbb/0x180 [18270.774997] [] ? hrtimer_get_res+0x50/0x50 [18270.782044] [] ? do_nanosleep+0x72/0xf0 [18270.788797] [] SyS_nanosleep+0x7e/0x90 [18270.795450] [] system_call_fastpath+0x16/0x1b [18270.802782] atd S 0000000000000000 0 1322 1 0x00000080 [18270.811326] ffff88083f97fe68 0000000000000082 ffff88042a68de20 ffff88083f97ffd8 [18270.820276] ffff88083f97ffd8 ffff88083f97ffd8 ffff88042a68de20 ffff88042a68de20 [18270.829218] ffff88083f97feb8 ffff88042a68de20 ffff88083f97ff58 0000000000000000 [18270.838157] Call Trace: [18270.841513] [] schedule+0x29/0x70 [18270.847686] [] do_nanosleep+0x86/0xf0 [18270.854245] [] hrtimer_nanosleep+0xbb/0x180 [18270.861383] [] ? hrtimer_get_res+0x50/0x50 [18270.868424] [] ? do_nanosleep+0x72/0xf0 [18270.875157] [] SyS_nanosleep+0x7e/0x90 [18270.881793] [] system_call_fastpath+0x16/0x1b [18270.889115] kworker/24:2 S ffff88082da16480 0 1325 2 0x00000080 [18270.897635] ffff88083fc77e50 0000000000000046 ffff880823a6edd0 ffff88083fc77fd8 [18270.906567] ffff88083fc77fd8 ffff88083fc77fd8 ffff880823a6edd0 ffff88082da164a0 [18270.915514] ffff88042bc577b0 ffff880823a6edd0 ffff88042bc57780 ffff88082da16480 [18270.924460] Call Trace: [18270.927807] [] schedule+0x29/0x70 [18270.933961] [] worker_thread+0x1d9/0x410 [18270.940795] [] ? rescuer_thread+0x460/0x460 [18270.947920] [] kthread+0xcf/0xe0 [18270.953987] [] ? kthread_create_on_node+0x140/0x140 [18270.961892] [] ret_from_fork+0x58/0x90 [18270.968537] [] ? kthread_create_on_node+0x140/0x140 [18270.976437] login S ffff88042a687ff0 0 1328 1 0x00000080 [18270.984956] ffff8804b9397e70 0000000000000082 ffff88042a688000 ffff8804b9397fd8 [18270.993890] ffff8804b9397fd8 ffff8804b9397fd8 ffff88042a688000 ffff8804b9397ef8 [18271.002808] ffff88042a6884f0 ffff88042a688000 ffff88042a688000 ffff88042a687ff0 [18271.011724] Call Trace: [18271.015063] [] schedule+0x29/0x70 [18271.021221] [] do_wait+0x1f3/0x250 [18271.027461] [] SyS_wait4+0x80/0x110 [18271.033800] [] ? task_stopped_code+0x60/0x60 [18271.041025] [] system_call_fastpath+0x16/0x1b [18271.048341] agetty S ffff880416375000 0 1329 1 0x00000080 [18271.056852] ffff88083fb3bcf0 0000000000000082 ffff88042a689f60 ffff88083fb3bfd8 [18271.065777] ffff88083fb3bfd8 ffff88083fb3bfd8 ffff88042a689f60 ffff880417ad6000 [18271.074706] 7fffffffffffffff ffff880416375400 ffff880416375400 ffff880416375000 [18271.083641] Call Trace: [18271.086986] [] schedule+0x29/0x70 [18271.093142] [] schedule_timeout+0x239/0x2c0 [18271.100276] [] ? get_work_pool+0x3e/0x70 [18271.107111] [] ? flush_work+0x15b/0x1c0 [18271.113845] [] ? set_room+0x68/0xc0 [18271.120201] [] n_tty_read+0x27c/0x990 [18271.126743] [] ? wake_up_state+0x20/0x20 [18271.133576] [] tty_read+0x8d/0x100 [18271.139821] [] vfs_read+0x9e/0x170 [18271.146053] [] SyS_read+0x7f/0xe0 [18271.152177] [] system_call_fastpath+0x16/0x1b [18271.159475] kworker/16:2 S ffff88042e216480 0 1341 2 0x00000080 [18271.167957] ffff8800b2c0fe50 0000000000000046 ffff8800b3cbde20 ffff8800b2c0ffd8 [18271.176868] ffff8800b2c0ffd8 ffff8800b2c0ffd8 ffff8800b3cbde20 ffff88042e2164a0 [18271.185761] ffff88042c790fb0 ffff8800b3cbde20 ffff88042c790f80 ffff88042e216480 [18271.194665] Call Trace: [18271.198003] [] schedule+0x29/0x70 [18271.204145] [] worker_thread+0x1d9/0x410 [18271.210970] [] ? rescuer_thread+0x460/0x460 [18271.218094] [] kthread+0xcf/0xe0 [18271.224148] [] ? kthread_create_on_node+0x140/0x140 [18271.232044] [] ret_from_fork+0x58/0x90 [18271.238659] [] ? kthread_create_on_node+0x140/0x140 [18271.246544] bioset S ffff88041b24c780 0 1381 2 0x00000080 [18271.255057] ffff880823e13e08 0000000000000046 ffff8800b3cb8fb0 ffff880823e13fd8 [18271.263973] ffff880823e13fd8 ffff880823e13fd8 ffff8800b3cb8fb0 ffff8800b3cb8fb0 [18271.272886] ffff88041b24c780 ffff8800b3936680 ffff88041b24c7b0 ffff88041b24c780 [18271.281784] Call Trace: [18271.285119] [] schedule+0x29/0x70 [18271.291280] [] rescuer_thread+0x359/0x460 [18271.298210] [] ? __schedule+0x3b0/0x990 [18271.304940] [] ? manage_workers.isra.24+0x2d0/0x2d0 [18271.312829] [] kthread+0xcf/0xe0 [18271.318873] [] ? kthread_create_on_node+0x140/0x140 [18271.326755] [] ret_from_fork+0x58/0x90 [18271.333380] [] ? kthread_create_on_node+0x140/0x140 [18271.341270] tuned S 0000000000000000 0 1673 1 0x00000080 [18271.349771] ffff880829cbf8e8 0000000000000086 ffff88083fde3ec0 ffff880829cbffd8 [18271.358692] ffff880829cbffd8 ffff880829cbffd8 ffff88083fde3ec0 ffff880829cbfac8 [18271.367617] 0000000005f5e100 0000000000000000 0000000000000000 0000000000000000 [18271.376548] Call Trace: [18271.379895] [] schedule+0x29/0x70 [18271.386067] [] schedule_hrtimeout_range_clock+0xb2/0x150 [18271.394472] [] ? hrtimer_get_res+0x50/0x50 [18271.401509] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [18271.410105] [] schedule_hrtimeout_range+0x13/0x20 [18271.417815] [] poll_schedule_timeout+0x55/0xb0 [18271.425238] [] do_select+0x6d1/0x7c0 [18271.431683] [] ? __inc_zone_page_state+0x32/0x60 [18271.439306] [] ? do_page_add_anon_rmap+0x9a/0xc0 [18271.446918] [] ? page_add_anon_rmap+0x10/0x20 [18271.454238] [] ? __inc_zone_page_state+0x32/0x60 [18271.461855] [] ? do_page_add_anon_rmap+0x9a/0xc0 [18271.469472] [] ? page_add_anon_rmap+0x10/0x20 [18271.476792] [] ? remove_migration_pte+0xc1/0x320 [18271.484417] [] ? do_pages_stat+0x1a0/0x1a0 [18271.491456] [] ? rmap_walk+0x1c4/0x250 [18271.498121] [] ? __mem_cgroup_uncharge_common+0xcd/0x2f0 [18271.506539] [] ? putback_lru_page+0x53/0xb0 [18271.513678] [] ? migrate_pages+0x43b/0x790 [18271.520710] [] ? should_numa_migrate_memory+0x5b/0x150 [18271.528911] [] core_sys_select+0x1db/0x300 [18271.535952] [] ? handle_mm_fault+0x6d7/0xfe0 [18271.543190] [] ? ktime_get_ts64+0x4c/0xf0 [18271.550126] [] ? ktime_get_ts64+0x4c/0xf0 [18271.557052] [] SyS_select+0xba/0x110 [18271.563492] [] system_call_fastpath+0x16/0x1b [18271.570806] gmain S ffff880826e3fba4 0 1940 1 0x00000080 [18271.579336] ffff880826e3fa30 0000000000000086 ffff880825ed0fb0 ffff880826e3ffd8 [18271.588272] ffff880826e3ffd8 ffff880826e3ffd8 ffff880825ed0fb0 0000000000000000 [18271.597198] ffff880826e3fc90 00000000fffffffc 0000000000000000 ffff880826e3fba4 [18271.606128] Call Trace: [18271.609474] [] schedule+0x29/0x70 [18271.615618] [] schedule_hrtimeout_range_clock+0x12d/0x150 [18271.624119] [] ? add_wait_queue+0x3c/0x50 [18271.631062] [] schedule_hrtimeout_range+0x13/0x20 [18271.638772] [] poll_schedule_timeout+0x55/0xb0 [18271.646200] [] do_sys_poll+0x4cd/0x580 [18271.652843] [] ? zone_statistics+0x89/0xa0 [18271.659874] [] ? zone_statistics+0x89/0xa0 [18271.666892] [] ? get_page_from_freelist+0x51a/0x9f0 [18271.674782] [] ? poll_select_copy_remaining+0x150/0x150 [18271.683066] [] ? filemap_fault+0x215/0x410 [18271.690106] [] ? ext4_filemap_fault+0x41/0x50 [ext4] [18271.698093] [] ? unlock_page+0x2b/0x30 [18271.704726] [] ? do_read_fault.isra.42+0xe4/0x130 [18271.712436] [] ? eventfd_ctx_read+0x67/0x210 [18271.719633] [] ? vfs_read+0xf7/0x170 [18271.726060] [] SyS_poll+0x74/0x110 [18271.732285] [] ? __audit_syscall_exit+0x1e6/0x280 [18271.739959] [] system_call_fastpath+0x16/0x1b [18271.747252] tuned S ffff88082bddfbac 0 1948 1 0x00000080 [18271.755733] ffff88082bddfa30 0000000000000086 ffff880825ed1f60 ffff88082bddffd8 [18271.764645] ffff88082bddffd8 ffff88082bddffd8 ffff880825ed1f60 0000000000000000 [18271.773561] ffff88082bddfc90 00000000fffffffc 0000000000000000 ffff88082bddfbac [18271.782504] Call Trace: [18271.785831] [] schedule+0x29/0x70 [18271.791970] [] schedule_hrtimeout_range_clock+0x12d/0x150 [18271.800454] [] ? add_wait_queue+0x3c/0x50 [18271.807387] [] ? add_wait_queue+0x3c/0x50 [18271.814308] [] schedule_hrtimeout_range+0x13/0x20 [18271.822008] [] poll_schedule_timeout+0x55/0xb0 [18271.829415] [] do_sys_poll+0x4cd/0x580 [18271.836060] [] ? dequeue_task_fair+0x41e/0x660 [18271.843471] [] ? sched_clock_cpu+0x85/0xc0 [18271.850501] [] ? __switch_to+0x15c/0x4c0 [18271.857322] [] ? __enqueue_entity+0x78/0x80 [18271.864440] [] ? poll_select_copy_remaining+0x150/0x150 [18271.872717] [] ? poll_select_copy_remaining+0x150/0x150 [18271.880985] [] ? ttwu_do_wakeup+0x19/0xd0 [18271.887893] [] ? ttwu_do_activate.constprop.91+0x5d/0x70 [18271.896261] [] ? try_to_wake_up+0x1c8/0x320 [18271.903383] [] ? wake_up_state+0x10/0x20 [18271.910210] [] ? wake_futex+0x66/0x80 [18271.916742] [] ? futex_wake+0x80/0x160 [18271.923356] [] ? do_futex+0x122/0x5b0 [18271.929855] [] ? wake_up_state+0x20/0x20 [18271.936651] [] SyS_poll+0x74/0x110 [18271.942857] [] ? __audit_syscall_exit+0x1e6/0x280 [18271.950525] [] system_call_fastpath+0x16/0x1b [18271.957796] tuned S 0000000000000000 0 1952 1 0x00000080 [18271.966281] ffff88082475f8e8 0000000000000086 ffff880825ed2f10 ffff88082475ffd8 [18271.975181] ffff88082475ffd8 ffff88082475ffd8 ffff880825ed2f10 ffff88082475fac8 [18271.984070] 00000000000f423f 0000000000000000 0000000000000000 0000000000000000 [18271.992951] Call Trace: [18271.996267] [] schedule+0x29/0x70 [18272.002390] [] schedule_hrtimeout_range_clock+0xb2/0x150 [18272.010751] [] ? hrtimer_get_res+0x50/0x50 [18272.017761] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [18272.026324] [] schedule_hrtimeout_range+0x13/0x20 [18272.034025] [] poll_schedule_timeout+0x55/0xb0 [18272.041422] [] do_select+0x6d1/0x7c0 [18272.047846] [] ? list_del+0xd/0x30 [18272.054081] [] ? __inc_zone_page_state+0x32/0x60 [18272.061665] [] ? do_page_add_anon_rmap+0x9a/0xc0 [18272.069249] [] ? page_add_anon_rmap+0x10/0x20 [18272.076552] [] ? remove_migration_pte+0xc1/0x320 [18272.084150] [] ? do_pages_stat+0x1a0/0x1a0 [18272.091153] [] ? rmap_walk+0x1c4/0x250 [18272.097759] [] ? __mem_cgroup_uncharge_common+0xcd/0x2f0 [18272.106111] [] ? putback_lru_page+0x53/0xb0 [18272.113216] [] ? migrate_pages+0x43b/0x790 [18272.120212] [] core_sys_select+0x1db/0x300 [18272.127198] [] ? handle_mm_fault+0x6d7/0xfe0 [18272.134378] [] ? ktime_get_ts64+0x4c/0xf0 [18272.141251] [] ? ktime_get_ts64+0x4c/0xf0 [18272.148115] [] SyS_select+0xba/0x110 [18272.154508] [] system_call_fastpath+0x16/0x1b [18272.161766] tuned S 0000000000000000 0 1963 1 0x00000080 [18272.170242] ffff880829c23db8 0000000000000086 ffff88082d54de20 ffff880829c23fd8 [18272.179129] ffff880829c23fd8 ffff880829c23fd8 ffff88082d54de20 0000000000000000 [18272.188011] 0000000000000000 ffff88082bfa33e0 ffff88082d54de20 0000000000000000 [18272.196878] Call Trace: [18272.200180] [] schedule+0x29/0x70 [18272.206302] [] schedule_hrtimeout_range_clock+0x12d/0x150 [18272.214749] [] ? wake_up_state+0x10/0x20 [18272.221559] [] ? wake_futex+0x66/0x80 [18272.228071] [] ? futex_wake+0x80/0x160 [18272.234673] [] schedule_hrtimeout_range+0x13/0x20 [18272.242342] [] ep_poll+0x23e/0x360 [18272.248562] [] ? __do_page_fault+0x171/0x450 [18272.255735] [] ? SyS_epoll_ctl+0x118/0xa20 [18272.262730] [] ? wake_up_state+0x20/0x20 [18272.269534] [] SyS_epoll_wait+0xed/0x120 [18272.276340] [] system_call_fastpath+0x16/0x1b [18272.283634] rsyslogd S 0000000000000000 0 1678 1 0x00000080 [18272.292145] ffff88082e5f38e8 0000000000000082 ffff880823a68000 ffff88082e5f3fd8 [18272.301051] ffff88082e5f3fd8 ffff88082e5f3fd8 ffff880823a68000 ffff88082e5f3ac8 [18272.309940] 0000000005f5e100 0000000000000000 0000000000000000 0000000000000000 [18272.318841] Call Trace: [18272.322168] [] schedule+0x29/0x70 [18272.328294] [] schedule_hrtimeout_range_clock+0xb2/0x150 [18272.336652] [] ? hrtimer_get_res+0x50/0x50 [18272.343664] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [18272.352224] [] schedule_hrtimeout_range+0x13/0x20 [18272.359908] [] poll_schedule_timeout+0x55/0xb0 [18272.367298] [] do_select+0x6d1/0x7c0 [18272.373713] [] ? ep_poll_callback+0xee/0x210 [18272.380904] [] ? __wake_up_sync_key+0x4f/0x60 [18272.388197] [] ? sock_def_readable+0x3a/0x70 [18272.395390] [] ? unix_dgram_sendmsg+0x3a8/0x720 [18272.402876] [] ? sock_sendmsg+0xb0/0xf0 [18272.409589] [] ? __switch_to+0x15c/0x4c0 [18272.416391] [] ? __enqueue_entity+0x78/0x80 [18272.423483] [] core_sys_select+0x1db/0x300 [18272.430483] [] ? __d_free+0x35/0x40 [18272.436792] [] ? d_free+0x60/0x70 [18272.442904] [] ? dentry_kill+0x14e/0x1b0 [18272.449693] [] ? mntput_no_expire+0x3e/0x120 [18272.456873] [] ? mntput+0x24/0x40 [18272.462981] [] ? __fput+0x183/0x260 [18272.469290] [] ? ktime_get_ts64+0x4c/0xf0 [18272.476175] [] SyS_select+0xba/0x110 [18272.482588] [] system_call_fastpath+0x16/0x1b [18272.489869] in:imjournal S ffff88082776fba4 0 1689 1 0x00000080 [18272.498367] ffff88082776fa30 0000000000000082 ffff88082c783ec0 ffff88082776ffd8 [18272.507290] ffff88082776ffd8 ffff88082776ffd8 ffff88082c783ec0 0000000000000000 [18272.516193] ffff88082776fc90 00000000fffffffc 0000000000000000 ffff88082776fba4 [18272.525086] Call Trace: [18272.528408] [] schedule+0x29/0x70 [18272.534535] [] schedule_hrtimeout_range_clock+0x12d/0x150 [18272.543007] [] ? ext4_free_blocks+0x67e/0xb80 [ext4] [18272.550977] [] ? add_wait_queue+0x3c/0x50 [18272.557884] [] schedule_hrtimeout_range+0x13/0x20 [18272.565574] [] poll_schedule_timeout+0x55/0xb0 [18272.572975] [] do_sys_poll+0x4cd/0x580 [18272.579604] [] ? jbd2_journal_release_jbd_inode+0x100/0x130 [jbd2] [18272.588952] [] ? __wake_up+0x44/0x50 [18272.595399] [] ? __enqueue_entity+0x78/0x80 [18272.602525] [] ? enqueue_entity+0x26c/0xb60 [18272.609645] [] ? enqueue_task_fair+0x208/0x6c0 [18272.617044] [] ? poll_select_copy_remaining+0x150/0x150 [18272.625338] [] ? ttwu_do_activate.constprop.91+0x5d/0x70 [18272.633710] [] ? try_to_wake_up+0x1c8/0x320 [18272.640816] [] ? __slab_free+0x81/0x2f0 [18272.647541] [] ? dentry_kill+0x14e/0x1b0 [18272.654356] [] ? wake_up_state+0x10/0x20 [18272.661158] [] ? wake_futex+0x66/0x80 [18272.667670] [] ? futex_wake_op+0x332/0x570 [18272.674661] [] ? do_futex+0x172/0x5b0 [18272.681171] [] ? wake_up_atomic_t+0x30/0x30 [18272.688250] [] SyS_poll+0x74/0x110 [18272.694466] [] ? __audit_syscall_exit+0x1e6/0x280 [18272.702131] [] system_call_fastpath+0x16/0x1b [18272.709398] rs:main Q:Reg S ffffc900037c3900 0 1690 1 0x00000080 [18272.717864] ffff8808267fbcd0 0000000000000082 ffff88082c784e70 ffff8808267fbfd8 [18272.726766] ffff8808267fbfd8 ffff8808267fbfd8 ffff88082c784e70 ffff88082c784e70 [18272.735672] ffff88082c784e70 0000000000000000 ffff8808267fbde0 ffffc900037c3900 [18272.744578] Call Trace: [18272.747890] [] schedule+0x29/0x70 [18272.754022] [] futex_wait_queue_me+0xc4/0x120 [18272.761331] [] futex_wait+0x179/0x280 [18272.767858] [] ? ext4_file_write+0xdb/0x470 [ext4] [18272.775646] [] do_futex+0xfe/0x5b0 [18272.781883] [] SyS_futex+0x80/0x180 [18272.788217] [] system_call_fastpath+0x16/0x1b [18272.795531] munged S ffff88042c2d7e30 0 1681 1 0x00000080 [18272.804051] ffff88042c2d7cc0 0000000000000082 ffff88042bae2f10 ffff88042c2d7fd8 [18272.812982] ffff88042c2d7fd8 ffff88042c2d7fd8 ffff88042bae2f10 ffff880414170490 [18272.821908] ffff880414170490 ffff88042c2d7e34 ffff880414170400 ffff88042c2d7e30 [18272.830842] Call Trace: [18272.834207] [] schedule+0x29/0x70 [18272.840381] [] schedule_timeout+0x239/0x2c0 [18272.847518] [] __skb_recv_datagram+0x3ca/0x460 [18272.854934] [] ? datagram_poll+0x100/0x100 [18272.861957] [] skb_recv_datagram+0x41/0x60 [18272.868972] [] unix_accept+0x97/0x170 [18272.875509] [] SYSC_accept4+0x103/0x220 [18272.882227] [] ? __sb_end_write+0x31/0x60 [18272.889132] [] ? vfs_write+0x172/0x1e0 [18272.895739] [] SyS_accept4+0xe/0x10 [18272.902067] [] SyS_accept+0x10/0x20 [18272.908381] [] system_call_fastpath+0x16/0x1b [18272.915661] munged S ffffc90003652b40 0 1683 1 0x00000080 [18272.924161] ffff88042d647cd0 0000000000000082 ffff88042e97bec0 ffff88042d647fd8 [18272.933051] ffff88042d647fd8 ffff88042d647fd8 ffff88042e97bec0 ffff88042e97bec0 [18272.941941] ffff88042e97bec0 ffff88042d647d60 ffff88042d647de0 ffffc90003652b40 [18272.950832] Call Trace: [18272.954157] [] schedule+0x29/0x70 [18272.960285] [] futex_wait_queue_me+0xc4/0x120 [18272.967591] [] futex_wait+0x179/0x280 [18272.974099] [] ? hrtimer_get_res+0x50/0x50 [18272.981103] [] ? futex_wait_queue_me+0xa2/0x120 [18272.988601] [] do_futex+0xfe/0x5b0 [18272.994824] [] SyS_futex+0x80/0x180 [18273.001154] [] system_call_fastpath+0x16/0x1b [18273.008456] munged S ffffc900035b9440 0 1684 1 0x00000080 [18273.016963] ffff88041929fcd0 0000000000000082 ffff88042d140000 ffff88041929ffd8 [18273.025886] ffff88041929ffd8 ffff88041929ffd8 ffff88042d140000 ffff88042d140000 [18273.034813] ffff88042d140000 0000000000000000 ffff88041929fde0 ffffc900035b9440 [18273.043740] Call Trace: [18273.047078] [] schedule+0x29/0x70 [18273.053228] [] futex_wait_queue_me+0xc4/0x120 [18273.060540] [] futex_wait+0x179/0x280 [18273.067067] [] ? dequeue_entity+0x11c/0x5d0 [18273.074182] [] ? dequeue_task_fair+0x41e/0x660 [18273.081585] [] do_futex+0xfe/0x5b0 [18273.087797] [] ? schedule+0x29/0x70 [18273.094107] [] ? recalc_sigpending+0x1b/0x50 [18273.101302] [] SyS_futex+0x80/0x180 [18273.107616] [] system_call_fastpath+0x16/0x1b [18273.114898] munged S ffffc900035b9440 0 1685 1 0x00000080 [18273.123394] ffff880419fe3cd0 0000000000000082 ffff88042d146dd0 ffff880419fe3fd8 [18273.132318] ffff880419fe3fd8 ffff880419fe3fd8 ffff88042d146dd0 ffff88042d146dd0 [18273.141223] ffff88042d146dd0 0000000000000000 ffff880419fe3de0 ffffc900035b9440 [18273.150132] Call Trace: [18273.153459] [] schedule+0x29/0x70 [18273.159605] [] futex_wait_queue_me+0xc4/0x120 [18273.166898] [] futex_wait+0x179/0x280 [18273.173409] [] do_futex+0xfe/0x5b0 [18273.179622] [] ? recalc_sigpending+0x1b/0x50 [18273.186815] [] SyS_futex+0x80/0x180 [18273.193134] [] system_call_fastpath+0x16/0x1b [18273.200431] xinetd S ffff880825703bb4 0 1687 1 0x00000080 [18273.208930] ffff880825703a30 0000000000000082 ffff88082c406dd0 ffff880825703fd8 [18273.217844] ffff880825703fd8 ffff880825703fd8 ffff88082c406dd0 0000000000000000 [18273.226766] ffff880825703c90 00000000fffffffc 0000000000000000 ffff880825703bb4 [18273.235691] Call Trace: [18273.239040] [] schedule+0x29/0x70 [18273.245206] [] schedule_hrtimeout_range_clock+0x12d/0x150 [18273.253709] [] ? add_wait_queue+0x3c/0x50 [18273.260668] [] ? add_wait_queue+0x3c/0x50 [18273.267593] [] schedule_hrtimeout_range+0x13/0x20 [18273.275292] [] poll_schedule_timeout+0x55/0xb0 [18273.282709] [] do_sys_poll+0x4cd/0x580 [18273.289345] [] ? __wake_up_common+0x58/0x90 [18273.296468] [] ? __wake_up_sync_key+0x4f/0x60 [18273.303772] [] ? sock_def_readable+0x3a/0x70 [18273.310974] [] ? unix_dgram_sendmsg+0x3a8/0x720 [18273.318495] [] ? poll_select_copy_remaining+0x150/0x150 [18273.326779] [] ? poll_select_copy_remaining+0x150/0x150 [18273.335062] [] ? poll_select_copy_remaining+0x150/0x150 [18273.343339] [] ? unlock_page+0x2b/0x30 [18273.349956] [] ? do_read_fault.isra.42+0xe4/0x130 [18273.357639] [] ? handle_mm_fault+0x6b1/0xfe0 [18273.364818] [] ? SYSC_sendto+0x121/0x1c0 [18273.371616] [] ? __do_page_fault+0x171/0x450 [18273.378804] [] SyS_poll+0x74/0x110 [18273.385007] [] ? page_fault+0x28/0x30 [18273.391501] [] system_call_fastpath+0x16/0x1b [18273.398772] sshd S 0000000000000005 0 1692 1 0x00000080 [18273.407266] ffff88042b71b8e8 0000000000000086 ffff88042cb00000 ffff88042b71bfd8 [18273.416168] ffff88042b71bfd8 ffff88042b71bfd8 ffff88042cb00000 0000000000000000 [18273.425065] ffff88042b71bad0 00000000fffffffc 0000000000000005 0000000000000005 [18273.433947] Call Trace: [18273.437276] [] schedule+0x29/0x70 [18273.443411] [] schedule_hrtimeout_range_clock+0x12d/0x150 [18273.451879] [] ? add_wait_queue+0x3c/0x50 [18273.458785] [] ? __pollwait+0x7f/0xf0 [18273.465312] [] schedule_hrtimeout_range+0x13/0x20 [18273.472985] [] poll_schedule_timeout+0x55/0xb0 [18273.480371] [] do_select+0x6d1/0x7c0 [18273.486787] [] ? poll_select_copy_remaining+0x150/0x150 [18273.495045] [] ? poll_select_copy_remaining+0x150/0x150 [18273.503305] [] ? put_dec+0x72/0x90 [18273.509512] [] ? number.isra.2+0x323/0x360 [18273.516503] [] ? fsnotify_clear_marks_by_inode+0xa7/0x140 [18273.524949] [] ? vsnprintf+0x441/0x6a0 [18273.531567] [] ? destroy_inode+0x38/0x60 [18273.538373] [] ? mix_pool_bytes.constprop.18+0x56/0xa0 [18273.546534] [] core_sys_select+0x1db/0x300 [18273.553525] [] ? wait_consider_task+0xa5/0xb80 [18273.560898] [] ? __restore_xstate_sig+0x99/0x5d0 [18273.568471] [] SyS_select+0xba/0x110 [18273.574868] [] system_call_fastpath+0x16/0x1b [18273.582137] automount S 0000000000000000 0 1694 1 0x00000080 [18273.590585] ffff880825a2fd90 0000000000000082 ffff88082d6f4e70 ffff880825a2ffd8 [18273.599473] ffff880825a2ffd8 ffff880825a2ffd8 ffff88082d6f4e70 ffff88082d6f4e70 [18273.608350] 7fffffffffffffff ffff880825a2fed0 ffff88082d6f4e70 0000000000000000 [18273.617233] Call Trace: [18273.620530] [] schedule+0x29/0x70 [18273.626640] [] schedule_timeout+0x239/0x2c0 [18273.633725] [] ? unlock_page+0x2b/0x30 [18273.640333] [] ? do_read_fault.isra.42+0xe4/0x130 [18273.647997] [] ? recalc_sigpending+0x1b/0x50 [18273.655186] [] schedule_timeout_interruptible+0x1e/0x20 [18273.663452] [] do_sigtimedwait+0x1c2/0x2a0 [18273.670457] [] SYSC_rt_sigtimedwait+0x85/0xf0 [18273.677755] [] ? __do_page_fault+0x171/0x450 [18273.684962] [] ? do_page_fault+0x35/0x90 [18273.691780] [] SyS_rt_sigtimedwait+0xe/0x10 [18273.698886] [] system_call_fastpath+0x16/0x1b [18273.706193] automount S ffffc90003932180 0 1698 1 0x00000080 [18273.714692] ffff88042d107cd0 0000000000000082 ffff88042b924e70 ffff88042d107fd8 [18273.723637] ffff88042d107fd8 ffff88042d107fd8 ffff88042b924e70 ffff88042b924e70 [18273.732572] ffff88042b924e70 ffff88042d107d60 ffff88042d107de0 ffffc90003932180 [18273.741516] Call Trace: [18273.744867] [] schedule+0x29/0x70 [18273.751016] [] futex_wait_queue_me+0xc4/0x120 [18273.758318] [] futex_wait+0x179/0x280 [18273.764848] [] ? hrtimer_get_res+0x50/0x50 [18273.771865] [] ? futex_wait_queue_me+0xa2/0x120 [18273.779362] [] do_futex+0xfe/0x5b0 [18273.785604] [] ? __do_page_fault+0x171/0x450 [18273.792800] [] SyS_futex+0x80/0x180 [18273.799123] [] system_call_fastpath+0x16/0x1b [18273.806420] automount S ffffc90003791700 0 1699 1 0x00000080 [18273.814907] ffff88042d16bcd0 0000000000000082 ffff88042b920000 ffff88042d16bfd8 [18273.823826] ffff88042d16bfd8 ffff88042d16bfd8 ffff88042b920000 ffff88042b920000 [18273.832740] ffff88042b920000 0000000000000000 ffff88042d16bde0 ffffc90003791700 [18273.841665] Call Trace: [18273.845006] [] schedule+0x29/0x70 [18273.851167] [] futex_wait_queue_me+0xc4/0x120 [18273.858501] [] futex_wait+0x179/0x280 [18273.865039] [] ? sched_slice.isra.55+0x66/0xc0 [18273.872447] [] ? get_futex_key+0x1c8/0x2c0 [18273.879486] [] do_futex+0xfe/0x5b0 [18273.885734] [] ? wake_up_new_task+0x10c/0x170 [18273.893049] [] ? do_fork+0xe8/0x2c0 [18273.899396] [] SyS_futex+0x80/0x180 [18273.905739] [] system_call_fastpath+0x16/0x1b [18273.913042] automount S ffff880418ffbbb4 0 1746 1 0x00000080 [18273.921558] ffff880418ffba30 0000000000000082 ffff88042bd03ec0 ffff880418ffbfd8 [18273.930474] ffff880418ffbfd8 ffff880418ffbfd8 ffff88042bd03ec0 0000000000000000 [18273.939402] ffff880418ffbc90 00000000fffffffc 0000000000000000 ffff880418ffbbb4 [18273.948344] Call Trace: [18273.951694] [] schedule+0x29/0x70 [18273.957845] [] schedule_hrtimeout_range_clock+0x12d/0x150 [18273.966341] [] ? add_wait_queue+0x3c/0x50 [18273.973279] [] schedule_hrtimeout_range+0x13/0x20 [18273.980987] [] poll_schedule_timeout+0x55/0xb0 [18273.988406] [] do_sys_poll+0x4cd/0x580 [18273.995044] [] ? __d_lookup+0x120/0x160 [18274.001776] [] ? mntput+0x24/0x40 [18274.007920] [] ? terminate_walk+0x51/0x60 [18274.014839] [] ? __enqueue_entity+0x78/0x80 [18274.021958] [] ? enqueue_entity+0x26c/0xb60 [18274.029075] [] ? enqueue_task_fair+0x208/0x6c0 [18274.036493] [] ? poll_select_copy_remaining+0x150/0x150 [18274.044780] [] ? poll_select_copy_remaining+0x150/0x150 [18274.053042] [] ? poll_select_copy_remaining+0x150/0x150 [18274.061302] [] ? unlock_page+0x2b/0x30 [18274.067906] [] ? do_read_fault.isra.42+0xe4/0x130 [18274.075591] [] ? handle_mm_fault+0x6b1/0xfe0 [18274.082772] [] ? do_futex+0x172/0x5b0 [18274.089275] [] ? __do_page_fault+0x171/0x450 [18274.096462] [] SyS_poll+0x74/0x110 [18274.102672] [] ? page_fault+0x28/0x30 [18274.109183] [] system_call_fastpath+0x16/0x1b [18274.116469] automount S ffff88042ae1fbb4 0 1791 1 0x00000080 [18274.124943] ffff88042ae1fa30 0000000000000082 ffff88042bacde20 ffff88042ae1ffd8 [18274.133852] ffff88042ae1ffd8 ffff88042ae1ffd8 ffff88042bacde20 0000000000000000 [18274.142755] ffff88042ae1fc90 00000000fffffffc 0000000000000000 ffff88042ae1fbb4 [18274.151670] Call Trace: [18274.154987] [] schedule+0x29/0x70 [18274.161116] [] schedule_hrtimeout_range_clock+0x12d/0x150 [18274.169585] [] ? add_wait_queue+0x3c/0x50 [18274.176503] [] schedule_hrtimeout_range+0x13/0x20 [18274.184190] [] poll_schedule_timeout+0x55/0xb0 [18274.191599] [] do_sys_poll+0x4cd/0x580 [18274.198200] [] ? __d_lookup+0x120/0x160 [18274.204909] [] ? mntput+0x24/0x40 [18274.211031] [] ? terminate_walk+0x51/0x60 [18274.217932] [] ? __enqueue_entity+0x78/0x80 [18274.225025] [] ? enqueue_entity+0x26c/0xb60 [18274.232112] [] ? pcpu_alloc+0x354/0x640 [18274.238809] [] ? enqueue_task_fair+0x208/0x6c0 [18274.246189] [] ? poll_select_copy_remaining+0x150/0x150 [18274.254460] [] ? poll_select_copy_remaining+0x150/0x150 [18274.262715] [] ? poll_select_copy_remaining+0x150/0x150 [18274.270945] [] ? wake_futex+0x66/0x80 [18274.277438] [] ? futex_wake_op+0x332/0x570 [18274.284403] [] ? do_futex+0x172/0x5b0 [18274.290872] [] ? autofs_dev_ioctl+0x13/0x20 [18274.297914] [] ? do_vfs_ioctl+0x2d5/0x4b0 [18274.304763] [] SyS_poll+0x74/0x110 [18274.310930] [] ? __audit_syscall_exit+0x1e6/0x280 [18274.318566] [] system_call_fastpath+0x16/0x1b [18274.325801] automount S ffff8804140f7bb4 0 1825 1 0x00000080 [18274.334249] ffff8804140f7a30 0000000000000082 ffff8800360d6dd0 ffff8804140f7fd8 [18274.343117] ffff8804140f7fd8 ffff8804140f7fd8 ffff8800360d6dd0 0000000000000000 [18274.351986] ffff8804140f7c90 00000000fffffffc 0000000000000000 ffff8804140f7bb4 [18274.360844] Call Trace: [18274.364124] [] schedule+0x29/0x70 [18274.370244] [] schedule_hrtimeout_range_clock+0x12d/0x150 [18274.378690] [] ? add_wait_queue+0x3c/0x50 [18274.385588] [] schedule_hrtimeout_range+0x13/0x20 [18274.393273] [] poll_schedule_timeout+0x55/0xb0 [18274.400662] [] do_sys_poll+0x4cd/0x580 [18274.407286] [] ? sched_clock+0x9/0x10 [18274.413800] [] ? sched_clock_cpu+0x85/0xc0 [18274.420789] [] ? __switch_to+0x15c/0x4c0 [18274.427594] [] ? timerqueue_del+0x24/0x70 [18274.434494] [] ? poll_select_copy_remaining+0x150/0x150 [18274.442756] [] ? poll_select_copy_remaining+0x150/0x150 [18274.451009] [] ? poll_select_copy_remaining+0x150/0x150 [18274.459263] [] ? hrtimer_get_res+0x50/0x50 [18274.466247] [] ? get_futex_key+0x1c8/0x2c0 [18274.473213] [] ? futex_wake+0x80/0x160 [18274.479785] [] ? do_futex+0x122/0x5b0 [18274.486256] [] ? wake_up_new_task+0x10c/0x170 [18274.493502] [] ? do_fork+0xe8/0x2c0 [18274.499769] [] SyS_poll+0x74/0x110 [18274.505933] [] ? __audit_syscall_exit+0x1e6/0x280 [18274.513565] [] system_call_fastpath+0x16/0x1b [18274.520797] automount S ffff8800b3f47bb4 0 1896 1 0x00000080 [18274.529247] ffff8800b3f47a30 0000000000000082 ffff88042a455e20 ffff8800b3f47fd8 [18274.538097] ffff8800b3f47fd8 ffff8800b3f47fd8 ffff88042a455e20 0000000000000000 [18274.546955] ffff8800b3f47c90 00000000fffffffc 0000000000000000 ffff8800b3f47bb4 [18274.555821] Call Trace: [18274.559104] [] schedule+0x29/0x70 [18274.565214] [] schedule_hrtimeout_range_clock+0x12d/0x150 [18274.573654] [] ? add_wait_queue+0x3c/0x50 [18274.580551] [] schedule_hrtimeout_range+0x13/0x20 [18274.588218] [] poll_schedule_timeout+0x55/0xb0 [18274.595598] [] do_sys_poll+0x4cd/0x580 [18274.602197] [] ? __d_lookup+0x120/0x160 [18274.608895] [] ? mntput+0x24/0x40 [18274.615005] [] ? terminate_walk+0x51/0x60 [18274.621902] [] ? __enqueue_entity+0x78/0x80 [18274.628993] [] ? enqueue_entity+0x26c/0xb60 [18274.636081] [] ? enqueue_task_fair+0x208/0x6c0 [18274.643466] [] ? poll_select_copy_remaining+0x150/0x150 [18274.651723] [] ? poll_select_copy_remaining+0x150/0x150 [18274.659970] [] ? poll_select_copy_remaining+0x150/0x150 [18274.668213] [] ? wake_futex+0x66/0x80 [18274.674688] [] ? futex_wake_op+0x332/0x570 [18274.681645] [] ? do_futex+0x172/0x5b0 [18274.688116] [] ? autofs_dev_ioctl+0x13/0x20 [18274.695169] [] ? do_vfs_ioctl+0x2d5/0x4b0 [18274.702017] [] SyS_poll+0x74/0x110 [18274.708182] [] ? __audit_syscall_exit+0x1e6/0x280 [18274.715797] [] system_call_fastpath+0x16/0x1b [18274.723037] automount S ffff8801726f7ba4 0 8921 1 0x00000080 [18274.731472] ffff8801726f7a30 0000000000000082 ffff8801710d0fb0 ffff8801726f7fd8 [18274.740336] ffff8801726f7fd8 ffff8801726f7fd8 ffff8801710d0fb0 0000000000000000 [18274.749199] ffff8801726f7c90 00000000fffffffc 0000000000000000 ffff8801726f7ba4 [18274.758048] Call Trace: [18274.761330] [] schedule+0x29/0x70 [18274.767429] [] schedule_hrtimeout_range_clock+0x12d/0x150 [18274.775862] [] ? add_wait_queue+0x3c/0x50 [18274.782733] [] schedule_hrtimeout_range+0x13/0x20 [18274.790405] [] poll_schedule_timeout+0x55/0xb0 [18274.797791] [] do_sys_poll+0x4cd/0x580 [18274.804407] [] ? memcg_check_events+0xbe/0x200 [18274.811793] [] ? poll_select_copy_remaining+0x150/0x150 [18274.820058] [] ? lru_cache_add+0xe/0x10 [18274.826765] [] ? page_add_new_anon_rmap+0x91/0x150 [18274.834561] [] ? wp_page_copy.isra.56+0x2e6/0x470 [18274.842260] [] ? do_wp_page+0xe5/0x530 [18274.848885] [] ? cpumask_next_and+0x35/0x50 [18274.855990] [] ? handle_mm_fault+0x72c/0xfe0 [18274.863200] [] ? recalc_sigpending+0x1b/0x50 [18274.870402] [] ? __set_task_blocked+0x41/0xa0 [18274.877684] [] SyS_poll+0x74/0x110 [18274.883898] [] ? __audit_syscall_exit+0x1e6/0x280 [18274.891575] [] system_call_fastpath+0x16/0x1b [18274.898850] kworker/30:1H S ffff88082db96840 0 1758 2 0x00000080 [18274.907328] ffff8808273abe50 0000000000000046 ffff88082d6f0fb0 ffff8808273abfd8 [18274.916233] ffff8808273abfd8 ffff8808273abfd8 ffff88082d6f0fb0 ffff88082db96860 [18274.925110] ffff88042bd16530 ffff88082d6f0fb0 ffff88042bd16500 ffff88082db96840 [18274.933991] Call Trace: [18274.937299] [] schedule+0x29/0x70 [18274.943428] [] worker_thread+0x1d9/0x410 [18274.950230] [] ? rescuer_thread+0x460/0x460 [18274.957330] [] kthread+0xcf/0xe0 [18274.963360] [] ? kthread_create_on_node+0x140/0x140 [18274.971242] [] ret_from_fork+0x58/0x90 [18274.977857] [] ? kthread_create_on_node+0x140/0x140 [18274.985741] master S ffff8800b2c07ea8 0 2314 1 0x00000080 [18274.994239] ffff8800b2c07db8 0000000000000086 ffff88042be69f60 ffff8800b2c07fd8 [18275.003171] ffff8800b2c07fd8 ffff8800b2c07fd8 ffff88042be69f60 ffff8800b2c07ea8 [18275.012107] 00000000039386ff 0000000000000000 ffff88042be69f60 ffff8800b2c07ea8 [18275.021034] Call Trace: [18275.024389] [] schedule+0x29/0x70 [18275.030558] [] schedule_hrtimeout_range_clock+0xb2/0x150 [18275.038963] [] ? hrtimer_get_res+0x50/0x50 [18275.046013] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [18275.054630] [] schedule_hrtimeout_range+0x13/0x20 [18275.062374] [] ep_poll+0x23e/0x360 [18275.068659] [] ? wake_up_state+0x20/0x20 [18275.075514] [] SyS_epoll_wait+0xed/0x120 [18275.082364] [] system_call_fastpath+0x16/0x1b [18275.089701] qmgr S ffff88082d6b7ea8 0 2355 2314 0x00000080 [18275.098237] ffff88082d6b7db8 0000000000000086 ffff88082a60ce70 ffff88082d6b7fd8 [18275.107180] ffff88082d6b7fd8 ffff88082d6b7fd8 ffff88082a60ce70 ffff88082d6b7ea8 [18275.116126] 0000000005f5e100 0000000000000000 ffff88082a60ce70 ffff88082d6b7ea8 [18275.125050] Call Trace: [18275.128392] [] schedule+0x29/0x70 [18275.134561] [] schedule_hrtimeout_range_clock+0xb2/0x150 [18275.142961] [] ? hrtimer_get_res+0x50/0x50 [18275.150005] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [18275.158605] [] schedule_hrtimeout_range+0x13/0x20 [18275.166328] [] ep_poll+0x23e/0x360 [18275.172595] [] ? do_vfs_ioctl+0x2d5/0x4b0 [18275.179548] [] ? wake_up_state+0x20/0x20 [18275.186390] [] SyS_epoll_wait+0xed/0x120 [18275.193238] [] system_call_fastpath+0x16/0x1b [18275.200578] gmond S ffff88082d5ffea8 0 2370 1 0x00000080 [18275.209127] ffff88082d5ffdb8 0000000000000086 ffff880826f8ce70 ffff88082d5fffd8 [18275.218073] ffff88082d5fffd8 ffff88082d5fffd8 ffff880826f8ce70 ffff88082d5ffea8 [18275.227009] 00000000000f423f 0000000000000000 ffff880826f8ce70 ffff88082d5ffea8 [18275.235946] Call Trace: [18275.239319] [] schedule+0x29/0x70 [18275.245502] [] schedule_hrtimeout_range_clock+0xb2/0x150 [18275.253912] [] ? hrtimer_get_res+0x50/0x50 [18275.260959] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [18275.269570] [] schedule_hrtimeout_range+0x13/0x20 [18275.277311] [] ep_poll+0x23e/0x360 [18275.283589] [] ? wake_up_state+0x20/0x20 [18275.290448] [] SyS_epoll_wait+0xed/0x120 [18275.297301] [] system_call_fastpath+0x16/0x1b [18275.304646] gmond S ffff8808242c3ea8 0 2464 1 0x00000080 [18275.313211] ffff8808242c3db8 0000000000000086 ffff88042bae5e20 ffff8808242c3fd8 [18275.322182] ffff8808242c3fd8 ffff8808242c3fd8 ffff88042bae5e20 ffff8808242c3ea8 [18275.331137] 000000000001869f 0000000000000000 ffff88042bae5e20 ffff8808242c3ea8 [18275.340083] Call Trace: [18275.343462] [] schedule+0x29/0x70 [18275.349642] [] schedule_hrtimeout_range_clock+0xb2/0x150 [18275.358055] [] ? hrtimer_get_res+0x50/0x50 [18275.365119] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [18275.373744] [] schedule_hrtimeout_range+0x13/0x20 [18275.381480] [] ep_poll+0x23e/0x360 [18275.387754] [] ? wake_up_state+0x20/0x20 [18275.394628] [] SyS_epoll_wait+0xed/0x120 [18275.401491] [] system_call_fastpath+0x16/0x1b [18275.408821] kworker/23:1H S ffff88042e3d6840 0 2924 2 0x00000080 [18275.417381] ffff88042b323e50 0000000000000046 ffff88042bb10fb0 ffff88042b323fd8 [18275.426354] ffff88042b323fd8 ffff88042b323fd8 ffff88042bb10fb0 ffff88042e3d6860 [18275.435326] ffff88042bc2d1b0 ffff88042bb10fb0 ffff88042bc2d180 ffff88042e3d6840 [18275.444288] Call Trace: [18275.447666] [] schedule+0x29/0x70 [18275.453836] [] worker_thread+0x1d9/0x410 [18275.460706] [] ? rescuer_thread+0x460/0x460 [18275.467855] [] kthread+0xcf/0xe0 [18275.473925] [] ? kthread_create_on_node+0x140/0x140 [18275.481834] [] ret_from_fork+0x58/0x90 [18275.488498] [] ? kthread_create_on_node+0x140/0x140 [18275.496417] kworker/23:2 S ffff88042e3d6480 0 2972 2 0x00000080 [18275.504953] ffff880412e6fe50 0000000000000046 ffff88042bacce70 ffff880412e6ffd8 [18275.513911] ffff880412e6ffd8 ffff880412e6ffd8 ffff88042bacce70 ffff88042e3d64a0 [18275.522854] ffff88042bc2cfb0 ffff88042bacce70 ffff88042bc2cf80 ffff88042e3d6480 [18275.531804] Call Trace: [18275.535177] [] schedule+0x29/0x70 [18275.541376] [] worker_thread+0x1d9/0x410 [18275.548243] [] ? rescuer_thread+0x460/0x460 [18275.555398] [] kthread+0xcf/0xe0 [18275.561476] [] ? kthread_create_on_node+0x140/0x140 [18275.569405] [] ret_from_fork+0x58/0x90 [18275.576053] [] ? kthread_create_on_node+0x140/0x140 [18275.583965] kworker/u484:1 S ffff88082a749000 0 3054 2 0x00000080 [18275.592507] ffff880412b73e50 0000000000000046 ffff880416300000 ffff880412b73fd8 [18275.601477] ffff880412b73fd8 ffff880412b73fd8 ffff880416300000 ffff88082a749020 [18275.610437] ffff88042c261230 ffff880416300000 ffff88042c261200 ffff88082a749000 [18275.619395] Call Trace: [18275.622751] [] schedule+0x29/0x70 [18275.628919] [] worker_thread+0x1d9/0x410 [18275.635766] [] ? rescuer_thread+0x460/0x460 [18275.642916] [] kthread+0xcf/0xe0 [18275.648988] [] ? kthread_create_on_node+0x140/0x140 [18275.656905] [] ret_from_fork+0x58/0x90 [18275.663560] [] ? kthread_create_on_node+0x140/0x140 [18275.671485] kworker/3:3 S ffff88042e0d6480 0 3056 2 0x00000080 [18275.680007] ffff880412f1fe50 0000000000000046 ffff880416301f60 ffff880412f1ffd8 [18275.688945] ffff880412f1ffd8 ffff880412f1ffd8 ffff880416301f60 ffff88042e0d64a0 [18275.697898] ffff88042fce3830 ffff880416301f60 ffff88042fce3800 ffff88042e0d6480 [18275.706847] Call Trace: [18275.710201] [] schedule+0x29/0x70 [18275.716375] [] worker_thread+0x1d9/0x410 [18275.723233] [] ? rescuer_thread+0x460/0x460 [18275.730381] [] kthread+0xcf/0xe0 [18275.736458] [] ? kthread_create_on_node+0x140/0x140 [18275.744382] [] ret_from_fork+0x58/0x90 [18275.751036] [] ? kthread_create_on_node+0x140/0x140 [18275.758948] spl_kmem_cache S ffff88081e697700 0 3082 2 0x00000080 [18275.767487] ffff8808257fbe08 0000000000000046 ffff880416305e20 ffff8808257fbfd8 [18275.776444] ffff8808257fbfd8 ffff8808257fbfd8 ffff880416305e20 ffff88042bc131e0 [18275.785416] ffff88081e697790 ffffffffa08254b0 ffff88081e6977b0 ffff88081e697700 [18275.794366] Call Trace: [18275.797740] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18275.805686] [] schedule+0x29/0x70 [18275.811875] [] taskq_thread+0x3fa/0x440 [spl] [18275.819238] [] ? wake_up_state+0x20/0x20 [18275.826106] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18275.834037] [] kthread+0xcf/0xe0 [18275.840134] [] ? kthread_create_on_node+0x140/0x140 [18275.848066] [] ret_from_fork+0x58/0x90 [18275.854733] [] ? kthread_create_on_node+0x140/0x140 [18275.862661] spl_kmem_cache S ffff88081e697700 0 3083 2 0x00000080 [18275.871222] ffff880825777e08 0000000000000046 ffff880416306dd0 ffff880825777fd8 [18275.880194] ffff880825777fd8 ffff880825777fd8 ffff880416306dd0 ffff88042bc13240 [18275.889168] ffff88081e697790 ffffffffa08254b0 ffff88081e6977b0 ffff88081e697700 [18275.898140] Call Trace: [18275.901521] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18275.909460] [] schedule+0x29/0x70 [18275.915677] [] taskq_thread+0x3fa/0x440 [spl] [18275.923048] [] ? wake_up_state+0x20/0x20 [18275.929934] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18275.937880] [] kthread+0xcf/0xe0 [18275.943974] [] ? kthread_create_on_node+0x140/0x140 [18275.951917] [] ret_from_fork+0x58/0x90 [18275.958605] [] ? kthread_create_on_node+0x140/0x140 [18275.966547] spl_kmem_cache S ffff88081e697700 0 3084 2 0x00000080 [18275.975116] ffff88082570be08 0000000000000046 ffff880416304e70 ffff88082570bfd8 [18275.984098] ffff88082570bfd8 ffff88082570bfd8 ffff880416304e70 ffff88042bc132a0 [18275.993071] ffff88081e697790 ffffffffa08254b0 ffff88081e6977b0 ffff88081e697700 [18276.002057] Call Trace: [18276.005471] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18276.013440] [] schedule+0x29/0x70 [18276.019667] [] taskq_thread+0x3fa/0x440 [spl] [18276.027042] [] ? wake_up_state+0x20/0x20 [18276.033930] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18276.041887] [] kthread+0xcf/0xe0 [18276.048001] [] ? kthread_create_on_node+0x140/0x140 [18276.055957] [] ret_from_fork+0x58/0x90 [18276.062646] [] ? kthread_create_on_node+0x140/0x140 [18276.070608] spl_kmem_cache S ffff88081e697700 0 3085 2 0x00000080 [18276.079199] ffff88082a39fe08 0000000000000046 ffff880416303ec0 ffff88082a39ffd8 [18276.088205] ffff88082a39ffd8 ffff88082a39ffd8 ffff880416303ec0 ffff88042bc13300 [18276.097221] ffff88081e697790 ffffffffa08254b0 ffff88081e6977b0 ffff88081e697700 [18276.106210] Call Trace: [18276.109631] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18276.117624] [] schedule+0x29/0x70 [18276.123845] [] taskq_thread+0x3fa/0x440 [spl] [18276.131238] [] ? wake_up_state+0x20/0x20 [18276.138149] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18276.146116] [] kthread+0xcf/0xe0 [18276.152236] [] ? kthread_create_on_node+0x140/0x140 [18276.160211] [] ret_from_fork+0x58/0x90 [18276.166916] [] ? kthread_create_on_node+0x140/0x140 [18276.174882] spl_system_task S ffff88081e695e00 0 3086 2 0x00000080 [18276.183471] ffff8808273dbe08 0000000000000046 ffff88082c89ce70 ffff8808273dbfd8 [18276.192469] ffff8808273dbfd8 ffff8808273dbfd8 ffff88082c89ce70 ffff88042bc13360 [18276.201472] ffff88081e695e90 ffffffffa08254b0 ffff88081e695eb0 ffff88081e695e00 [18276.210477] Call Trace: [18276.213896] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18276.221866] [] schedule+0x29/0x70 [18276.228088] [] taskq_thread+0x3fa/0x440 [spl] [18276.235486] [] ? wake_up_state+0x20/0x20 [18276.242389] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18276.250354] [] kthread+0xcf/0xe0 [18276.256465] [] ? kthread_create_on_node+0x140/0x140 [18276.264426] [] ret_from_fork+0x58/0x90 [18276.271118] [] ? kthread_create_on_node+0x140/0x140 [18276.279073] spl_system_task S ffff88081e695e00 0 3087 2 0x00000080 [18276.287670] ffff8808273ebe08 0000000000000046 ffff88082c89bec0 ffff8808273ebfd8 [18276.296687] ffff8808273ebfd8 ffff8808273ebfd8 ffff88082c89bec0 ffff88042bc133c0 [18276.305698] ffff88081e695e90 ffffffffa08254b0 ffff88081e695eb0 ffff88081e695e00 [18276.314700] Call Trace: [18276.318111] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18276.326085] [] schedule+0x29/0x70 [18276.332307] [] taskq_thread+0x3fa/0x440 [spl] [18276.339709] [] ? wake_up_state+0x20/0x20 [18276.346625] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18276.354594] [] kthread+0xcf/0xe0 [18276.360710] [] ? kthread_create_on_node+0x140/0x140 [18276.368668] [] ret_from_fork+0x58/0x90 [18276.375363] [] ? kthread_create_on_node+0x140/0x140 [18276.383322] spl_system_task S ffff88081e695e00 0 3088 2 0x00000080 [18276.391905] ffff880823383e08 0000000000000046 ffff88082a405e20 ffff880823383fd8 [18276.400909] ffff880823383fd8 ffff880823383fd8 ffff88082a405e20 ffff88042bc13420 [18276.409908] ffff88081e695e90 ffffffffa08254b0 ffff88081e695eb0 ffff88081e695e00 [18276.418899] Call Trace: [18276.422306] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18276.430284] [] schedule+0x29/0x70 [18276.436502] [] taskq_thread+0x3fa/0x440 [spl] [18276.443889] [] ? wake_up_state+0x20/0x20 [18276.450806] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18276.458784] [] kthread+0xcf/0xe0 [18276.464897] [] ? kthread_create_on_node+0x140/0x140 [18276.472854] [] ret_from_fork+0x58/0x90 [18276.479564] [] ? kthread_create_on_node+0x140/0x140 [18276.487528] spl_system_task S ffff88081e695e00 0 3089 2 0x00000080 [18276.496097] ffff88083fd37e08 0000000000000046 ffff88082a402f10 ffff88083fd37fd8 [18276.505092] ffff88083fd37fd8 ffff88083fd37fd8 ffff88082a402f10 ffff88042bc13480 [18276.514082] ffff88081e695e90 ffffffffa08254b0 ffff88081e695eb0 ffff88081e695e00 [18276.523080] Call Trace: [18276.526493] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18276.534468] [] schedule+0x29/0x70 [18276.540697] [] taskq_thread+0x3fa/0x440 [spl] [18276.548086] [] ? wake_up_state+0x20/0x20 [18276.554994] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18276.562963] [] kthread+0xcf/0xe0 [18276.569075] [] ? kthread_create_on_node+0x140/0x140 [18276.577033] [] ret_from_fork+0x58/0x90 [18276.583744] [] ? kthread_create_on_node+0x140/0x140 [18276.591709] spl_system_task S ffff88081e695e00 0 3090 2 0x00000080 [18276.600286] ffff88082d4efe08 0000000000000046 ffff88082a404e70 ffff88082d4effd8 [18276.609291] ffff88082d4effd8 ffff88082d4effd8 ffff88082a404e70 ffff88042bc134e0 [18276.618285] ffff88081e695e90 ffffffffa08254b0 ffff88081e695eb0 ffff88081e695e00 [18276.627284] Call Trace: [18276.630697] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18276.638671] [] schedule+0x29/0x70 [18276.644890] [] taskq_thread+0x3fa/0x440 [spl] [18276.652286] [] ? wake_up_state+0x20/0x20 [18276.659204] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18276.667168] [] kthread+0xcf/0xe0 [18276.673273] [] ? kthread_create_on_node+0x140/0x140 [18276.681237] [] ret_from_fork+0x58/0x90 [18276.687934] [] ? kthread_create_on_node+0x140/0x140 [18276.695892] spl_system_task S ffff88081e695e00 0 3091 2 0x00000080 [18276.704471] ffff88082739be08 0000000000000046 ffff88082a406dd0 ffff88082739bfd8 [18276.713476] ffff88082739bfd8 ffff88082739bfd8 ffff88082a406dd0 ffff88042bc13540 [18276.722475] ffff88081e695e90 ffffffffa08254b0 ffff88081e695eb0 ffff88081e695e00 [18276.731468] Call Trace: [18276.734875] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18276.742848] [] schedule+0x29/0x70 [18276.749066] [] taskq_thread+0x3fa/0x440 [spl] [18276.756460] [] ? wake_up_state+0x20/0x20 [18276.763363] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18276.771336] [] kthread+0xcf/0xe0 [18276.777453] [] ? kthread_create_on_node+0x140/0x140 [18276.785421] [] ret_from_fork+0x58/0x90 [18276.792112] [] ? kthread_create_on_node+0x140/0x140 [18276.800064] spl_system_task S ffff88081e695e00 0 3092 2 0x00000080 [18276.808655] ffff880826267e08 0000000000000046 ffff88082a403ec0 ffff880826267fd8 [18276.817660] ffff880826267fd8 ffff880826267fd8 ffff88082a403ec0 ffff88042bc135a0 [18276.826661] ffff88081e695e90 ffffffffa08254b0 ffff88081e695eb0 ffff88081e695e00 [18276.835669] Call Trace: [18276.839080] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18276.847054] [] schedule+0x29/0x70 [18276.853274] [] taskq_thread+0x3fa/0x440 [spl] [18276.860675] [] ? wake_up_state+0x20/0x20 [18276.867577] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18276.875542] [] kthread+0xcf/0xe0 [18276.881664] [] ? kthread_create_on_node+0x140/0x140 [18276.889629] [] ret_from_fork+0x58/0x90 [18276.896330] [] ? kthread_create_on_node+0x140/0x140 [18276.904294] spl_system_task S ffff88081e695e00 0 3093 2 0x00000080 [18276.912866] ffff8808253b3e08 0000000000000046 ffff880824a7edd0 ffff8808253b3fd8 [18276.921870] ffff8808253b3fd8 ffff8808253b3fd8 ffff880824a7edd0 ffff88042bc13600 [18276.930844] ffff88081e695e90 ffffffffa08254b0 ffff88081e695eb0 ffff88081e695e00 [18276.939843] Call Trace: [18276.943265] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18276.951241] [] schedule+0x29/0x70 [18276.957463] [] taskq_thread+0x3fa/0x440 [spl] [18276.964852] [] ? wake_up_state+0x20/0x20 [18276.971761] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18276.979726] [] kthread+0xcf/0xe0 [18276.985831] [] ? kthread_create_on_node+0x140/0x140 [18276.993800] [] ret_from_fork+0x58/0x90 [18277.000496] [] ? kthread_create_on_node+0x140/0x140 [18277.008457] spl_system_task S ffff88081e695e00 0 3094 2 0x00000080 [18277.017019] ffff88081e70be08 0000000000000046 ffff880824a7de20 ffff88081e70bfd8 [18277.026019] ffff88081e70bfd8 ffff88081e70bfd8 ffff880824a7de20 ffff88042bc13660 [18277.035019] ffff88081e695e90 ffffffffa08254b0 ffff88081e695eb0 ffff88081e695e00 [18277.044006] Call Trace: [18277.047414] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18277.055392] [] schedule+0x29/0x70 [18277.061612] [] taskq_thread+0x3fa/0x440 [spl] [18277.068998] [] ? wake_up_state+0x20/0x20 [18277.075902] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18277.083857] [] kthread+0xcf/0xe0 [18277.089969] [] ? kthread_create_on_node+0x140/0x140 [18277.097926] [] ret_from_fork+0x58/0x90 [18277.104628] [] ? kthread_create_on_node+0x140/0x140 [18277.112594] spl_system_task S ffff88081e695e00 0 3095 2 0x00000080 [18277.121174] ffff88082dbffe08 0000000000000046 ffff880824a7ce70 ffff88082dbfffd8 [18277.130177] ffff88082dbfffd8 ffff88082dbfffd8 ffff880824a7ce70 ffff88042bc136c0 [18277.139178] ffff88081e695e90 ffffffffa08254b0 ffff88081e695eb0 ffff88081e695e00 [18277.148179] Call Trace: [18277.151583] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18277.159563] [] schedule+0x29/0x70 [18277.165793] [] taskq_thread+0x3fa/0x440 [spl] [18277.173177] [] ? wake_up_state+0x20/0x20 [18277.180070] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18277.188025] [] kthread+0xcf/0xe0 [18277.194132] [] ? kthread_create_on_node+0x140/0x140 [18277.202088] [] ret_from_fork+0x58/0x90 [18277.208792] [] ? kthread_create_on_node+0x140/0x140 [18277.216756] spl_system_task S ffff88081e695e00 0 3096 2 0x00000080 [18277.225349] ffff88081e6d7e08 0000000000000046 ffff880824a7bec0 ffff88081e6d7fd8 [18277.234339] ffff88081e6d7fd8 ffff88081e6d7fd8 ffff880824a7bec0 ffff88042bc13720 [18277.243331] ffff88081e695e90 ffffffffa08254b0 ffff88081e695eb0 ffff88081e695e00 [18277.252334] Call Trace: [18277.255759] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18277.263739] [] schedule+0x29/0x70 [18277.269955] [] taskq_thread+0x3fa/0x440 [spl] [18277.277351] [] ? wake_up_state+0x20/0x20 [18277.284248] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18277.292210] [] kthread+0xcf/0xe0 [18277.298321] [] ? kthread_create_on_node+0x140/0x140 [18277.306275] [] ret_from_fork+0x58/0x90 [18277.312970] [] ? kthread_create_on_node+0x140/0x140 [18277.320921] spl_system_task S ffff88081e695e00 0 3097 2 0x00000080 [18277.329507] ffff88082ab47e08 0000000000000046 ffff880824a7af10 ffff88082ab47fd8 [18277.338519] ffff88082ab47fd8 ffff88082ab47fd8 ffff880824a7af10 ffff88042bc13780 [18277.347528] ffff88081e695e90 ffffffffa08254b0 ffff88081e695eb0 ffff88081e695e00 [18277.356526] Call Trace: [18277.359929] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18277.367891] [] schedule+0x29/0x70 [18277.374107] [] taskq_thread+0x3fa/0x440 [spl] [18277.381494] [] ? wake_up_state+0x20/0x20 [18277.388404] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18277.396374] [] kthread+0xcf/0xe0 [18277.402484] [] ? kthread_create_on_node+0x140/0x140 [18277.410442] [] ret_from_fork+0x58/0x90 [18277.417136] [] ? kthread_create_on_node+0x140/0x140 [18277.425092] spl_system_task S ffff88081e695e00 0 3098 2 0x00000080 [18277.433678] ffff880826b6fe08 0000000000000046 ffff880824a79f60 ffff880826b6ffd8 [18277.442678] ffff880826b6ffd8 ffff880826b6ffd8 ffff880824a79f60 ffff88042bc137e0 [18277.451675] ffff88081e695e90 ffffffffa08254b0 ffff88081e695eb0 ffff88081e695e00 [18277.460675] Call Trace: [18277.464084] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18277.472052] [] schedule+0x29/0x70 [18277.478265] [] taskq_thread+0x3fa/0x440 [spl] [18277.485659] [] ? wake_up_state+0x20/0x20 [18277.492568] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18277.500538] [] kthread+0xcf/0xe0 [18277.506658] [] ? kthread_create_on_node+0x140/0x140 [18277.514615] [] ret_from_fork+0x58/0x90 [18277.521295] [] ? kthread_create_on_node+0x140/0x140 [18277.529265] spl_system_task S ffff88081e695e00 0 3099 2 0x00000080 [18277.537845] ffff880824ba3e08 0000000000000046 ffff880824a78fb0 ffff880824ba3fd8 [18277.546845] ffff880824ba3fd8 ffff880824ba3fd8 ffff880824a78fb0 ffff88042bc13840 [18277.555833] ffff88081e695e90 ffffffffa08254b0 ffff88081e695eb0 ffff88081e695e00 [18277.564842] Call Trace: [18277.568250] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18277.576225] [] schedule+0x29/0x70 [18277.582441] [] taskq_thread+0x3fa/0x440 [spl] [18277.589839] [] ? wake_up_state+0x20/0x20 [18277.596746] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18277.604708] [] kthread+0xcf/0xe0 [18277.610834] [] ? kthread_create_on_node+0x140/0x140 [18277.618794] [] ret_from_fork+0x58/0x90 [18277.625496] [] ? kthread_create_on_node+0x140/0x140 [18277.633457] spl_system_task S ffff88081e695e00 0 3100 2 0x00000080 [18277.642021] ffff88082d9ffe08 0000000000000046 ffff880824a78000 ffff88082d9fffd8 [18277.651020] ffff88082d9fffd8 ffff88082d9fffd8 ffff880824a78000 ffff88042bc138a0 [18277.660020] ffff88081e695e90 ffffffffa08254b0 ffff88081e695eb0 ffff88081e695e00 [18277.669013] Call Trace: [18277.672421] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18277.680391] [] schedule+0x29/0x70 [18277.686618] [] taskq_thread+0x3fa/0x440 [spl] [18277.694000] [] ? wake_up_state+0x20/0x20 [18277.700898] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18277.708863] [] kthread+0xcf/0xe0 [18277.714973] [] ? kthread_create_on_node+0x140/0x140 [18277.722931] [] ret_from_fork+0x58/0x90 [18277.729629] [] ? kthread_create_on_node+0x140/0x140 [18277.737590] spl_system_task S ffff88081e695e00 0 3101 2 0x00000080 [18277.746156] ffff8808276afe08 0000000000000046 ffff88082a638fb0 ffff8808276affd8 [18277.755129] ffff8808276affd8 ffff8808276affd8 ffff88082a638fb0 ffff88042bc13900 [18277.764127] ffff88081e695e90 ffffffffa08254b0 ffff88081e695eb0 ffff88081e695e00 [18277.773120] Call Trace: [18277.776531] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18277.784508] [] schedule+0x29/0x70 [18277.790725] [] taskq_thread+0x3fa/0x440 [spl] [18277.798109] [] ? wake_up_state+0x20/0x20 [18277.805007] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18277.812953] [] kthread+0xcf/0xe0 [18277.819061] [] ? kthread_create_on_node+0x140/0x140 [18277.827016] [] ret_from_fork+0x58/0x90 [18277.833698] [] ? kthread_create_on_node+0x140/0x140 [18277.841654] spl_system_task S ffff88081e695e00 0 3102 2 0x00000080 [18277.850243] ffff8808266d7e08 0000000000000046 ffff88082a63de20 ffff8808266d7fd8 [18277.859244] ffff8808266d7fd8 ffff8808266d7fd8 ffff88082a63de20 ffff88042bc13960 [18277.868230] ffff88081e695e90 ffffffffa08254b0 ffff88081e695eb0 ffff88081e695e00 [18277.877228] Call Trace: [18277.880647] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18277.888631] [] schedule+0x29/0x70 [18277.894853] [] taskq_thread+0x3fa/0x440 [spl] [18277.902241] [] ? wake_up_state+0x20/0x20 [18277.909142] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18277.917104] [] kthread+0xcf/0xe0 [18277.923223] [] ? kthread_create_on_node+0x140/0x140 [18277.931178] [] ret_from_fork+0x58/0x90 [18277.937871] [] ? kthread_create_on_node+0x140/0x140 [18277.945836] spl_system_task S ffff88081e695e00 0 3103 2 0x00000080 [18277.954407] ffff88082daffe08 0000000000000046 ffff88082a63af10 ffff88082dafffd8 [18277.963412] ffff88082dafffd8 ffff88082dafffd8 ffff88082a63af10 ffff88042bc139c0 [18277.972406] ffff88081e695e90 ffffffffa08254b0 ffff88081e695eb0 ffff88081e695e00 [18277.981409] Call Trace: [18277.984837] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18277.992821] [] schedule+0x29/0x70 [18277.999039] [] taskq_thread+0x3fa/0x440 [spl] [18278.006425] [] ? wake_up_state+0x20/0x20 [18278.013332] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18278.021299] [] kthread+0xcf/0xe0 [18278.027425] [] ? kthread_create_on_node+0x140/0x140 [18278.035391] [] ret_from_fork+0x58/0x90 [18278.042086] [] ? kthread_create_on_node+0x140/0x140 [18278.050036] spl_system_task S ffff88081e695e00 0 3104 2 0x00000080 [18278.058635] ffff88082c7e3e08 0000000000000046 ffff88082d6f6dd0 ffff88082c7e3fd8 [18278.067647] ffff88082c7e3fd8 ffff88082c7e3fd8 ffff88082d6f6dd0 ffff88042bc13a20 [18278.076660] ffff88081e695e90 ffffffffa08254b0 ffff88081e695eb0 ffff88081e695e00 [18278.085660] Call Trace: [18278.089074] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18278.097042] [] schedule+0x29/0x70 [18278.103260] [] taskq_thread+0x3fa/0x440 [spl] [18278.110657] [] ? wake_up_state+0x20/0x20 [18278.117562] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18278.125527] [] kthread+0xcf/0xe0 [18278.131632] [] ? kthread_create_on_node+0x140/0x140 [18278.139590] [] ret_from_fork+0x58/0x90 [18278.146296] [] ? kthread_create_on_node+0x140/0x140 [18278.154256] spl_system_task S ffff88081e695e00 0 3105 2 0x00000080 [18278.162838] ffff880829af7e08 0000000000000046 ffff88082d6f1f60 ffff880829af7fd8 [18278.171844] ffff880829af7fd8 ffff880829af7fd8 ffff88082d6f1f60 ffff88042bc13a80 [18278.180843] ffff88081e695e90 ffffffffa08254b0 ffff88081e695eb0 ffff88081e695e00 [18278.189849] Call Trace: [18278.193267] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18278.201246] [] schedule+0x29/0x70 [18278.207465] [] taskq_thread+0x3fa/0x440 [spl] [18278.214848] [] ? wake_up_state+0x20/0x20 [18278.221758] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18278.229736] [] kthread+0xcf/0xe0 [18278.235858] [] ? kthread_create_on_node+0x140/0x140 [18278.243832] [] ret_from_fork+0x58/0x90 [18278.250537] [] ? kthread_create_on_node+0x140/0x140 [18278.258505] spl_system_task S ffff88081e695e00 0 3106 2 0x00000080 [18278.267073] ffff88082aa93e08 0000000000000046 ffff88082d6f3ec0 ffff88082aa93fd8 [18278.276076] ffff88082aa93fd8 ffff88082aa93fd8 ffff88082d6f3ec0 ffff88042bc13ae0 [18278.285079] ffff88081e695e90 ffffffffa08254b0 ffff88081e695eb0 ffff88081e695e00 [18278.294074] Call Trace: [18278.297486] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18278.305459] [] schedule+0x29/0x70 [18278.311679] [] taskq_thread+0x3fa/0x440 [spl] [18278.319064] [] ? wake_up_state+0x20/0x20 [18278.325958] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18278.333915] [] kthread+0xcf/0xe0 [18278.340031] [] ? finish_task_switch+0x53/0x180 [18278.347502] [] ? kthread_create_on_node+0x140/0x140 [18278.355466] [] ret_from_fork+0x58/0x90 [18278.362151] [] ? kthread_create_on_node+0x140/0x140 [18278.370109] spl_system_task S ffff88081e695e00 0 3107 2 0x00000080 [18278.378677] ffff880826acbe08 0000000000000046 ffff88082d6f5e20 ffff880826acbfd8 [18278.387680] ffff880826acbfd8 ffff880826acbfd8 ffff88082d6f5e20 ffff88042bc13b40 [18278.396684] ffff88081e695e90 ffffffffa08254b0 ffff88081e695eb0 ffff88081e695e00 [18278.405683] Call Trace: [18278.409074] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18278.417037] [] schedule+0x29/0x70 [18278.423254] [] taskq_thread+0x3fa/0x440 [spl] [18278.430648] [] ? wake_up_state+0x20/0x20 [18278.437550] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18278.445516] [] kthread+0xcf/0xe0 [18278.451637] [] ? kthread_create_on_node+0x140/0x140 [18278.459593] [] ret_from_fork+0x58/0x90 [18278.466293] [] ? kthread_create_on_node+0x140/0x140 [18278.474255] spl_system_task S ffff88081e695e00 0 3108 2 0x00000080 [18278.482831] ffff880827b9be08 0000000000000046 ffff88082d6f2f10 ffff880827b9bfd8 [18278.491824] ffff880827b9bfd8 ffff880827b9bfd8 ffff88082d6f2f10 ffff88042bc13ba0 [18278.500835] ffff88081e695e90 ffffffffa08254b0 ffff88081e695eb0 ffff88081e695e00 [18278.509831] Call Trace: [18278.513232] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18278.521191] [] schedule+0x29/0x70 [18278.527412] [] taskq_thread+0x3fa/0x440 [spl] [18278.534798] [] ? wake_up_state+0x20/0x20 [18278.541695] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18278.549662] [] kthread+0xcf/0xe0 [18278.555785] [] ? kthread_create_on_node+0x140/0x140 [18278.563739] [] ret_from_fork+0x58/0x90 [18278.570434] [] ? kthread_create_on_node+0x140/0x140 [18278.578391] spl_system_task S ffff88081e695e00 0 3109 2 0x00000080 [18278.586957] ffff880826b63e08 0000000000000046 ffff88082d6f0000 ffff880826b63fd8 [18278.595952] ffff880826b63fd8 ffff880826b63fd8 ffff88082d6f0000 ffff88042bc13c00 [18278.604940] ffff88081e695e90 ffffffffa08254b0 ffff88081e695eb0 ffff88081e695e00 [18278.613945] Call Trace: [18278.617349] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18278.625316] [] schedule+0x29/0x70 [18278.631537] [] taskq_thread+0x3fa/0x440 [spl] [18278.638921] [] ? wake_up_state+0x20/0x20 [18278.645815] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18278.653772] [] kthread+0xcf/0xe0 [18278.659885] [] ? kthread_create_on_node+0x140/0x140 [18278.667839] [] ret_from_fork+0x58/0x90 [18278.674540] [] ? kthread_create_on_node+0x140/0x140 [18278.682504] spl_system_task S ffff88081e695e00 0 3110 2 0x00000080 [18278.691073] ffff880827bafe08 0000000000000046 ffff88082bcb3ec0 ffff880827baffd8 [18278.700065] ffff880827baffd8 ffff880827baffd8 ffff88082bcb3ec0 ffff88042bc13c60 [18278.709060] ffff88081e695e90 ffffffffa08254b0 ffff88081e695eb0 ffff88081e695e00 [18278.718051] Call Trace: [18278.721461] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18278.729440] [] schedule+0x29/0x70 [18278.735665] [] taskq_thread+0x3fa/0x440 [spl] [18278.743050] [] ? wake_up_state+0x20/0x20 [18278.749959] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18278.757916] [] kthread+0xcf/0xe0 [18278.764030] [] ? kthread_create_on_node+0x140/0x140 [18278.771980] [] ret_from_fork+0x58/0x90 [18278.778674] [] ? kthread_create_on_node+0x140/0x140 [18278.786636] spl_system_task S ffff88081e695e00 0 3111 2 0x00000080 [18278.795207] ffff880827657e08 0000000000000046 ffff88082bcb0fb0 ffff880827657fd8 [18278.804185] ffff880827657fd8 ffff880827657fd8 ffff88082bcb0fb0 ffff88042bc13cc0 [18278.813176] ffff88081e695e90 ffffffffa08254b0 ffff88081e695eb0 ffff88081e695e00 [18278.822169] Call Trace: [18278.825584] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18278.833557] [] schedule+0x29/0x70 [18278.839789] [] taskq_thread+0x3fa/0x440 [spl] [18278.847170] [] ? wake_up_state+0x20/0x20 [18278.854071] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18278.862040] [] kthread+0xcf/0xe0 [18278.868151] [] ? kthread_create_on_node+0x140/0x140 [18278.876109] [] ret_from_fork+0x58/0x90 [18278.882814] [] ? kthread_create_on_node+0x140/0x140 [18278.890782] spl_system_task S ffff88081e695e00 0 3112 2 0x00000080 [18278.899369] ffff88082ab4be08 0000000000000046 ffff88082e5b8000 ffff88082ab4bfd8 [18278.908369] ffff88082ab4bfd8 ffff88082ab4bfd8 ffff88082e5b8000 ffff88042bc13d20 [18278.917365] ffff88081e695e90 ffffffffa08254b0 ffff88081e695eb0 ffff88081e695e00 [18278.926363] Call Trace: [18278.929796] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18278.937779] [] schedule+0x29/0x70 [18278.943998] [] taskq_thread+0x3fa/0x440 [spl] [18278.951386] [] ? wake_up_state+0x20/0x20 [18278.958291] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18278.966267] [] kthread+0xcf/0xe0 [18278.972378] [] ? kthread_create_on_node+0x140/0x140 [18278.980334] [] ret_from_fork+0x58/0x90 [18278.987036] [] ? kthread_create_on_node+0x140/0x140 [18278.994994] spl_system_task S ffff88081e695e00 0 3113 2 0x00000080 [18279.003577] ffff880824bbfe08 0000000000000046 ffff88082e5b9f60 ffff880824bbffd8 [18279.012580] ffff880824bbffd8 ffff880824bbffd8 ffff88082e5b9f60 ffff88042bc13d80 [18279.021584] ffff88081e695e90 ffffffffa08254b0 ffff88081e695eb0 ffff88081e695e00 [18279.030580] Call Trace: [18279.033988] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18279.041954] [] schedule+0x29/0x70 [18279.048172] [] taskq_thread+0x3fa/0x440 [spl] [18279.055566] [] ? wake_up_state+0x20/0x20 [18279.062467] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18279.070429] [] kthread+0xcf/0xe0 [18279.076549] [] ? kthread_create_on_node+0x140/0x140 [18279.084506] [] ret_from_fork+0x58/0x90 [18279.091193] [] ? kthread_create_on_node+0x140/0x140 [18279.099145] spl_system_task S ffff88081e695e00 0 3114 2 0x00000080 [18279.107738] ffff88081e753e08 0000000000000046 ffff88082e5baf10 ffff88081e753fd8 [18279.116750] ffff88081e753fd8 ffff88081e753fd8 ffff88082e5baf10 ffff88042bc13de0 [18279.125754] ffff88081e695e90 ffffffffa08254b0 ffff88081e695eb0 ffff88081e695e00 [18279.134758] Call Trace: [18279.138167] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18279.146138] [] schedule+0x29/0x70 [18279.152366] [] taskq_thread+0x3fa/0x440 [spl] [18279.159762] [] ? wake_up_state+0x20/0x20 [18279.166662] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18279.174623] [] kthread+0xcf/0xe0 [18279.180745] [] ? kthread_create_on_node+0x140/0x140 [18279.188708] [] ret_from_fork+0x58/0x90 [18279.195399] [] ? kthread_create_on_node+0x140/0x140 [18279.203353] spl_system_task S ffff88081e695e00 0 3115 2 0x00000080 [18279.211932] ffff880824317e08 0000000000000046 ffff88082e5bbec0 ffff880824317fd8 [18279.220933] ffff880824317fd8 ffff880824317fd8 ffff88082e5bbec0 ffff88042bc13e40 [18279.229929] ffff88081e695e90 ffffffffa08254b0 ffff88081e695eb0 ffff88081e695e00 [18279.238919] Call Trace: [18279.242337] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18279.250307] [] schedule+0x29/0x70 [18279.256540] [] taskq_thread+0x3fa/0x440 [spl] [18279.263925] [] ? wake_up_state+0x20/0x20 [18279.270827] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18279.278790] [] kthread+0xcf/0xe0 [18279.284908] [] ? kthread_create_on_node+0x140/0x140 [18279.292872] [] ret_from_fork+0x58/0x90 [18279.299566] [] ? kthread_create_on_node+0x140/0x140 [18279.307518] spl_system_task S ffff88081e695e00 0 3116 2 0x00000080 [18279.316088] ffff88082475be08 0000000000000046 ffff88082e5bce70 ffff88082475bfd8 [18279.325082] ffff88082475bfd8 ffff88082475bfd8 ffff88082e5bce70 ffff88042bc13ea0 [18279.334076] ffff88081e695e90 ffffffffa08254b0 ffff88081e695eb0 ffff88081e695e00 [18279.343072] Call Trace: [18279.346490] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18279.354459] [] schedule+0x29/0x70 [18279.360688] [] taskq_thread+0x3fa/0x440 [spl] [18279.368071] [] ? wake_up_state+0x20/0x20 [18279.374970] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18279.382928] [] kthread+0xcf/0xe0 [18279.389036] [] ? kthread_create_on_node+0x140/0x140 [18279.396990] [] ret_from_fork+0x58/0x90 [18279.403692] [] ? kthread_create_on_node+0x140/0x140 [18279.411659] spl_system_task S ffff88081e695e00 0 3117 2 0x00000080 [18279.420229] ffff880827b8fe08 0000000000000046 ffff88082e5bedd0 ffff880827b8ffd8 [18279.429212] ffff880827b8ffd8 ffff880827b8ffd8 ffff88082e5bedd0 ffff88042bc13f00 [18279.438196] ffff88081e695e90 ffffffffa08254b0 ffff88081e695eb0 ffff88081e695e00 [18279.447186] Call Trace: [18279.450604] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18279.458585] [] schedule+0x29/0x70 [18279.464809] [] taskq_thread+0x3fa/0x440 [spl] [18279.472194] [] ? wake_up_state+0x20/0x20 [18279.479094] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18279.487053] [] kthread+0xcf/0xe0 [18279.493162] [] ? kthread_create_on_node+0x140/0x140 [18279.501117] [] ret_from_fork+0x58/0x90 [18279.507824] [] ? kthread_create_on_node+0x140/0x140 [18279.515781] spl_system_task S ffff88081e695e00 0 3118 2 0x00000080 [18279.524369] ffff88082d2e3e08 0000000000000046 ffff88082e5bde20 ffff88082d2e3fd8 [18279.533368] ffff88082d2e3fd8 ffff88082d2e3fd8 ffff88082e5bde20 ffff88042bc13f60 [18279.542377] ffff88081e695e90 ffffffffa08254b0 ffff88081e695eb0 ffff88081e695e00 [18279.551371] Call Trace: [18279.554788] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18279.562773] [] schedule+0x29/0x70 [18279.568989] [] taskq_thread+0x3fa/0x440 [spl] [18279.576383] [] ? wake_up_state+0x20/0x20 [18279.583283] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18279.591238] [] kthread+0xcf/0xe0 [18279.597355] [] ? kthread_create_on_node+0x140/0x140 [18279.605321] [] ret_from_fork+0x58/0x90 [18279.612008] [] ? kthread_create_on_node+0x140/0x140 [18279.619965] spl_system_task S ffff88081e695e00 0 3119 2 0x00000080 [18279.628561] ffff88082aec3e08 0000000000000046 ffff88082d5c0000 ffff88082aec3fd8 [18279.637568] ffff88082aec3fd8 ffff88082aec3fd8 ffff88082d5c0000 ffff88042bc130c0 [18279.646563] ffff88081e695e90 ffffffffa08254b0 ffff88081e695eb0 ffff88081e695e00 [18279.655569] Call Trace: [18279.658974] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18279.666945] [] schedule+0x29/0x70 [18279.673158] [] taskq_thread+0x3fa/0x440 [spl] [18279.680544] [] ? wake_up_state+0x20/0x20 [18279.687446] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18279.695405] [] kthread+0xcf/0xe0 [18279.701532] [] ? kthread_create_on_node+0x140/0x140 [18279.709496] [] ret_from_fork+0x58/0x90 [18279.716182] [] ? kthread_create_on_node+0x140/0x140 [18279.724136] spl_system_task S ffff88081e695e00 0 3120 2 0x00000080 [18279.732735] ffff88082676fe08 0000000000000046 ffff88082d5c0fb0 ffff88082676ffd8 [18279.741725] ffff88082676ffd8 ffff88082676ffd8 ffff88082d5c0fb0 ffff88042bc13060 [18279.750709] ffff88081e695e90 ffffffffa08254b0 ffff88081e695eb0 ffff88081e695e00 [18279.759712] Call Trace: [18279.763124] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18279.771096] [] schedule+0x29/0x70 [18279.777312] [] taskq_thread+0x3fa/0x440 [spl] [18279.784704] [] ? wake_up_state+0x20/0x20 [18279.791604] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18279.799574] [] kthread+0xcf/0xe0 [18279.805697] [] ? kthread_create_on_node+0x140/0x140 [18279.813656] [] ret_from_fork+0x58/0x90 [18279.820350] [] ? kthread_create_on_node+0x140/0x140 [18279.828308] spl_system_task S ffff88081e695e00 0 3121 2 0x00000080 [18279.836895] ffff880825fd7e08 0000000000000046 ffff88082d5c1f60 ffff880825fd7fd8 [18279.845894] ffff880825fd7fd8 ffff880825fd7fd8 ffff88082d5c1f60 ffff88042bcad480 [18279.854899] ffff88081e695e90 ffffffffa08254b0 ffff88081e695eb0 ffff88081e695e00 [18279.863900] Call Trace: [18279.867308] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18279.875283] [] schedule+0x29/0x70 [18279.881504] [] taskq_thread+0x3fa/0x440 [spl] [18279.888905] [] ? wake_up_state+0x20/0x20 [18279.895821] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18279.903788] [] kthread+0xcf/0xe0 [18279.909906] [] ? kthread_create_on_node+0x140/0x140 [18279.917871] [] ret_from_fork+0x58/0x90 [18279.924574] [] ? kthread_create_on_node+0x140/0x140 [18279.932543] spl_system_task S ffff88081e695e00 0 3122 2 0x00000080 [18279.941114] ffff88082664be08 0000000000000046 ffff88082d5c2f10 ffff88082664bfd8 [18279.950110] ffff88082664bfd8 ffff88082664bfd8 ffff88082d5c2f10 ffff88042bcade40 [18279.959107] ffff88081e695e90 ffffffffa08254b0 ffff88081e695eb0 ffff88081e695e00 [18279.968103] Call Trace: [18279.971519] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18279.979488] [] schedule+0x29/0x70 [18279.985708] [] taskq_thread+0x3fa/0x440 [spl] [18279.993088] [] ? wake_up_state+0x20/0x20 [18279.999982] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18280.007943] [] kthread+0xcf/0xe0 [18280.014057] [] ? kthread_create_on_node+0x140/0x140 [18280.022015] [] ret_from_fork+0x58/0x90 [18280.028708] [] ? kthread_create_on_node+0x140/0x140 [18280.036662] spl_system_task S ffff88081e695e00 0 3123 2 0x00000080 [18280.045231] ffff880827bb3e08 0000000000000046 ffff88082d5c3ec0 ffff880827bb3fd8 [18280.054221] ffff880827bb3fd8 ffff880827bb3fd8 ffff88082d5c3ec0 ffff88042bcad840 [18280.063210] ffff88081e695e90 ffffffffa08254b0 ffff88081e695eb0 ffff88081e695e00 [18280.072205] Call Trace: [18280.075619] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18280.083591] [] schedule+0x29/0x70 [18280.089821] [] taskq_thread+0x3fa/0x440 [spl] [18280.097208] [] ? wake_up_state+0x20/0x20 [18280.104109] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18280.112086] [] kthread+0xcf/0xe0 [18280.118198] [] ? kthread_create_on_node+0x140/0x140 [18280.126149] [] ret_from_fork+0x58/0x90 [18280.132853] [] ? kthread_create_on_node+0x140/0x140 [18280.140810] spl_system_task S ffff88081e695e00 0 3124 2 0x00000080 [18280.149404] ffff880826abbe08 0000000000000046 ffff88082d5c4e70 ffff880826abbfd8 [18280.158407] ffff880826abbfd8 ffff880826abbfd8 ffff88082d5c4e70 ffff88042bcad120 [18280.167411] ffff88081e695e90 ffffffffa08254b0 ffff88081e695eb0 ffff88081e695e00 [18280.176409] Call Trace: [18280.179814] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18280.187790] [] schedule+0x29/0x70 [18280.194005] [] taskq_thread+0x3fa/0x440 [spl] [18280.201395] [] ? wake_up_state+0x20/0x20 [18280.208312] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18280.216273] [] kthread+0xcf/0xe0 [18280.222388] [] ? kthread_create_on_node+0x140/0x140 [18280.230353] [] ret_from_fork+0x58/0x90 [18280.237055] [] ? kthread_create_on_node+0x140/0x140 [18280.245019] spl_system_task S ffff88081e695e00 0 3125 2 0x00000080 [18280.253616] ffff88082debfe08 0000000000000046 ffff88082d5c5e20 ffff88082debffd8 [18280.262622] ffff88082debffd8 ffff88082debffd8 ffff88082d5c5e20 ffff88042bcada80 [18280.271630] ffff88081e695e90 ffffffffa08254b0 ffff88081e695eb0 ffff88081e695e00 [18280.280646] Call Trace: [18280.284055] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18280.292036] [] schedule+0x29/0x70 [18280.298254] [] taskq_thread+0x3fa/0x440 [spl] [18280.305645] [] ? wake_up_state+0x20/0x20 [18280.312543] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18280.320516] [] kthread+0xcf/0xe0 [18280.326638] [] ? kthread_create_on_node+0x140/0x140 [18280.334601] [] ret_from_fork+0x58/0x90 [18280.341302] [] ? kthread_create_on_node+0x140/0x140 [18280.349261] spl_system_task S ffff88081e695e00 0 3126 2 0x00000080 [18280.357842] ffff88082623be08 0000000000000046 ffff88082d5c6dd0 ffff88082623bfd8 [18280.366852] ffff88082623bfd8 ffff88082623bfd8 ffff88082d5c6dd0 ffff88042bcad2a0 [18280.375854] ffff88081e695e90 ffffffffa08254b0 ffff88081e695eb0 ffff88081e695e00 [18280.384855] Call Trace: [18280.388268] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18280.396241] [] schedule+0x29/0x70 [18280.402450] [] taskq_thread+0x3fa/0x440 [spl] [18280.409836] [] ? wake_up_state+0x20/0x20 [18280.416748] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18280.424711] [] kthread+0xcf/0xe0 [18280.430835] [] ? kthread_create_on_node+0x140/0x140 [18280.438803] [] ret_from_fork+0x58/0x90 [18280.445494] [] ? kthread_create_on_node+0x140/0x140 [18280.453456] spl_system_task S ffff88081e695e00 0 3127 2 0x00000080 [18280.462031] ffff88082e0b3e08 0000000000000046 ffff88082aba0000 ffff88082e0b3fd8 [18280.471030] ffff88082e0b3fd8 ffff88082e0b3fd8 ffff88082aba0000 ffff88042bcad180 [18280.480029] ffff88081e695e90 ffffffffa08254b0 ffff88081e695eb0 ffff88081e695e00 [18280.489027] Call Trace: [18280.492436] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18280.500416] [] schedule+0x29/0x70 [18280.506642] [] taskq_thread+0x3fa/0x440 [spl] [18280.514028] [] ? wake_up_state+0x20/0x20 [18280.520949] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18280.528922] [] kthread+0xcf/0xe0 [18280.535034] [] ? kthread_create_on_node+0x140/0x140 [18280.542992] [] ret_from_fork+0x58/0x90 [18280.549694] [] ? kthread_create_on_node+0x140/0x140 [18280.557659] spl_system_task S ffff88081e695e00 0 3128 2 0x00000080 [18280.566227] ffff88082e3b7e08 0000000000000046 ffff88082aba0fb0 ffff88082e3b7fd8 [18280.575229] ffff88082e3b7fd8 ffff88082e3b7fd8 ffff88082aba0fb0 ffff88042bcada20 [18280.584221] ffff88081e695e90 ffffffffa08254b0 ffff88081e695eb0 ffff88081e695e00 [18280.593225] Call Trace: [18280.596643] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18280.604624] [] schedule+0x29/0x70 [18280.610854] [] taskq_thread+0x3fa/0x440 [spl] [18280.618239] [] ? wake_up_state+0x20/0x20 [18280.625140] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18280.633098] [] kthread+0xcf/0xe0 [18280.639204] [] ? kthread_create_on_node+0x140/0x140 [18280.647162] [] ret_from_fork+0x58/0x90 [18280.653868] [] ? kthread_create_on_node+0x140/0x140 [18280.661836] spl_system_task S ffff88081e695e00 0 3129 2 0x00000080 [18280.670424] ffff88082cc03e08 0000000000000046 ffff88082aba1f60 ffff88082cc03fd8 [18280.679420] ffff88082cc03fd8 ffff88082cc03fd8 ffff88082aba1f60 ffff88042bcadba0 [18280.688411] ffff88081e695e90 ffffffffa08254b0 ffff88081e695eb0 ffff88081e695e00 [18280.697421] Call Trace: [18280.700825] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18280.708799] [] schedule+0x29/0x70 [18280.715031] [] taskq_thread+0x3fa/0x440 [spl] [18280.722428] [] ? wake_up_state+0x20/0x20 [18280.729332] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18280.737291] [] kthread+0xcf/0xe0 [18280.743411] [] ? kthread_create_on_node+0x140/0x140 [18280.751378] [] ret_from_fork+0x58/0x90 [18280.758064] [] ? kthread_create_on_node+0x140/0x140 [18280.766017] spl_system_task S ffff88081e695e00 0 3130 2 0x00000080 [18280.774598] ffff88082afb7e08 0000000000000046 ffff88082aba2f10 ffff88082afb7fd8 [18280.783610] ffff88082afb7fd8 ffff88082afb7fd8 ffff88082aba2f10 ffff88042bcad8a0 [18280.792625] ffff88081e695e90 ffffffffa08254b0 ffff88081e695eb0 ffff88081e695e00 [18280.801633] Call Trace: [18280.805040] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18280.813012] [] schedule+0x29/0x70 [18280.819225] [] taskq_thread+0x3fa/0x440 [spl] [18280.826622] [] ? wake_up_state+0x20/0x20 [18280.833523] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18280.841490] [] kthread+0xcf/0xe0 [18280.847606] [] ? kthread_create_on_node+0x140/0x140 [18280.855574] [] ret_from_fork+0x58/0x90 [18280.862268] [] ? kthread_create_on_node+0x140/0x140 [18280.870227] spl_system_task S ffff88081e695e00 0 3131 2 0x00000080 [18280.878808] ffff88082431fe08 0000000000000046 ffff88082aba3ec0 ffff88082431ffd8 [18280.887819] ffff88082431ffd8 ffff88082431ffd8 ffff88082aba3ec0 ffff88042bcadea0 [18280.896820] ffff88081e695e90 ffffffffa08254b0 ffff88081e695eb0 ffff88081e695e00 [18280.905825] Call Trace: [18280.909234] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18280.917209] [] schedule+0x29/0x70 [18280.923435] [] taskq_thread+0x3fa/0x440 [spl] [18280.930832] [] ? wake_up_state+0x20/0x20 [18280.937735] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18280.945699] [] kthread+0xcf/0xe0 [18280.951816] [] ? kthread_create_on_node+0x140/0x140 [18280.959782] [] ret_from_fork+0x58/0x90 [18280.966479] [] ? kthread_create_on_node+0x140/0x140 [18280.974439] spl_system_task S ffff88081e695e00 0 3132 2 0x00000080 [18280.983003] ffff880824323e08 0000000000000046 ffff88082aba4e70 ffff880824323fd8 [18280.992000] ffff880824323fd8 ffff880824323fd8 ffff88082aba4e70 ffff88042bcad420 [18281.000993] ffff88081e695e90 ffffffffa08254b0 ffff88081e695eb0 ffff88081e695e00 [18281.009990] Call Trace: [18281.013409] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18281.021394] [] schedule+0x29/0x70 [18281.027616] [] taskq_thread+0x3fa/0x440 [spl] [18281.035009] [] ? wake_up_state+0x20/0x20 [18281.041920] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18281.049891] [] kthread+0xcf/0xe0 [18281.056005] [] ? kthread_create_on_node+0x140/0x140 [18281.063978] [] ret_from_fork+0x58/0x90 [18281.070674] [] ? kthread_create_on_node+0x140/0x140 [18281.078644] spl_system_task S ffff88081e695e00 0 3133 2 0x00000080 [18281.087226] ffff880829c6be08 0000000000000046 ffff88082aba5e20 ffff880829c6bfd8 [18281.096208] ffff880829c6bfd8 ffff880829c6bfd8 ffff88082aba5e20 ffff88042bcadc00 [18281.105198] ffff88081e695e90 ffffffffa08254b0 ffff88081e695eb0 ffff88081e695e00 [18281.114196] Call Trace: [18281.117620] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18281.125604] [] schedule+0x29/0x70 [18281.131833] [] taskq_thread+0x3fa/0x440 [spl] [18281.139213] [] ? wake_up_state+0x20/0x20 [18281.146107] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18281.154064] [] kthread+0xcf/0xe0 [18281.160176] [] ? kthread_create_on_node+0x140/0x140 [18281.168134] [] ret_from_fork+0x58/0x90 [18281.174829] [] ? kthread_create_on_node+0x140/0x140 [18281.182785] spl_system_task S ffff88081e695e00 0 3134 2 0x00000080 [18281.191361] ffff88082af6fe08 0000000000000046 ffff88082aba6dd0 ffff88082af6ffd8 [18281.200362] ffff88082af6ffd8 ffff88082af6ffd8 ffff88082aba6dd0 ffff88042bcad600 [18281.209356] ffff88081e695e90 ffffffffa08254b0 ffff88081e695eb0 ffff88081e695e00 [18281.218347] Call Trace: [18281.221748] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18281.229726] [] schedule+0x29/0x70 [18281.235959] [] taskq_thread+0x3fa/0x440 [spl] [18281.243353] [] ? wake_up_state+0x20/0x20 [18281.250254] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18281.258217] [] kthread+0xcf/0xe0 [18281.264340] [] ? kthread_create_on_node+0x140/0x140 [18281.272292] [] ret_from_fork+0x58/0x90 [18281.278987] [] ? kthread_create_on_node+0x140/0x140 [18281.286946] spl_system_task S ffff88081e695e00 0 3135 2 0x00000080 [18281.295522] ffff880826fffe08 0000000000000046 ffff880825780000 ffff880826ffffd8 [18281.304536] ffff880826ffffd8 ffff880826ffffd8 ffff880825780000 ffff88042bcad6c0 [18281.313537] ffff88081e695e90 ffffffffa08254b0 ffff88081e695eb0 ffff88081e695e00 [18281.322544] Call Trace: [18281.325964] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18281.333937] [] schedule+0x29/0x70 [18281.340147] [] taskq_thread+0x3fa/0x440 [spl] [18281.347536] [] ? wake_up_state+0x20/0x20 [18281.354438] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18281.362400] [] kthread+0xcf/0xe0 [18281.368512] [] ? kthread_create_on_node+0x140/0x140 [18281.376470] [] ret_from_fork+0x58/0x90 [18281.383166] [] ? kthread_create_on_node+0x140/0x140 [18281.391122] spl_system_task S ffff88081e695e00 0 3136 2 0x00000080 [18281.399702] ffff88082db37e08 0000000000000046 ffff880825780fb0 ffff88082db37fd8 [18281.408715] ffff88082db37fd8 ffff88082db37fd8 ffff880825780fb0 ffff88042bcadcc0 [18281.417723] ffff88081e695e90 ffffffffa08254b0 ffff88081e695eb0 ffff88081e695e00 [18281.426723] Call Trace: [18281.430131] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18281.438104] [] schedule+0x29/0x70 [18281.444322] [] taskq_thread+0x3fa/0x440 [spl] [18281.451715] [] ? wake_up_state+0x20/0x20 [18281.458615] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18281.466584] [] kthread+0xcf/0xe0 [18281.472705] [] ? kthread_create_on_node+0x140/0x140 [18281.480670] [] ret_from_fork+0x58/0x90 [18281.487365] [] ? kthread_create_on_node+0x140/0x140 [18281.495332] spl_system_task S ffff88081e695e00 0 3137 2 0x00000080 [18281.503913] ffff880829f0be08 0000000000000046 ffff880825781f60 ffff880829f0bfd8 [18281.512927] ffff880829f0bfd8 ffff880829f0bfd8 ffff880825781f60 ffff88042bcadb40 [18281.521924] ffff88081e695e90 ffffffffa08254b0 ffff88081e695eb0 ffff88081e695e00 [18281.530923] Call Trace: [18281.534330] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18281.542301] [] schedule+0x29/0x70 [18281.548520] [] taskq_thread+0x3fa/0x440 [spl] [18281.555909] [] ? wake_up_state+0x20/0x20 [18281.562815] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18281.570776] [] kthread+0xcf/0xe0 [18281.576899] [] ? kthread_create_on_node+0x140/0x140 [18281.584865] [] ret_from_fork+0x58/0x90 [18281.591566] [] ? kthread_create_on_node+0x140/0x140 [18281.599531] spl_system_task S ffff88081e695e00 0 3138 2 0x00000080 [18281.608099] ffff88082c3cbe08 0000000000000046 ffff880825782f10 ffff88082c3cbfd8 [18281.617093] ffff88082c3cbfd8 ffff88082c3cbfd8 ffff880825782f10 ffff88042bcadde0 [18281.626094] ffff88081e695e90 ffffffffa08254b0 ffff88081e695eb0 ffff88081e695e00 [18281.635083] Call Trace: [18281.638501] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18281.646473] [] schedule+0x29/0x70 [18281.652691] [] taskq_thread+0x3fa/0x440 [spl] [18281.660077] [] ? wake_up_state+0x20/0x20 [18281.666987] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18281.674959] [] kthread+0xcf/0xe0 [18281.681068] [] ? kthread_create_on_node+0x140/0x140 [18281.689024] [] ret_from_fork+0x58/0x90 [18281.695727] [] ? kthread_create_on_node+0x140/0x140 [18281.703693] spl_system_task S ffff88081e695e00 0 3139 2 0x00000080 [18281.712265] ffff8808266ebe08 0000000000000046 ffff880825783ec0 ffff8808266ebfd8 [18281.721261] ffff8808266ebfd8 ffff8808266ebfd8 ffff880825783ec0 ffff88042bcad7e0 [18281.730265] ffff88081e695e90 ffffffffa08254b0 ffff88081e695eb0 ffff88081e695e00 [18281.739255] Call Trace: [18281.742659] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18281.750636] [] schedule+0x29/0x70 [18281.756864] [] taskq_thread+0x3fa/0x440 [spl] [18281.764250] [] ? wake_up_state+0x20/0x20 [18281.771151] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18281.779108] [] kthread+0xcf/0xe0 [18281.785211] [] ? kthread_create_on_node+0x140/0x140 [18281.793171] [] ret_from_fork+0x58/0x90 [18281.799873] [] ? kthread_create_on_node+0x140/0x140 [18281.807839] spl_system_task S ffff88081e695e00 0 3140 2 0x00000080 [18281.816430] ffff8808266c3e08 0000000000000046 ffff880825784e70 ffff8808266c3fd8 [18281.825431] ffff8808266c3fd8 ffff8808266c3fd8 ffff880825784e70 ffff88042bcad4e0 [18281.834436] ffff88081e695e90 ffffffffa08254b0 ffff88081e695eb0 ffff88081e695e00 [18281.843433] Call Trace: [18281.846844] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18281.854835] [] schedule+0x29/0x70 [18281.861050] [] taskq_thread+0x3fa/0x440 [spl] [18281.868435] [] ? wake_up_state+0x20/0x20 [18281.875338] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18281.883298] [] kthread+0xcf/0xe0 [18281.889412] [] ? kthread_create_on_node+0x140/0x140 [18281.897384] [] ret_from_fork+0x58/0x90 [18281.904080] [] ? kthread_create_on_node+0x140/0x140 [18281.912030] spl_system_task S ffff88081e695e00 0 3141 2 0x00000080 [18281.920627] ffff88082e57fe08 0000000000000046 ffff880825785e20 ffff88082e57ffd8 [18281.929626] ffff88082e57ffd8 ffff88082e57ffd8 ffff880825785e20 ffff88042bcadae0 [18281.938627] ffff88081e695e90 ffffffffa08254b0 ffff88081e695eb0 ffff88081e695e00 [18281.947611] Call Trace: [18281.951022] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18281.959006] [] schedule+0x29/0x70 [18281.965222] [] taskq_thread+0x3fa/0x440 [spl] [18281.972617] [] ? wake_up_state+0x20/0x20 [18281.979517] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18281.987486] [] kthread+0xcf/0xe0 [18281.993607] [] ? kthread_create_on_node+0x140/0x140 [18282.001564] [] ret_from_fork+0x58/0x90 [18282.008256] [] ? kthread_create_on_node+0x140/0x140 [18282.016211] spl_system_task S ffff88081e695e00 0 3142 2 0x00000080 [18282.024802] ffff88082bf03e08 0000000000000046 ffff880825786dd0 ffff88082bf03fd8 [18282.033816] ffff88082bf03fd8 ffff88082bf03fd8 ffff880825786dd0 ffff88042bcad300 [18282.042806] ffff88081e695e90 ffffffffa08254b0 ffff88081e695eb0 ffff88081e695e00 [18282.051803] Call Trace: [18282.055215] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18282.063186] [] schedule+0x29/0x70 [18282.069418] [] taskq_thread+0x3fa/0x440 [spl] [18282.076814] [] ? wake_up_state+0x20/0x20 [18282.083714] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18282.091674] [] kthread+0xcf/0xe0 [18282.097794] [] ? kthread_create_on_node+0x140/0x140 [18282.105754] [] ret_from_fork+0x58/0x90 [18282.112442] [] ? kthread_create_on_node+0x140/0x140 [18282.120399] spl_system_task S ffff88081e695e00 0 3143 2 0x00000080 [18282.128991] ffff880827f5fe08 0000000000000046 ffff8808257c0000 ffff880827f5ffd8 [18282.137982] ffff880827f5ffd8 ffff880827f5ffd8 ffff8808257c0000 ffff88042bcad1e0 [18282.146982] ffff88081e695e90 ffffffffa08254b0 ffff88081e695eb0 ffff88081e695e00 [18282.155975] Call Trace: [18282.159375] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18282.167338] [] schedule+0x29/0x70 [18282.173552] [] taskq_thread+0x3fa/0x440 [spl] [18282.180952] [] ? wake_up_state+0x20/0x20 [18282.187843] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18282.195810] [] kthread+0xcf/0xe0 [18282.201923] [] ? kthread_create_on_node+0x140/0x140 [18282.209872] [] ret_from_fork+0x58/0x90 [18282.216565] [] ? kthread_create_on_node+0x140/0x140 [18282.224534] spl_system_task S ffff88081e695e00 0 3144 2 0x00000080 [18282.233112] ffff880823e17e08 0000000000000046 ffff8808257c0fb0 ffff880823e17fd8 [18282.242095] ffff880823e17fd8 ffff880823e17fd8 ffff8808257c0fb0 ffff88042bcad9c0 [18282.251086] ffff88081e695e90 ffffffffa08254b0 ffff88081e695eb0 ffff88081e695e00 [18282.260086] Call Trace: [18282.263494] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18282.271463] [] schedule+0x29/0x70 [18282.277685] [] taskq_thread+0x3fa/0x440 [spl] [18282.285076] [] ? wake_up_state+0x20/0x20 [18282.291976] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18282.299951] [] kthread+0xcf/0xe0 [18282.306059] [] ? kthread_create_on_node+0x140/0x140 [18282.314018] [] ret_from_fork+0x58/0x90 [18282.320719] [] ? kthread_create_on_node+0x140/0x140 [18282.328671] spl_system_task S ffff88081e695e00 0 3145 2 0x00000080 [18282.337235] ffff88081c33fe08 0000000000000046 ffff8808257c1f60 ffff88081c33ffd8 [18282.346225] ffff88081c33ffd8 ffff88081c33ffd8 ffff8808257c1f60 ffff88042bcad780 [18282.355226] ffff88081e695e90 ffffffffa08254b0 ffff88081e695eb0 ffff88081e695e00 [18282.364217] Call Trace: [18282.367621] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18282.375592] [] schedule+0x29/0x70 [18282.381811] [] taskq_thread+0x3fa/0x440 [spl] [18282.389193] [] ? wake_up_state+0x20/0x20 [18282.396089] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18282.404049] [] kthread+0xcf/0xe0 [18282.410159] [] ? kthread_create_on_node+0x140/0x140 [18282.418114] [] ret_from_fork+0x58/0x90 [18282.424816] [] ? kthread_create_on_node+0x140/0x140 [18282.432773] spl_system_task S ffff88081e695e00 0 3146 2 0x00000080 [18282.441343] ffff8808296dfe08 0000000000000046 ffff8808257c2f10 ffff8808296dffd8 [18282.450338] ffff8808296dffd8 ffff8808296dffd8 ffff8808257c2f10 ffff88042bcad900 [18282.459321] ffff88081e695e90 ffffffffa08254b0 ffff88081e695eb0 ffff88081e695e00 [18282.468304] Call Trace: [18282.471717] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18282.479701] [] schedule+0x29/0x70 [18282.485917] [] taskq_thread+0x3fa/0x440 [spl] [18282.493294] [] ? wake_up_state+0x20/0x20 [18282.500193] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18282.508153] [] kthread+0xcf/0xe0 [18282.514262] [] ? kthread_create_on_node+0x140/0x140 [18282.522219] [] ret_from_fork+0x58/0x90 [18282.528927] [] ? kthread_create_on_node+0x140/0x140 [18282.536886] spl_system_task S ffff88081e695e00 0 3147 2 0x00000080 [18282.545470] ffff88082d83be08 0000000000000046 ffff8808257c3ec0 ffff88082d83bfd8 [18282.554471] ffff88082d83bfd8 ffff88082d83bfd8 ffff8808257c3ec0 ffff88042bcadf00 [18282.563461] ffff88081e695e90 ffffffffa08254b0 ffff88081e695eb0 ffff88081e695e00 [18282.572452] Call Trace: [18282.575865] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18282.583843] [] schedule+0x29/0x70 [18282.590060] [] taskq_thread+0x3fa/0x440 [spl] [18282.597444] [] ? wake_up_state+0x20/0x20 [18282.604342] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18282.612296] [] kthread+0xcf/0xe0 [18282.618415] [] ? kthread_create_on_node+0x140/0x140 [18282.626381] [] ret_from_fork+0x58/0x90 [18282.633081] [] ? kthread_create_on_node+0x140/0x140 [18282.641037] spl_system_task S ffff88081e695e00 0 3148 2 0x00000080 [18282.649627] ffff88082432be08 0000000000000046 ffff8808257c4e70 ffff88082432bfd8 [18282.658636] ffff88082432bfd8 ffff88082432bfd8 ffff8808257c4e70 ffff88042bcad3c0 [18282.667645] ffff88081e695e90 ffffffffa08254b0 ffff88081e695eb0 ffff88081e695e00 [18282.676660] Call Trace: [18282.680066] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18282.688036] [] schedule+0x29/0x70 [18282.694253] [] taskq_thread+0x3fa/0x440 [spl] [18282.701642] [] ? wake_up_state+0x20/0x20 [18282.708542] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18282.716508] [] kthread+0xcf/0xe0 [18282.722633] [] ? kthread_create_on_node+0x140/0x140 [18282.730598] [] ret_from_fork+0x58/0x90 [18282.737291] [] ? kthread_create_on_node+0x140/0x140 [18282.745242] spl_system_task S ffff88081e695e00 0 3149 2 0x00000080 [18282.753827] ffff88082d607e08 0000000000000046 ffff8808257c5e20 ffff88082d607fd8 [18282.762819] ffff88082d607fd8 ffff88082d607fd8 ffff8808257c5e20 ffff88042bcadf60 [18282.771822] ffff88081e695e90 ffffffffa08254b0 ffff88081e695eb0 ffff88081e695e00 [18282.780817] Call Trace: [18282.784224] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18282.792194] [] schedule+0x29/0x70 [18282.798414] [] taskq_thread+0x3fa/0x440 [spl] [18282.805803] [] ? wake_up_state+0x20/0x20 [18282.812704] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18282.820674] [] kthread+0xcf/0xe0 [18282.826792] [] ? kthread_create_on_node+0x140/0x140 [18282.834759] [] ret_from_fork+0x58/0x90 [18282.841451] [] ? kthread_create_on_node+0x140/0x140 [18282.849410] spl_dynamic_tas S ffff88081e694200 0 3150 2 0x00000080 [18282.857992] ffff8808296f3e08 0000000000000046 ffff8808257c6dd0 ffff8808296f3fd8 [18282.866993] ffff8808296f3fd8 ffff8808296f3fd8 ffff8808257c6dd0 ffff88042bcad5a0 [18282.875998] ffff88081e694290 ffffffffa08254b0 ffff88081e6942b0 ffff88081e694200 [18282.884994] Call Trace: [18282.888404] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18282.896381] [] schedule+0x29/0x70 [18282.902607] [] taskq_thread+0x3fa/0x440 [spl] [18282.909998] [] ? wake_up_state+0x20/0x20 [18282.916891] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18282.924860] [] kthread+0xcf/0xe0 [18282.930972] [] ? kthread_create_on_node+0x140/0x140 [18282.938937] [] ret_from_fork+0x58/0x90 [18282.945640] [] ? kthread_create_on_node+0x140/0x140 [18282.953598] dbu_evict S ffff88081e695900 0 3155 2 0x00000080 [18282.962180] ffff88082d023e08 0000000000000046 ffff88082d270000 ffff88082d023fd8 [18282.971172] ffff88082d023fd8 ffff88082d023fd8 ffff88082d270000 ffff880826ff6e40 [18282.980173] ffff88081e695990 ffffffffa08254b0 ffff88081e6959b0 ffff88081e695900 [18282.989159] Call Trace: [18282.992564] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18283.000549] [] schedule+0x29/0x70 [18283.006767] [] taskq_thread+0x3fa/0x440 [spl] [18283.014152] [] ? wake_up_state+0x20/0x20 [18283.021054] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18283.029014] [] kthread+0xcf/0xe0 [18283.035123] [] ? kthread_create_on_node+0x140/0x140 [18283.043079] [] ret_from_fork+0x58/0x90 [18283.049783] [] ? kthread_create_on_node+0x140/0x140 [18283.057737] arc_prune S ffff88081e694d00 0 3156 2 0x00000080 [18283.066303] ffff880825fc7e08 0000000000000046 ffff88082d270fb0 ffff880825fc7fd8 [18283.075297] ffff880825fc7fd8 ffff880825fc7fd8 ffff88082d270fb0 ffff880826ff6de0 [18283.084285] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18283.093270] Call Trace: [18283.096684] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18283.104654] [] schedule+0x29/0x70 [18283.110873] [] taskq_thread+0x3fa/0x440 [spl] [18283.118251] [] ? wake_up_state+0x20/0x20 [18283.125145] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18283.133106] [] kthread+0xcf/0xe0 [18283.139217] [] ? kthread_create_on_node+0x140/0x140 [18283.147181] [] ret_from_fork+0x58/0x90 [18283.153872] [] ? kthread_create_on_node+0x140/0x140 [18283.161845] arc_prune S ffff88081e694d00 0 3157 2 0x00000080 [18283.170427] ffff8808266e3e08 0000000000000046 ffff88082d271f60 ffff8808266e3fd8 [18283.179430] ffff8808266e3fd8 ffff8808266e3fd8 ffff88082d271f60 ffff880826ff6d80 [18283.188433] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18283.197416] Call Trace: [18283.200822] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18283.208795] [] schedule+0x29/0x70 [18283.215026] [] taskq_thread+0x3fa/0x440 [spl] [18283.222414] [] ? wake_up_state+0x20/0x20 [18283.229309] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18283.237270] [] kthread+0xcf/0xe0 [18283.243378] [] ? kthread_create_on_node+0x140/0x140 [18283.251329] [] ret_from_fork+0x58/0x90 [18283.258021] [] ? kthread_create_on_node+0x140/0x140 [18283.265986] arc_prune S ffff88081e694d00 0 3158 2 0x00000080 [18283.274577] ffff8808266dfe08 0000000000000046 ffff88082d272f10 ffff8808266dffd8 [18283.283570] ffff8808266dffd8 ffff8808266dffd8 ffff88082d272f10 ffff880826ff6d20 [18283.292565] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18283.301566] Call Trace: [18283.304968] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18283.312939] [] schedule+0x29/0x70 [18283.319149] [] taskq_thread+0x3fa/0x440 [spl] [18283.326523] [] ? wake_up_state+0x20/0x20 [18283.333427] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18283.341388] [] kthread+0xcf/0xe0 [18283.347506] [] ? kthread_create_on_node+0x140/0x140 [18283.355464] [] ret_from_fork+0x58/0x90 [18283.362157] [] ? kthread_create_on_node+0x140/0x140 [18283.370121] arc_prune S ffff88081e694d00 0 3159 2 0x00000080 [18283.378692] ffff8808256b3e08 0000000000000046 ffff88082d273ec0 ffff8808256b3fd8 [18283.387686] ffff8808256b3fd8 ffff8808256b3fd8 ffff88082d273ec0 ffff880826ff6cc0 [18283.396696] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18283.405685] Call Trace: [18283.409094] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18283.417071] [] schedule+0x29/0x70 [18283.423285] [] taskq_thread+0x3fa/0x440 [spl] [18283.430673] [] ? wake_up_state+0x20/0x20 [18283.437571] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18283.445528] [] kthread+0xcf/0xe0 [18283.451647] [] ? kthread_create_on_node+0x140/0x140 [18283.459613] [] ret_from_fork+0x58/0x90 [18283.466304] [] ? kthread_create_on_node+0x140/0x140 [18283.474259] arc_prune S ffff88081e694d00 0 3160 2 0x00000080 [18283.482846] ffff88082d267e08 0000000000000046 ffff88082d274e70 ffff88082d267fd8 [18283.491854] ffff88082d267fd8 ffff88082d267fd8 ffff88082d274e70 ffff880826ff6c60 [18283.500860] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18283.509868] Call Trace: [18283.513273] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18283.521236] [] schedule+0x29/0x70 [18283.527464] [] taskq_thread+0x3fa/0x440 [spl] [18283.534844] [] ? wake_up_state+0x20/0x20 [18283.541750] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18283.549717] [] kthread+0xcf/0xe0 [18283.555840] [] ? kthread_create_on_node+0x140/0x140 [18283.563804] [] ret_from_fork+0x58/0x90 [18283.570499] [] ? kthread_create_on_node+0x140/0x140 [18283.578457] arc_prune S ffff88081e694d00 0 3161 2 0x00000080 [18283.587037] ffff88082bddbe08 0000000000000046 ffff88082d275e20 ffff88082bddbfd8 [18283.596040] ffff88082bddbfd8 ffff88082bddbfd8 ffff88082d275e20 ffff880826ff6c00 [18283.605047] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18283.614048] Call Trace: [18283.617451] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18283.625429] [] schedule+0x29/0x70 [18283.631653] [] taskq_thread+0x3fa/0x440 [spl] [18283.639038] [] ? wake_up_state+0x20/0x20 [18283.645946] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18283.653909] [] kthread+0xcf/0xe0 [18283.660035] [] ? kthread_create_on_node+0x140/0x140 [18283.667997] [] ret_from_fork+0x58/0x90 [18283.674697] [] ? kthread_create_on_node+0x140/0x140 [18283.682662] arc_prune S ffff88081e694d00 0 3162 2 0x00000080 [18283.691230] ffff8808253ffe08 0000000000000046 ffff88082d276dd0 ffff8808253fffd8 [18283.700224] ffff8808253fffd8 ffff8808253fffd8 ffff88082d276dd0 ffff880826ff6ba0 [18283.709218] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18283.718218] Call Trace: [18283.721634] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18283.729602] [] schedule+0x29/0x70 [18283.735829] [] taskq_thread+0x3fa/0x440 [spl] [18283.743203] [] ? wake_up_state+0x20/0x20 [18283.750102] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18283.758069] [] kthread+0xcf/0xe0 [18283.764172] [] ? kthread_create_on_node+0x140/0x140 [18283.772119] [] ret_from_fork+0x58/0x90 [18283.778822] [] ? kthread_create_on_node+0x140/0x140 [18283.786781] arc_prune S ffff88081e694d00 0 3163 2 0x00000080 [18283.795344] ffff88082bd63e08 0000000000000046 ffff88083fe10000 ffff88082bd63fd8 [18283.804339] ffff88082bd63fd8 ffff88082bd63fd8 ffff88083fe10000 ffff880826ff6b40 [18283.813339] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18283.822323] Call Trace: [18283.825740] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18283.833719] [] schedule+0x29/0x70 [18283.839937] [] taskq_thread+0x3fa/0x440 [spl] [18283.847321] [] ? wake_up_state+0x20/0x20 [18283.854222] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18283.862185] [] kthread+0xcf/0xe0 [18283.868294] [] ? kthread_create_on_node+0x140/0x140 [18283.876248] [] ret_from_fork+0x58/0x90 [18283.882938] [] ? kthread_create_on_node+0x140/0x140 [18283.890894] arc_prune S ffff88081e694d00 0 3164 2 0x00000080 [18283.899476] ffff88082c277e08 0000000000000046 ffff88083fe10fb0 ffff88082c277fd8 [18283.908467] ffff88082c277fd8 ffff88082c277fd8 ffff88083fe10fb0 ffff880826ff6ae0 [18283.917453] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18283.926446] Call Trace: [18283.929854] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18283.937824] [] schedule+0x29/0x70 [18283.944053] [] taskq_thread+0x3fa/0x440 [spl] [18283.951439] [] ? wake_up_state+0x20/0x20 [18283.958338] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18283.966294] [] kthread+0xcf/0xe0 [18283.972397] [] ? kthread_create_on_node+0x140/0x140 [18283.980350] [] ret_from_fork+0x58/0x90 [18283.987051] [] ? kthread_create_on_node+0x140/0x140 [18283.995017] arc_prune S ffff88081e694d00 0 3165 2 0x00000080 [18284.003603] ffff880826a3fe08 0000000000000046 ffff88083fe11f60 ffff880826a3ffd8 [18284.012602] ffff880826a3ffd8 ffff880826a3ffd8 ffff88083fe11f60 ffff880826ff6a80 [18284.021602] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18284.030591] Call Trace: [18284.034008] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18284.041986] [] schedule+0x29/0x70 [18284.048202] [] taskq_thread+0x3fa/0x440 [spl] [18284.055601] [] ? wake_up_state+0x20/0x20 [18284.062501] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18284.070463] [] kthread+0xcf/0xe0 [18284.076570] [] ? kthread_create_on_node+0x140/0x140 [18284.084547] [] ret_from_fork+0x58/0x90 [18284.091235] [] ? kthread_create_on_node+0x140/0x140 [18284.099195] arc_prune S ffff88081e694d00 0 3166 2 0x00000080 [18284.107786] ffff880827f6be08 0000000000000046 ffff88083fe12f10 ffff880827f6bfd8 [18284.116802] ffff880827f6bfd8 ffff880827f6bfd8 ffff88083fe12f10 ffff880826ff6a20 [18284.125807] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18284.134814] Call Trace: [18284.138224] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18284.146193] [] schedule+0x29/0x70 [18284.152400] [] taskq_thread+0x3fa/0x440 [spl] [18284.159791] [] ? wake_up_state+0x20/0x20 [18284.166699] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18284.174658] [] kthread+0xcf/0xe0 [18284.180775] [] ? kthread_create_on_node+0x140/0x140 [18284.188743] [] ret_from_fork+0x58/0x90 [18284.195430] [] ? kthread_create_on_node+0x140/0x140 [18284.203381] arc_prune S ffff88081e694d00 0 3167 2 0x00000080 [18284.211961] ffff88082e033e08 0000000000000046 ffff88083fe13ec0 ffff88082e033fd8 [18284.220963] ffff88082e033fd8 ffff88082e033fd8 ffff88083fe13ec0 ffff880826ff69c0 [18284.229968] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18284.238965] Call Trace: [18284.242362] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18284.250327] [] schedule+0x29/0x70 [18284.256540] [] taskq_thread+0x3fa/0x440 [spl] [18284.263937] [] ? wake_up_state+0x20/0x20 [18284.270839] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18284.278800] [] kthread+0xcf/0xe0 [18284.284921] [] ? kthread_create_on_node+0x140/0x140 [18284.292876] [] ret_from_fork+0x58/0x90 [18284.299567] [] ? kthread_create_on_node+0x140/0x140 [18284.307523] arc_prune S ffff88081e694d00 0 3168 2 0x00000080 [18284.316081] ffff880827e4be08 0000000000000046 ffff88083fe14e70 ffff880827e4bfd8 [18284.325082] ffff880827e4bfd8 ffff880827e4bfd8 ffff88083fe14e70 ffff880826ff6960 [18284.334081] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18284.343075] Call Trace: [18284.346488] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18284.354475] [] schedule+0x29/0x70 [18284.360705] [] taskq_thread+0x3fa/0x440 [spl] [18284.368096] [] ? wake_up_state+0x20/0x20 [18284.374993] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18284.382960] [] kthread+0xcf/0xe0 [18284.389083] [] ? kthread_create_on_node+0x140/0x140 [18284.397053] [] ret_from_fork+0x58/0x90 [18284.403746] [] ? kthread_create_on_node+0x140/0x140 [18284.411704] arc_prune S ffff88081e694d00 0 3169 2 0x00000080 [18284.420268] ffff880824bdbe08 0000000000000046 ffff88083fe15e20 ffff880824bdbfd8 [18284.429267] ffff880824bdbfd8 ffff880824bdbfd8 ffff88083fe15e20 ffff880826ff6900 [18284.438259] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18284.447256] Call Trace: [18284.450666] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18284.458638] [] schedule+0x29/0x70 [18284.464860] [] taskq_thread+0x3fa/0x440 [spl] [18284.472238] [] ? wake_up_state+0x20/0x20 [18284.479129] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18284.487093] [] kthread+0xcf/0xe0 [18284.493192] [] ? kthread_create_on_node+0x140/0x140 [18284.501153] [] ret_from_fork+0x58/0x90 [18284.507853] [] ? kthread_create_on_node+0x140/0x140 [18284.515816] arc_prune S ffff88081e694d00 0 3170 2 0x00000080 [18284.524389] ffff8808233efe08 0000000000000046 ffff88083fe16dd0 ffff8808233effd8 [18284.533373] ffff8808233effd8 ffff8808233effd8 ffff88083fe16dd0 ffff880826ff68a0 [18284.542362] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18284.551361] Call Trace: [18284.554771] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18284.562742] [] schedule+0x29/0x70 [18284.568959] [] taskq_thread+0x3fa/0x440 [spl] [18284.576341] [] ? wake_up_state+0x20/0x20 [18284.583238] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18284.591195] [] kthread+0xcf/0xe0 [18284.597297] [] ? kthread_create_on_node+0x140/0x140 [18284.605247] [] ret_from_fork+0x58/0x90 [18284.611948] [] ? kthread_create_on_node+0x140/0x140 [18284.619917] arc_prune S ffff88081e694d00 0 3171 2 0x00000080 [18284.628496] ffff880829707e08 0000000000000046 ffff880825f88000 ffff880829707fd8 [18284.637502] ffff880829707fd8 ffff880829707fd8 ffff880825f88000 ffff880826ff6840 [18284.646509] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18284.655485] Call Trace: [18284.658905] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18284.666886] [] schedule+0x29/0x70 [18284.673104] [] taskq_thread+0x3fa/0x440 [spl] [18284.680502] [] ? wake_up_state+0x20/0x20 [18284.687400] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18284.695350] [] kthread+0xcf/0xe0 [18284.701467] [] ? kthread_create_on_node+0x140/0x140 [18284.709415] [] ret_from_fork+0x58/0x90 [18284.716106] [] ? kthread_create_on_node+0x140/0x140 [18284.724072] arc_prune S ffff88081e694d00 0 3172 2 0x00000080 [18284.732642] ffff880826f3fe08 0000000000000046 ffff880825f88fb0 ffff880826f3ffd8 [18284.741638] ffff880826f3ffd8 ffff880826f3ffd8 ffff880825f88fb0 ffff880826ff67e0 [18284.750637] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18284.759644] Call Trace: [18284.763053] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18284.771035] [] schedule+0x29/0x70 [18284.777249] [] taskq_thread+0x3fa/0x440 [spl] [18284.784638] [] ? wake_up_state+0x20/0x20 [18284.791545] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18284.799496] [] kthread+0xcf/0xe0 [18284.805617] [] ? kthread_create_on_node+0x140/0x140 [18284.813576] [] ret_from_fork+0x58/0x90 [18284.820267] [] ? kthread_create_on_node+0x140/0x140 [18284.828222] arc_prune S ffff88081e694d00 0 3173 2 0x00000080 [18284.836794] ffff880825ef3e08 0000000000000046 ffff880825f89f60 ffff880825ef3fd8 [18284.845799] ffff880825ef3fd8 ffff880825ef3fd8 ffff880825f89f60 ffff880826ff6780 [18284.854802] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18284.863808] Call Trace: [18284.867216] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18284.875175] [] schedule+0x29/0x70 [18284.881385] [] taskq_thread+0x3fa/0x440 [spl] [18284.888766] [] ? wake_up_state+0x20/0x20 [18284.895668] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18284.903625] [] kthread+0xcf/0xe0 [18284.909737] [] ? kthread_create_on_node+0x140/0x140 [18284.917708] [] ret_from_fork+0x58/0x90 [18284.924392] [] ? kthread_create_on_node+0x140/0x140 [18284.932349] arc_prune S ffff88081e694d00 0 3174 2 0x00000080 [18284.940923] ffff88082578fe08 0000000000000046 ffff880825f8af10 ffff88082578ffd8 [18284.949917] ffff88082578ffd8 ffff88082578ffd8 ffff880825f8af10 ffff880826ff6720 [18284.958905] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18284.967898] Call Trace: [18284.971295] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18284.979260] [] schedule+0x29/0x70 [18284.985485] [] taskq_thread+0x3fa/0x440 [spl] [18284.992868] [] ? wake_up_state+0x20/0x20 [18284.999772] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18285.007738] [] kthread+0xcf/0xe0 [18285.013852] [] ? kthread_create_on_node+0x140/0x140 [18285.021804] [] ret_from_fork+0x58/0x90 [18285.028494] [] ? kthread_create_on_node+0x140/0x140 [18285.036448] arc_prune S ffff88081e694d00 0 3175 2 0x00000080 [18285.045029] ffff880827bcbe08 0000000000000046 ffff880825f8bec0 ffff880827bcbfd8 [18285.054038] ffff880827bcbfd8 ffff880827bcbfd8 ffff880825f8bec0 ffff880826ff66c0 [18285.063035] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18285.072046] Call Trace: [18285.075456] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18285.083427] [] schedule+0x29/0x70 [18285.089640] [] taskq_thread+0x3fa/0x440 [spl] [18285.097040] [] ? wake_up_state+0x20/0x20 [18285.103958] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18285.111933] [] kthread+0xcf/0xe0 [18285.118049] [] ? kthread_create_on_node+0x140/0x140 [18285.126009] [] ret_from_fork+0x58/0x90 [18285.132703] [] ? kthread_create_on_node+0x140/0x140 [18285.140661] arc_prune S ffff88081e694d00 0 3176 2 0x00000080 [18285.149228] ffff8808233dfe08 0000000000000046 ffff880825f8ce70 ffff8808233dffd8 [18285.158219] ffff8808233dffd8 ffff8808233dffd8 ffff880825f8ce70 ffff880826ff6660 [18285.167216] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18285.176208] Call Trace: [18285.179630] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18285.187601] [] schedule+0x29/0x70 [18285.193814] [] taskq_thread+0x3fa/0x440 [spl] [18285.201205] [] ? wake_up_state+0x20/0x20 [18285.208105] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18285.216081] [] kthread+0xcf/0xe0 [18285.222190] [] ? kthread_create_on_node+0x140/0x140 [18285.230147] [] ret_from_fork+0x58/0x90 [18285.236850] [] ? kthread_create_on_node+0x140/0x140 [18285.244813] arc_prune S ffff88081e694d00 0 3177 2 0x00000080 [18285.253380] ffff88082e2f3e08 0000000000000046 ffff880825f8de20 ffff88082e2f3fd8 [18285.262381] ffff88082e2f3fd8 ffff88082e2f3fd8 ffff880825f8de20 ffff880826ff6600 [18285.271383] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18285.280375] Call Trace: [18285.283789] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18285.291760] [] schedule+0x29/0x70 [18285.297981] [] taskq_thread+0x3fa/0x440 [spl] [18285.305361] [] ? wake_up_state+0x20/0x20 [18285.312257] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18285.320217] [] kthread+0xcf/0xe0 [18285.326324] [] ? kthread_create_on_node+0x140/0x140 [18285.334280] [] ret_from_fork+0x58/0x90 [18285.340962] [] ? kthread_create_on_node+0x140/0x140 [18285.348918] arc_prune S ffff88081e694d00 0 3178 2 0x00000080 [18285.357504] ffff88083fb3fe08 0000000000000046 ffff880825f8edd0 ffff88083fb3ffd8 [18285.366494] ffff88083fb3ffd8 ffff88083fb3ffd8 ffff880825f8edd0 ffff880826ff65a0 [18285.375497] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18285.384485] Call Trace: [18285.387893] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18285.395868] [] schedule+0x29/0x70 [18285.402092] [] taskq_thread+0x3fa/0x440 [spl] [18285.409478] [] ? wake_up_state+0x20/0x20 [18285.416378] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18285.424339] [] kthread+0xcf/0xe0 [18285.430442] [] ? kthread_create_on_node+0x140/0x140 [18285.438382] [] ret_from_fork+0x58/0x90 [18285.445082] [] ? kthread_create_on_node+0x140/0x140 [18285.453044] arc_prune S ffff88081e694d00 0 3179 2 0x00000080 [18285.461637] ffff8808256bbe08 0000000000000046 ffff8808263b8000 ffff8808256bbfd8 [18285.470620] ffff8808256bbfd8 ffff8808256bbfd8 ffff8808263b8000 ffff880826ff6540 [18285.479616] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18285.488621] Call Trace: [18285.492036] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18285.500005] [] schedule+0x29/0x70 [18285.506218] [] taskq_thread+0x3fa/0x440 [spl] [18285.513603] [] ? wake_up_state+0x20/0x20 [18285.520505] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18285.528463] [] kthread+0xcf/0xe0 [18285.534571] [] ? kthread_create_on_node+0x140/0x140 [18285.542539] [] ret_from_fork+0x58/0x90 [18285.549238] [] ? kthread_create_on_node+0x140/0x140 [18285.557194] arc_prune S ffff88081e694d00 0 3180 2 0x00000080 [18285.565775] ffff880829bf3e08 0000000000000046 ffff8808263b8fb0 ffff880829bf3fd8 [18285.574771] ffff880829bf3fd8 ffff880829bf3fd8 ffff8808263b8fb0 ffff880826ff64e0 [18285.583776] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18285.592778] Call Trace: [18285.596185] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18285.604150] [] schedule+0x29/0x70 [18285.610363] [] taskq_thread+0x3fa/0x440 [spl] [18285.617754] [] ? wake_up_state+0x20/0x20 [18285.624654] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18285.632616] [] kthread+0xcf/0xe0 [18285.638726] [] ? kthread_create_on_node+0x140/0x140 [18285.646681] [] ret_from_fork+0x58/0x90 [18285.653371] [] ? kthread_create_on_node+0x140/0x140 [18285.661325] arc_prune S ffff88081e694d00 0 3181 2 0x00000080 [18285.669899] ffff88083fdcfe08 0000000000000046 ffff8808263b9f60 ffff88083fdcffd8 [18285.678921] ffff88083fdcffd8 ffff88083fdcffd8 ffff8808263b9f60 ffff880826ff6480 [18285.687922] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18285.696918] Call Trace: [18285.700332] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18285.708302] [] schedule+0x29/0x70 [18285.714526] [] taskq_thread+0x3fa/0x440 [spl] [18285.721918] [] ? wake_up_state+0x20/0x20 [18285.728816] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18285.736767] [] kthread+0xcf/0xe0 [18285.742875] [] ? kthread_create_on_node+0x140/0x140 [18285.750841] [] ret_from_fork+0x58/0x90 [18285.757530] [] ? kthread_create_on_node+0x140/0x140 [18285.765492] arc_prune S ffff88081e694d00 0 3182 2 0x00000080 [18285.774057] ffff880829a63e08 0000000000000046 ffff8808263baf10 ffff880829a63fd8 [18285.783067] ffff880829a63fd8 ffff880829a63fd8 ffff8808263baf10 ffff880826ff6420 [18285.792077] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18285.801082] Call Trace: [18285.804494] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18285.812467] [] schedule+0x29/0x70 [18285.818682] [] taskq_thread+0x3fa/0x440 [spl] [18285.826076] [] ? wake_up_state+0x20/0x20 [18285.832978] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18285.840939] [] kthread+0xcf/0xe0 [18285.847061] [] ? kthread_create_on_node+0x140/0x140 [18285.855025] [] ret_from_fork+0x58/0x90 [18285.861727] [] ? kthread_create_on_node+0x140/0x140 [18285.869692] arc_prune S ffff88081e694d00 0 3183 2 0x00000080 [18285.878261] ffff88083f687e08 0000000000000046 ffff8808263bbec0 ffff88083f687fd8 [18285.887261] ffff88083f687fd8 ffff88083f687fd8 ffff8808263bbec0 ffff880826ff63c0 [18285.896256] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18285.905242] Call Trace: [18285.908662] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18285.916635] [] schedule+0x29/0x70 [18285.922859] [] taskq_thread+0x3fa/0x440 [spl] [18285.930244] [] ? wake_up_state+0x20/0x20 [18285.937149] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18285.945117] [] kthread+0xcf/0xe0 [18285.951220] [] ? kthread_create_on_node+0x140/0x140 [18285.959176] [] ret_from_fork+0x58/0x90 [18285.965870] [] ? kthread_create_on_node+0x140/0x140 [18285.973836] arc_prune S ffff88081e694d00 0 3184 2 0x00000080 [18285.982400] ffff88083fa7be08 0000000000000046 ffff8808263bce70 ffff88083fa7bfd8 [18285.991391] ffff88083fa7bfd8 ffff88083fa7bfd8 ffff8808263bce70 ffff880826ff6360 [18286.000374] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18286.009359] Call Trace: [18286.012772] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18286.020741] [] schedule+0x29/0x70 [18286.026968] [] taskq_thread+0x3fa/0x440 [spl] [18286.034346] [] ? wake_up_state+0x20/0x20 [18286.041247] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18286.049210] [] kthread+0xcf/0xe0 [18286.055319] [] ? kthread_create_on_node+0x140/0x140 [18286.063276] [] ret_from_fork+0x58/0x90 [18286.069971] [] ? kthread_create_on_node+0x140/0x140 [18286.077928] arc_prune S ffff88081e694d00 0 3185 2 0x00000080 [18286.086500] ffff880822ecbe08 0000000000000046 ffff8808263bde20 ffff880822ecbfd8 [18286.095500] ffff880822ecbfd8 ffff880822ecbfd8 ffff8808263bde20 ffff880826ff6300 [18286.104505] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18286.113501] Call Trace: [18286.116907] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18286.124886] [] schedule+0x29/0x70 [18286.131102] [] taskq_thread+0x3fa/0x440 [spl] [18286.138486] [] ? wake_up_state+0x20/0x20 [18286.145386] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18286.153345] [] kthread+0xcf/0xe0 [18286.159452] [] ? kthread_create_on_node+0x140/0x140 [18286.167408] [] ret_from_fork+0x58/0x90 [18286.174114] [] ? kthread_create_on_node+0x140/0x140 [18286.182087] arc_prune S ffff88081e694d00 0 3186 2 0x00000080 [18286.190661] ffff88082bf7be08 0000000000000046 ffff8808263bedd0 ffff88082bf7bfd8 [18286.199664] ffff88082bf7bfd8 ffff88082bf7bfd8 ffff8808263bedd0 ffff880826ff62a0 [18286.208667] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18286.217652] Call Trace: [18286.221061] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18286.229031] [] schedule+0x29/0x70 [18286.235245] [] taskq_thread+0x3fa/0x440 [spl] [18286.242630] [] ? wake_up_state+0x20/0x20 [18286.249539] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18286.257515] [] kthread+0xcf/0xe0 [18286.263626] [] ? kthread_create_on_node+0x140/0x140 [18286.271590] [] ret_from_fork+0x58/0x90 [18286.278273] [] ? kthread_create_on_node+0x140/0x140 [18286.286230] arc_prune S ffff88081e694d00 0 3187 2 0x00000080 [18286.294814] ffff88082daf3e08 0000000000000046 ffff880825230000 ffff88082daf3fd8 [18286.303807] ffff88082daf3fd8 ffff88082daf3fd8 ffff880825230000 ffff880826ff6240 [18286.312810] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18286.321792] Call Trace: [18286.325190] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18286.333161] [] schedule+0x29/0x70 [18286.339372] [] taskq_thread+0x3fa/0x440 [spl] [18286.346765] [] ? wake_up_state+0x20/0x20 [18286.353665] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18286.361626] [] kthread+0xcf/0xe0 [18286.367740] [] ? kthread_create_on_node+0x140/0x140 [18286.375703] [] ret_from_fork+0x58/0x90 [18286.382394] [] ? kthread_create_on_node+0x140/0x140 [18286.390343] arc_prune S ffff88081e694d00 0 3188 2 0x00000080 [18286.398919] ffff880826ab3e08 0000000000000046 ffff880825230fb0 ffff880826ab3fd8 [18286.407901] ffff880826ab3fd8 ffff880826ab3fd8 ffff880825230fb0 ffff880826ff61e0 [18286.416904] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18286.425907] Call Trace: [18286.429307] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18286.437274] [] schedule+0x29/0x70 [18286.443482] [] taskq_thread+0x3fa/0x440 [spl] [18286.450876] [] ? wake_up_state+0x20/0x20 [18286.457781] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18286.465740] [] kthread+0xcf/0xe0 [18286.471857] [] ? kthread_create_on_node+0x140/0x140 [18286.479820] [] ret_from_fork+0x58/0x90 [18286.486511] [] ? kthread_create_on_node+0x140/0x140 [18286.494467] arc_prune S ffff88081e694d00 0 3189 2 0x00000080 [18286.503037] ffff880826fdfe08 0000000000000046 ffff880825231f60 ffff880826fdffd8 [18286.512033] ffff880826fdffd8 ffff880826fdffd8 ffff880825231f60 ffff880826ff6180 [18286.521027] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18286.530022] Call Trace: [18286.533430] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18286.541398] [] schedule+0x29/0x70 [18286.547619] [] taskq_thread+0x3fa/0x440 [spl] [18286.555016] [] ? wake_up_state+0x20/0x20 [18286.561914] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18286.569875] [] kthread+0xcf/0xe0 [18286.575984] [] ? kthread_create_on_node+0x140/0x140 [18286.583951] [] ret_from_fork+0x58/0x90 [18286.590653] [] ? kthread_create_on_node+0x140/0x140 [18286.598608] arc_prune S ffff88081e694d00 0 3190 2 0x00000080 [18286.607192] ffff880827a67e08 0000000000000046 ffff880825232f10 ffff880827a67fd8 [18286.616186] ffff880827a67fd8 ffff880827a67fd8 ffff880825232f10 ffff880826ff6120 [18286.625187] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18286.634174] Call Trace: [18286.637589] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18286.645559] [] schedule+0x29/0x70 [18286.651782] [] taskq_thread+0x3fa/0x440 [spl] [18286.659172] [] ? wake_up_state+0x20/0x20 [18286.666079] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18286.674030] [] kthread+0xcf/0xe0 [18286.680141] [] ? kthread_create_on_node+0x140/0x140 [18286.688104] [] ret_from_fork+0x58/0x90 [18286.694792] [] ? kthread_create_on_node+0x140/0x140 [18286.702748] arc_prune S ffff88081e694d00 0 3191 2 0x00000080 [18286.711314] ffff880827a73e08 0000000000000046 ffff880825233ec0 ffff880827a73fd8 [18286.720306] ffff880827a73fd8 ffff880827a73fd8 ffff880825233ec0 ffff880412a55120 [18286.729309] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18286.738311] Call Trace: [18286.741729] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18286.749706] [] schedule+0x29/0x70 [18286.755931] [] taskq_thread+0x3fa/0x440 [spl] [18286.763318] [] ? wake_up_state+0x20/0x20 [18286.770216] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18286.778179] [] kthread+0xcf/0xe0 [18286.784289] [] ? kthread_create_on_node+0x140/0x140 [18286.792252] [] ret_from_fork+0x58/0x90 [18286.798946] [] ? kthread_create_on_node+0x140/0x140 [18286.806895] arc_prune S ffff88081e694d00 0 3192 2 0x00000080 [18286.815454] ffff880827f43e08 0000000000000046 ffff880825234e70 ffff880827f43fd8 [18286.824448] ffff880827f43fd8 ffff880827f43fd8 ffff880825234e70 ffff880412a55180 [18286.833442] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18286.842438] Call Trace: [18286.845857] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18286.853840] [] schedule+0x29/0x70 [18286.860079] [] taskq_thread+0x3fa/0x440 [spl] [18286.867464] [] ? wake_up_state+0x20/0x20 [18286.874363] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18286.882320] [] kthread+0xcf/0xe0 [18286.888429] [] ? kthread_create_on_node+0x140/0x140 [18286.896387] [] ret_from_fork+0x58/0x90 [18286.903081] [] ? kthread_create_on_node+0x140/0x140 [18286.911054] arc_prune S ffff88081e694d00 0 3193 2 0x00000080 [18286.919630] ffff880827f17e08 0000000000000046 ffff880825235e20 ffff880827f17fd8 [18286.928634] ffff880827f17fd8 ffff880827f17fd8 ffff880825235e20 ffff880412a551e0 [18286.937647] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18286.946638] Call Trace: [18286.950052] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18286.958033] [] schedule+0x29/0x70 [18286.964242] [] taskq_thread+0x3fa/0x440 [spl] [18286.971628] [] ? wake_up_state+0x20/0x20 [18286.978536] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18286.986493] [] kthread+0xcf/0xe0 [18286.992604] [] ? kthread_create_on_node+0x140/0x140 [18287.000571] [] ret_from_fork+0x58/0x90 [18287.007275] [] ? kthread_create_on_node+0x140/0x140 [18287.015227] arc_prune S ffff88081e694d00 0 3194 2 0x00000080 [18287.023815] ffff88082907fe08 0000000000000046 ffff880825236dd0 ffff88082907ffd8 [18287.032821] ffff88082907ffd8 ffff88082907ffd8 ffff880825236dd0 ffff880412a55240 [18287.041839] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18287.050850] Call Trace: [18287.054257] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18287.062224] [] schedule+0x29/0x70 [18287.068444] [] taskq_thread+0x3fa/0x440 [spl] [18287.075840] [] ? wake_up_state+0x20/0x20 [18287.082739] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18287.090704] [] kthread+0xcf/0xe0 [18287.096817] [] ? kthread_create_on_node+0x140/0x140 [18287.104776] [] ret_from_fork+0x58/0x90 [18287.111472] [] ? kthread_create_on_node+0x140/0x140 [18287.119424] arc_prune S ffff88081e694d00 0 3195 2 0x00000080 [18287.128004] ffff88082765be08 0000000000000046 ffff880825f28000 ffff88082765bfd8 [18287.137011] ffff88082765bfd8 ffff88082765bfd8 ffff880825f28000 ffff880412a552a0 [18287.146017] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18287.155012] Call Trace: [18287.158417] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18287.166385] [] schedule+0x29/0x70 [18287.172608] [] taskq_thread+0x3fa/0x440 [spl] [18287.179996] [] ? wake_up_state+0x20/0x20 [18287.186902] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18287.194873] [] kthread+0xcf/0xe0 [18287.200995] [] ? kthread_create_on_node+0x140/0x140 [18287.208947] [] ret_from_fork+0x58/0x90 [18287.215653] [] ? kthread_create_on_node+0x140/0x140 [18287.223621] arc_prune S ffff88081e694d00 0 3196 2 0x00000080 [18287.232195] ffff88082bec3e08 0000000000000046 ffff880825f28fb0 ffff88082bec3fd8 [18287.241195] ffff88082bec3fd8 ffff88082bec3fd8 ffff880825f28fb0 ffff880412a55300 [18287.250191] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18287.259189] Call Trace: [18287.262596] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18287.270568] [] schedule+0x29/0x70 [18287.276786] [] taskq_thread+0x3fa/0x440 [spl] [18287.284172] [] ? wake_up_state+0x20/0x20 [18287.291080] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18287.299056] [] kthread+0xcf/0xe0 [18287.305176] [] ? kthread_create_on_node+0x140/0x140 [18287.313142] [] ret_from_fork+0x58/0x90 [18287.319847] [] ? kthread_create_on_node+0x140/0x140 [18287.327813] arc_prune S ffff88081e694d00 0 3197 2 0x00000080 [18287.336383] ffff880827353e08 0000000000000046 ffff880825f29f60 ffff880827353fd8 [18287.345362] ffff880827353fd8 ffff880827353fd8 ffff880825f29f60 ffff880412a55360 [18287.354361] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18287.363334] Call Trace: [18287.366745] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18287.374722] [] schedule+0x29/0x70 [18287.380944] [] taskq_thread+0x3fa/0x440 [spl] [18287.388328] [] ? wake_up_state+0x20/0x20 [18287.395230] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18287.403204] [] kthread+0xcf/0xe0 [18287.409314] [] ? kthread_create_on_node+0x140/0x140 [18287.417269] [] ret_from_fork+0x58/0x90 [18287.423971] [] ? kthread_create_on_node+0x140/0x140 [18287.431934] arc_prune S ffff88081e694d00 0 3198 2 0x00000080 [18287.440502] ffff880826e43e08 0000000000000046 ffff880825f2af10 ffff880826e43fd8 [18287.449481] ffff880826e43fd8 ffff880826e43fd8 ffff880825f2af10 ffff880412a553c0 [18287.458470] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18287.467448] Call Trace: [18287.470867] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18287.478842] [] schedule+0x29/0x70 [18287.485060] [] taskq_thread+0x3fa/0x440 [spl] [18287.492447] [] ? wake_up_state+0x20/0x20 [18287.499348] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18287.507308] [] kthread+0xcf/0xe0 [18287.513421] [] ? kthread_create_on_node+0x140/0x140 [18287.521371] [] ret_from_fork+0x58/0x90 [18287.528073] [] ? kthread_create_on_node+0x140/0x140 [18287.536041] arc_prune S ffff88081e694d00 0 3199 2 0x00000080 [18287.544628] ffff88082bc03e08 0000000000000046 ffff880825f2bec0 ffff88082bc03fd8 [18287.553629] ffff88082bc03fd8 ffff88082bc03fd8 ffff880825f2bec0 ffff880412a55420 [18287.562627] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18287.571619] Call Trace: [18287.575032] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18287.583004] [] schedule+0x29/0x70 [18287.589219] [] taskq_thread+0x3fa/0x440 [spl] [18287.596605] [] ? wake_up_state+0x20/0x20 [18287.603506] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18287.611461] [] kthread+0xcf/0xe0 [18287.617580] [] ? kthread_create_on_node+0x140/0x140 [18287.625539] [] ret_from_fork+0x58/0x90 [18287.632234] [] ? kthread_create_on_node+0x140/0x140 [18287.640197] arc_prune S ffff88081e694d00 0 3200 2 0x00000080 [18287.648793] ffff88081bae3e08 0000000000000046 ffff880825f2ce70 ffff88081bae3fd8 [18287.657811] ffff88081bae3fd8 ffff88081bae3fd8 ffff880825f2ce70 ffff880412a55480 [18287.666818] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18287.675824] Call Trace: [18287.679229] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18287.687216] [] schedule+0x29/0x70 [18287.693434] [] taskq_thread+0x3fa/0x440 [spl] [18287.700818] [] ? wake_up_state+0x20/0x20 [18287.707722] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18287.715692] [] kthread+0xcf/0xe0 [18287.721821] [] ? kthread_create_on_node+0x140/0x140 [18287.729789] [] ret_from_fork+0x58/0x90 [18287.736477] [] ? kthread_create_on_node+0x140/0x140 [18287.744435] arc_prune S ffff88081e694d00 0 3201 2 0x00000080 [18287.753016] ffff88082af17e08 0000000000000046 ffff880825f2de20 ffff88082af17fd8 [18287.762022] ffff88082af17fd8 ffff88082af17fd8 ffff880825f2de20 ffff880412a554e0 [18287.771029] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18287.780036] Call Trace: [18287.783445] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18287.791410] [] schedule+0x29/0x70 [18287.797635] [] taskq_thread+0x3fa/0x440 [spl] [18287.805021] [] ? wake_up_state+0x20/0x20 [18287.811930] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18287.819905] [] kthread+0xcf/0xe0 [18287.826014] [] ? kthread_create_on_node+0x140/0x140 [18287.833980] [] ret_from_fork+0x58/0x90 [18287.840686] [] ? kthread_create_on_node+0x140/0x140 [18287.848655] arc_prune S ffff88081e694d00 0 3202 2 0x00000080 [18287.857227] ffff88082d34be08 0000000000000046 ffff880825f2edd0 ffff88082d34bfd8 [18287.866218] ffff88082d34bfd8 ffff88082d34bfd8 ffff880825f2edd0 ffff880412a55540 [18287.875212] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18287.884207] Call Trace: [18287.887620] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18287.895595] [] schedule+0x29/0x70 [18287.901820] [] taskq_thread+0x3fa/0x440 [spl] [18287.909217] [] ? wake_up_state+0x20/0x20 [18287.916128] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18287.924113] [] kthread+0xcf/0xe0 [18287.930228] [] ? kthread_create_on_node+0x140/0x140 [18287.938186] [] ret_from_fork+0x58/0x90 [18287.944891] [] ? kthread_create_on_node+0x140/0x140 [18287.952857] arc_prune S ffff88081e694d00 0 3203 2 0x00000080 [18287.961434] ffff88082e93fe08 0000000000000046 ffff88082bcc8000 ffff88082e93ffd8 [18287.970424] ffff88082e93ffd8 ffff88082e93ffd8 ffff88082bcc8000 ffff880412a555a0 [18287.979424] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18287.988414] Call Trace: [18287.991828] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18287.999809] [] schedule+0x29/0x70 [18288.006031] [] taskq_thread+0x3fa/0x440 [spl] [18288.013420] [] ? wake_up_state+0x20/0x20 [18288.020322] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18288.028284] [] kthread+0xcf/0xe0 [18288.034398] [] ? kthread_create_on_node+0x140/0x140 [18288.042348] [] ret_from_fork+0x58/0x90 [18288.049045] [] ? kthread_create_on_node+0x140/0x140 [18288.057006] arc_prune S ffff88081e694d00 0 3204 2 0x00000080 [18288.065582] ffff88082aea3e08 0000000000000046 ffff88082bcc8fb0 ffff88082aea3fd8 [18288.074578] ffff88082aea3fd8 ffff88082aea3fd8 ffff88082bcc8fb0 ffff880412a55600 [18288.083573] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18288.092575] Call Trace: [18288.095982] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18288.103956] [] schedule+0x29/0x70 [18288.110180] [] taskq_thread+0x3fa/0x440 [spl] [18288.117569] [] ? wake_up_state+0x20/0x20 [18288.124463] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18288.132426] [] kthread+0xcf/0xe0 [18288.138535] [] ? kthread_create_on_node+0x140/0x140 [18288.146492] [] ret_from_fork+0x58/0x90 [18288.153193] [] ? kthread_create_on_node+0x140/0x140 [18288.161155] arc_prune S ffff88081e694d00 0 3205 2 0x00000080 [18288.169739] ffff880826303e08 0000000000000046 ffff88082bcc9f60 ffff880826303fd8 [18288.178738] ffff880826303fd8 ffff880826303fd8 ffff88082bcc9f60 ffff880412a55660 [18288.187741] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18288.196748] Call Trace: [18288.200176] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18288.208162] [] schedule+0x29/0x70 [18288.214379] [] taskq_thread+0x3fa/0x440 [spl] [18288.221773] [] ? wake_up_state+0x20/0x20 [18288.228677] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18288.236649] [] kthread+0xcf/0xe0 [18288.242752] [] ? kthread_create_on_node+0x140/0x140 [18288.250710] [] ret_from_fork+0x58/0x90 [18288.257403] [] ? kthread_create_on_node+0x140/0x140 [18288.265366] arc_prune S ffff88081e694d00 0 3206 2 0x00000080 [18288.273962] ffff88083fedbe08 0000000000000046 ffff88082bccaf10 ffff88083fedbfd8 [18288.282963] ffff88083fedbfd8 ffff88083fedbfd8 ffff88082bccaf10 ffff880412a556c0 [18288.291968] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18288.300964] Call Trace: [18288.304372] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18288.312342] [] schedule+0x29/0x70 [18288.318555] [] taskq_thread+0x3fa/0x440 [spl] [18288.325941] [] ? wake_up_state+0x20/0x20 [18288.332843] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18288.340806] [] kthread+0xcf/0xe0 [18288.346926] [] ? kthread_create_on_node+0x140/0x140 [18288.354889] [] ret_from_fork+0x58/0x90 [18288.361583] [] ? kthread_create_on_node+0x140/0x140 [18288.369537] arc_prune S ffff88081e694d00 0 3207 2 0x00000080 [18288.378118] ffff8808277b3e08 0000000000000046 ffff88082bccbec0 ffff8808277b3fd8 [18288.387109] ffff8808277b3fd8 ffff8808277b3fd8 ffff88082bccbec0 ffff880412a55720 [18288.396112] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18288.405115] Call Trace: [18288.408526] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18288.416499] [] schedule+0x29/0x70 [18288.422711] [] taskq_thread+0x3fa/0x440 [spl] [18288.430113] [] ? wake_up_state+0x20/0x20 [18288.437022] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18288.444997] [] kthread+0xcf/0xe0 [18288.451116] [] ? kthread_create_on_node+0x140/0x140 [18288.459081] [] ret_from_fork+0x58/0x90 [18288.465775] [] ? kthread_create_on_node+0x140/0x140 [18288.473736] arc_prune S ffff88081e694d00 0 3208 2 0x00000080 [18288.482322] ffff88082632fe08 0000000000000046 ffff88082bccce70 ffff88082632ffd8 [18288.491322] ffff88082632ffd8 ffff88082632ffd8 ffff88082bccce70 ffff880412a55780 [18288.500314] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18288.509303] Call Trace: [18288.512715] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18288.520697] [] schedule+0x29/0x70 [18288.526916] [] taskq_thread+0x3fa/0x440 [spl] [18288.534305] [] ? wake_up_state+0x20/0x20 [18288.541213] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18288.549185] [] kthread+0xcf/0xe0 [18288.555305] [] ? kthread_create_on_node+0x140/0x140 [18288.563264] [] ret_from_fork+0x58/0x90 [18288.569960] [] ? kthread_create_on_node+0x140/0x140 [18288.577926] arc_prune S ffff88081e694d00 0 3209 2 0x00000080 [18288.586500] ffff88082e4b7e08 0000000000000046 ffff88082bccde20 ffff88082e4b7fd8 [18288.595506] ffff88082e4b7fd8 ffff88082e4b7fd8 ffff88082bccde20 ffff880412a557e0 [18288.604496] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18288.613491] Call Trace: [18288.616903] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18288.624875] [] schedule+0x29/0x70 [18288.631096] [] taskq_thread+0x3fa/0x440 [spl] [18288.638476] [] ? wake_up_state+0x20/0x20 [18288.645381] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18288.653343] [] kthread+0xcf/0xe0 [18288.659455] [] ? kthread_create_on_node+0x140/0x140 [18288.667407] [] ret_from_fork+0x58/0x90 [18288.674101] [] ? kthread_create_on_node+0x140/0x140 [18288.682062] arc_prune S ffff88081e694d00 0 3210 2 0x00000080 [18288.690654] ffff88082dabfe08 0000000000000046 ffff88082bccedd0 ffff88082dabffd8 [18288.699665] ffff88082dabffd8 ffff88082dabffd8 ffff88082bccedd0 ffff880412a55840 [18288.708657] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18288.717650] Call Trace: [18288.721073] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18288.729064] [] schedule+0x29/0x70 [18288.735283] [] taskq_thread+0x3fa/0x440 [spl] [18288.742670] [] ? wake_up_state+0x20/0x20 [18288.749568] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18288.757514] [] kthread+0xcf/0xe0 [18288.763618] [] ? kthread_create_on_node+0x140/0x140 [18288.771578] [] ret_from_fork+0x58/0x90 [18288.778272] [] ? kthread_create_on_node+0x140/0x140 [18288.786231] arc_prune S ffff88081e694d00 0 3211 2 0x00000080 [18288.794815] ffff88082bd73e08 0000000000000046 ffff880827758000 ffff88082bd73fd8 [18288.803818] ffff88082bd73fd8 ffff88082bd73fd8 ffff880827758000 ffff880412a558a0 [18288.812822] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18288.821817] Call Trace: [18288.825237] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18288.833210] [] schedule+0x29/0x70 [18288.839426] [] taskq_thread+0x3fa/0x440 [spl] [18288.846810] [] ? wake_up_state+0x20/0x20 [18288.853720] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18288.861690] [] kthread+0xcf/0xe0 [18288.867804] [] ? kthread_create_on_node+0x140/0x140 [18288.875772] [] ret_from_fork+0x58/0x90 [18288.882466] [] ? kthread_create_on_node+0x140/0x140 [18288.890423] arc_prune S ffff88081e694d00 0 3212 2 0x00000080 [18288.899006] ffff88082c793e08 0000000000000046 ffff880827758fb0 ffff88082c793fd8 [18288.908018] ffff88082c793fd8 ffff88082c793fd8 ffff880827758fb0 ffff880412a55900 [18288.917024] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18288.926051] Call Trace: [18288.929466] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18288.937433] [] schedule+0x29/0x70 [18288.943661] [] taskq_thread+0x3fa/0x440 [spl] [18288.951040] [] ? wake_up_state+0x20/0x20 [18288.957948] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18288.965915] [] kthread+0xcf/0xe0 [18288.972032] [] ? kthread_create_on_node+0x140/0x140 [18288.979985] [] ret_from_fork+0x58/0x90 [18288.986675] [] ? kthread_create_on_node+0x140/0x140 [18288.994633] arc_prune S ffff88081e694d00 0 3213 2 0x00000080 [18289.003222] ffff880825effe08 0000000000000046 ffff880827759f60 ffff880825efffd8 [18289.012229] ffff880825efffd8 ffff880825efffd8 ffff880827759f60 ffff880412a55960 [18289.021234] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18289.030239] Call Trace: [18289.033649] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18289.041630] [] schedule+0x29/0x70 [18289.047847] [] taskq_thread+0x3fa/0x440 [spl] [18289.055239] [] ? wake_up_state+0x20/0x20 [18289.062148] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18289.070119] [] kthread+0xcf/0xe0 [18289.076239] [] ? kthread_create_on_node+0x140/0x140 [18289.084205] [] ret_from_fork+0x58/0x90 [18289.090910] [] ? kthread_create_on_node+0x140/0x140 [18289.098868] arc_prune S ffff88081e694d00 0 3214 2 0x00000080 [18289.107434] ffff8808273d7e08 0000000000000046 ffff88082775af10 ffff8808273d7fd8 [18289.116423] ffff8808273d7fd8 ffff8808273d7fd8 ffff88082775af10 ffff880412a559c0 [18289.125422] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18289.134411] Call Trace: [18289.137818] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18289.145797] [] schedule+0x29/0x70 [18289.152022] [] taskq_thread+0x3fa/0x440 [spl] [18289.159409] [] ? wake_up_state+0x20/0x20 [18289.166309] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18289.174273] [] kthread+0xcf/0xe0 [18289.180375] [] ? kthread_create_on_node+0x140/0x140 [18289.188333] [] ret_from_fork+0x58/0x90 [18289.195028] [] ? kthread_create_on_node+0x140/0x140 [18289.202993] arc_prune S ffff88081e694d00 0 3215 2 0x00000080 [18289.211569] ffff8808273e7e08 0000000000000046 ffff88082775bec0 ffff8808273e7fd8 [18289.220558] ffff8808273e7fd8 ffff8808273e7fd8 ffff88082775bec0 ffff880412a55a20 [18289.229545] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18289.238535] Call Trace: [18289.241959] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18289.249942] [] schedule+0x29/0x70 [18289.256175] [] taskq_thread+0x3fa/0x440 [spl] [18289.263563] [] ? wake_up_state+0x20/0x20 [18289.270460] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18289.278418] [] kthread+0xcf/0xe0 [18289.284529] [] ? kthread_create_on_node+0x140/0x140 [18289.292486] [] ret_from_fork+0x58/0x90 [18289.299176] [] ? kthread_create_on_node+0x140/0x140 [18289.307151] arc_prune S ffff88081e694d00 0 3216 2 0x00000080 [18289.315746] ffff88082a67be08 0000000000000046 ffff88082775ce70 ffff88082a67bfd8 [18289.324743] ffff88082a67bfd8 ffff88082a67bfd8 ffff88082775ce70 ffff880412a55a80 [18289.333732] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18289.342726] Call Trace: [18289.346135] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18289.354108] [] schedule+0x29/0x70 [18289.360324] [] taskq_thread+0x3fa/0x440 [spl] [18289.367709] [] ? wake_up_state+0x20/0x20 [18289.374621] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18289.382579] [] kthread+0xcf/0xe0 [18289.388700] [] ? kthread_create_on_node+0x140/0x140 [18289.396668] [] ret_from_fork+0x58/0x90 [18289.403363] [] ? kthread_create_on_node+0x140/0x140 [18289.411319] arc_prune S ffff88081e694d00 0 3217 2 0x00000080 [18289.419900] ffff88082a77be08 0000000000000046 ffff88082775de20 ffff88082a77bfd8 [18289.428912] ffff88082a77bfd8 ffff88082a77bfd8 ffff88082775de20 ffff880412a55ae0 [18289.437918] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18289.446926] Call Trace: [18289.450335] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18289.458308] [] schedule+0x29/0x70 [18289.464525] [] taskq_thread+0x3fa/0x440 [spl] [18289.471921] [] ? wake_up_state+0x20/0x20 [18289.478820] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18289.486780] [] kthread+0xcf/0xe0 [18289.492900] [] ? kthread_create_on_node+0x140/0x140 [18289.500868] [] ret_from_fork+0x58/0x90 [18289.507563] [] ? kthread_create_on_node+0x140/0x140 [18289.515524] arc_prune S ffff88081e694d00 0 3218 2 0x00000080 [18289.524115] ffff8808262abe08 0000000000000046 ffff88082775edd0 ffff8808262abfd8 [18289.533122] ffff8808262abfd8 ffff8808262abfd8 ffff88082775edd0 ffff880412a55b40 [18289.542138] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18289.551135] Call Trace: [18289.554542] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18289.562515] [] schedule+0x29/0x70 [18289.568743] [] taskq_thread+0x3fa/0x440 [spl] [18289.576140] [] ? wake_up_state+0x20/0x20 [18289.583041] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18289.591001] [] kthread+0xcf/0xe0 [18289.597126] [] ? kthread_create_on_node+0x140/0x140 [18289.605093] [] ret_from_fork+0x58/0x90 [18289.611788] [] ? kthread_create_on_node+0x140/0x140 [18289.619748] arc_prune S ffff88081e694d00 0 3219 2 0x00000080 [18289.628317] ffff88082a363e08 0000000000000046 ffff8808252c8000 ffff88082a363fd8 [18289.637329] ffff88082a363fd8 ffff88082a363fd8 ffff8808252c8000 ffff880412a55ba0 [18289.646326] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18289.655329] Call Trace: [18289.658741] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18289.666722] [] schedule+0x29/0x70 [18289.672954] [] taskq_thread+0x3fa/0x440 [spl] [18289.680339] [] ? wake_up_state+0x20/0x20 [18289.687250] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18289.695223] [] kthread+0xcf/0xe0 [18289.701342] [] ? kthread_create_on_node+0x140/0x140 [18289.709300] [] ret_from_fork+0x58/0x90 [18289.716002] [] ? kthread_create_on_node+0x140/0x140 [18289.723977] arc_prune S ffff88081e694d00 0 3220 2 0x00000080 [18289.732546] ffff88082e23be08 0000000000000046 ffff8808252c8fb0 ffff88082e23bfd8 [18289.741547] ffff88082e23bfd8 ffff88082e23bfd8 ffff8808252c8fb0 ffff880412a55c00 [18289.750542] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18289.759538] Call Trace: [18289.762949] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18289.770921] [] schedule+0x29/0x70 [18289.777138] [] taskq_thread+0x3fa/0x440 [spl] [18289.784522] [] ? wake_up_state+0x20/0x20 [18289.791421] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18289.799389] [] kthread+0xcf/0xe0 [18289.805499] [] ? kthread_create_on_node+0x140/0x140 [18289.813456] [] ret_from_fork+0x58/0x90 [18289.820157] [] ? kthread_create_on_node+0x140/0x140 [18289.828123] arc_prune S ffff88081e694d00 0 3221 2 0x00000080 [18289.836703] ffff88082e1bfe08 0000000000000046 ffff8808252c9f60 ffff88082e1bffd8 [18289.845700] ffff88082e1bffd8 ffff88082e1bffd8 ffff8808252c9f60 ffff880412a55c60 [18289.854692] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18289.863682] Call Trace: [18289.867104] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18289.875084] [] schedule+0x29/0x70 [18289.881303] [] taskq_thread+0x3fa/0x440 [spl] [18289.888682] [] ? wake_up_state+0x20/0x20 [18289.895590] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18289.903550] [] kthread+0xcf/0xe0 [18289.909666] [] ? finish_task_switch+0x53/0x180 [18289.917136] [] ? kthread_create_on_node+0x140/0x140 [18289.925098] [] ret_from_fork+0x58/0x90 [18289.931800] [] ? kthread_create_on_node+0x140/0x140 [18289.939757] arc_prune S ffff88081e694d00 0 3222 2 0x00000080 [18289.948316] ffff88082d333e08 0000000000000046 ffff8808252caf10 ffff88082d333fd8 [18289.957318] ffff88082d333fd8 ffff88082d333fd8 ffff8808252caf10 ffff880412a55cc0 [18289.966304] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18289.975292] Call Trace: [18289.978702] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18289.986674] [] schedule+0x29/0x70 [18289.992910] [] taskq_thread+0x3fa/0x440 [spl] [18290.000293] [] ? wake_up_state+0x20/0x20 [18290.007188] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18290.015159] [] kthread+0xcf/0xe0 [18290.021275] [] ? kthread_create_on_node+0x140/0x140 [18290.029238] [] ret_from_fork+0x58/0x90 [18290.035940] [] ? kthread_create_on_node+0x140/0x140 [18290.043907] arc_prune S ffff88081e694d00 0 3223 2 0x00000080 [18290.052465] ffff88082c607e08 0000000000000046 ffff8808252cbec0 ffff88082c607fd8 [18290.061457] ffff88082c607fd8 ffff88082c607fd8 ffff8808252cbec0 ffff880412a55d20 [18290.070448] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18290.079438] Call Trace: [18290.082838] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18290.090804] [] schedule+0x29/0x70 [18290.097025] [] taskq_thread+0x3fa/0x440 [spl] [18290.104405] [] ? wake_up_state+0x20/0x20 [18290.111298] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18290.119265] [] kthread+0xcf/0xe0 [18290.125374] [] ? kthread_create_on_node+0x140/0x140 [18290.133324] [] ret_from_fork+0x58/0x90 [18290.140026] [] ? kthread_create_on_node+0x140/0x140 [18290.147981] arc_prune S ffff88081e694d00 0 3224 2 0x00000080 [18290.156539] ffff88082c233e08 0000000000000046 ffff8808252cce70 ffff88082c233fd8 [18290.165515] ffff88082c233fd8 ffff88082c233fd8 ffff8808252cce70 ffff880412a55d80 [18290.174509] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18290.183499] Call Trace: [18290.186905] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18290.194869] [] schedule+0x29/0x70 [18290.201100] [] taskq_thread+0x3fa/0x440 [spl] [18290.208489] [] ? wake_up_state+0x20/0x20 [18290.215384] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18290.223346] [] kthread+0xcf/0xe0 [18290.229457] [] ? kthread_create_on_node+0x140/0x140 [18290.237415] [] ret_from_fork+0x58/0x90 [18290.244118] [] ? kthread_create_on_node+0x140/0x140 [18290.252079] arc_prune S ffff88081e694d00 0 3225 2 0x00000080 [18290.260652] ffff88083fe0be08 0000000000000046 ffff8808252cde20 ffff88083fe0bfd8 [18290.269653] ffff88083fe0bfd8 ffff88083fe0bfd8 ffff8808252cde20 ffff880412a55de0 [18290.278643] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18290.287653] Call Trace: [18290.291059] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18290.299040] [] schedule+0x29/0x70 [18290.305270] [] taskq_thread+0x3fa/0x440 [spl] [18290.312670] [] ? wake_up_state+0x20/0x20 [18290.319563] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18290.327523] [] kthread+0xcf/0xe0 [18290.333633] [] ? kthread_create_on_node+0x140/0x140 [18290.341591] [] ret_from_fork+0x58/0x90 [18290.348298] [] ? kthread_create_on_node+0x140/0x140 [18290.356262] arc_prune S ffff88081e694d00 0 3226 2 0x00000080 [18290.364840] ffff880827ac7e08 0000000000000046 ffff8808252cedd0 ffff880827ac7fd8 [18290.373846] ffff880827ac7fd8 ffff880827ac7fd8 ffff8808252cedd0 ffff880412a55e40 [18290.382850] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18290.391847] Call Trace: [18290.395258] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18290.403242] [] schedule+0x29/0x70 [18290.409457] [] taskq_thread+0x3fa/0x440 [spl] [18290.416850] [] ? wake_up_state+0x20/0x20 [18290.423752] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18290.431712] [] kthread+0xcf/0xe0 [18290.437825] [] ? kthread_create_on_node+0x140/0x140 [18290.445771] [] ret_from_fork+0x58/0x90 [18290.452465] [] ? kthread_create_on_node+0x140/0x140 [18290.460422] arc_prune S ffff88081e694d00 0 3227 2 0x00000080 [18290.469009] ffff880827acbe08 0000000000000046 ffff880825620000 ffff880827acbfd8 [18290.478008] ffff880827acbfd8 ffff880827acbfd8 ffff880825620000 ffff8800b168d000 [18290.487003] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18290.495999] Call Trace: [18290.499404] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18290.507373] [] schedule+0x29/0x70 [18290.513589] [] taskq_thread+0x3fa/0x440 [spl] [18290.520986] [] ? wake_up_state+0x20/0x20 [18290.527888] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18290.535858] [] kthread+0xcf/0xe0 [18290.541977] [] ? kthread_create_on_node+0x140/0x140 [18290.549942] [] ret_from_fork+0x58/0x90 [18290.556631] [] ? kthread_create_on_node+0x140/0x140 [18290.564581] arc_prune S ffff88081e694d00 0 3228 2 0x00000080 [18290.573160] ffff88082d9f7e08 0000000000000046 ffff880825620fb0 ffff88082d9f7fd8 [18290.582173] ffff88082d9f7fd8 ffff88082d9f7fd8 ffff880825620fb0 ffff8800b168d060 [18290.591170] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18290.600182] Call Trace: [18290.603590] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18290.611556] [] schedule+0x29/0x70 [18290.617778] [] taskq_thread+0x3fa/0x440 [spl] [18290.625172] [] ? wake_up_state+0x20/0x20 [18290.632080] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18290.640045] [] kthread+0xcf/0xe0 [18290.646163] [] ? kthread_create_on_node+0x140/0x140 [18290.654127] [] ret_from_fork+0x58/0x90 [18290.660813] [] ? kthread_create_on_node+0x140/0x140 [18290.668779] arc_prune S ffff88081e694d00 0 3229 2 0x00000080 [18290.677346] ffff8808273c3e08 0000000000000046 ffff880825621f60 ffff8808273c3fd8 [18290.686343] ffff8808273c3fd8 ffff8808273c3fd8 ffff880825621f60 ffff8800b168d0c0 [18290.695334] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18290.704334] Call Trace: [18290.707752] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18290.715734] [] schedule+0x29/0x70 [18290.721945] [] taskq_thread+0x3fa/0x440 [spl] [18290.729330] [] ? wake_up_state+0x20/0x20 [18290.736243] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18290.744205] [] kthread+0xcf/0xe0 [18290.750316] [] ? kthread_create_on_node+0x140/0x140 [18290.758276] [] ret_from_fork+0x58/0x90 [18290.764971] [] ? kthread_create_on_node+0x140/0x140 [18290.772929] arc_prune S ffff88081e694d00 0 3230 2 0x00000080 [18290.781495] ffff88082a7f3e08 0000000000000046 ffff880825622f10 ffff88082a7f3fd8 [18290.790488] ffff88082a7f3fd8 ffff88082a7f3fd8 ffff880825622f10 ffff8800b168d120 [18290.799483] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18290.808482] Call Trace: [18290.811897] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18290.819882] [] schedule+0x29/0x70 [18290.826105] [] taskq_thread+0x3fa/0x440 [spl] [18290.833492] [] ? wake_up_state+0x20/0x20 [18290.840394] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18290.848355] [] kthread+0xcf/0xe0 [18290.854465] [] ? kthread_create_on_node+0x140/0x140 [18290.862421] [] ret_from_fork+0x58/0x90 [18290.869124] [] ? kthread_create_on_node+0x140/0x140 [18290.877095] arc_prune S ffff88081e694d00 0 3231 2 0x00000080 [18290.885681] ffff88082977be08 0000000000000046 ffff880825623ec0 ffff88082977bfd8 [18290.894656] ffff88082977bfd8 ffff88082977bfd8 ffff880825623ec0 ffff8800b168d180 [18290.903635] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18290.912629] Call Trace: [18290.916051] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18290.924040] [] schedule+0x29/0x70 [18290.930275] [] taskq_thread+0x3fa/0x440 [spl] [18290.937661] [] ? wake_up_state+0x20/0x20 [18290.944557] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18290.952519] [] kthread+0xcf/0xe0 [18290.958630] [] ? kthread_create_on_node+0x140/0x140 [18290.966588] [] ret_from_fork+0x58/0x90 [18290.973288] [] ? kthread_create_on_node+0x140/0x140 [18290.981260] arc_prune S ffff88081e694d00 0 3232 2 0x00000080 [18290.989849] ffff88082e533e08 0000000000000046 ffff880825624e70 ffff88082e533fd8 [18290.998858] ffff88082e533fd8 ffff88082e533fd8 ffff880825624e70 ffff8800b168d1e0 [18291.007851] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18291.016828] Call Trace: [18291.020235] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18291.028207] [] schedule+0x29/0x70 [18291.034425] [] taskq_thread+0x3fa/0x440 [spl] [18291.041813] [] ? wake_up_state+0x20/0x20 [18291.048731] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18291.056694] [] kthread+0xcf/0xe0 [18291.062808] [] ? kthread_create_on_node+0x140/0x140 [18291.070754] [] ret_from_fork+0x58/0x90 [18291.077455] [] ? kthread_create_on_node+0x140/0x140 [18291.085414] arc_prune S ffff88081e694d00 0 3233 2 0x00000080 [18291.094000] ffff88082d493e08 0000000000000046 ffff880825625e20 ffff88082d493fd8 [18291.103007] ffff88082d493fd8 ffff88082d493fd8 ffff880825625e20 ffff8800b168d240 [18291.112015] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18291.121013] Call Trace: [18291.124420] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18291.132393] [] schedule+0x29/0x70 [18291.138611] [] taskq_thread+0x3fa/0x440 [spl] [18291.146000] [] ? wake_up_state+0x20/0x20 [18291.152906] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18291.160865] [] kthread+0xcf/0xe0 [18291.166977] [] ? kthread_create_on_node+0x140/0x140 [18291.174942] [] ret_from_fork+0x58/0x90 [18291.181637] [] ? kthread_create_on_node+0x140/0x140 [18291.189582] arc_prune S ffff88081e694d00 0 3234 2 0x00000080 [18291.198176] ffff880826397e08 0000000000000046 ffff880825626dd0 ffff880826397fd8 [18291.207178] ffff880826397fd8 ffff880826397fd8 ffff880825626dd0 ffff8800b168d2a0 [18291.216176] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18291.225169] Call Trace: [18291.228569] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18291.236541] [] schedule+0x29/0x70 [18291.242759] [] taskq_thread+0x3fa/0x440 [spl] [18291.250146] [] ? wake_up_state+0x20/0x20 [18291.257055] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18291.265018] [] kthread+0xcf/0xe0 [18291.271124] [] ? kthread_create_on_node+0x140/0x140 [18291.279085] [] ret_from_fork+0x58/0x90 [18291.285779] [] ? kthread_create_on_node+0x140/0x140 [18291.293747] arc_prune S ffff88081e694d00 0 3235 2 0x00000080 [18291.302327] ffff8808263dbe08 0000000000000046 ffff88082df38000 ffff8808263dbfd8 [18291.311331] ffff8808263dbfd8 ffff8808263dbfd8 ffff88082df38000 ffff8800b168d300 [18291.320337] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18291.329326] Call Trace: [18291.332751] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18291.340718] [] schedule+0x29/0x70 [18291.346951] [] taskq_thread+0x3fa/0x440 [spl] [18291.354344] [] ? wake_up_state+0x20/0x20 [18291.361260] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18291.369232] [] kthread+0xcf/0xe0 [18291.375345] [] ? kthread_create_on_node+0x140/0x140 [18291.383305] [] ret_from_fork+0x58/0x90 [18291.390008] [] ? kthread_create_on_node+0x140/0x140 [18291.397963] arc_prune S ffff88081e694d00 0 3236 2 0x00000080 [18291.406532] ffff880829d4fe08 0000000000000046 ffff88082df38fb0 ffff880829d4ffd8 [18291.415530] ffff880829d4ffd8 ffff880829d4ffd8 ffff88082df38fb0 ffff8800b168d360 [18291.424533] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18291.433527] Call Trace: [18291.436939] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18291.444912] [] schedule+0x29/0x70 [18291.451128] [] taskq_thread+0x3fa/0x440 [spl] [18291.458513] [] ? wake_up_state+0x20/0x20 [18291.465415] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18291.473378] [] kthread+0xcf/0xe0 [18291.479489] [] ? kthread_create_on_node+0x140/0x140 [18291.487446] [] ret_from_fork+0x58/0x90 [18291.494134] [] ? kthread_create_on_node+0x140/0x140 [18291.502098] arc_prune S ffff88081e694d00 0 3237 2 0x00000080 [18291.510672] ffff88083fa2fe08 0000000000000046 ffff88082df39f60 ffff88083fa2ffd8 [18291.519685] ffff88083fa2ffd8 ffff88083fa2ffd8 ffff88082df39f60 ffff8800b168d3c0 [18291.528673] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18291.537654] Call Trace: [18291.541073] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18291.549052] [] schedule+0x29/0x70 [18291.555275] [] taskq_thread+0x3fa/0x440 [spl] [18291.562655] [] ? wake_up_state+0x20/0x20 [18291.569556] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18291.577512] [] kthread+0xcf/0xe0 [18291.583614] [] ? kthread_create_on_node+0x140/0x140 [18291.591572] [] ret_from_fork+0x58/0x90 [18291.598276] [] ? kthread_create_on_node+0x140/0x140 [18291.606244] arc_prune S ffff88081e694d00 0 3238 2 0x00000080 [18291.614830] ffff880824b4be08 0000000000000046 ffff88082df3af10 ffff880824b4bfd8 [18291.623825] ffff880824b4bfd8 ffff880824b4bfd8 ffff88082df3af10 ffff8800b168d420 [18291.632813] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18291.641812] Call Trace: [18291.645225] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18291.653198] [] schedule+0x29/0x70 [18291.659427] [] taskq_thread+0x3fa/0x440 [spl] [18291.666822] [] ? wake_up_state+0x20/0x20 [18291.673725] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18291.681688] [] kthread+0xcf/0xe0 [18291.687803] [] ? kthread_create_on_node+0x140/0x140 [18291.695762] [] ret_from_fork+0x58/0x90 [18291.702457] [] ? kthread_create_on_node+0x140/0x140 [18291.710402] arc_prune S ffff88081e694d00 0 3239 2 0x00000080 [18291.718984] ffff88081d603e08 0000000000000046 ffff88082df3bec0 ffff88081d603fd8 [18291.727987] ffff88081d603fd8 ffff88081d603fd8 ffff88082df3bec0 ffff8800b168d480 [18291.736998] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18291.746020] Call Trace: [18291.749428] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18291.757401] [] schedule+0x29/0x70 [18291.763605] [] taskq_thread+0x3fa/0x440 [spl] [18291.771001] [] ? wake_up_state+0x20/0x20 [18291.777903] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18291.785865] [] kthread+0xcf/0xe0 [18291.791989] [] ? kthread_create_on_node+0x140/0x140 [18291.799950] [] ret_from_fork+0x58/0x90 [18291.806645] [] ? kthread_create_on_node+0x140/0x140 [18291.814605] arc_prune S ffff88081e694d00 0 3240 2 0x00000080 [18291.823184] ffff88082362be08 0000000000000046 ffff88082df3ce70 ffff88082362bfd8 [18291.832195] ffff88082362bfd8 ffff88082362bfd8 ffff88082df3ce70 ffff8800b168d4e0 [18291.841199] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18291.850217] Call Trace: [18291.853629] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18291.861593] [] schedule+0x29/0x70 [18291.867820] [] taskq_thread+0x3fa/0x440 [spl] [18291.875214] [] ? wake_up_state+0x20/0x20 [18291.882111] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18291.890074] [] kthread+0xcf/0xe0 [18291.896194] [] ? kthread_create_on_node+0x140/0x140 [18291.904150] [] ret_from_fork+0x58/0x90 [18291.910839] [] ? kthread_create_on_node+0x140/0x140 [18291.918794] arc_prune S ffff88081e694d00 0 3241 2 0x00000080 [18291.927374] ffff88082abfbe08 0000000000000046 ffff88082df3de20 ffff88082abfbfd8 [18291.936374] ffff88082abfbfd8 ffff88082abfbfd8 ffff88082df3de20 ffff8800b168d540 [18291.945367] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18291.954366] Call Trace: [18291.957780] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18291.965759] [] schedule+0x29/0x70 [18291.971976] [] taskq_thread+0x3fa/0x440 [spl] [18291.979374] [] ? wake_up_state+0x20/0x20 [18291.986284] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18291.994249] [] kthread+0xcf/0xe0 [18292.000350] [] ? kthread_create_on_node+0x140/0x140 [18292.008321] [] ret_from_fork+0x58/0x90 [18292.015024] [] ? kthread_create_on_node+0x140/0x140 [18292.022982] arc_prune S ffff88081e694d00 0 3242 2 0x00000080 [18292.031561] ffff880825ee3e08 0000000000000046 ffff88082df3edd0 ffff880825ee3fd8 [18292.040553] ffff880825ee3fd8 ffff880825ee3fd8 ffff88082df3edd0 ffff8800b168d5a0 [18292.049555] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18292.058548] Call Trace: [18292.061966] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18292.069948] [] schedule+0x29/0x70 [18292.076167] [] taskq_thread+0x3fa/0x440 [spl] [18292.083553] [] ? wake_up_state+0x20/0x20 [18292.090449] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18292.098413] [] kthread+0xcf/0xe0 [18292.104515] [] ? kthread_create_on_node+0x140/0x140 [18292.112473] [] ret_from_fork+0x58/0x90 [18292.119171] [] ? kthread_create_on_node+0x140/0x140 [18292.127144] arc_prune S ffff88081e694d00 0 3243 2 0x00000080 [18292.135736] ffff880826327e08 0000000000000046 ffff880826740000 ffff880826327fd8 [18292.144735] ffff880826327fd8 ffff880826327fd8 ffff880826740000 ffff8800b168d600 [18292.153733] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18292.162717] Call Trace: [18292.166126] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18292.174102] [] schedule+0x29/0x70 [18292.180321] [] taskq_thread+0x3fa/0x440 [spl] [18292.187719] [] ? wake_up_state+0x20/0x20 [18292.194620] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18292.202583] [] kthread+0xcf/0xe0 [18292.208696] [] ? kthread_create_on_node+0x140/0x140 [18292.216649] [] ret_from_fork+0x58/0x90 [18292.223350] [] ? kthread_create_on_node+0x140/0x140 [18292.231310] arc_prune S ffff88081e694d00 0 3244 2 0x00000080 [18292.239901] ffff88082568be08 0000000000000046 ffff880826740fb0 ffff88082568bfd8 [18292.248886] ffff88082568bfd8 ffff88082568bfd8 ffff880826740fb0 ffff8800b168d660 [18292.257888] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18292.266881] Call Trace: [18292.270301] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18292.278283] [] schedule+0x29/0x70 [18292.284498] [] taskq_thread+0x3fa/0x440 [spl] [18292.291885] [] ? wake_up_state+0x20/0x20 [18292.298788] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18292.306753] [] kthread+0xcf/0xe0 [18292.312874] [] ? kthread_create_on_node+0x140/0x140 [18292.320848] [] ret_from_fork+0x58/0x90 [18292.327544] [] ? kthread_create_on_node+0x140/0x140 [18292.335508] arc_prune S ffff88081e694d00 0 3245 2 0x00000080 [18292.344096] ffff88083fea7e08 0000000000000046 ffff880826741f60 ffff88083fea7fd8 [18292.353103] ffff88083fea7fd8 ffff88083fea7fd8 ffff880826741f60 ffff8800b168d6c0 [18292.362112] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18292.371113] Call Trace: [18292.374520] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18292.382492] [] schedule+0x29/0x70 [18292.388712] [] taskq_thread+0x3fa/0x440 [spl] [18292.396098] [] ? wake_up_state+0x20/0x20 [18292.403006] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18292.410976] [] kthread+0xcf/0xe0 [18292.417088] [] ? kthread_create_on_node+0x140/0x140 [18292.425055] [] ret_from_fork+0x58/0x90 [18292.431748] [] ? kthread_create_on_node+0x140/0x140 [18292.439711] arc_prune S ffff88081e694d00 0 3246 2 0x00000080 [18292.448300] ffff880829d2be08 0000000000000046 ffff880826742f10 ffff880829d2bfd8 [18292.457311] ffff880829d2bfd8 ffff880829d2bfd8 ffff880826742f10 ffff8800b168d720 [18292.466320] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18292.475322] Call Trace: [18292.478738] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18292.486707] [] schedule+0x29/0x70 [18292.492928] [] taskq_thread+0x3fa/0x440 [spl] [18292.500325] [] ? wake_up_state+0x20/0x20 [18292.507232] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18292.515205] [] kthread+0xcf/0xe0 [18292.521329] [] ? kthread_create_on_node+0x140/0x140 [18292.529303] [] ret_from_fork+0x58/0x90 [18292.536003] [] ? kthread_create_on_node+0x140/0x140 [18292.543959] arc_prune S ffff88081e694d00 0 3247 2 0x00000080 [18292.552520] ffff8808276a7e08 0000000000000046 ffff880826743ec0 ffff8808276a7fd8 [18292.561517] ffff8808276a7fd8 ffff8808276a7fd8 ffff880826743ec0 ffff8800b168d780 [18292.570514] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18292.579499] Call Trace: [18292.582918] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18292.590898] [] schedule+0x29/0x70 [18292.597122] [] taskq_thread+0x3fa/0x440 [spl] [18292.604509] [] ? wake_up_state+0x20/0x20 [18292.611405] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18292.619372] [] kthread+0xcf/0xe0 [18292.625484] [] ? kthread_create_on_node+0x140/0x140 [18292.633446] [] ret_from_fork+0x58/0x90 [18292.640138] [] ? kthread_create_on_node+0x140/0x140 [18292.648106] arc_prune S ffff88081e694d00 0 3248 2 0x00000080 [18292.656676] ffff880829767e08 0000000000000046 ffff880826744e70 ffff880829767fd8 [18292.665663] ffff880829767fd8 ffff880829767fd8 ffff880826744e70 ffff8800b168d7e0 [18292.674660] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18292.683653] Call Trace: [18292.687061] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18292.695034] [] schedule+0x29/0x70 [18292.701252] [] taskq_thread+0x3fa/0x440 [spl] [18292.708640] [] ? wake_up_state+0x20/0x20 [18292.715540] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18292.723503] [] kthread+0xcf/0xe0 [18292.729607] [] ? kthread_create_on_node+0x140/0x140 [18292.737559] [] ret_from_fork+0x58/0x90 [18292.744262] [] ? kthread_create_on_node+0x140/0x140 [18292.752222] arc_prune S ffff88081e694d00 0 3249 2 0x00000080 [18292.760812] ffff88082979be08 0000000000000046 ffff880826745e20 ffff88082979bfd8 [18292.769796] ffff88082979bfd8 ffff88082979bfd8 ffff880826745e20 ffff8800b168d840 [18292.778805] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18292.787801] Call Trace: [18292.791208] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18292.799194] [] schedule+0x29/0x70 [18292.805411] [] taskq_thread+0x3fa/0x440 [spl] [18292.812805] [] ? wake_up_state+0x20/0x20 [18292.819701] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18292.827662] [] kthread+0xcf/0xe0 [18292.833772] [] ? kthread_create_on_node+0x140/0x140 [18292.841742] [] ret_from_fork+0x58/0x90 [18292.848440] [] ? kthread_create_on_node+0x140/0x140 [18292.856395] arc_prune S ffff88081e694d00 0 3250 2 0x00000080 [18292.864968] ffff88082c707e08 0000000000000046 ffff880826746dd0 ffff88082c707fd8 [18292.873976] ffff88082c707fd8 ffff88082c707fd8 ffff880826746dd0 ffff8800b168d8a0 [18292.882968] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18292.891952] Call Trace: [18292.895380] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18292.903358] [] schedule+0x29/0x70 [18292.909574] [] taskq_thread+0x3fa/0x440 [spl] [18292.916962] [] ? wake_up_state+0x20/0x20 [18292.923872] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18292.931834] [] kthread+0xcf/0xe0 [18292.937952] [] ? kthread_create_on_node+0x140/0x140 [18292.945918] [] ret_from_fork+0x58/0x90 [18292.952611] [] ? kthread_create_on_node+0x140/0x140 [18292.960569] arc_prune S ffff88081e694d00 0 3251 2 0x00000080 [18292.969159] ffff8808253bfe08 0000000000000046 ffff88082ab60000 ffff8808253bffd8 [18292.978167] ffff8808253bffd8 ffff8808253bffd8 ffff88082ab60000 ffff8800b168d900 [18292.987155] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18292.996139] Call Trace: [18292.999545] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18293.007516] [] schedule+0x29/0x70 [18293.013735] [] taskq_thread+0x3fa/0x440 [spl] [18293.021124] [] ? wake_up_state+0x20/0x20 [18293.028034] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18293.036003] [] kthread+0xcf/0xe0 [18293.042123] [] ? kthread_create_on_node+0x140/0x140 [18293.050081] [] ret_from_fork+0x58/0x90 [18293.056782] [] ? kthread_create_on_node+0x140/0x140 [18293.064758] arc_prune S ffff88081e694d00 0 3252 2 0x00000080 [18293.073345] ffff88082bef7e08 0000000000000046 ffff88082ab60fb0 ffff88082bef7fd8 [18293.082349] ffff88082bef7fd8 ffff88082bef7fd8 ffff88082ab60fb0 ffff8800b168d960 [18293.091356] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18293.100356] Call Trace: [18293.103779] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18293.111759] [] schedule+0x29/0x70 [18293.117995] [] taskq_thread+0x3fa/0x440 [spl] [18293.125393] [] ? wake_up_state+0x20/0x20 [18293.132311] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18293.140277] [] kthread+0xcf/0xe0 [18293.146394] [] ? kthread_create_on_node+0x140/0x140 [18293.154358] [] ret_from_fork+0x58/0x90 [18293.161057] [] ? kthread_create_on_node+0x140/0x140 [18293.169024] arc_prune S ffff88081e694d00 0 3253 2 0x00000080 [18293.177582] ffff880829f8fe08 0000000000000046 ffff88082ab61f60 ffff880829f8ffd8 [18293.186577] ffff880829f8ffd8 ffff880829f8ffd8 ffff88082ab61f60 ffff8800b168d9c0 [18293.195565] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18293.204554] Call Trace: [18293.207967] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18293.215940] [] schedule+0x29/0x70 [18293.222157] [] taskq_thread+0x3fa/0x440 [spl] [18293.229548] [] ? wake_up_state+0x20/0x20 [18293.236443] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18293.244400] [] kthread+0xcf/0xe0 [18293.250517] [] ? kthread_create_on_node+0x140/0x140 [18293.258472] [] ret_from_fork+0x58/0x90 [18293.265168] [] ? kthread_create_on_node+0x140/0x140 [18293.273134] arc_prune S ffff88081e694d00 0 3254 2 0x00000080 [18293.281700] ffff88082d6afe08 0000000000000046 ffff88082ab62f10 ffff88082d6affd8 [18293.290687] ffff88082d6affd8 ffff88082d6affd8 ffff88082ab62f10 ffff8800b168da20 [18293.299679] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18293.308669] Call Trace: [18293.312071] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18293.320059] [] schedule+0x29/0x70 [18293.326286] [] taskq_thread+0x3fa/0x440 [spl] [18293.333673] [] ? wake_up_state+0x20/0x20 [18293.340577] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18293.348540] [] kthread+0xcf/0xe0 [18293.354650] [] ? kthread_create_on_node+0x140/0x140 [18293.362609] [] ret_from_fork+0x58/0x90 [18293.369313] [] ? kthread_create_on_node+0x140/0x140 [18293.377278] arc_prune S ffff88081e694d00 0 3255 2 0x00000080 [18293.385865] ffff88082d8ffe08 0000000000000046 ffff88082ab63ec0 ffff88082d8fffd8 [18293.394862] ffff88082d8fffd8 ffff88082d8fffd8 ffff88082ab63ec0 ffff8800b168da80 [18293.403862] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18293.412859] Call Trace: [18293.416264] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18293.424239] [] schedule+0x29/0x70 [18293.430455] [] taskq_thread+0x3fa/0x440 [spl] [18293.437851] [] ? wake_up_state+0x20/0x20 [18293.444756] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18293.452718] [] kthread+0xcf/0xe0 [18293.458828] [] ? kthread_create_on_node+0x140/0x140 [18293.466792] [] ret_from_fork+0x58/0x90 [18293.473484] [] ? kthread_create_on_node+0x140/0x140 [18293.481440] arc_prune S ffff88081e694d00 0 3256 2 0x00000080 [18293.490040] ffff88082dc3fe08 0000000000000046 ffff88082ab64e70 ffff88082dc3ffd8 [18293.499041] ffff88082dc3ffd8 ffff88082dc3ffd8 ffff88082ab64e70 ffff8800b168dae0 [18293.508037] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18293.517059] Call Trace: [18293.520471] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18293.528443] [] schedule+0x29/0x70 [18293.534661] [] taskq_thread+0x3fa/0x440 [spl] [18293.542055] [] ? wake_up_state+0x20/0x20 [18293.548958] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18293.556916] [] kthread+0xcf/0xe0 [18293.563022] [] ? kthread_create_on_node+0x140/0x140 [18293.570974] [] ret_from_fork+0x58/0x90 [18293.577662] [] ? kthread_create_on_node+0x140/0x140 [18293.585614] arc_prune S ffff88081e694d00 0 3257 2 0x00000080 [18293.594187] ffff88083fecfe08 0000000000000046 ffff88082ab65e20 ffff88083fecffd8 [18293.603174] ffff88083fecffd8 ffff88083fecffd8 ffff88082ab65e20 ffff8800b168db40 [18293.612166] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18293.621156] Call Trace: [18293.624557] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18293.632523] [] schedule+0x29/0x70 [18293.638734] [] taskq_thread+0x3fa/0x440 [spl] [18293.646129] [] ? wake_up_state+0x20/0x20 [18293.653038] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18293.660998] [] kthread+0xcf/0xe0 [18293.667120] [] ? kthread_create_on_node+0x140/0x140 [18293.675085] [] ret_from_fork+0x58/0x90 [18293.681765] [] ? kthread_create_on_node+0x140/0x140 [18293.689724] arc_prune S ffff88081e694d00 0 3258 2 0x00000080 [18293.698310] ffff88082e4bbe08 0000000000000046 ffff88082ab66dd0 ffff88082e4bbfd8 [18293.707314] ffff88082e4bbfd8 ffff88082e4bbfd8 ffff88082ab66dd0 ffff8800b168dba0 [18293.716320] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18293.725319] Call Trace: [18293.728730] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18293.736701] [] schedule+0x29/0x70 [18293.742918] [] taskq_thread+0x3fa/0x440 [spl] [18293.750315] [] ? wake_up_state+0x20/0x20 [18293.757219] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18293.765184] [] kthread+0xcf/0xe0 [18293.771296] [] ? kthread_create_on_node+0x140/0x140 [18293.779259] [] ret_from_fork+0x58/0x90 [18293.785962] [] ? kthread_create_on_node+0x140/0x140 [18293.793930] arc_prune S ffff88081e694d00 0 3259 2 0x00000080 [18293.802509] ffff88082dd37e08 0000000000000046 ffff880826be0000 ffff88082dd37fd8 [18293.811498] ffff88082dd37fd8 ffff88082dd37fd8 ffff880826be0000 ffff8800b168dc00 [18293.820484] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18293.829481] Call Trace: [18293.832903] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18293.840875] [] schedule+0x29/0x70 [18293.847103] [] taskq_thread+0x3fa/0x440 [spl] [18293.854483] [] ? wake_up_state+0x20/0x20 [18293.861406] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18293.869385] [] kthread+0xcf/0xe0 [18293.875489] [] ? kthread_create_on_node+0x140/0x140 [18293.883449] [] ret_from_fork+0x58/0x90 [18293.890158] [] ? kthread_create_on_node+0x140/0x140 [18293.898123] arc_prune S ffff88081e694d00 0 3260 2 0x00000080 [18293.906694] ffff880829ac3e08 0000000000000046 ffff880826be0fb0 ffff880829ac3fd8 [18293.915684] ffff880829ac3fd8 ffff880829ac3fd8 ffff880826be0fb0 ffff8800b168dc60 [18293.924671] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18293.933662] Call Trace: [18293.937085] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18293.945068] [] schedule+0x29/0x70 [18293.951295] [] taskq_thread+0x3fa/0x440 [spl] [18293.958680] [] ? wake_up_state+0x20/0x20 [18293.965583] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18293.973538] [] kthread+0xcf/0xe0 [18293.979633] [] ? kthread_create_on_node+0x140/0x140 [18293.987588] [] ret_from_fork+0x58/0x90 [18293.994297] [] ? kthread_create_on_node+0x140/0x140 [18294.002267] arc_prune S ffff88081e694d00 0 3261 2 0x00000080 [18294.010840] ffff88082ae6be08 0000000000000046 ffff880826be1f60 ffff88082ae6bfd8 [18294.019848] ffff88082ae6bfd8 ffff88082ae6bfd8 ffff880826be1f60 ffff8800b168dcc0 [18294.028842] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18294.037849] Call Trace: [18294.041267] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18294.049250] [] schedule+0x29/0x70 [18294.055460] [] taskq_thread+0x3fa/0x440 [spl] [18294.062838] [] ? wake_up_state+0x20/0x20 [18294.069736] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18294.077698] [] kthread+0xcf/0xe0 [18294.083822] [] ? kthread_create_on_node+0x140/0x140 [18294.091787] [] ret_from_fork+0x58/0x90 [18294.098480] [] ? kthread_create_on_node+0x140/0x140 [18294.106442] arc_prune S ffff88081e694d00 0 3262 2 0x00000080 [18294.115010] ffff88082a6afe08 0000000000000046 ffff880826be2f10 ffff88082a6affd8 [18294.124011] ffff88082a6affd8 ffff88082a6affd8 ffff880826be2f10 ffff8800b168dd20 [18294.133015] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18294.142014] Call Trace: [18294.145431] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18294.153409] [] schedule+0x29/0x70 [18294.159624] [] taskq_thread+0x3fa/0x440 [spl] [18294.167010] [] ? wake_up_state+0x20/0x20 [18294.173912] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18294.181875] [] kthread+0xcf/0xe0 [18294.187987] [] ? kthread_create_on_node+0x140/0x140 [18294.195942] [] ret_from_fork+0x58/0x90 [18294.202620] [] ? kthread_create_on_node+0x140/0x140 [18294.210573] arc_prune S ffff88081e694d00 0 3263 2 0x00000080 [18294.219155] ffff88082e33be08 0000000000000046 ffff880826be3ec0 ffff88082e33bfd8 [18294.228153] ffff88082e33bfd8 ffff88082e33bfd8 ffff880826be3ec0 ffff8800b168dd80 [18294.237162] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18294.246164] Call Trace: [18294.249568] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18294.257538] [] schedule+0x29/0x70 [18294.263746] [] taskq_thread+0x3fa/0x440 [spl] [18294.271125] [] ? wake_up_state+0x20/0x20 [18294.278028] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18294.285992] [] kthread+0xcf/0xe0 [18294.292106] [] ? kthread_create_on_node+0x140/0x140 [18294.300066] [] ret_from_fork+0x58/0x90 [18294.306753] [] ? kthread_create_on_node+0x140/0x140 [18294.314705] arc_prune S ffff88081e694d00 0 3264 2 0x00000080 [18294.323275] ffff88082e7b7e08 0000000000000046 ffff880826be4e70 ffff88082e7b7fd8 [18294.332275] ffff88082e7b7fd8 ffff88082e7b7fd8 ffff880826be4e70 ffff8800b168dde0 [18294.341268] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18294.350273] Call Trace: [18294.353669] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18294.361633] [] schedule+0x29/0x70 [18294.367853] [] taskq_thread+0x3fa/0x440 [spl] [18294.375235] [] ? wake_up_state+0x20/0x20 [18294.382139] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18294.390105] [] kthread+0xcf/0xe0 [18294.396210] [] ? kthread_create_on_node+0x140/0x140 [18294.404170] [] ret_from_fork+0x58/0x90 [18294.410859] [] ? kthread_create_on_node+0x140/0x140 [18294.418810] arc_prune S ffff88081e694d00 0 3265 2 0x00000080 [18294.427379] ffff880829c1fe08 0000000000000046 ffff880826be5e20 ffff880829c1ffd8 [18294.436372] ffff880829c1ffd8 ffff880829c1ffd8 ffff880826be5e20 ffff8800b168de40 [18294.445359] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18294.454340] Call Trace: [18294.457737] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18294.465703] [] schedule+0x29/0x70 [18294.471922] [] taskq_thread+0x3fa/0x440 [spl] [18294.479303] [] ? wake_up_state+0x20/0x20 [18294.486207] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18294.494171] [] kthread+0xcf/0xe0 [18294.500284] [] ? kthread_create_on_node+0x140/0x140 [18294.508235] [] ret_from_fork+0x58/0x90 [18294.514924] [] ? kthread_create_on_node+0x140/0x140 [18294.522878] arc_prune S ffff88081e694d00 0 3266 2 0x00000080 [18294.531435] ffff88082ddfbe08 0000000000000046 ffff880826be6dd0 ffff88082ddfbfd8 [18294.540420] ffff88082ddfbfd8 ffff88082ddfbfd8 ffff880826be6dd0 ffff8800b168dea0 [18294.549408] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18294.558400] Call Trace: [18294.561810] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18294.569785] [] schedule+0x29/0x70 [18294.576006] [] taskq_thread+0x3fa/0x440 [spl] [18294.583389] [] ? wake_up_state+0x20/0x20 [18294.590294] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18294.598261] [] kthread+0xcf/0xe0 [18294.604376] [] ? kthread_create_on_node+0x140/0x140 [18294.612337] [] ret_from_fork+0x58/0x90 [18294.619026] [] ? kthread_create_on_node+0x140/0x140 [18294.626979] arc_prune S ffff88081e694d00 0 3267 2 0x00000080 [18294.635553] ffff88082be07e08 0000000000000046 ffff88082e9f0000 ffff88082be07fd8 [18294.644538] ffff88082be07fd8 ffff88082be07fd8 ffff88082e9f0000 ffff8800b168df00 [18294.653521] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18294.662507] Call Trace: [18294.665909] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18294.673876] [] schedule+0x29/0x70 [18294.680090] [] taskq_thread+0x3fa/0x440 [spl] [18294.687472] [] ? wake_up_state+0x20/0x20 [18294.694371] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18294.702329] [] kthread+0xcf/0xe0 [18294.708436] [] ? kthread_create_on_node+0x140/0x140 [18294.716391] [] ret_from_fork+0x58/0x90 [18294.723090] [] ? kthread_create_on_node+0x140/0x140 [18294.731058] arc_prune S ffff88081e694d00 0 3268 2 0x00000080 [18294.739622] ffff88082dab3e08 0000000000000046 ffff88082e9f0fb0 ffff88082dab3fd8 [18294.748622] ffff88082dab3fd8 ffff88082dab3fd8 ffff88082e9f0fb0 ffff8800b168df60 [18294.757619] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18294.766620] Call Trace: [18294.770030] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18294.778001] [] schedule+0x29/0x70 [18294.784226] [] taskq_thread+0x3fa/0x440 [spl] [18294.791615] [] ? wake_up_state+0x20/0x20 [18294.798519] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18294.806481] [] kthread+0xcf/0xe0 [18294.812590] [] ? kthread_create_on_node+0x140/0x140 [18294.820548] [] ret_from_fork+0x58/0x90 [18294.827251] [] ? kthread_create_on_node+0x140/0x140 [18294.835217] arc_prune S ffff88081e694d00 0 3269 2 0x00000080 [18294.843783] ffff88083fbffe08 0000000000000046 ffff88082e9f1f60 ffff88083fbfffd8 [18294.852798] ffff88083fbfffd8 ffff88083fbfffd8 ffff88082e9f1f60 ffff8800b1698000 [18294.861789] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18294.870774] Call Trace: [18294.874178] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18294.882150] [] schedule+0x29/0x70 [18294.888380] [] taskq_thread+0x3fa/0x440 [spl] [18294.895764] [] ? wake_up_state+0x20/0x20 [18294.902657] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18294.910619] [] kthread+0xcf/0xe0 [18294.916730] [] ? kthread_create_on_node+0x140/0x140 [18294.924689] [] ret_from_fork+0x58/0x90 [18294.931384] [] ? kthread_create_on_node+0x140/0x140 [18294.939338] arc_prune S ffff88081e694d00 0 3270 2 0x00000080 [18294.947922] ffff88082a68be08 0000000000000046 ffff88082e9f2f10 ffff88082a68bfd8 [18294.956920] ffff88082a68bfd8 ffff88082a68bfd8 ffff88082e9f2f10 ffff8800b1698060 [18294.965915] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18294.974909] Call Trace: [18294.978333] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18294.986306] [] schedule+0x29/0x70 [18294.992517] [] taskq_thread+0x3fa/0x440 [spl] [18294.999900] [] ? wake_up_state+0x20/0x20 [18295.006802] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18295.014764] [] kthread+0xcf/0xe0 [18295.020884] [] ? kthread_create_on_node+0x140/0x140 [18295.028851] [] ret_from_fork+0x58/0x90 [18295.035548] [] ? kthread_create_on_node+0x140/0x140 [18295.043508] arc_prune S ffff88081e694d00 0 3271 2 0x00000080 [18295.052092] ffff880829a73e08 0000000000000046 ffff88082e9f3ec0 ffff880829a73fd8 [18295.061094] ffff880829a73fd8 ffff880829a73fd8 ffff88082e9f3ec0 ffff8800b16980c0 [18295.070104] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18295.079110] Call Trace: [18295.082513] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18295.090491] [] schedule+0x29/0x70 [18295.096702] [] taskq_thread+0x3fa/0x440 [spl] [18295.104100] [] ? wake_up_state+0x20/0x20 [18295.111005] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18295.118973] [] kthread+0xcf/0xe0 [18295.125088] [] ? kthread_create_on_node+0x140/0x140 [18295.133054] [] ret_from_fork+0x58/0x90 [18295.139749] [] ? kthread_create_on_node+0x140/0x140 [18295.147705] arc_prune S ffff88081e694d00 0 3272 2 0x00000080 [18295.156295] ffff88082dc37e08 0000000000000046 ffff88082e9f4e70 ffff88082dc37fd8 [18295.165301] ffff88082dc37fd8 ffff88082dc37fd8 ffff88082e9f4e70 ffff8800b1698120 [18295.174300] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18295.183288] Call Trace: [18295.186696] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18295.194667] [] schedule+0x29/0x70 [18295.200886] [] taskq_thread+0x3fa/0x440 [spl] [18295.208271] [] ? wake_up_state+0x20/0x20 [18295.215178] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18295.223138] [] kthread+0xcf/0xe0 [18295.229263] [] ? kthread_create_on_node+0x140/0x140 [18295.237223] [] ret_from_fork+0x58/0x90 [18295.243935] [] ? kthread_create_on_node+0x140/0x140 [18295.251904] arc_prune S ffff88081e694d00 0 3273 2 0x00000080 [18295.260483] ffff88082e83fe08 0000000000000046 ffff88082e9f5e20 ffff88082e83ffd8 [18295.269478] ffff88082e83ffd8 ffff88082e83ffd8 ffff88082e9f5e20 ffff8800b1698180 [18295.278467] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18295.287452] Call Trace: [18295.290861] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18295.298841] [] schedule+0x29/0x70 [18295.305059] [] taskq_thread+0x3fa/0x440 [spl] [18295.312451] [] ? wake_up_state+0x20/0x20 [18295.319362] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18295.327327] [] kthread+0xcf/0xe0 [18295.333453] [] ? kthread_create_on_node+0x140/0x140 [18295.341425] [] ret_from_fork+0x58/0x90 [18295.348120] [] ? kthread_create_on_node+0x140/0x140 [18295.356077] arc_prune S ffff88081e694d00 0 3274 2 0x00000080 [18295.364644] ffff88082e0f7e08 0000000000000046 ffff88082e9f6dd0 ffff88082e0f7fd8 [18295.373636] ffff88082e0f7fd8 ffff88082e0f7fd8 ffff88082e9f6dd0 ffff8800b16981e0 [18295.382618] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18295.391614] Call Trace: [18295.395025] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18295.402995] [] schedule+0x29/0x70 [18295.409211] [] taskq_thread+0x3fa/0x440 [spl] [18295.416593] [] ? wake_up_state+0x20/0x20 [18295.423487] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18295.431457] [] kthread+0xcf/0xe0 [18295.437569] [] ? kthread_create_on_node+0x140/0x140 [18295.445531] [] ret_from_fork+0x58/0x90 [18295.452227] [] ? kthread_create_on_node+0x140/0x140 [18295.460192] arc_prune S ffff88081e694d00 0 3275 2 0x00000080 [18295.468763] ffff88082bdcfe08 0000000000000046 ffff88083f7a8000 ffff88082bdcffd8 [18295.477750] ffff88082bdcffd8 ffff88082bdcffd8 ffff88083f7a8000 ffff8800b1698240 [18295.486745] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18295.495737] Call Trace: [18295.499149] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18295.507121] [] schedule+0x29/0x70 [18295.513344] [] taskq_thread+0x3fa/0x440 [spl] [18295.520728] [] ? wake_up_state+0x20/0x20 [18295.527628] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18295.535587] [] kthread+0xcf/0xe0 [18295.541691] [] ? kthread_create_on_node+0x140/0x140 [18295.549634] [] ret_from_fork+0x58/0x90 [18295.556335] [] ? kthread_create_on_node+0x140/0x140 [18295.564307] arc_prune S ffff88081e694d00 0 3276 2 0x00000080 [18295.572895] ffff88082c6abe08 0000000000000046 ffff88083f7a8fb0 ffff88082c6abfd8 [18295.581899] ffff88082c6abfd8 ffff88082c6abfd8 ffff88083f7a8fb0 ffff8800b16982a0 [18295.590897] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18295.599893] Call Trace: [18295.603307] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18295.611278] [] schedule+0x29/0x70 [18295.617487] [] taskq_thread+0x3fa/0x440 [spl] [18295.624873] [] ? wake_up_state+0x20/0x20 [18295.631780] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18295.639743] [] kthread+0xcf/0xe0 [18295.645856] [] ? kthread_create_on_node+0x140/0x140 [18295.653814] [] ret_from_fork+0x58/0x90 [18295.660508] [] ? kthread_create_on_node+0x140/0x140 [18295.668471] arc_prune S ffff88081e694d00 0 3277 2 0x00000080 [18295.677047] ffff88082d103e08 0000000000000046 ffff88083f7a9f60 ffff88082d103fd8 [18295.686047] ffff88082d103fd8 ffff88082d103fd8 ffff88083f7a9f60 ffff8800b1698300 [18295.695037] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18295.704021] Call Trace: [18295.707439] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18295.715425] [] schedule+0x29/0x70 [18295.721640] [] taskq_thread+0x3fa/0x440 [spl] [18295.729033] [] ? wake_up_state+0x20/0x20 [18295.735937] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18295.743898] [] kthread+0xcf/0xe0 [18295.750015] [] ? kthread_create_on_node+0x140/0x140 [18295.757973] [] ret_from_fork+0x58/0x90 [18295.764660] [] ? kthread_create_on_node+0x140/0x140 [18295.772613] arc_prune S ffff88081e694d00 0 3278 2 0x00000080 [18295.781199] ffff88082dcf7e08 0000000000000046 ffff88083f7aaf10 ffff88082dcf7fd8 [18295.790203] ffff88082dcf7fd8 ffff88082dcf7fd8 ffff88083f7aaf10 ffff8800b1698360 [18295.799192] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18295.808208] Call Trace: [18295.811612] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18295.819582] [] schedule+0x29/0x70 [18295.825797] [] taskq_thread+0x3fa/0x440 [spl] [18295.833194] [] ? wake_up_state+0x20/0x20 [18295.840095] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18295.848067] [] kthread+0xcf/0xe0 [18295.854187] [] ? kthread_create_on_node+0x140/0x140 [18295.862152] [] ret_from_fork+0x58/0x90 [18295.868846] [] ? kthread_create_on_node+0x140/0x140 [18295.876803] arc_prune S ffff88081e694d00 0 3279 2 0x00000080 [18295.885387] ffff88082e03fe08 0000000000000046 ffff88083f7abec0 ffff88082e03ffd8 [18295.894386] ffff88082e03ffd8 ffff88082e03ffd8 ffff88083f7abec0 ffff8800b16983c0 [18295.903395] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18295.912394] Call Trace: [18295.915799] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18295.923772] [] schedule+0x29/0x70 [18295.929998] [] taskq_thread+0x3fa/0x440 [spl] [18295.937391] [] ? wake_up_state+0x20/0x20 [18295.944294] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18295.952258] [] kthread+0xcf/0xe0 [18295.958370] [] ? kthread_create_on_node+0x140/0x140 [18295.966327] [] ret_from_fork+0x58/0x90 [18295.973021] [] ? kthread_create_on_node+0x140/0x140 [18295.980979] arc_prune S ffff88081e694d00 0 3280 2 0x00000080 [18295.989566] ffff88082c977e08 0000000000000046 ffff88083f7ace70 ffff88082c977fd8 [18295.998556] ffff88082c977fd8 ffff88082c977fd8 ffff88083f7ace70 ffff8800b1698420 [18296.007556] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18296.016553] Call Trace: [18296.019958] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18296.027931] [] schedule+0x29/0x70 [18296.034137] [] taskq_thread+0x3fa/0x440 [spl] [18296.041525] [] ? wake_up_state+0x20/0x20 [18296.048435] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18296.056410] [] kthread+0xcf/0xe0 [18296.062514] [] ? kthread_create_on_node+0x140/0x140 [18296.070478] [] ret_from_fork+0x58/0x90 [18296.077181] [] ? kthread_create_on_node+0x140/0x140 [18296.085133] arc_prune S ffff88081e694d00 0 3281 2 0x00000080 [18296.093700] ffff88082a7ebe08 0000000000000046 ffff88083f7ade20 ffff88082a7ebfd8 [18296.102696] ffff88082a7ebfd8 ffff88082a7ebfd8 ffff88083f7ade20 ffff8800b1698480 [18296.111699] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18296.120699] Call Trace: [18296.124112] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18296.132085] [] schedule+0x29/0x70 [18296.138305] [] taskq_thread+0x3fa/0x440 [spl] [18296.145689] [] ? wake_up_state+0x20/0x20 [18296.152588] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18296.160547] [] kthread+0xcf/0xe0 [18296.166655] [] ? kthread_create_on_node+0x140/0x140 [18296.174618] [] ret_from_fork+0x58/0x90 [18296.181310] [] ? kthread_create_on_node+0x140/0x140 [18296.189273] arc_prune S ffff88081e694d00 0 3282 2 0x00000080 [18296.197851] ffff88082dff7e08 0000000000000046 ffff88083f7aedd0 ffff88082dff7fd8 [18296.206837] ffff88082dff7fd8 ffff88082dff7fd8 ffff88083f7aedd0 ffff8800b16984e0 [18296.215839] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18296.224823] Call Trace: [18296.228244] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18296.236224] [] schedule+0x29/0x70 [18296.242446] [] taskq_thread+0x3fa/0x440 [spl] [18296.249824] [] ? wake_up_state+0x20/0x20 [18296.256732] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18296.264692] [] kthread+0xcf/0xe0 [18296.270803] [] ? kthread_create_on_node+0x140/0x140 [18296.278760] [] ret_from_fork+0x58/0x90 [18296.285467] [] ? kthread_create_on_node+0x140/0x140 [18296.293440] arc_prune S ffff88081e694d00 0 3283 2 0x00000080 [18296.302021] ffff88082e1ffe08 0000000000000046 ffff880826690000 ffff88082e1fffd8 [18296.311005] ffff88082e1fffd8 ffff88082e1fffd8 ffff880826690000 ffff8800b1698540 [18296.320002] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18296.329008] Call Trace: [18296.332423] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18296.340402] [] schedule+0x29/0x70 [18296.346616] [] taskq_thread+0x3fa/0x440 [spl] [18296.353997] [] ? wake_up_state+0x20/0x20 [18296.360913] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18296.368883] [] kthread+0xcf/0xe0 [18296.374991] [] ? kthread_create_on_node+0x140/0x140 [18296.382954] [] ret_from_fork+0x58/0x90 [18296.389647] [] ? kthread_create_on_node+0x140/0x140 [18296.397605] arc_prune S ffff88081e694d00 0 3284 2 0x00000080 [18296.406191] ffff88082e777e08 0000000000000046 ffff880826690fb0 ffff88082e777fd8 [18296.415196] ffff88082e777fd8 ffff88082e777fd8 ffff880826690fb0 ffff8800b16985a0 [18296.424195] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18296.433187] Call Trace: [18296.436593] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18296.444556] [] schedule+0x29/0x70 [18296.450766] [] taskq_thread+0x3fa/0x440 [spl] [18296.458150] [] ? wake_up_state+0x20/0x20 [18296.465050] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18296.473030] [] kthread+0xcf/0xe0 [18296.479152] [] ? kthread_create_on_node+0x140/0x140 [18296.487095] [] ret_from_fork+0x58/0x90 [18296.493786] [] ? kthread_create_on_node+0x140/0x140 [18296.501735] arc_prune S ffff88081e694d00 0 3285 2 0x00000080 [18296.510328] ffff8808297dfe08 0000000000000046 ffff880826691f60 ffff8808297dffd8 [18296.519338] ffff8808297dffd8 ffff8808297dffd8 ffff880826691f60 ffff8800b1698600 [18296.528339] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18296.537348] Call Trace: [18296.540762] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18296.548730] [] schedule+0x29/0x70 [18296.554946] [] taskq_thread+0x3fa/0x440 [spl] [18296.562327] [] ? wake_up_state+0x20/0x20 [18296.569234] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18296.577202] [] kthread+0xcf/0xe0 [18296.583313] [] ? kthread_create_on_node+0x140/0x140 [18296.591263] [] ret_from_fork+0x58/0x90 [18296.597963] [] ? kthread_create_on_node+0x140/0x140 [18296.605921] arc_prune S ffff88081e694d00 0 3286 2 0x00000080 [18296.614509] ffff88082a6d7e08 0000000000000046 ffff880826692f10 ffff88082a6d7fd8 [18296.623516] ffff88082a6d7fd8 ffff88082a6d7fd8 ffff880826692f10 ffff8800b1698660 [18296.632507] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18296.641504] Call Trace: [18296.644910] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18296.652892] [] schedule+0x29/0x70 [18296.659113] [] taskq_thread+0x3fa/0x440 [spl] [18296.666507] [] ? wake_up_state+0x20/0x20 [18296.673419] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18296.681378] [] kthread+0xcf/0xe0 [18296.687491] [] ? kthread_create_on_node+0x140/0x140 [18296.695454] [] ret_from_fork+0x58/0x90 [18296.702146] [] ? kthread_create_on_node+0x140/0x140 [18296.710104] arc_prune S ffff88081e694d00 0 3287 2 0x00000080 [18296.718670] ffff880829bc7e08 0000000000000046 ffff880826693ec0 ffff880829bc7fd8 [18296.727670] ffff880829bc7fd8 ffff880829bc7fd8 ffff880826693ec0 ffff8800b16986c0 [18296.736669] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18296.745673] Call Trace: [18296.749084] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18296.757060] [] schedule+0x29/0x70 [18296.763283] [] taskq_thread+0x3fa/0x440 [spl] [18296.770661] [] ? wake_up_state+0x20/0x20 [18296.777558] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18296.785522] [] kthread+0xcf/0xe0 [18296.791614] [] ? kthread_create_on_node+0x140/0x140 [18296.799573] [] ret_from_fork+0x58/0x90 [18296.806269] [] ? kthread_create_on_node+0x140/0x140 [18296.814227] arc_prune S ffff88081e694d00 0 3288 2 0x00000080 [18296.822792] ffff880829fb3e08 0000000000000046 ffff880826694e70 ffff880829fb3fd8 [18296.831787] ffff880829fb3fd8 ffff880829fb3fd8 ffff880826694e70 ffff8800b1698720 [18296.840787] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18296.849771] Call Trace: [18296.853172] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18296.861142] [] schedule+0x29/0x70 [18296.867358] [] taskq_thread+0x3fa/0x440 [spl] [18296.874741] [] ? wake_up_state+0x20/0x20 [18296.881638] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18296.889590] [] kthread+0xcf/0xe0 [18296.895693] [] ? kthread_create_on_node+0x140/0x140 [18296.903648] [] ret_from_fork+0x58/0x90 [18296.910338] [] ? kthread_create_on_node+0x140/0x140 [18296.918295] arc_prune S ffff88081e694d00 0 3289 2 0x00000080 [18296.926884] ffff88082afe3e08 0000000000000046 ffff880826695e20 ffff88082afe3fd8 [18296.935889] ffff88082afe3fd8 ffff88082afe3fd8 ffff880826695e20 ffff8800b1698780 [18296.944881] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18296.953871] Call Trace: [18296.957273] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18296.965243] [] schedule+0x29/0x70 [18296.971463] [] taskq_thread+0x3fa/0x440 [spl] [18296.978842] [] ? wake_up_state+0x20/0x20 [18296.985741] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18296.993700] [] kthread+0xcf/0xe0 [18296.999809] [] ? kthread_create_on_node+0x140/0x140 [18297.007765] [] ret_from_fork+0x58/0x90 [18297.014464] [] ? kthread_create_on_node+0x140/0x140 [18297.022428] arc_prune S ffff88081e694d00 0 3290 2 0x00000080 [18297.031010] ffff880829733e08 0000000000000046 ffff880826696dd0 ffff880829733fd8 [18297.040006] ffff880829733fd8 ffff880829733fd8 ffff880826696dd0 ffff8800b16987e0 [18297.048999] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18297.057985] Call Trace: [18297.061388] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18297.069357] [] schedule+0x29/0x70 [18297.075570] [] taskq_thread+0x3fa/0x440 [spl] [18297.082956] [] ? wake_up_state+0x20/0x20 [18297.089847] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18297.097806] [] kthread+0xcf/0xe0 [18297.103914] [] ? kthread_create_on_node+0x140/0x140 [18297.111876] [] ret_from_fork+0x58/0x90 [18297.118570] [] ? kthread_create_on_node+0x140/0x140 [18297.126537] arc_prune S ffff88081e694d00 0 3291 2 0x00000080 [18297.135122] ffff88082d33fe08 0000000000000046 ffff88082d158000 ffff88082d33ffd8 [18297.144134] ffff88082d33ffd8 ffff88082d33ffd8 ffff88082d158000 ffff8800b1698840 [18297.153137] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18297.162139] Call Trace: [18297.165550] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18297.173526] [] schedule+0x29/0x70 [18297.179741] [] taskq_thread+0x3fa/0x440 [spl] [18297.187127] [] ? wake_up_state+0x20/0x20 [18297.194038] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18297.202009] [] kthread+0xcf/0xe0 [18297.208120] [] ? kthread_create_on_node+0x140/0x140 [18297.216078] [] ret_from_fork+0x58/0x90 [18297.222771] [] ? kthread_create_on_node+0x140/0x140 [18297.230728] arc_prune S ffff88081e694d00 0 3292 2 0x00000080 [18297.239314] ffff88082de3be08 0000000000000046 ffff88082d158fb0 ffff88082de3bfd8 [18297.248322] ffff88082de3bfd8 ffff88082de3bfd8 ffff88082d158fb0 ffff8800b16988a0 [18297.257330] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18297.266339] Call Trace: [18297.269747] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18297.277719] [] schedule+0x29/0x70 [18297.283935] [] taskq_thread+0x3fa/0x440 [spl] [18297.291335] [] ? wake_up_state+0x20/0x20 [18297.298240] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18297.306209] [] kthread+0xcf/0xe0 [18297.312323] [] ? kthread_create_on_node+0x140/0x140 [18297.320289] [] ret_from_fork+0x58/0x90 [18297.326984] [] ? kthread_create_on_node+0x140/0x140 [18297.334949] arc_prune S ffff88081e694d00 0 3293 2 0x00000080 [18297.343529] ffff880829f9fe08 0000000000000046 ffff88082d159f60 ffff880829f9ffd8 [18297.352538] ffff880829f9ffd8 ffff880829f9ffd8 ffff88082d159f60 ffff8800b1698900 [18297.361541] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18297.370532] Call Trace: [18297.373949] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18297.381925] [] schedule+0x29/0x70 [18297.388148] [] taskq_thread+0x3fa/0x440 [spl] [18297.395547] [] ? wake_up_state+0x20/0x20 [18297.402451] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18297.410414] [] kthread+0xcf/0xe0 [18297.416540] [] ? kthread_create_on_node+0x140/0x140 [18297.424513] [] ret_from_fork+0x58/0x90 [18297.431205] [] ? kthread_create_on_node+0x140/0x140 [18297.439170] arc_prune S ffff88081e694d00 0 3294 2 0x00000080 [18297.447728] ffff88082a35fe08 0000000000000046 ffff88082d15af10 ffff88082a35ffd8 [18297.456731] ffff88082a35ffd8 ffff88082a35ffd8 ffff88082d15af10 ffff8800b1698960 [18297.465718] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18297.474717] Call Trace: [18297.478140] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18297.479869] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] Message from[18297.479896] Modules linked in: rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support syslogd@soak-8 at May 9 23:14:09 ... kernel mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes :NMI watchdog: BUG: soft lockup - CPU#24 stuck f[18297.479912] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 or 22s! [OI_scru[18297.479913] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 b:5551] [18297.479913] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [18297.479933] RIP: 0010:[] [] osd_inode_iteration+0x269/0xcc0 [osd_ldiskfs] [18297.479934] RSP: 0018:ffff880703603d18 EFLAGS: 00000202 [18297.479934] RAX: 00000000000072ce RBX: 0000000000000001 RCX: ffff88042bd78800 [18297.479935] RDX: 00000000000072ce RSI: ffff880419dc3a00 RDI: ffff88042bd79800 [18297.479936] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [18297.479945] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [18297.479945] R13: ffffea00027e3c80 R14: 000000009f8f2001 R15: ffff880703603d08 [18297.479946] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [18297.479947] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18297.479948] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [18297.479948] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [18297.479949] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [18297.479949] Stack: [18297.479950] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [18297.479951] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [18297.479952] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [18297.479952] Call Trace: [18297.479960] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [18297.479966] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [18297.479972] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [18297.479974] [] ? wake_up_state+0x20/0x20 [18297.479980] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [18297.479982] [] kthread+0xcf/0xe0 [18297.479984] [] ? kthread_create_on_node+0x140/0x140 [18297.479985] [] ret_from_fork+0x58/0x90 [18297.479986] [] ? kthread_create_on_node+0x140/0x140 [18297.479996] Code: 88 e8 dc 2d f5 ff 48 85 c0 48 89 45 90 0f 84 b3 08 00 00 44 8b 7d a0 48 8b 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 e8 77 5b f4 ff <42> 8d 14 38 48 8b 45 88 48 8b 80 a8 03 00 00 48 3b 50 20 0f 83 [18297.906787] [] schedule+0x29/0x70 [18297.913315] [] taskq_thread+0x3fa/0x440 [spl] [18297.921006] [] ? wake_up_state+0x20/0x20 [18297.928202] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18297.936468] [] kthread+0xcf/0xe0 [18297.942876] [] ? kthread_create_on_node+0x140/0x140 [18297.951102] [] ret_from_fork+0x58/0x90 [18297.958062] [] ? kthread_create_on_node+0x140/0x140 [18297.966267] arc_prune S ffff88081e694d00 0 3295 2 0x00000080 [18297.975066] ffff88082c673e08 0000000000000046 ffff88082d15bec0 ffff88082c673fd8 [18297.984265] ffff88082c673fd8 ffff88082c673fd8 ffff88082d15bec0 ffff8800b16989c0 [18297.993442] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18298.002596] Call Trace: [18298.006145] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18298.014238] [] schedule+0x29/0x70 [18298.020581] [] taskq_thread+0x3fa/0x440 [spl] [18298.028053] [] ? wake_up_state+0x20/0x20 [18298.035032] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18298.043052] [] kthread+0xcf/0xe0 [18298.049222] [] ? kthread_create_on_node+0x140/0x140 [18298.057220] [] ret_from_fork+0x58/0x90 [18298.063937] [] ? kthread_create_on_node+0x140/0x140 [18298.071919] arc_prune S ffff88081e694d00 0 3296 2 0x00000080 [18298.080530] ffff88082c5cbe08 0000000000000046 ffff88082d15ce70 ffff88082c5cbfd8 [18298.089562] ffff88082c5cbfd8 ffff88082c5cbfd8 ffff88082d15ce70 ffff8800b1698a20 [18298.098581] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18298.107595] Call Trace: [18298.111027] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18298.119036] [] schedule+0x29/0x70 [18298.125288] [] taskq_thread+0x3fa/0x440 [spl] [18298.132697] [] ? wake_up_state+0x20/0x20 [18298.139631] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18298.147627] [] kthread+0xcf/0xe0 [18298.153762] [] ? kthread_create_on_node+0x140/0x140 [18298.161751] [] ret_from_fork+0x58/0x90 [18298.168478] [] ? kthread_create_on_node+0x140/0x140 [18298.176463] arc_prune S ffff88081e694d00 0 3297 2 0x00000080 [18298.185077] ffff880829f87e08 0000000000000046 ffff88082d15de20 ffff880829f87fd8 [18298.194092] ffff880829f87fd8 ffff880829f87fd8 ffff88082d15de20 ffff8800b1698a80 [18298.203111] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18298.212144] Call Trace: [18298.215576] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18298.223586] [] schedule+0x29/0x70 [18298.229833] [] taskq_thread+0x3fa/0x440 [spl] [18298.237249] [] ? wake_up_state+0x20/0x20 [18298.244177] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18298.252156] [] kthread+0xcf/0xe0 [18298.258280] [] ? kthread_create_on_node+0x140/0x140 [18298.266243] [] ret_from_fork+0x58/0x90 [18298.272945] [] ? kthread_create_on_node+0x140/0x140 [18298.280905] arc_prune S ffff88081e694d00 0 3298 2 0x00000080 [18298.289494] ffff88082d5f7e08 0000000000000046 ffff88082d15edd0 ffff88082d5f7fd8 [18298.298495] ffff88082d5f7fd8 ffff88082d5f7fd8 ffff88082d15edd0 ffff8800b1698ae0 [18298.307505] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18298.316507] Call Trace: [18298.319923] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18298.327890] [] schedule+0x29/0x70 [18298.334116] [] taskq_thread+0x3fa/0x440 [spl] [18298.341510] [] ? wake_up_state+0x20/0x20 [18298.348420] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18298.356397] [] kthread+0xcf/0xe0 [18298.362512] [] ? kthread_create_on_node+0x140/0x140 [18298.370481] [] ret_from_fork+0x58/0x90 [18298.377176] [] ? kthread_create_on_node+0x140/0x140 [18298.385125] arc_prune S ffff88081e694d00 0 3299 2 0x00000080 [18298.393695] ffff880829a57e08 0000000000000046 ffff8808262c0000 ffff880829a57fd8 [18298.402698] ffff880829a57fd8 ffff880829a57fd8 ffff8808262c0000 ffff8800b1698b40 [18298.411694] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18298.420684] Call Trace: [18298.424093] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18298.432076] [] schedule+0x29/0x70 [18298.438296] [] taskq_thread+0x3fa/0x440 [spl] [18298.445680] [] ? wake_up_state+0x20/0x20 [18298.452594] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18298.460558] [] kthread+0xcf/0xe0 [18298.466659] [] ? kthread_create_on_node+0x140/0x140 [18298.474616] [] ret_from_fork+0x58/0x90 [18298.481315] [] ? kthread_create_on_node+0x140/0x140 [18298.489280] arc_prune S ffff88081e694d00 0 3300 2 0x00000080 [18298.497846] ffff880829befe08 0000000000000046 ffff8808262c0fb0 ffff880829beffd8 [18298.506837] ffff880829beffd8 ffff880829beffd8 ffff8808262c0fb0 ffff8800b1698ba0 [18298.515824] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18298.524803] Call Trace: [18298.528215] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18298.536195] [] schedule+0x29/0x70 [18298.542421] [] taskq_thread+0x3fa/0x440 [spl] [18298.549801] [] ? wake_up_state+0x20/0x20 [18298.556694] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18298.564655] [] kthread+0xcf/0xe0 [18298.570767] [] ? kthread_create_on_node+0x140/0x140 [18298.578723] [] ret_from_fork+0x58/0x90 [18298.585427] [] ? kthread_create_on_node+0x140/0x140 [18298.593383] arc_prune S ffff88081e694d00 0 3301 2 0x00000080 [18298.601968] ffff88082ca67e08 0000000000000046 ffff8808262c1f60 ffff88082ca67fd8 [18298.610975] ffff88082ca67fd8 ffff88082ca67fd8 ffff8808262c1f60 ffff8800b1698c00 [18298.619985] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18298.628962] Call Trace: [18298.632376] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18298.640342] [] schedule+0x29/0x70 [18298.646573] [] taskq_thread+0x3fa/0x440 [spl] [18298.653965] [] ? wake_up_state+0x20/0x20 [18298.660856] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18298.668817] [] kthread+0xcf/0xe0 [18298.674935] [] ? kthread_create_on_node+0x140/0x140 [18298.682885] [] ret_from_fork+0x58/0x90 [18298.689581] [] ? kthread_create_on_node+0x140/0x140 [18298.697556] arc_prune S ffff88081e694d00 0 3302 2 0x00000080 [18298.706146] ffff880826e33e08 0000000000000046 ffff8808262c2f10 ffff880826e33fd8 [18298.715149] ffff880826e33fd8 ffff880826e33fd8 ffff8808262c2f10 ffff8800b1698c60 [18298.724137] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18298.733134] Call Trace: [18298.736536] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18298.744507] [] schedule+0x29/0x70 [18298.750718] [] taskq_thread+0x3fa/0x440 [spl] [18298.758112] [] ? wake_up_state+0x20/0x20 [18298.765011] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18298.772973] [] kthread+0xcf/0xe0 [18298.779092] [] ? kthread_create_on_node+0x140/0x140 [18298.787046] [] ret_from_fork+0x58/0x90 [18298.793738] [] ? kthread_create_on_node+0x140/0x140 [18298.801695] arc_prune S ffff88081e694d00 0 3303 2 0x00000080 [18298.810288] ffff88082764be08 0000000000000046 ffff8808262c3ec0 ffff88082764bfd8 [18298.819292] ffff88082764bfd8 ffff88082764bfd8 ffff8808262c3ec0 ffff8800b1698cc0 [18298.828284] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18298.837268] Call Trace: [18298.840682] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18298.848660] [] schedule+0x29/0x70 [18298.854876] [] taskq_thread+0x3fa/0x440 [spl] [18298.862268] [] ? wake_up_state+0x20/0x20 [18298.869167] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18298.877128] [] kthread+0xcf/0xe0 [18298.883242] [] ? kthread_create_on_node+0x140/0x140 [18298.891210] [] ret_from_fork+0x58/0x90 [18298.897905] [] ? kthread_create_on_node+0x140/0x140 [18298.905854] arc_prune S ffff88081e694d00 0 3304 2 0x00000080 [18298.914441] ffff88083f8f7e08 0000000000000046 ffff8808262c4e70 ffff88083f8f7fd8 [18298.923452] ffff88083f8f7fd8 ffff88083f8f7fd8 ffff8808262c4e70 ffff8800b1698d20 [18298.932465] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18298.941471] Call Trace: [18298.944886] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18298.952857] [] schedule+0x29/0x70 [18298.959070] [] taskq_thread+0x3fa/0x440 [spl] [18298.966463] [] ? wake_up_state+0x20/0x20 [18298.973362] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18298.981337] [] kthread+0xcf/0xe0 [18298.987457] [] ? kthread_create_on_node+0x140/0x140 [18298.995421] [] ret_from_fork+0x58/0x90 [18299.002117] [] ? kthread_create_on_node+0x140/0x140 [18299.010070] arc_prune S ffff88081e694d00 0 3305 2 0x00000080 [18299.018632] ffff88082530fe08 0000000000000046 ffff8808262c5e20 ffff88082530ffd8 [18299.027630] ffff88082530ffd8 ffff88082530ffd8 ffff8808262c5e20 ffff8800b1698d80 [18299.036628] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18299.045623] Call Trace: [18299.049036] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18299.057012] [] schedule+0x29/0x70 [18299.063232] [] taskq_thread+0x3fa/0x440 [spl] [18299.070610] [] ? wake_up_state+0x20/0x20 [18299.077527] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18299.085505] [] kthread+0xcf/0xe0 [18299.091617] [] ? kthread_create_on_node+0x140/0x140 [18299.099580] [] ret_from_fork+0x58/0x90 [18299.106282] [] ? kthread_create_on_node+0x140/0x140 [18299.114243] arc_prune S ffff88081e694d00 0 3306 2 0x00000080 [18299.122812] ffff880829ec7e08 0000000000000046 ffff8808262c6dd0 ffff880829ec7fd8 [18299.131816] ffff880829ec7fd8 ffff880829ec7fd8 ffff8808262c6dd0 ffff8800b1698de0 [18299.140813] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18299.149805] Call Trace: [18299.153225] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18299.161193] [] schedule+0x29/0x70 [18299.167420] [] taskq_thread+0x3fa/0x440 [spl] [18299.174796] [] ? wake_up_state+0x20/0x20 [18299.181696] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18299.189658] [] kthread+0xcf/0xe0 [18299.195766] [] ? kthread_create_on_node+0x140/0x140 [18299.203717] [] ret_from_fork+0x58/0x90 [18299.210423] [] ? kthread_create_on_node+0x140/0x140 [18299.218395] arc_prune S ffff88081e694d00 0 3307 2 0x00000080 [18299.226971] ffff88082531fe08 0000000000000046 ffff88083f6f0000 ffff88082531ffd8 [18299.235973] ffff88082531ffd8 ffff88082531ffd8 ffff88083f6f0000 ffff8800b1698e40 [18299.244963] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18299.253959] Call Trace: [18299.257356] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18299.265335] [] schedule+0x29/0x70 [18299.271561] [] taskq_thread+0x3fa/0x440 [spl] [18299.278944] [] ? wake_up_state+0x20/0x20 [18299.285845] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18299.293806] [] kthread+0xcf/0xe0 [18299.299915] [] ? kthread_create_on_node+0x140/0x140 [18299.307871] [] ret_from_fork+0x58/0x90 [18299.314579] [] ? kthread_create_on_node+0x140/0x140 [18299.322557] arc_prune S ffff88081e694d00 0 3308 2 0x00000080 [18299.331137] ffff880825303e08 0000000000000046 ffff88083f6f0fb0 ffff880825303fd8 [18299.340145] ffff880825303fd8 ffff880825303fd8 ffff88083f6f0fb0 ffff8800b1698ea0 [18299.349147] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18299.358132] Call Trace: [18299.361557] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18299.369524] [] schedule+0x29/0x70 [18299.375738] [] taskq_thread+0x3fa/0x440 [spl] [18299.383120] [] ? wake_up_state+0x20/0x20 [18299.390022] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18299.397988] [] kthread+0xcf/0xe0 [18299.404099] [] ? kthread_create_on_node+0x140/0x140 [18299.412053] [] ret_from_fork+0x58/0x90 [18299.418747] [] ? kthread_create_on_node+0x140/0x140 [18299.426705] arc_prune S ffff88081e694d00 0 3309 2 0x00000080 [18299.435293] ffff880825687e08 0000000000000046 ffff88083f6f1f60 ffff880825687fd8 [18299.444281] ffff880825687fd8 ffff880825687fd8 ffff88083f6f1f60 ffff8800b1698f00 [18299.453273] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18299.462273] Call Trace: [18299.465676] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18299.473644] [] schedule+0x29/0x70 [18299.479857] [] taskq_thread+0x3fa/0x440 [spl] [18299.487249] [] ? wake_up_state+0x20/0x20 [18299.494143] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18299.502104] [] kthread+0xcf/0xe0 [18299.508213] [] ? kthread_create_on_node+0x140/0x140 [18299.516175] [] ret_from_fork+0x58/0x90 [18299.522861] [] ? kthread_create_on_node+0x140/0x140 [18299.530818] arc_prune S ffff88081e694d00 0 3310 2 0x00000080 [18299.539391] ffff88082e17be08 0000000000000046 ffff88083f6f2f10 ffff88082e17bfd8 [18299.548402] ffff88082e17bfd8 ffff88082e17bfd8 ffff88083f6f2f10 ffff8800b1698f60 [18299.557421] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18299.566413] Call Trace: [18299.569815] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18299.577779] [] schedule+0x29/0x70 [18299.583996] [] taskq_thread+0x3fa/0x440 [spl] [18299.591385] [] ? wake_up_state+0x20/0x20 [18299.598287] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18299.606254] [] kthread+0xcf/0xe0 [18299.612372] [] ? kthread_create_on_node+0x140/0x140 [18299.620338] [] ret_from_fork+0x58/0x90 [18299.627033] [] ? kthread_create_on_node+0x140/0x140 [18299.634989] arc_prune S ffff88081e694d00 0 3311 2 0x00000080 [18299.643576] ffff880827b7be08 0000000000000046 ffff88083f6f3ec0 ffff880827b7bfd8 [18299.652579] ffff880827b7bfd8 ffff880827b7bfd8 ffff88083f6f3ec0 ffff8800b1699000 [18299.661577] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18299.670581] Call Trace: [18299.673989] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18299.681955] [] schedule+0x29/0x70 [18299.688172] [] taskq_thread+0x3fa/0x440 [spl] [18299.695570] [] ? wake_up_state+0x20/0x20 [18299.702471] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18299.710433] [] kthread+0xcf/0xe0 [18299.716558] [] ? kthread_create_on_node+0x140/0x140 [18299.724532] [] ret_from_fork+0x58/0x90 [18299.731226] [] ? kthread_create_on_node+0x140/0x140 [18299.739198] arc_prune S ffff88081e694d00 0 3312 2 0x00000080 [18299.747773] ffff88083feb7e08 0000000000000046 ffff88083f6f4e70 ffff88083feb7fd8 [18299.756767] ffff88083feb7fd8 ffff88083feb7fd8 ffff88083f6f4e70 ffff8800b1699060 [18299.765762] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18299.774750] Call Trace: [18299.778168] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18299.786149] [] schedule+0x29/0x70 [18299.792365] [] taskq_thread+0x3fa/0x440 [spl] [18299.799752] [] ? wake_up_state+0x20/0x20 [18299.806653] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18299.814629] [] kthread+0xcf/0xe0 [18299.820741] [] ? kthread_create_on_node+0x140/0x140 [18299.828700] [] ret_from_fork+0x58/0x90 [18299.835395] [] ? kthread_create_on_node+0x140/0x140 [18299.843357] arc_prune S ffff88081e694d00 0 3313 2 0x00000080 [18299.851917] ffff88082471be08 0000000000000046 ffff88083f6f5e20 ffff88082471bfd8 [18299.860907] ffff88082471bfd8 ffff88082471bfd8 ffff88083f6f5e20 ffff8800b16990c0 [18299.869903] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18299.878897] Call Trace: [18299.882319] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18299.890298] [] schedule+0x29/0x70 [18299.896523] [] taskq_thread+0x3fa/0x440 [spl] [18299.903901] [] ? wake_up_state+0x20/0x20 [18299.910802] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18299.918764] [] kthread+0xcf/0xe0 [18299.924875] [] ? kthread_create_on_node+0x140/0x140 [18299.932832] [] ret_from_fork+0x58/0x90 [18299.939523] [] ? kthread_create_on_node+0x140/0x140 [18299.947482] arc_prune S ffff88081e694d00 0 3314 2 0x00000080 [18299.956069] ffff880824713e08 0000000000000046 ffff88083f6f6dd0 ffff880824713fd8 [18299.965067] ffff880824713fd8 ffff880824713fd8 ffff88083f6f6dd0 ffff8800b1699120 [18299.974072] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18299.983058] Call Trace: [18299.986470] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18299.994439] [] schedule+0x29/0x70 [18300.000655] [] taskq_thread+0x3fa/0x440 [spl] [18300.008055] [] ? wake_up_state+0x20/0x20 [18300.014948] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18300.022904] [] kthread+0xcf/0xe0 [18300.029018] [] ? kthread_create_on_node+0x140/0x140 [18300.036968] [] ret_from_fork+0x58/0x90 [18300.043663] [] ? kthread_create_on_node+0x140/0x140 [18300.051629] arc_prune S ffff88081e694d00 0 3315 2 0x00000080 [18300.060218] ffff88082af5fe08 0000000000000046 ffff880824e70000 ffff88082af5ffd8 [18300.069227] ffff88082af5ffd8 ffff88082af5ffd8 ffff880824e70000 ffff8800b1699180 [18300.078212] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18300.087215] Call Trace: [18300.090628] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18300.098602] [] schedule+0x29/0x70 [18300.104818] [] taskq_thread+0x3fa/0x440 [spl] [18300.112208] [] ? wake_up_state+0x20/0x20 [18300.119116] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18300.127081] [] kthread+0xcf/0xe0 [18300.133204] [] ? kthread_create_on_node+0x140/0x140 [18300.141177] [] ret_from_fork+0x58/0x90 [18300.147872] [] ? kthread_create_on_node+0x140/0x140 [18300.155818] arc_prune S ffff88081e694d00 0 3316 2 0x00000080 [18300.164398] ffff88082562be08 0000000000000046 ffff880824e70fb0 ffff88082562bfd8 [18300.173404] ffff88082562bfd8 ffff88082562bfd8 ffff880824e70fb0 ffff8800b16991e0 [18300.182415] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18300.191438] Call Trace: [18300.194851] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18300.202830] [] schedule+0x29/0x70 [18300.209057] [] taskq_thread+0x3fa/0x440 [spl] [18300.216454] [] ? wake_up_state+0x20/0x20 [18300.223364] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18300.231330] [] kthread+0xcf/0xe0 [18300.237456] [] ? kthread_create_on_node+0x140/0x140 [18300.245413] [] ret_from_fork+0x58/0x90 [18300.252115] [] ? kthread_create_on_node+0x140/0x140 [18300.260074] arc_prune S ffff88081e694d00 0 3317 2 0x00000080 [18300.268646] ffff880823fd7e08 0000000000000046 ffff880824e71f60 ffff880823fd7fd8 [18300.277645] ffff880823fd7fd8 ffff880823fd7fd8 ffff880824e71f60 ffff8800b1699240 [18300.286642] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18300.295648] Call Trace: [18300.299063] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18300.307041] [] schedule+0x29/0x70 [18300.313277] [] taskq_thread+0x3fa/0x440 [spl] [18300.320666] [] ? wake_up_state+0x20/0x20 [18300.327576] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18300.335544] [] kthread+0xcf/0xe0 [18300.341653] [] ? kthread_create_on_node+0x140/0x140 [18300.349625] [] ret_from_fork+0x58/0x90 [18300.356328] [] ? kthread_create_on_node+0x140/0x140 [18300.364299] arc_prune S ffff88081e694d00 0 3318 2 0x00000080 [18300.372873] ffff880823fdbe08 0000000000000046 ffff880824e72f10 ffff880823fdbfd8 [18300.381862] ffff880823fdbfd8 ffff880823fdbfd8 ffff880824e72f10 ffff8800b16992a0 [18300.390857] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18300.399848] Call Trace: [18300.403269] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18300.411240] [] schedule+0x29/0x70 [18300.417472] [] taskq_thread+0x3fa/0x440 [spl] [18300.424858] [] ? wake_up_state+0x20/0x20 [18300.431757] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18300.439719] [] kthread+0xcf/0xe0 [18300.445829] [] ? kthread_create_on_node+0x140/0x140 [18300.453788] [] ret_from_fork+0x58/0x90 [18300.460482] [] ? kthread_create_on_node+0x140/0x140 [18300.468442] arc_prune S ffff88081e694d00 0 3319 2 0x00000080 [18300.477017] ffff880823fe7e08 0000000000000046 ffff880824e73ec0 ffff880823fe7fd8 [18300.486015] ffff880823fe7fd8 ffff880823fe7fd8 ffff880824e73ec0 ffff8800b1699300 [18300.495008] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18300.504002] Call Trace: [18300.507413] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18300.515380] [] schedule+0x29/0x70 [18300.521609] [] taskq_thread+0x3fa/0x440 [spl] [18300.528994] [] ? wake_up_state+0x20/0x20 [18300.535888] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18300.543842] [] kthread+0xcf/0xe0 [18300.549953] [] ? kthread_create_on_node+0x140/0x140 [18300.557911] [] ret_from_fork+0x58/0x90 [18300.564620] [] ? kthread_create_on_node+0x140/0x140 [18300.572590] arc_prune S ffff88081e694d00 0 3320 2 0x00000080 [18300.581172] ffff88082deffe08 0000000000000046 ffff880824e74e70 ffff88082defffd8 [18300.590170] ffff88082defffd8 ffff88082defffd8 ffff880824e74e70 ffff8800b1699360 [18300.599172] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18300.608163] Call Trace: [18300.611580] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18300.619567] [] schedule+0x29/0x70 [18300.625782] [] taskq_thread+0x3fa/0x440 [spl] [18300.633176] [] ? wake_up_state+0x20/0x20 [18300.640086] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18300.648045] [] kthread+0xcf/0xe0 [18300.654166] [] ? kthread_create_on_node+0x140/0x140 [18300.662123] [] ret_from_fork+0x58/0x90 [18300.668810] [] ? kthread_create_on_node+0x140/0x140 [18300.676767] arc_prune S ffff88081e694d00 0 3321 2 0x00000080 [18300.685352] ffff88082523be08 0000000000000046 ffff880824e75e20 ffff88082523bfd8 [18300.694347] ffff88082523bfd8 ffff88082523bfd8 ffff880824e75e20 ffff8800b16993c0 [18300.703344] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18300.712346] Call Trace: [18300.715752] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18300.723724] [] schedule+0x29/0x70 [18300.729936] [] taskq_thread+0x3fa/0x440 [spl] [18300.737332] [] ? wake_up_state+0x20/0x20 [18300.744233] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18300.752196] [] kthread+0xcf/0xe0 [18300.758305] [] ? kthread_create_on_node+0x140/0x140 [18300.766257] [] ret_from_fork+0x58/0x90 [18300.772951] [] ? kthread_create_on_node+0x140/0x140 [18300.780908] arc_prune S ffff88081e694d00 0 3322 2 0x00000080 [18300.789492] ffff88081ba43e08 0000000000000046 ffff880824e76dd0 ffff88081ba43fd8 [18300.798499] ffff88081ba43fd8 ffff88081ba43fd8 ffff880824e76dd0 ffff8800b1699420 [18300.807500] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18300.816513] Call Trace: [18300.819926] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18300.827899] [] schedule+0x29/0x70 [18300.834126] [] taskq_thread+0x3fa/0x440 [spl] [18300.841515] [] ? wake_up_state+0x20/0x20 [18300.848419] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18300.856390] [] kthread+0xcf/0xe0 [18300.862509] [] ? kthread_create_on_node+0x140/0x140 [18300.870474] [] ret_from_fork+0x58/0x90 [18300.877176] [] ? kthread_create_on_node+0x140/0x140 [18300.885135] arc_prune S ffff88081e694d00 0 3323 2 0x00000080 [18300.893701] ffff8808242f7e08 0000000000000046 ffff880824e10000 ffff8808242f7fd8 [18300.902700] ffff8808242f7fd8 ffff8808242f7fd8 ffff880824e10000 ffff8800b1699480 [18300.911691] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18300.920690] Call Trace: [18300.924106] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18300.932092] [] schedule+0x29/0x70 [18300.938310] [] taskq_thread+0x3fa/0x440 [spl] [18300.945686] [] ? wake_up_state+0x20/0x20 [18300.952589] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18300.960567] [] kthread+0xcf/0xe0 [18300.966678] [] ? kthread_create_on_node+0x140/0x140 [18300.974645] [] ret_from_fork+0x58/0x90 [18300.981350] [] ? kthread_create_on_node+0x140/0x140 [18300.989308] arc_prune S ffff88081e694d00 0 3324 2 0x00000080 [18300.997878] ffff8808272d3e08 0000000000000046 ffff880824e10fb0 ffff8808272d3fd8 [18301.006870] ffff8808272d3fd8 ffff8808272d3fd8 ffff880824e10fb0 ffff8800b16994e0 [18301.015866] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18301.024861] Call Trace: [18301.028283] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18301.036266] [] schedule+0x29/0x70 [18301.042491] [] taskq_thread+0x3fa/0x440 [spl] [18301.049871] [] ? wake_up_state+0x20/0x20 [18301.056771] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18301.064732] [] kthread+0xcf/0xe0 [18301.070843] [] ? kthread_create_on_node+0x140/0x140 [18301.078799] [] ret_from_fork+0x58/0x90 [18301.085501] [] ? kthread_create_on_node+0x140/0x140 [18301.093470] arc_prune S ffff88081e694d00 0 3325 2 0x00000080 [18301.102049] ffff88083f6d3e08 0000000000000046 ffff880824e11f60 ffff88083f6d3fd8 [18301.111040] ffff88083f6d3fd8 ffff88083f6d3fd8 ffff880824e11f60 ffff8800b1699540 [18301.120045] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18301.129026] Call Trace: [18301.132429] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18301.140407] [] schedule+0x29/0x70 [18301.146638] [] taskq_thread+0x3fa/0x440 [spl] [18301.154032] [] ? wake_up_state+0x20/0x20 [18301.160933] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18301.168894] [] kthread+0xcf/0xe0 [18301.175002] [] ? kthread_create_on_node+0x140/0x140 [18301.182964] [] ret_from_fork+0x58/0x90 [18301.189671] [] ? kthread_create_on_node+0x140/0x140 [18301.197643] arc_prune S ffff88081e694d00 0 3326 2 0x00000080 [18301.206230] ffff8808263d3e08 0000000000000046 ffff880824e12f10 ffff8808263d3fd8 [18301.215236] ffff8808263d3fd8 ffff8808263d3fd8 ffff880824e12f10 ffff8800b16995a0 [18301.224247] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18301.233251] Call Trace: [18301.236671] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18301.244664] [] schedule+0x29/0x70 [18301.250874] [] taskq_thread+0x3fa/0x440 [spl] [18301.258251] [] ? wake_up_state+0x20/0x20 [18301.265151] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18301.273111] [] kthread+0xcf/0xe0 [18301.279227] [] ? kthread_create_on_node+0x140/0x140 [18301.287189] [] ret_from_fork+0x58/0x90 [18301.293880] [] ? kthread_create_on_node+0x140/0x140 [18301.301831] arc_prune S ffff88081e694d00 0 3327 2 0x00000080 [18301.310412] ffff880826bebe08 0000000000000046 ffff880824e13ec0 ffff880826bebfd8 [18301.319413] ffff880826bebfd8 ffff880826bebfd8 ffff880824e13ec0 ffff8800b1699600 [18301.328401] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18301.337395] Call Trace: [18301.340800] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18301.348763] [] schedule+0x29/0x70 [18301.354974] [] taskq_thread+0x3fa/0x440 [spl] [18301.362352] [] ? wake_up_state+0x20/0x20 [18301.369259] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18301.377223] [] kthread+0xcf/0xe0 [18301.383329] [] ? kthread_create_on_node+0x140/0x140 [18301.391287] [] ret_from_fork+0x58/0x90 [18301.397980] [] ? kthread_create_on_node+0x140/0x140 [18301.405929] arc_prune S ffff88081e694d00 0 3328 2 0x00000080 [18301.414514] ffff88081c357e08 0000000000000046 ffff880824e14e70 ffff88081c357fd8 [18301.423515] ffff88081c357fd8 ffff88081c357fd8 ffff880824e14e70 ffff8800b1699660 [18301.432511] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18301.441517] Call Trace: [18301.444928] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18301.452891] [] schedule+0x29/0x70 [18301.459103] [] taskq_thread+0x3fa/0x440 [spl] [18301.466496] [] ? wake_up_state+0x20/0x20 [18301.473407] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18301.481368] [] kthread+0xcf/0xe0 [18301.487476] [] ? kthread_create_on_node+0x140/0x140 [18301.495444] [] ret_from_fork+0x58/0x90 [18301.502138] [] ? kthread_create_on_node+0x140/0x140 [18301.510104] arc_prune S ffff88081e694d00 0 3329 2 0x00000080 [18301.518680] ffff88082e9fbe08 0000000000000046 ffff880824e15e20 ffff88082e9fbfd8 [18301.527686] ffff88082e9fbfd8 ffff88082e9fbfd8 ffff880824e15e20 ffff8800b16996c0 [18301.536692] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18301.545688] Call Trace: [18301.549101] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18301.557074] [] schedule+0x29/0x70 [18301.563306] [] taskq_thread+0x3fa/0x440 [spl] [18301.570691] [] ? wake_up_state+0x20/0x20 [18301.577609] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18301.585585] [] kthread+0xcf/0xe0 [18301.591709] [] ? kthread_create_on_node+0x140/0x140 [18301.599679] [] ret_from_fork+0x58/0x90 [18301.606372] [] ? kthread_create_on_node+0x140/0x140 [18301.614331] arc_prune S ffff88081e694d00 0 3330 2 0x00000080 [18301.622902] ffff88083fab7e08 0000000000000046 ffff880824e16dd0 ffff88083fab7fd8 [18301.631901] ffff88083fab7fd8 ffff88083fab7fd8 ffff880824e16dd0 ffff8800b1699720 [18301.640885] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18301.649881] Call Trace: [18301.653293] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18301.661266] [] schedule+0x29/0x70 [18301.667491] [] taskq_thread+0x3fa/0x440 [spl] [18301.674876] [] ? wake_up_state+0x20/0x20 [18301.681786] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18301.689753] [] kthread+0xcf/0xe0 [18301.695857] [] ? kthread_create_on_node+0x140/0x140 [18301.703805] [] ret_from_fork+0x58/0x90 [18301.710508] [] ? kthread_create_on_node+0x140/0x140 [18301.718474] arc_prune S ffff88081e694d00 0 3331 2 0x00000080 [18301.727058] ffff88082d623e08 0000000000000046 ffff880829058000 ffff88082d623fd8 [18301.736063] ffff88082d623fd8 ffff88082d623fd8 ffff880829058000 ffff8800b1699780 [18301.745063] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18301.754050] Call Trace: [18301.757462] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18301.765436] [] schedule+0x29/0x70 [18301.771656] [] taskq_thread+0x3fa/0x440 [spl] [18301.779041] [] ? wake_up_state+0x20/0x20 [18301.785941] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18301.793901] [] kthread+0xcf/0xe0 [18301.800011] [] ? kthread_create_on_node+0x140/0x140 [18301.807967] [] ret_from_fork+0x58/0x90 [18301.814659] [] ? kthread_create_on_node+0x140/0x140 [18301.822620] arc_prune S ffff88081e694d00 0 3332 2 0x00000080 [18301.831200] ffff88083fb63e08 0000000000000046 ffff880829058fb0 ffff88083fb63fd8 [18301.840184] ffff88083fb63fd8 ffff88083fb63fd8 ffff880829058fb0 ffff8800b16997e0 [18301.849185] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18301.858170] Call Trace: [18301.861583] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18301.869563] [] schedule+0x29/0x70 [18301.875777] [] taskq_thread+0x3fa/0x440 [spl] [18301.883168] [] ? wake_up_state+0x20/0x20 [18301.890075] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18301.898034] [] kthread+0xcf/0xe0 [18301.904144] [] ? kthread_create_on_node+0x140/0x140 [18301.912107] [] ret_from_fork+0x58/0x90 [18301.918790] [] ? kthread_create_on_node+0x140/0x140 [18301.926760] arc_prune S ffff88081e694d00 0 3333 2 0x00000080 [18301.935348] ffff88082a5c3e08 0000000000000046 ffff880829059f60 ffff88082a5c3fd8 [18301.944354] ffff88082a5c3fd8 ffff88082a5c3fd8 ffff880829059f60 ffff8800b1699840 [18301.953370] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18301.962367] Call Trace: [18301.965771] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18301.973739] [] schedule+0x29/0x70 [18301.979953] [] taskq_thread+0x3fa/0x440 [spl] [18301.987354] [] ? wake_up_state+0x20/0x20 [18301.994254] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18302.002215] [] kthread+0xcf/0xe0 [18302.008324] [] ? kthread_create_on_node+0x140/0x140 [18302.016279] [] ret_from_fork+0x58/0x90 [18302.022970] [] ? kthread_create_on_node+0x140/0x140 [18302.030924] arc_prune S ffff88081e694d00 0 3334 2 0x00000080 [18302.039501] ffff88082c367e08 0000000000000046 ffff88082905af10 ffff88082c367fd8 [18302.048498] ffff88082c367fd8 ffff88082c367fd8 ffff88082905af10 ffff8800b16998a0 [18302.057504] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18302.066509] Call Trace: [18302.069918] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18302.077891] [] schedule+0x29/0x70 [18302.084120] [] taskq_thread+0x3fa/0x440 [spl] [18302.091505] [] ? wake_up_state+0x20/0x20 [18302.098417] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18302.106380] [] kthread+0xcf/0xe0 [18302.112494] [] ? kthread_create_on_node+0x140/0x140 [18302.120461] [] ret_from_fork+0x58/0x90 [18302.127169] [] ? kthread_create_on_node+0x140/0x140 [18302.135130] arc_prune S ffff88081e694d00 0 3335 2 0x00000080 [18302.143711] ffff880824ecbe08 0000000000000046 ffff88082905bec0 ffff880824ecbfd8 [18302.152707] ffff880824ecbfd8 ffff880824ecbfd8 ffff88082905bec0 ffff8800b1699900 [18302.161696] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18302.170696] Call Trace: [18302.174113] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18302.182093] [] schedule+0x29/0x70 [18302.188325] [] taskq_thread+0x3fa/0x440 [spl] [18302.195711] [] ? wake_up_state+0x20/0x20 [18302.202617] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18302.210579] [] kthread+0xcf/0xe0 [18302.216695] [] ? kthread_create_on_node+0x140/0x140 [18302.224660] [] ret_from_fork+0x58/0x90 [18302.231361] [] ? kthread_create_on_node+0x140/0x140 [18302.239328] arc_prune S ffff88081e694d00 0 3336 2 0x00000080 [18302.247896] ffff88082c217e08 0000000000000046 ffff88082905ce70 ffff88082c217fd8 [18302.256890] ffff88082c217fd8 ffff88082c217fd8 ffff88082905ce70 ffff8800b1699960 [18302.265886] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18302.274877] Call Trace: [18302.278288] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18302.286262] [] schedule+0x29/0x70 [18302.292485] [] taskq_thread+0x3fa/0x440 [spl] [18302.299871] [] ? wake_up_state+0x20/0x20 [18302.306779] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18302.314743] [] kthread+0xcf/0xe0 [18302.320857] [] ? kthread_create_on_node+0x140/0x140 [18302.328810] [] ret_from_fork+0x58/0x90 [18302.335506] [] ? kthread_create_on_node+0x140/0x140 [18302.343471] arc_prune S ffff88081e694d00 0 3337 2 0x00000080 [18302.352031] ffff88083f7c7e08 0000000000000046 ffff88082905de20 ffff88083f7c7fd8 [18302.361010] ffff88083f7c7fd8 ffff88083f7c7fd8 ffff88082905de20 ffff8800b16999c0 [18302.369987] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18302.378982] Call Trace: [18302.382404] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18302.390388] [] schedule+0x29/0x70 [18302.396620] [] taskq_thread+0x3fa/0x440 [spl] [18302.404003] [] ? wake_up_state+0x20/0x20 [18302.410907] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18302.418869] [] kthread+0xcf/0xe0 [18302.424981] [] ? kthread_create_on_node+0x140/0x140 [18302.432931] [] ret_from_fork+0x58/0x90 [18302.439634] [] ? kthread_create_on_node+0x140/0x140 [18302.447595] arc_prune S ffff88081e694d00 0 3338 2 0x00000080 [18302.456174] ffff88083f7a7e08 0000000000000046 ffff88082905edd0 ffff88083f7a7fd8 [18302.465186] ffff88083f7a7fd8 ffff88083f7a7fd8 ffff88082905edd0 ffff8800b1699a20 [18302.474188] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18302.483181] Call Trace: [18302.486581] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18302.494554] [] schedule+0x29/0x70 [18302.500763] [] taskq_thread+0x3fa/0x440 [spl] [18302.508148] [] ? wake_up_state+0x20/0x20 [18302.515041] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18302.522999] [] kthread+0xcf/0xe0 [18302.529125] [] ? kthread_create_on_node+0x140/0x140 [18302.537090] [] ret_from_fork+0x58/0x90 [18302.543784] [] ? kthread_create_on_node+0x140/0x140 [18302.551745] arc_prune S ffff88081e694d00 0 3339 2 0x00000080 [18302.560330] ffff880824fbbe08 0000000000000046 ffff88082d2c8000 ffff880824fbbfd8 [18302.569324] ffff880824fbbfd8 ffff880824fbbfd8 ffff88082d2c8000 ffff8800b1699a80 [18302.578333] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18302.587345] Call Trace: [18302.590747] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18302.598726] [] schedule+0x29/0x70 [18302.604943] [] taskq_thread+0x3fa/0x440 [spl] [18302.612337] [] ? wake_up_state+0x20/0x20 [18302.619249] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18302.627214] [] kthread+0xcf/0xe0 [18302.633335] [] ? kthread_create_on_node+0x140/0x140 [18302.641299] [] ret_from_fork+0x58/0x90 [18302.647994] [] ? kthread_create_on_node+0x140/0x140 [18302.655940] arc_prune S ffff88081e694d00 0 3340 2 0x00000080 [18302.664535] ffff880826f47e08 0000000000000046 ffff88082d2c8fb0 ffff880826f47fd8 [18302.673542] ffff880826f47fd8 ffff880826f47fd8 ffff88082d2c8fb0 ffff8800b1699ae0 [18302.682552] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18302.691564] Call Trace: [18302.694977] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18302.702947] [] schedule+0x29/0x70 [18302.709165] [] taskq_thread+0x3fa/0x440 [spl] [18302.716561] [] ? wake_up_state+0x20/0x20 [18302.723471] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18302.731443] [] kthread+0xcf/0xe0 [18302.737557] [] ? kthread_create_on_node+0x140/0x140 [18302.745525] [] ret_from_fork+0x58/0x90 [18302.752220] [] ? kthread_create_on_node+0x140/0x140 [18302.760178] arc_prune S ffff88081e694d00 0 3341 2 0x00000080 [18302.768750] ffff880824e4be08 0000000000000046 ffff88082d2c9f60 ffff880824e4bfd8 [18302.777744] ffff880824e4bfd8 ffff880824e4bfd8 ffff88082d2c9f60 ffff8800b1699b40 [18302.786745] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18302.795743] Call Trace: [18302.799156] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18302.807140] [] schedule+0x29/0x70 [18302.813358] [] taskq_thread+0x3fa/0x440 [spl] [18302.820752] [] ? wake_up_state+0x20/0x20 [18302.827662] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18302.835638] [] kthread+0xcf/0xe0 [18302.841747] [] ? kthread_create_on_node+0x140/0x140 [18302.849713] [] ret_from_fork+0x58/0x90 [18302.856402] [] ? kthread_create_on_node+0x140/0x140 [18302.864361] arc_prune S ffff88081e694d00 0 3342 2 0x00000080 [18302.872941] ffff880824e47e08 0000000000000046 ffff88082d2caf10 ffff880824e47fd8 [18302.881944] ffff880824e47fd8 ffff880824e47fd8 ffff88082d2caf10 ffff8800b1699ba0 [18302.890944] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18302.899944] Call Trace: [18302.903350] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18302.911330] [] schedule+0x29/0x70 [18302.917554] [] taskq_thread+0x3fa/0x440 [spl] [18302.924938] [] ? wake_up_state+0x20/0x20 [18302.931836] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18302.939796] [] kthread+0xcf/0xe0 [18302.945909] [] ? kthread_create_on_node+0x140/0x140 [18302.953871] [] ret_from_fork+0x58/0x90 [18302.960564] [] ? kthread_create_on_node+0x140/0x140 [18302.968540] arc_prune S ffff88081e694d00 0 3343 2 0x00000080 [18302.977125] ffff880826b77e08 0000000000000046 ffff88082d2cbec0 ffff880826b77fd8 [18302.986130] ffff880826b77fd8 ffff880826b77fd8 ffff88082d2cbec0 ffff8800b1699c00 [18302.995140] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18303.004135] Call Trace: [18303.007543] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18303.015508] [] schedule+0x29/0x70 [18303.021732] [] taskq_thread+0x3fa/0x440 [spl] [18303.029118] [] ? wake_up_state+0x20/0x20 [18303.036012] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18303.043971] [] kthread+0xcf/0xe0 [18303.050082] [] ? kthread_create_on_node+0x140/0x140 [18303.058039] [] ret_from_fork+0x58/0x90 [18303.064737] [] ? kthread_create_on_node+0x140/0x140 [18303.072703] arc_prune S ffff88081e694d00 0 3344 2 0x00000080 [18303.081288] ffff88082d163e08 0000000000000046 ffff88082d2cce70 ffff88082d163fd8 [18303.090289] ffff88082d163fd8 ffff88082d163fd8 ffff88082d2cce70 ffff8800b1699c60 [18303.099285] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18303.108277] Call Trace: [18303.111691] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18303.119667] [] schedule+0x29/0x70 [18303.125881] [] taskq_thread+0x3fa/0x440 [spl] [18303.133267] [] ? wake_up_state+0x20/0x20 [18303.140168] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18303.148131] [] kthread+0xcf/0xe0 [18303.154251] [] ? kthread_create_on_node+0x140/0x140 [18303.162221] [] ret_from_fork+0x58/0x90 [18303.168908] [] ? kthread_create_on_node+0x140/0x140 [18303.176865] arc_prune S ffff88081e694d00 0 3345 2 0x00000080 [18303.185471] ffff88082e677e08 0000000000000046 ffff88082d2cde20 ffff88082e677fd8 [18303.194488] ffff88082e677fd8 ffff88082e677fd8 ffff88082d2cde20 ffff8800b1699cc0 [18303.203489] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18303.212505] Call Trace: [18303.215908] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18303.223876] [] schedule+0x29/0x70 [18303.230097] [] taskq_thread+0x3fa/0x440 [spl] [18303.237490] [] ? wake_up_state+0x20/0x20 [18303.244400] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18303.252369] [] kthread+0xcf/0xe0 [18303.258484] [] ? kthread_create_on_node+0x140/0x140 [18303.266448] [] ret_from_fork+0x58/0x90 [18303.273134] [] ? kthread_create_on_node+0x140/0x140 [18303.281100] arc_prune S ffff88081e694d00 0 3346 2 0x00000080 [18303.289687] ffff88082e63fe08 0000000000000046 ffff88082d2cedd0 ffff88082e63ffd8 [18303.298693] ffff88082e63ffd8 ffff88082e63ffd8 ffff88082d2cedd0 ffff8800b1699d20 [18303.307698] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18303.316699] Call Trace: [18303.320107] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18303.328076] [] schedule+0x29/0x70 [18303.334293] [] taskq_thread+0x3fa/0x440 [spl] [18303.341699] [] ? wake_up_state+0x20/0x20 [18303.348601] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18303.356562] [] kthread+0xcf/0xe0 [18303.362685] [] ? kthread_create_on_node+0x140/0x140 [18303.370656] [] ret_from_fork+0x58/0x90 [18303.377358] [] ? kthread_create_on_node+0x140/0x140 [18303.385314] arc_prune S ffff88081e694d00 0 3347 2 0x00000080 [18303.393883] ffff880827f97e08 0000000000000046 ffff880829450000 ffff880827f97fd8 [18303.402876] ffff880827f97fd8 ffff880827f97fd8 ffff880829450000 ffff8800b1699d80 [18303.411873] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18303.420865] Call Trace: [18303.424271] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18303.432239] [] schedule+0x29/0x70 [18303.438458] [] taskq_thread+0x3fa/0x440 [spl] [18303.445841] [] ? wake_up_state+0x20/0x20 [18303.452746] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18303.460722] [] kthread+0xcf/0xe0 [18303.466832] [] ? kthread_create_on_node+0x140/0x140 [18303.474788] [] ret_from_fork+0x58/0x90 [18303.481482] [] ? kthread_create_on_node+0x140/0x140 [18303.489447] arc_prune S ffff88081e694d00 0 3348 2 0x00000080 [18303.498003] ffff8808272a3e08 0000000000000046 ffff880829450fb0 ffff8808272a3fd8 [18303.506981] ffff8808272a3fd8 ffff8808272a3fd8 ffff880829450fb0 ffff8800b1699de0 [18303.515954] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18303.524956] Call Trace: [18303.528370] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18303.536350] [] schedule+0x29/0x70 [18303.542574] [] taskq_thread+0x3fa/0x440 [spl] [18303.549956] [] ? wake_up_state+0x20/0x20 [18303.556856] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18303.564815] [] kthread+0xcf/0xe0 [18303.570925] [] ? kthread_create_on_node+0x140/0x140 [18303.578883] [] ret_from_fork+0x58/0x90 [18303.585584] [] ? kthread_create_on_node+0x140/0x140 [18303.593548] arc_prune S ffff88081e694d00 0 3349 2 0x00000080 [18303.602140] ffff88082d14fe08 0000000000000046 ffff880829451f60 ffff88082d14ffd8 [18303.611130] ffff88082d14ffd8 ffff88082d14ffd8 ffff880829451f60 ffff8800b1699e40 [18303.620142] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18303.629135] Call Trace: [18303.632544] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18303.640514] [] schedule+0x29/0x70 [18303.646738] [] taskq_thread+0x3fa/0x440 [spl] [18303.654123] [] ? wake_up_state+0x20/0x20 [18303.661024] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18303.668978] [] kthread+0xcf/0xe0 [18303.675087] [] ? kthread_create_on_node+0x140/0x140 [18303.683042] [] ret_from_fork+0x58/0x90 [18303.689741] [] ? kthread_create_on_node+0x140/0x140 [18303.697703] arc_prune S ffff88081e694d00 0 3350 2 0x00000080 [18303.706289] ffff88082d157e08 0000000000000046 ffff880829452f10 ffff88082d157fd8 [18303.715294] ffff88082d157fd8 ffff88082d157fd8 ffff880829452f10 ffff8800b1699ea0 [18303.724298] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18303.733302] Call Trace: [18303.736721] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18303.744703] [] schedule+0x29/0x70 [18303.750906] [] taskq_thread+0x3fa/0x440 [spl] [18303.758295] [] ? wake_up_state+0x20/0x20 [18303.765196] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18303.773167] [] kthread+0xcf/0xe0 [18303.779276] [] ? kthread_create_on_node+0x140/0x140 [18303.787236] [] ret_from_fork+0x58/0x90 [18303.793927] [] ? kthread_create_on_node+0x140/0x140 [18303.801880] arc_prune S ffff88081e694d00 0 3351 2 0x00000080 [18303.810469] ffff88082e633e08 0000000000000046 ffff880829453ec0 ffff88082e633fd8 [18303.819472] ffff88082e633fd8 ffff88082e633fd8 ffff880829453ec0 ffff8800b1699f00 [18303.828484] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18303.837491] Call Trace: [18303.840904] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18303.848874] [] schedule+0x29/0x70 [18303.855083] [] taskq_thread+0x3fa/0x440 [spl] [18303.862476] [] ? wake_up_state+0x20/0x20 [18303.869377] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18303.877340] [] kthread+0xcf/0xe0 [18303.883450] [] ? kthread_create_on_node+0x140/0x140 [18303.891408] [] ret_from_fork+0x58/0x90 [18303.898101] [] ? kthread_create_on_node+0x140/0x140 [18303.906057] arc_prune S ffff88081e694d00 0 3352 2 0x00000080 [18303.914637] ffff88082d4cfe08 0000000000000046 ffff880829454e70 ffff88082d4cffd8 [18303.923646] ffff88082d4cffd8 ffff88082d4cffd8 ffff880829454e70 ffff8800b1699f60 [18303.932642] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18303.941638] Call Trace: [18303.945038] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18303.953008] [] schedule+0x29/0x70 [18303.959224] [] taskq_thread+0x3fa/0x440 [spl] [18303.966621] [] ? wake_up_state+0x20/0x20 [18303.973520] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18303.981474] [] kthread+0xcf/0xe0 [18303.987581] [] ? kthread_create_on_node+0x140/0x140 [18303.995545] [] ret_from_fork+0x58/0x90 [18304.002238] [] ? kthread_create_on_node+0x140/0x140 [18304.010196] arc_prune S ffff88081e694d00 0 3353 2 0x00000080 [18304.018760] ffff88081e77be08 0000000000000046 ffff880829455e20 ffff88081e77bfd8 [18304.027765] ffff88081e77bfd8 ffff88081e77bfd8 ffff880829455e20 ffff8800b169a000 [18304.036765] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18304.045764] Call Trace: [18304.049180] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18304.057150] [] schedule+0x29/0x70 [18304.063372] [] taskq_thread+0x3fa/0x440 [spl] [18304.070761] [] ? wake_up_state+0x20/0x20 [18304.077656] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18304.085615] [] kthread+0xcf/0xe0 [18304.091723] [] ? kthread_create_on_node+0x140/0x140 [18304.099684] [] ret_from_fork+0x58/0x90 [18304.106384] [] ? kthread_create_on_node+0x140/0x140 [18304.114350] arc_prune S ffff88081e694d00 0 3354 2 0x00000080 [18304.122918] ffff88081baefe08 0000000000000046 ffff880829456dd0 ffff88081baeffd8 [18304.131915] ffff88081baeffd8 ffff88081baeffd8 ffff880829456dd0 ffff8800b169a060 [18304.140898] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18304.149888] Call Trace: [18304.153302] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18304.161278] [] schedule+0x29/0x70 [18304.167500] [] taskq_thread+0x3fa/0x440 [spl] [18304.174882] [] ? wake_up_state+0x20/0x20 [18304.181770] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18304.189742] [] kthread+0xcf/0xe0 [18304.195848] [] ? kthread_create_on_node+0x140/0x140 [18304.203803] [] ret_from_fork+0x58/0x90 [18304.210501] [] ? kthread_create_on_node+0x140/0x140 [18304.218454] arc_prune S ffff88081e694d00 0 3355 2 0x00000080 [18304.227010] ffff88082abcbe08 0000000000000046 ffff880825ab8000 ffff88082abcbfd8 [18304.235998] ffff88082abcbfd8 ffff88082abcbfd8 ffff880825ab8000 ffff8800b169a0c0 [18304.244995] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18304.253983] Call Trace: [18304.257402] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18304.265375] [] schedule+0x29/0x70 [18304.271606] [] taskq_thread+0x3fa/0x440 [spl] [18304.278984] [] ? wake_up_state+0x20/0x20 [18304.285883] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18304.293843] [] kthread+0xcf/0xe0 [18304.299947] [] ? kthread_create_on_node+0x140/0x140 [18304.307903] [] ret_from_fork+0x58/0x90 [18304.314598] [] ? kthread_create_on_node+0x140/0x140 [18304.322562] arc_prune S ffff88081e694d00 0 3356 2 0x00000080 [18304.331144] ffff880827e5fe08 0000000000000046 ffff880825ab8fb0 ffff880827e5ffd8 [18304.340138] ffff880827e5ffd8 ffff880827e5ffd8 ffff880825ab8fb0 ffff8800b169a120 [18304.349126] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18304.358115] Call Trace: [18304.361528] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18304.369502] [] schedule+0x29/0x70 [18304.375717] [] taskq_thread+0x3fa/0x440 [spl] [18304.383100] [] ? wake_up_state+0x20/0x20 [18304.390001] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18304.397961] [] kthread+0xcf/0xe0 [18304.404072] [] ? kthread_create_on_node+0x140/0x140 [18304.412029] [] ret_from_fork+0x58/0x90 [18304.418731] [] ? kthread_create_on_node+0x140/0x140 [18304.426690] arc_prune S ffff88081e694d00 0 3357 2 0x00000080 [18304.435281] ffff880829b0be08 0000000000000046 ffff880825ab9f60 ffff880829b0bfd8 [18304.444281] ffff880829b0bfd8 ffff880829b0bfd8 ffff880825ab9f60 ffff8800b169a180 [18304.453276] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18304.462279] Call Trace: [18304.465706] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18304.473693] [] schedule+0x29/0x70 [18304.479913] [] taskq_thread+0x3fa/0x440 [spl] [18304.487307] [] ? wake_up_state+0x20/0x20 [18304.494207] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18304.502180] [] kthread+0xcf/0xe0 [18304.508297] [] ? kthread_create_on_node+0x140/0x140 [18304.516253] [] ret_from_fork+0x58/0x90 [18304.522938] [] ? kthread_create_on_node+0x140/0x140 [18304.530894] arc_prune S ffff88081e694d00 0 3358 2 0x00000080 [18304.539483] ffff880829a47e08 0000000000000046 ffff880825abaf10 ffff880829a47fd8 [18304.548478] ffff880829a47fd8 ffff880829a47fd8 ffff880825abaf10 ffff8800b169a1e0 [18304.557492] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18304.566502] Call Trace: [18304.569906] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18304.577879] [] schedule+0x29/0x70 [18304.584095] [] taskq_thread+0x3fa/0x440 [spl] [18304.591485] [] ? wake_up_state+0x20/0x20 [18304.598385] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18304.606344] [] kthread+0xcf/0xe0 [18304.612460] [] ? kthread_create_on_node+0x140/0x140 [18304.620426] [] ret_from_fork+0x58/0x90 [18304.627111] [] ? kthread_create_on_node+0x140/0x140 [18304.635061] arc_prune S ffff88081e694d00 0 3359 2 0x00000080 [18304.643633] ffff880829a4be08 0000000000000046 ffff880825abbec0 ffff880829a4bfd8 [18304.652636] ffff880829a4bfd8 ffff880829a4bfd8 ffff880825abbec0 ffff8800b169a240 [18304.661635] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18304.670642] Call Trace: [18304.674058] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18304.682029] [] schedule+0x29/0x70 [18304.688254] [] taskq_thread+0x3fa/0x440 [spl] [18304.695648] [] ? wake_up_state+0x20/0x20 [18304.702550] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18304.710511] [] kthread+0xcf/0xe0 [18304.716634] [] ? kthread_create_on_node+0x140/0x140 [18304.724605] [] ret_from_fork+0x58/0x90 [18304.731300] [] ? kthread_create_on_node+0x140/0x140 [18304.739257] arc_prune S ffff88081e694d00 0 3360 2 0x00000080 [18304.747827] ffff8808266cbe08 0000000000000046 ffff880825abce70 ffff8808266cbfd8 [18304.756822] ffff8808266cbfd8 ffff8808266cbfd8 ffff880825abce70 ffff8800b169a2a0 [18304.765819] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18304.774809] Call Trace: [18304.778220] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18304.786198] [] schedule+0x29/0x70 [18304.792420] [] taskq_thread+0x3fa/0x440 [spl] [18304.799819] [] ? wake_up_state+0x20/0x20 [18304.806734] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18304.814701] [] kthread+0xcf/0xe0 [18304.820812] [] ? kthread_create_on_node+0x140/0x140 [18304.828769] [] ret_from_fork+0x58/0x90 [18304.835463] [] ? kthread_create_on_node+0x140/0x140 [18304.843423] arc_prune S ffff88081e694d00 0 3361 2 0x00000080 [18304.851999] ffff880824e5be08 0000000000000046 ffff880825abde20 ffff880824e5bfd8 [18304.860998] ffff880824e5bfd8 ffff880824e5bfd8 ffff880825abde20 ffff8800b169a300 [18304.870002] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18304.878990] Call Trace: [18304.882404] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18304.890378] [] schedule+0x29/0x70 [18304.896604] [] taskq_thread+0x3fa/0x440 [spl] [18304.903990] [] ? wake_up_state+0x20/0x20 [18304.910894] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18304.918864] [] kthread+0xcf/0xe0 [18304.924975] [] ? finish_task_switch+0x53/0x180 [18304.932452] [] ? kthread_create_on_node+0x140/0x140 [18304.940416] [] ret_from_fork+0x58/0x90 [18304.947110] [] ? kthread_create_on_node+0x140/0x140 [18304.955061] arc_prune S ffff88081e694d00 0 3362 2 0x00000080 [18304.963640] ffff88083f8ffe08 0000000000000046 ffff880825abedd0 ffff88083f8fffd8 [18304.972647] ffff88083f8fffd8 ffff88083f8fffd8 ffff880825abedd0 ffff8800b169a360 [18304.981654] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18304.990649] Call Trace: [18304.994060] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18305.002021] [] schedule+0x29/0x70 [18305.008251] [] taskq_thread+0x3fa/0x440 [spl] [18305.015635] [] ? wake_up_state+0x20/0x20 [18305.022538] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18305.030504] [] kthread+0xcf/0xe0 [18305.036619] [] ? kthread_create_on_node+0x140/0x140 [18305.044578] [] ret_from_fork+0x58/0x90 [18305.051275] [] ? kthread_create_on_node+0x140/0x140 [18305.059233] arc_prune S ffff88081e694d00 0 3363 2 0x00000080 [18305.067800] ffff88083f89fe08 0000000000000046 ffff880829680000 ffff88083f89ffd8 [18305.076807] ffff88083f89ffd8 ffff88083f89ffd8 ffff880829680000 ffff8800b169a3c0 [18305.085802] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18305.094803] Call Trace: [18305.098199] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18305.106178] [] schedule+0x29/0x70 [18305.112398] [] taskq_thread+0x3fa/0x440 [spl] [18305.119788] [] ? wake_up_state+0x20/0x20 [18305.126696] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18305.134659] [] kthread+0xcf/0xe0 [18305.140782] [] ? kthread_create_on_node+0x140/0x140 [18305.148747] [] ret_from_fork+0x58/0x90 [18305.155440] [] ? kthread_create_on_node+0x140/0x140 [18305.163401] arc_prune S ffff88081e694d00 0 3364 2 0x00000080 [18305.171978] ffff88083f8e7e08 0000000000000046 ffff880829680fb0 ffff88083f8e7fd8 [18305.180974] ffff88083f8e7fd8 ffff88083f8e7fd8 ffff880829680fb0 ffff8800b169a420 [18305.189980] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18305.198976] Call Trace: [18305.202372] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18305.210332] [] schedule+0x29/0x70 [18305.216551] [] taskq_thread+0x3fa/0x440 [spl] [18305.223925] [] ? wake_up_state+0x20/0x20 [18305.230814] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18305.238775] [] kthread+0xcf/0xe0 [18305.244882] [] ? kthread_create_on_node+0x140/0x140 [18305.252836] [] ret_from_fork+0x58/0x90 [18305.259536] [] ? kthread_create_on_node+0x140/0x140 [18305.267502] arc_prune S ffff88081e694d00 0 3365 2 0x00000080 [18305.276067] ffff88082d683e08 0000000000000046 ffff880829681f60 ffff88082d683fd8 [18305.285053] ffff88082d683fd8 ffff88082d683fd8 ffff880829681f60 ffff8800b169a480 [18305.294046] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18305.303036] Call Trace: [18305.306453] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18305.314430] [] schedule+0x29/0x70 [18305.320659] [] taskq_thread+0x3fa/0x440 [spl] [18305.328035] [] ? wake_up_state+0x20/0x20 [18305.334932] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18305.342894] [] kthread+0xcf/0xe0 [18305.349003] [] ? kthread_create_on_node+0x140/0x140 [18305.356958] [] ret_from_fork+0x58/0x90 [18305.363648] [] ? kthread_create_on_node+0x140/0x140 [18305.371602] arc_prune S ffff88081e694d00 0 3366 2 0x00000080 [18305.380181] ffff880829433e08 0000000000000046 ffff880829682f10 ffff880829433fd8 [18305.389193] ffff880829433fd8 ffff880829433fd8 ffff880829682f10 ffff8800b169a4e0 [18305.398200] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18305.407201] Call Trace: [18305.410609] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18305.418580] [] schedule+0x29/0x70 [18305.424809] [] taskq_thread+0x3fa/0x440 [spl] [18305.432191] [] ? wake_up_state+0x20/0x20 [18305.439091] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18305.447052] [] kthread+0xcf/0xe0 [18305.453163] [] ? kthread_create_on_node+0x140/0x140 [18305.461110] [] ret_from_fork+0x58/0x90 [18305.467805] [] ? kthread_create_on_node+0x140/0x140 [18305.475765] arc_prune S ffff88081e694d00 0 3367 2 0x00000080 [18305.484346] ffff88082942fe08 0000000000000046 ffff880829683ec0 ffff88082942ffd8 [18305.493343] ffff88082942ffd8 ffff88082942ffd8 ffff880829683ec0 ffff8800b169a540 [18305.502329] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18305.511323] Call Trace: [18305.514732] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18305.522710] [] schedule+0x29/0x70 [18305.528927] [] taskq_thread+0x3fa/0x440 [spl] [18305.536320] [] ? wake_up_state+0x20/0x20 [18305.543217] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18305.551192] [] kthread+0xcf/0xe0 [18305.557311] [] ? kthread_create_on_node+0x140/0x140 [18305.565265] [] ret_from_fork+0x58/0x90 [18305.571959] [] ? kthread_create_on_node+0x140/0x140 [18305.579911] arc_prune S ffff88081e694d00 0 3368 2 0x00000080 [18305.588497] ffff88083f8d7e08 0000000000000046 ffff880829684e70 ffff88083f8d7fd8 [18305.597520] ffff88083f8d7fd8 ffff88083f8d7fd8 ffff880829684e70 ffff8800b169a5a0 [18305.606522] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18305.615521] Call Trace: [18305.618934] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18305.626905] [] schedule+0x29/0x70 [18305.633119] [] taskq_thread+0x3fa/0x440 [spl] [18305.640505] [] ? wake_up_state+0x20/0x20 [18305.647409] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18305.655371] [] kthread+0xcf/0xe0 [18305.661491] [] ? kthread_create_on_node+0x140/0x140 [18305.669457] [] ret_from_fork+0x58/0x90 [18305.676151] [] ? kthread_create_on_node+0x140/0x140 [18305.684108] arc_prune S ffff88081e694d00 0 3369 2 0x00000080 [18305.692684] ffff88083f8dbe08 0000000000000046 ffff880829685e20 ffff88083f8dbfd8 [18305.701697] ffff88083f8dbfd8 ffff88083f8dbfd8 ffff880829685e20 ffff8800b169a600 [18305.710702] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18305.719690] Call Trace: [18305.723099] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18305.731065] [] schedule+0x29/0x70 [18305.737289] [] taskq_thread+0x3fa/0x440 [spl] [18305.744679] [] ? wake_up_state+0x20/0x20 [18305.751579] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18305.759549] [] kthread+0xcf/0xe0 [18305.765659] [] ? kthread_create_on_node+0x140/0x140 [18305.773621] [] ret_from_fork+0x58/0x90 [18305.780329] [] ? kthread_create_on_node+0x140/0x140 [18305.788294] arc_prune S ffff88081e694d00 0 3370 2 0x00000080 [18305.796867] ffff88082421be08 0000000000000046 ffff880829686dd0 ffff88082421bfd8 [18305.805860] ffff88082421bfd8 ffff88082421bfd8 ffff880829686dd0 ffff8800b169a660 [18305.814858] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18305.823844] Call Trace: [18305.827249] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18305.835226] [] schedule+0x29/0x70 [18305.841450] [] taskq_thread+0x3fa/0x440 [spl] [18305.848833] [] ? wake_up_state+0x20/0x20 [18305.855736] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18305.863701] [] kthread+0xcf/0xe0 [18305.869816] [] ? kthread_create_on_node+0x140/0x140 [18305.877787] [] ret_from_fork+0x58/0x90 [18305.884485] [] ? kthread_create_on_node+0x140/0x140 [18305.892451] arc_prune S ffff88081e694d00 0 3371 2 0x00000080 [18305.901017] ffff880824217e08 0000000000000046 ffff88081bb48000 ffff880824217fd8 [18305.910023] ffff880824217fd8 ffff880824217fd8 ffff88081bb48000 ffff8800b169a6c0 [18305.919021] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18305.928028] Call Trace: [18305.931446] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18305.939417] [] schedule+0x29/0x70 [18305.945640] [] taskq_thread+0x3fa/0x440 [spl] [18305.953022] [] ? wake_up_state+0x20/0x20 [18305.959924] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18305.967884] [] kthread+0xcf/0xe0 [18305.973993] [] ? kthread_create_on_node+0x140/0x140 [18305.981943] [] ret_from_fork+0x58/0x90 [18305.988644] [] ? kthread_create_on_node+0x140/0x140 [18305.996603] arc_prune S ffff88081e694d00 0 3372 2 0x00000080 [18306.005179] ffff88082abd7e08 0000000000000046 ffff88081bb48fb0 ffff88082abd7fd8 [18306.014187] ffff88082abd7fd8 ffff88082abd7fd8 ffff88081bb48fb0 ffff8800b169a720 [18306.023194] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18306.032185] Call Trace: [18306.035600] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18306.043570] [] schedule+0x29/0x70 [18306.049795] [] taskq_thread+0x3fa/0x440 [spl] [18306.057180] [] ? wake_up_state+0x20/0x20 [18306.064081] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18306.072043] [] kthread+0xcf/0xe0 [18306.078153] [] ? kthread_create_on_node+0x140/0x140 [18306.086108] [] ret_from_fork+0x58/0x90 [18306.092806] [] ? kthread_create_on_node+0x140/0x140 [18306.100779] arc_prune S ffff88081e694d00 0 3373 2 0x00000080 [18306.109351] ffff88081e7b7e08 0000000000000046 ffff88081bb49f60 ffff88081e7b7fd8 [18306.118359] ffff88081e7b7fd8 ffff88081e7b7fd8 ffff88081bb49f60 ffff8800b169a780 [18306.127352] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18306.136338] Call Trace: [18306.139756] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18306.147743] [] schedule+0x29/0x70 [18306.153963] [] taskq_thread+0x3fa/0x440 [spl] [18306.161355] [] ? wake_up_state+0x20/0x20 [18306.168266] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18306.176227] [] kthread+0xcf/0xe0 [18306.182336] [] ? kthread_create_on_node+0x140/0x140 [18306.190294] [] ret_from_fork+0x58/0x90 [18306.196993] [] ? kthread_create_on_node+0x140/0x140 [18306.204950] arc_prune S ffff88081e694d00 0 3374 2 0x00000080 [18306.213544] ffff88082c32fe08 0000000000000046 ffff88081bb4af10 ffff88082c32ffd8 [18306.222552] ffff88082c32ffd8 ffff88082c32ffd8 ffff88081bb4af10 ffff8800b169a7e0 [18306.231563] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18306.240571] Call Trace: [18306.243975] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18306.251944] [] schedule+0x29/0x70 [18306.258160] [] taskq_thread+0x3fa/0x440 [spl] [18306.265560] [] ? wake_up_state+0x20/0x20 [18306.272449] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18306.280418] [] kthread+0xcf/0xe0 [18306.286537] [] ? kthread_create_on_node+0x140/0x140 [18306.294502] [] ret_from_fork+0x58/0x90 [18306.301198] [] ? kthread_create_on_node+0x140/0x140 [18306.309153] arc_prune S ffff88081e694d00 0 3375 2 0x00000080 [18306.317732] ffff88082d063e08 0000000000000046 ffff88081bb4bec0 ffff88082d063fd8 [18306.326751] ffff88082d063fd8 ffff88082d063fd8 ffff88081bb4bec0 ffff8800b169a840 [18306.335765] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18306.344775] Call Trace: [18306.348182] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18306.356146] [] schedule+0x29/0x70 [18306.362372] [] taskq_thread+0x3fa/0x440 [spl] [18306.369772] [] ? wake_up_state+0x20/0x20 [18306.376663] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18306.384632] [] kthread+0xcf/0xe0 [18306.390756] [] ? kthread_create_on_node+0x140/0x140 [18306.398723] [] ret_from_fork+0x58/0x90 [18306.405417] [] ? kthread_create_on_node+0x140/0x140 [18306.413374] arc_prune S ffff88081e694d00 0 3376 2 0x00000080 [18306.421948] ffff88082af23e08 0000000000000046 ffff88081bb4ce70 ffff88082af23fd8 [18306.430945] ffff88082af23fd8 ffff88082af23fd8 ffff88081bb4ce70 ffff8800b169a8a0 [18306.439928] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18306.448925] Call Trace: [18306.452343] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18306.460312] [] schedule+0x29/0x70 [18306.466542] [] taskq_thread+0x3fa/0x440 [spl] [18306.473926] [] ? wake_up_state+0x20/0x20 [18306.480841] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18306.488794] [] kthread+0xcf/0xe0 [18306.494912] [] ? kthread_create_on_node+0x140/0x140 [18306.502867] [] ret_from_fork+0x58/0x90 [18306.509561] [] ? kthread_create_on_node+0x140/0x140 [18306.517522] arc_prune S ffff88081e694d00 0 3377 2 0x00000080 [18306.526095] ffff880825763e08 0000000000000046 ffff88081bb4de20 ffff880825763fd8 [18306.535073] ffff880825763fd8 ffff880825763fd8 ffff88081bb4de20 ffff8800b169a900 [18306.544067] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18306.553066] Call Trace: [18306.556480] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18306.564450] [] schedule+0x29/0x70 [18306.570675] [] taskq_thread+0x3fa/0x440 [spl] [18306.578060] [] ? wake_up_state+0x20/0x20 [18306.584959] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18306.592921] [] kthread+0xcf/0xe0 [18306.599030] [] ? kthread_create_on_node+0x140/0x140 [18306.606984] [] ret_from_fork+0x58/0x90 [18306.613676] [] ? kthread_create_on_node+0x140/0x140 [18306.621637] arc_prune S ffff88081e694d00 0 3378 2 0x00000080 [18306.630227] ffff88082628fe08 0000000000000046 ffff88081bb4edd0 ffff88082628ffd8 [18306.639226] ffff88082628ffd8 ffff88082628ffd8 ffff88081bb4edd0 ffff8800b169a960 [18306.648215] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18306.657198] Call Trace: [18306.660605] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18306.668571] [] schedule+0x29/0x70 [18306.674801] [] taskq_thread+0x3fa/0x440 [spl] [18306.682178] [] ? wake_up_state+0x20/0x20 [18306.689073] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18306.697032] [] kthread+0xcf/0xe0 [18306.703142] [] ? kthread_create_on_node+0x140/0x140 [18306.711098] [] ret_from_fork+0x58/0x90 [18306.717807] [] ? kthread_create_on_node+0x140/0x140 [18306.725787] arc_prune S ffff88081e694d00 0 3379 2 0x00000080 [18306.734371] ffff88082944fe08 0000000000000046 ffff880826798000 ffff88082944ffd8 [18306.743359] ffff88082944ffd8 ffff88082944ffd8 ffff880826798000 ffff8800b169a9c0 [18306.752356] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18306.761356] Call Trace: [18306.764781] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18306.772767] [] schedule+0x29/0x70 [18306.778990] [] taskq_thread+0x3fa/0x440 [spl] [18306.786383] [] ? wake_up_state+0x20/0x20 [18306.793292] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18306.801264] [] kthread+0xcf/0xe0 [18306.807373] [] ? kthread_create_on_node+0x140/0x140 [18306.815332] [] ret_from_fork+0x58/0x90 [18306.822027] [] ? kthread_create_on_node+0x140/0x140 [18306.829988] arc_prune S ffff88081e694d00 0 3380 2 0x00000080 [18306.838577] ffff8808256ebe08 0000000000000046 ffff880826798fb0 ffff8808256ebfd8 [18306.847591] ffff8808256ebfd8 ffff8808256ebfd8 ffff880826798fb0 ffff8800b169aa20 [18306.856597] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18306.865599] Call Trace: [18306.868997] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18306.876965] [] schedule+0x29/0x70 [18306.883174] [] taskq_thread+0x3fa/0x440 [spl] [18306.890560] [] ? wake_up_state+0x20/0x20 [18306.897463] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18306.905433] [] kthread+0xcf/0xe0 [18306.911544] [] ? kthread_create_on_node+0x140/0x140 [18306.919506] [] ret_from_fork+0x58/0x90 [18306.926192] [] ? kthread_create_on_node+0x140/0x140 [18306.934134] arc_prune S ffff88081e694d00 0 3381 2 0x00000080 [18306.942703] ffff8808252a3e08 0000000000000046 ffff880826799f60 ffff8808252a3fd8 [18306.951695] ffff8808252a3fd8 ffff8808252a3fd8 ffff880826799f60 ffff8800b169aa80 [18306.960701] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18306.969700] Call Trace: [18306.973106] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18306.981077] [] schedule+0x29/0x70 [18306.987297] [] taskq_thread+0x3fa/0x440 [spl] [18306.994682] [] ? wake_up_state+0x20/0x20 [18307.001590] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18307.009561] [] kthread+0xcf/0xe0 [18307.015674] [] ? kthread_create_on_node+0x140/0x140 [18307.023634] [] ret_from_fork+0x58/0x90 [18307.030327] [] ? kthread_create_on_node+0x140/0x140 [18307.038286] arc_prune S ffff88081e694d00 0 3382 2 0x00000080 [18307.046874] ffff88082945be08 0000000000000046 ffff88082679af10 ffff88082945bfd8 [18307.055877] ffff88082945bfd8 ffff88082945bfd8 ffff88082679af10 ffff8800b169aae0 [18307.064870] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18307.073873] Call Trace: [18307.077292] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18307.085281] [] schedule+0x29/0x70 [18307.091498] [] taskq_thread+0x3fa/0x440 [spl] [18307.098889] [] ? wake_up_state+0x20/0x20 [18307.105792] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18307.113765] [] kthread+0xcf/0xe0 [18307.119887] [] ? kthread_create_on_node+0x140/0x140 [18307.127847] [] ret_from_fork+0x58/0x90 [18307.134554] [] ? kthread_create_on_node+0x140/0x140 [18307.142512] arc_prune S ffff88081e694d00 0 3383 2 0x00000080 [18307.151072] ffff8808276bfe08 0000000000000046 ffff88082679bec0 ffff8808276bffd8 [18307.160054] ffff8808276bffd8 ffff8808276bffd8 ffff88082679bec0 ffff8800b169ab40 [18307.169045] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18307.178041] Call Trace: [18307.181449] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18307.189423] [] schedule+0x29/0x70 [18307.195639] [] taskq_thread+0x3fa/0x440 [spl] [18307.203023] [] ? wake_up_state+0x20/0x20 [18307.209921] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18307.217888] [] kthread+0xcf/0xe0 [18307.224001] [] ? kthread_create_on_node+0x140/0x140 [18307.231961] [] ret_from_fork+0x58/0x90 [18307.238665] [] ? kthread_create_on_node+0x140/0x140 [18307.246632] arc_prune S ffff88081e694d00 0 3384 2 0x00000080 [18307.255199] ffff880823e23e08 0000000000000046 ffff88082679ce70 ffff880823e23fd8 [18307.264191] ffff880823e23fd8 ffff880823e23fd8 ffff88082679ce70 ffff8800b169aba0 [18307.273187] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18307.282160] Call Trace: [18307.285568] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18307.293539] [] schedule+0x29/0x70 [18307.299759] [] taskq_thread+0x3fa/0x440 [spl] [18307.307144] [] ? wake_up_state+0x20/0x20 [18307.314043] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18307.322001] [] kthread+0xcf/0xe0 [18307.328104] [] ? kthread_create_on_node+0x140/0x140 [18307.336058] [] ret_from_fork+0x58/0x90 [18307.342753] [] ? kthread_create_on_node+0x140/0x140 [18307.350710] arc_prune S ffff88081e694d00 0 3385 2 0x00000080 [18307.359296] ffff880823e1fe08 0000000000000046 ffff88082679de20 ffff880823e1ffd8 [18307.368307] ffff880823e1ffd8 ffff880823e1ffd8 ffff88082679de20 ffff8800b169ac00 [18307.377302] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18307.386299] Call Trace: [18307.389717] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18307.397696] [] schedule+0x29/0x70 [18307.403912] [] taskq_thread+0x3fa/0x440 [spl] [18307.411302] [] ? wake_up_state+0x20/0x20 [18307.418202] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18307.426161] [] kthread+0xcf/0xe0 [18307.432281] [] ? kthread_create_on_node+0x140/0x140 [18307.440253] [] ret_from_fork+0x58/0x90 [18307.446939] [] ? kthread_create_on_node+0x140/0x140 [18307.454889] arc_prune S ffff88081e694d00 0 3386 2 0x00000080 [18307.463473] ffff88081bb9fe08 0000000000000046 ffff88082679edd0 ffff88081bb9ffd8 [18307.472456] ffff88081bb9ffd8 ffff88081bb9ffd8 ffff88082679edd0 ffff8800b169ac60 [18307.481459] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18307.490455] Call Trace: [18307.493869] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18307.501851] [] schedule+0x29/0x70 [18307.508064] [] taskq_thread+0x3fa/0x440 [spl] [18307.515449] [] ? wake_up_state+0x20/0x20 [18307.522365] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18307.530336] [] kthread+0xcf/0xe0 [18307.536445] [] ? kthread_create_on_node+0x140/0x140 [18307.544400] [] ret_from_fork+0x58/0x90 [18307.551092] [] ? kthread_create_on_node+0x140/0x140 [18307.559046] arc_prune S ffff88081e694d00 0 3387 2 0x00000080 [18307.567641] ffff88082433be08 0000000000000046 ffff880825248000 ffff88082433bfd8 [18307.576654] ffff88082433bfd8 ffff88082433bfd8 ffff880825248000 ffff8800b169acc0 [18307.585660] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18307.594670] Call Trace: [18307.598079] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18307.606048] [] schedule+0x29/0x70 [18307.612269] [] taskq_thread+0x3fa/0x440 [spl] [18307.619653] [] ? wake_up_state+0x20/0x20 [18307.626563] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18307.634531] [] kthread+0xcf/0xe0 [18307.640649] [] ? kthread_create_on_node+0x140/0x140 [18307.648613] [] ret_from_fork+0x58/0x90 [18307.655292] [] ? kthread_create_on_node+0x140/0x140 [18307.663257] arc_prune S ffff88081e694d00 0 3388 2 0x00000080 [18307.671833] ffff88083f8b3e08 0000000000000046 ffff880825248fb0 ffff88083f8b3fd8 [18307.680846] ffff88083f8b3fd8 ffff88083f8b3fd8 ffff880825248fb0 ffff8800b169ad20 [18307.689855] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18307.698851] Call Trace: [18307.702265] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18307.710235] [] schedule+0x29/0x70 [18307.716451] [] taskq_thread+0x3fa/0x440 [spl] [18307.723848] [] ? wake_up_state+0x20/0x20 [18307.730756] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18307.738729] [] kthread+0xcf/0xe0 [18307.744841] [] ? kthread_create_on_node+0x140/0x140 [18307.752812] [] ret_from_fork+0x58/0x90 [18307.759506] [] ? kthread_create_on_node+0x140/0x140 [18307.767468] arc_prune S ffff88081e694d00 0 3389 2 0x00000080 [18307.776035] ffff880829023e08 0000000000000046 ffff880825249f60 ffff880829023fd8 [18307.785034] ffff880829023fd8 ffff880829023fd8 ffff880825249f60 ffff8800b169ad80 [18307.794023] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18307.803025] Call Trace: [18307.806433] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18307.814404] [] schedule+0x29/0x70 [18307.820621] [] taskq_thread+0x3fa/0x440 [spl] [18307.828006] [] ? wake_up_state+0x20/0x20 [18307.834908] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18307.842877] [] kthread+0xcf/0xe0 [18307.848985] [] ? kthread_create_on_node+0x140/0x140 [18307.856941] [] ret_from_fork+0x58/0x90 [18307.863622] [] ? kthread_create_on_node+0x140/0x140 [18307.871588] arc_prune S ffff88081e694d00 0 3390 2 0x00000080 [18307.880158] ffff88083f8cfe08 0000000000000046 ffff88082524af10 ffff88083f8cffd8 [18307.889147] ffff88083f8cffd8 ffff88083f8cffd8 ffff88082524af10 ffff8800b169ade0 [18307.898133] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18307.907120] Call Trace: [18307.910533] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18307.918511] [] schedule+0x29/0x70 [18307.924749] [] taskq_thread+0x3fa/0x440 [spl] [18307.932135] [] ? wake_up_state+0x20/0x20 [18307.939036] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18307.946996] [] kthread+0xcf/0xe0 [18307.953106] [] ? kthread_create_on_node+0x140/0x140 [18307.961057] [] ret_from_fork+0x58/0x90 [18307.967758] [] ? kthread_create_on_node+0x140/0x140 [18307.975716] arc_prune S ffff88081e694d00 0 3391 2 0x00000080 [18307.984295] ffff8808283bbe08 0000000000000046 ffff88082524bec0 ffff8808283bbfd8 [18307.993296] ffff8808283bbfd8 ffff8808283bbfd8 ffff88082524bec0 ffff8800b169ae40 [18308.002295] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18308.011282] Call Trace: [18308.014688] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18308.022668] [] schedule+0x29/0x70 [18308.028886] [] taskq_thread+0x3fa/0x440 [spl] [18308.036270] [] ? wake_up_state+0x20/0x20 [18308.043169] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18308.051131] [] kthread+0xcf/0xe0 [18308.057239] [] ? kthread_create_on_node+0x140/0x140 [18308.065187] [] ret_from_fork+0x58/0x90 [18308.071901] [] ? kthread_create_on_node+0x140/0x140 [18308.079874] arc_prune S ffff88081e694d00 0 3392 2 0x00000080 [18308.088463] ffff880825263e08 0000000000000046 ffff88082524ce70 ffff880825263fd8 [18308.097465] ffff880825263fd8 ffff880825263fd8 ffff88082524ce70 ffff8800b169aea0 [18308.106466] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18308.115473] Call Trace: [18308.118889] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18308.126869] [] schedule+0x29/0x70 [18308.133083] [] taskq_thread+0x3fa/0x440 [spl] [18308.140483] [] ? wake_up_state+0x20/0x20 [18308.147384] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18308.155352] [] kthread+0xcf/0xe0 [18308.161467] [] ? kthread_create_on_node+0x140/0x140 [18308.169421] [] ret_from_fork+0x58/0x90 [18308.176114] [] ? kthread_create_on_node+0x140/0x140 [18308.184072] arc_prune S ffff88081e694d00 0 3393 2 0x00000080 [18308.192658] ffff88083f8afe08 0000000000000046 ffff88082524de20 ffff88083f8affd8 [18308.201658] ffff88083f8affd8 ffff88083f8affd8 ffff88082524de20 ffff8800b169af00 [18308.210667] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18308.219676] Call Trace: [18308.223079] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18308.231049] [] schedule+0x29/0x70 [18308.237271] [] taskq_thread+0x3fa/0x440 [spl] [18308.244664] [] ? wake_up_state+0x20/0x20 [18308.251564] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18308.259522] [] kthread+0xcf/0xe0 [18308.265633] [] ? kthread_create_on_node+0x140/0x140 [18308.273594] [] ret_from_fork+0x58/0x90 [18308.280292] [] ? kthread_create_on_node+0x140/0x140 [18308.288242] arc_prune S ffff88081e694d00 0 3394 2 0x00000080 [18308.296825] ffff880829e67e08 0000000000000046 ffff88082524edd0 ffff880829e67fd8 [18308.305834] ffff880829e67fd8 ffff880829e67fd8 ffff88082524edd0 ffff8800b169af60 [18308.314839] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18308.323827] Call Trace: [18308.327232] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18308.335202] [] schedule+0x29/0x70 [18308.341417] [] taskq_thread+0x3fa/0x440 [spl] [18308.348809] [] ? wake_up_state+0x20/0x20 [18308.355710] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18308.363671] [] kthread+0xcf/0xe0 [18308.369792] [] ? kthread_create_on_node+0x140/0x140 [18308.377757] [] ret_from_fork+0x58/0x90 [18308.384448] [] ? kthread_create_on_node+0x140/0x140 [18308.392405] arc_prune S ffff88081e694d00 0 3395 2 0x00000080 [18308.400996] ffff88082aa6fe08 0000000000000046 ffff88083f840000 ffff88082aa6ffd8 [18308.409990] ffff88082aa6ffd8 ffff88082aa6ffd8 ffff88083f840000 ffff8800b169b000 [18308.418986] ffff88081e694d90 ffffffffa08254b0 ffff88081e694db0 ffff88081e694d00 [18308.427983] Call Trace: [18308.431399] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18308.439378] [] schedule+0x29/0x70 [18308.445603] [] taskq_thread+0x3fa/0x440 [spl] [18308.452981] [] ? wake_up_state+0x20/0x20 [18308.459898] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18308.467873] [] kthread+0xcf/0xe0 [18308.473977] [] ? kthread_create_on_node+0x140/0x140 [18308.481928] [] ret_from_fork+0x58/0x90 [18308.488621] [] ? kthread_create_on_node+0x140/0x140 [18308.496576] arc_reclaim S 000000000000f888 0 3396 2 0x00000080 [18308.505143] ffff88082d61fd30 0000000000000046 ffff88083f840fb0 ffff88082d61ffd8 [18308.514130] ffff88082d61ffd8 ffff88082d61ffd8 ffff88083f840fb0 ffff8804b96ec000 [18308.523122] ffff88082d61fd60 000000010112c7bf ffff8804b96ec000 000000000000f888 [18308.532120] Call Trace: [18308.535527] [] schedule+0x29/0x70 [18308.541757] [] schedule_timeout+0x174/0x2c0 [18308.548939] [] ? internal_add_timer+0x70/0x70 [18308.556310] [] ? prepare_to_wait_exclusive+0x56/0x90 [18308.564377] [] __cv_timedwait_common+0xba/0x160 [spl] [18308.572531] [] ? wake_up_atomic_t+0x30/0x30 [18308.579713] [] __cv_timedwait_sig+0x13/0x20 [spl] [18308.587506] [] arc_reclaim_thread+0x109/0x210 [zfs] [18308.595473] [] ? arc_shrink+0xc0/0xc0 [zfs] [18308.602668] [] thread_generic_wrapper+0x71/0x80 [spl] [18308.610817] [] ? __thread_exit+0x20/0x20 [spl] [18308.618280] [] kthread+0xcf/0xe0 [18308.624387] [] ? kthread_create_on_node+0x140/0x140 [18308.632344] [] ret_from_fork+0x58/0x90 [18308.639029] [] ? kthread_create_on_node+0x140/0x140 [18308.646974] arc_user_evicts S 000000000000f888 0 3397 2 0x00000080 [18308.655540] ffff8808242dfd40 0000000000000046 ffff88083f841f60 ffff8808242dffd8 [18308.664530] ffff8808242dffd8 ffff8808242dffd8 ffff88083f841f60 ffff8804b9068000 [18308.673506] ffff8808242dfd70 000000010112c7bf ffff8804b9068000 000000000000f888 [18308.682484] Call Trace: [18308.685874] [] schedule+0x29/0x70 [18308.692060] [] schedule_timeout+0x174/0x2c0 [18308.699221] [] ? internal_add_timer+0x70/0x70 [18308.706584] [] ? prepare_to_wait_exclusive+0x56/0x90 [18308.714634] [] __cv_timedwait_common+0xba/0x160 [spl] [18308.722788] [] ? wake_up_atomic_t+0x30/0x30 [18308.729956] [] ? hdr_recl+0x50/0x50 [zfs] [18308.736927] [] __cv_timedwait_sig+0x13/0x20 [spl] [18308.744677] [] arc_user_evicts_thread+0xad/0x110 [zfs] [18308.752907] [] thread_generic_wrapper+0x71/0x80 [spl] [18308.761031] [] ? __thread_exit+0x20/0x20 [spl] [18308.768467] [] kthread+0xcf/0xe0 [18308.774551] [] ? kthread_create_on_node+0x140/0x140 [18308.782485] [] ret_from_fork+0x58/0x90 [18308.789137] [] ? kthread_create_on_node+0x140/0x140 [18308.797052] l2arc_feed S 000000000000f888 0 3398 2 0x00000080 [18308.805602] ffff88083f87bc70 0000000000000046 ffff88083f842f10 ffff88083f87bfd8 [18308.814572] ffff88083f87bfd8 ffff88083f87bfd8 ffff88083f842f10 ffff880179130000 [18308.823533] ffff88083f87bca0 000000010112c75c ffff880179130000 000000000000f888 [18308.832487] Call Trace: [18308.835862] [] schedule+0x29/0x70 [18308.842049] [] schedule_timeout+0x174/0x2c0 [18308.849205] [] ? internal_add_timer+0x70/0x70 [18308.856555] [] ? prepare_to_wait_exclusive+0x56/0x90 [18308.864582] [] __cv_timedwait_common+0xba/0x160 [spl] [18308.872689] [] ? wake_up_atomic_t+0x30/0x30 [18308.879856] [] ? l2arc_evict+0x280/0x280 [zfs] [18308.887299] [] __cv_timedwait_sig+0x13/0x20 [spl] [18308.895033] [] l2arc_feed_thread+0x78/0xbe0 [zfs] [18308.902769] [] ? sched_clock_cpu+0x85/0xc0 [18308.909829] [] ? l2arc_evict+0x280/0x280 [zfs] [18308.917271] [] ? spl_kmem_free+0x2a/0x40 [spl] [18308.924716] [] ? kfree+0x103/0x140 [18308.930987] [] ? l2arc_evict+0x280/0x280 [zfs] [18308.938424] [] thread_generic_wrapper+0x71/0x80 [spl] [18308.946547] [] ? __thread_exit+0x20/0x20 [spl] [18308.953977] [] kthread+0xcf/0xe0 [18308.960046] [] ? kthread_create_on_node+0x140/0x140 [18308.967959] [] ret_from_fork+0x58/0x90 [18308.974619] [] ? kthread_create_on_node+0x140/0x140 [18308.982544] z_unmount S ffff88081e694f00 0 3399 2 0x00000080 [18308.991058] ffff8808242e7e08 0000000000000046 ffff88083f843ec0 ffff8808242e7fd8 [18309.000007] ffff8808242e7fd8 ffff8808242e7fd8 ffff88083f843ec0 ffff8800b169b060 [18309.008960] ffff88081e694f90 ffffffffa08254b0 ffff88081e694fb0 ffff88081e694f00 [18309.017904] Call Trace: [18309.021265] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18309.029194] [] schedule+0x29/0x70 [18309.035375] [] taskq_thread+0x3fa/0x440 [spl] [18309.042716] [] ? wake_up_state+0x20/0x20 [18309.049583] [] ? taskq_thread_spawn+0x60/0x60 [spl] [18309.057510] [] kthread+0xcf/0xe0 [18309.063593] [] ? kthread_create_on_node+0x140/0x140 [18309.071522] [] ret_from_fork+0x58/0x90 [18309.078174] [] ? kthread_create_on_node+0x140/0x140 [18309.086090] cfs_rh_00 S ffff88042bc2ce30 0 3411 2 0x00000080 [18309.094645] ffff880826763e28 0000000000000046 ffff88083f844e70 ffff880826763fd8 [18309.103615] ffff880826763fd8 ffff880826763fd8 ffff88083f844e70 ffff88042bc2ce00 [18309.112577] ffff88042bc2ce18 ffff88042bc2ce10 0000000000000000 ffff88042bc2ce30 [18309.121550] Call Trace: [18309.124920] [] schedule+0x29/0x70 [18309.131121] [] cfs_wi_scheduler+0x2f8/0x440 [libcfs] [18309.139151] [] ? wake_up_atomic_t+0x30/0x30 [18309.146320] [] ? cfs_wi_schedule+0x190/0x190 [libcfs] [18309.154449] [] kthread+0xcf/0xe0 [18309.160535] [] ? kthread_create_on_node+0x140/0x140 [18309.168471] [] ret_from_fork+0x58/0x90 [18309.175142] [] ? kthread_create_on_node+0x140/0x140 [18309.183063] cfs_rh_01 S ffff88042bc2ce30 0 3412 2 0x00000080 [18309.191604] ffff8808273f3e28 0000000000000046 ffff88083f845e20 ffff8808273f3fd8 [18309.200571] ffff8808273f3fd8 ffff8808273f3fd8 ffff88083f845e20 ffff88042bc2ce00 [18309.209549] ffff88042bc2ce18 ffff88042bc2ce10 0000000000000000 ffff88042bc2ce30 [18309.218521] Call Trace: [18309.221904] [] schedule+0x29/0x70 [18309.228104] [] cfs_wi_scheduler+0x2f8/0x440 [libcfs] [18309.236149] [] ? wake_up_atomic_t+0x30/0x30 [18309.243329] [] ? cfs_wi_schedule+0x190/0x190 [libcfs] [18309.251483] [] kthread+0xcf/0xe0 [18309.257587] [] ? kthread_create_on_node+0x140/0x140 [18309.265529] [] ret_from_fork+0x58/0x90 [18309.272219] [] ? kthread_create_on_node+0x140/0x140 [18309.280173] cfs_rh_02 S ffff88042bc2ce30 0 3413 2 0x00000080 [18309.288757] ffff88082a757e28 0000000000000046 ffff88083f846dd0 ffff88082a757fd8 [18309.297753] ffff88082a757fd8 ffff88082a757fd8 ffff88083f846dd0 ffff88042bc2ce00 [18309.306745] ffff88042bc2ce18 ffff88042bc2ce10 0000000000000000 ffff88042bc2ce30 [18309.315757] Call Trace: [18309.319169] [] schedule+0x29/0x70 [18309.325403] [] cfs_wi_scheduler+0x2f8/0x440 [libcfs] [18309.333480] [] ? wake_up_atomic_t+0x30/0x30 [18309.340679] [] ? cfs_wi_schedule+0x190/0x190 [libcfs] [18309.348840] [] kthread+0xcf/0xe0 [18309.354971] [] ? kthread_create_on_node+0x140/0x140 [18309.362943] [] ret_from_fork+0x58/0x90 [18309.369638] [] ? kthread_create_on_node+0x140/0x140 [18309.377596] cfs_rh_03 S ffff88042bc2ce30 0 3414 2 0x00000080 [18309.386163] ffff88081badbe28 0000000000000046 ffff88082d220000 ffff88081badbfd8 [18309.395156] ffff88081badbfd8 ffff88081badbfd8 ffff88082d220000 ffff88042bc2ce00 [18309.404144] ffff88042bc2ce18 ffff88042bc2ce10 0000000000000000 ffff88042bc2ce30 [18309.413126] Call Trace: [18309.416533] [] schedule+0x29/0x70 [18309.422766] [] cfs_wi_scheduler+0x2f8/0x440 [libcfs] [18309.430838] [] ? wake_up_atomic_t+0x30/0x30 [18309.438037] [] ? cfs_wi_schedule+0x190/0x190 [libcfs] [18309.446194] [] kthread+0xcf/0xe0 [18309.452305] [] ? kthread_create_on_node+0x140/0x140 [18309.460265] [] ret_from_fork+0x58/0x90 [18309.466977] [] ? kthread_create_on_node+0x140/0x140 [18309.474939] obd_zombid S 0000000000000000 0 3419 2 0x00000080 [18309.483534] ffff8808247ebe40 0000000000000046 ffff88082d222f10 ffff8808247ebfd8 [18309.492525] ffff8808247ebfd8 ffff8808247ebfd8 ffff88082d222f10 ffff88082d222f10 [18309.501520] ffff88082d222f10 ffff88082d222f10 0000000000000000 0000000000000000 [18309.510505] Call Trace: [18309.513916] [] schedule+0x29/0x70 [18309.520173] [] obd_zombie_impexp_thread+0xd5/0x1c0 [obdclass] [18309.529122] [] ? wake_up_state+0x20/0x20 [18309.536039] [] ? obd_zombie_impexp_cull+0x930/0x930 [obdclass] [18309.545079] [] kthread+0xcf/0xe0 [18309.551190] [] ? kthread_create_on_node+0x140/0x140 [18309.559160] [] ret_from_fork+0x58/0x90 [18309.565876] [] ? kthread_create_on_node+0x140/0x140 [18309.573841] ptlrpc_hr00_000 S 00000000ffffffff 0 3422 2 0x00000080 [18309.582427] ffff880824bd7dc0 0000000000000046 ffff88082d223ec0 ffff880824bd7fd8 [18309.591435] ffff880824bd7fd8 ffff880824bd7fd8 ffff88082d223ec0 ffff88040fc17608 [18309.600436] 0000000000000000 ffff88082d223ec0 ffff88040fc17600 00000000ffffffff [18309.609433] Call Trace: [18309.612853] [] schedule+0x29/0x70 [18309.619155] [] ptlrpc_hr_main+0x675/0x8f0 [ptlrpc] [18309.627035] [] ? sched_clock_cpu+0x85/0xc0 [18309.634137] [] ? wake_up_state+0x20/0x20 [18309.641061] [] ? ptlrpc_svcpt_stop_threads+0x590/0x590 [ptlrpc] [18309.650195] [] kthread+0xcf/0xe0 [18309.656322] [] ? finish_task_switch+0x53/0x180 [18309.663820] [] ? kthread_create_on_node+0x140/0x140 [18309.671788] [] ret_from_fork+0x58/0x90 [18309.678504] [] ? kthread_create_on_node+0x140/0x140 [18309.686468] ptlrpc_hr00_001 S 00000000ffffffff 0 3423 2 0x00000080 [18309.695044] ffff8808283abdc0 0000000000000046 ffff88082d224e70 ffff8808283abfd8 [18309.704045] ffff8808283abfd8 ffff8808283abfd8 ffff88082d224e70 ffff88040fc17640 [18309.713046] 0000000000000000 ffff88082d224e70 ffff88040fc17638 00000000ffffffff [18309.722043] Call Trace: [18309.725459] [] schedule+0x29/0x70 [18309.731725] [] ptlrpc_hr_main+0x675/0x8f0 [ptlrpc] [18309.739612] [] ? sched_clock_cpu+0x85/0xc0 [18309.746706] [] ? wake_up_state+0x20/0x20 [18309.753639] [] ? ptlrpc_svcpt_stop_threads+0x590/0x590 [ptlrpc] [18309.762782] [] kthread+0xcf/0xe0 [18309.768909] [] ? kthread_create_on_node+0x140/0x140 [18309.776892] [] ret_from_fork+0x58/0x90 [18309.783609] [] ? kthread_create_on_node+0x140/0x140 [18309.791589] ptlrpc_hr00_002 S 00000000ffffffff 0 3424 2 0x00000080 [18309.800171] ffff8808262cbdc0 0000000000000046 ffff88082d225e20 ffff8808262cbfd8 [18309.809172] ffff8808262cbfd8 ffff8808262cbfd8 ffff88082d225e20 ffff88040fc17678 [18309.818168] 0000000000000000 ffff88082d225e20 ffff88040fc17670 00000000ffffffff [18309.827167] Call Trace: [18309.830590] [] schedule+0x29/0x70 [18309.836858] [] ptlrpc_hr_main+0x675/0x8f0 [ptlrpc] [18309.844746] [] ? sched_clock_cpu+0x85/0xc0 [18309.851854] [] ? wake_up_state+0x20/0x20 [18309.858794] [] ? ptlrpc_svcpt_stop_threads+0x590/0x590 [ptlrpc] [18309.867946] [] kthread+0xcf/0xe0 [18309.874077] [] ? kthread_create_on_node+0x140/0x140 [18309.882045] [] ret_from_fork+0x58/0x90 [18309.888752] [] ? kthread_create_on_node+0x140/0x140 [18309.896725] ptlrpc_hr00_003 S 00000000ffffffff 0 3425 2 0x00000080 [18309.905299] ffff88082460fdc0 0000000000000046 ffff88082d226dd0 ffff88082460ffd8 [18309.914307] ffff88082460ffd8 ffff88082460ffd8 ffff88082d226dd0 ffff88040fc176b0 [18309.923309] 0000000000000000 ffff88082d226dd0 ffff88040fc176a8 00000000ffffffff [18309.932319] Call Trace: [18309.935733] [] schedule+0x29/0x70 [18309.941978] [] ptlrpc_hr_main+0x675/0x8f0 [ptlrpc] [18309.949866] [] ? sched_clock_cpu+0x85/0xc0 [18309.956971] [] ? wake_up_state+0x20/0x20 [18309.963896] [] ? ptlrpc_svcpt_stop_threads+0x590/0x590 [ptlrpc] [18309.973031] [] kthread+0xcf/0xe0 [18309.979155] [] ? kthread_create_on_node+0x140/0x140 [18309.987128] [] ret_from_fork+0x58/0x90 [18309.993842] [] ? kthread_create_on_node+0x140/0x140 [18310.001818] ptlrpc_hr00_004 S 00000000ffffffff 0 3426 2 0x00000080 [18310.010418] ffff88082572fdc0 0000000000000046 ffff88082d221f60 ffff88082572ffd8 [18310.019415] ffff88082572ffd8 ffff88082572ffd8 ffff88082d221f60 ffff88040fc176e8 [18310.028417] 0000000000000000 ffff88082d221f60 ffff88040fc176e0 00000000ffffffff [18310.037415] Call Trace: [18310.040829] [] schedule+0x29/0x70 [18310.047084] [] ptlrpc_hr_main+0x675/0x8f0 [ptlrpc] [18310.054964] [] ? sched_clock_cpu+0x85/0xc0 [18310.062072] [] ? wake_up_state+0x20/0x20 [18310.069009] [] ? ptlrpc_svcpt_stop_threads+0x590/0x590 [ptlrpc] [18310.078137] [] kthread+0xcf/0xe0 [18310.084260] [] ? kthread_create_on_node+0x140/0x140 [18310.092231] [] ret_from_fork+0x58/0x90 [18310.098953] [] ? kthread_create_on_node+0x140/0x140 [18310.106932] ptlrpc_hr00_005 S 00000000ffffffff 0 3427 2 0x00000080 [18310.115528] ffff880825a67dc0 0000000000000046 ffff88082d220fb0 ffff880825a67fd8 [18310.124542] ffff880825a67fd8 ffff880825a67fd8 ffff88082d220fb0 ffff88040fc17720 [18310.133551] 0000000000000000 ffff88082d220fb0 ffff88040fc17718 00000000ffffffff [18310.142558] Call Trace: [18310.145978] [] schedule+0x29/0x70 [18310.152228] [] ptlrpc_hr_main+0x675/0x8f0 [ptlrpc] [18310.160101] [] ? sched_clock_cpu+0x85/0xc0 [18310.167197] [] ? wake_up_state+0x20/0x20 [18310.174121] [] ? ptlrpc_svcpt_stop_threads+0x590/0x590 [ptlrpc] [18310.183254] [] kthread+0xcf/0xe0 [18310.189394] [] ? kthread_create_on_node+0x140/0x140 [18310.197386] [] ret_from_fork+0x58/0x90 [18310.204092] [] ? kthread_create_on_node+0x140/0x140 [18310.212062] ptlrpc_hr00_006 S 00000000ffffffff 0 3428 2 0x00000080 [18310.220653] ffff88081e7abdc0 0000000000000046 ffff8808297f8000 ffff88081e7abfd8 [18310.229661] ffff88081e7abfd8 ffff88081e7abfd8 ffff8808297f8000 ffff88040fc17758 [18310.238662] 0000000000000000 ffff8808297f8000 ffff88040fc17750 00000000ffffffff [18310.247665] Call Trace: [18310.251085] [] schedule+0x29/0x70 [18310.257336] [] ptlrpc_hr_main+0x675/0x8f0 [ptlrpc] [18310.265204] [] ? sched_clock_cpu+0x85/0xc0 [18310.272299] [] ? wake_up_state+0x20/0x20 [18310.279217] [] ? ptlrpc_svcpt_stop_threads+0x590/0x590 [ptlrpc] [18310.288350] [] kthread+0xcf/0xe0 [18310.294474] [] ? kthread_create_on_node+0x140/0x140 [18310.302443] [] ret_from_fork+0x58/0x90 [18310.309157] [] ? kthread_create_on_node+0x140/0x140 [18310.317136] ptlrpc_hr00_007 S 00000000ffffffff 0 3429 2 0x00000080 [18310.325727] ffff88082a3b3dc0 0000000000000046 ffff8808297f8fb0 ffff88082a3b3fd8 [18310.334729] ffff88082a3b3fd8 ffff88082a3b3fd8 ffff8808297f8fb0 ffff88040fc17790 [18310.343744] 0000000000000000 ffff8808297f8fb0 ffff88040fc17788 00000000ffffffff [18310.352756] Call Trace: [18310.356180] [] schedule+0x29/0x70 [18310.362430] [] ptlrpc_hr_main+0x675/0x8f0 [ptlrpc] [18310.370302] [] ? sched_clock_cpu+0x85/0xc0 [18310.377404] [] ? wake_up_state+0x20/0x20 [18310.384334] [] ? ptlrpc_svcpt_stop_threads+0x590/0x590 [ptlrpc] [18310.393473] [] kthread+0xcf/0xe0 [18310.399606] [] ? kthread_create_on_node+0x140/0x140 [18310.407579] [] ret_from_fork+0x58/0x90 [18310.414278] [] ? kthread_create_on_node+0x140/0x140 [18310.422251] ptlrpc_hr01_000 S 00000000ffffffff 0 3430 2 0x00000080 [18310.430834] ffff880829a87dc0 0000000000000046 ffff8808297f9f60 ffff880829a87fd8 [18310.439828] ffff880829a87fd8 ffff880829a87fd8 ffff8808297f9f60 ffff88082abac208 [18310.448842] 0000000000000000 ffff8808297f9f60 ffff88082abac200 00000000ffffffff [18310.457847] Call Trace: [18310.461265] [] schedule+0x29/0x70 [18310.467514] [] ptlrpc_hr_main+0x675/0x8f0 [ptlrpc] [18310.475387] [] ? sched_clock_cpu+0x85/0xc0 [18310.482496] [] ? wake_up_state+0x20/0x20 [18310.489409] [] ? ptlrpc_svcpt_stop_threads+0x590/0x590 [ptlrpc] [18310.498541] [] kthread+0xcf/0xe0 [18310.504664] [] ? kthread_create_on_node+0x140/0x140 [18310.512643] [] ret_from_fork+0x58/0x90 [18310.519356] [] ? kthread_create_on_node+0x140/0x140 [18310.527324] ptlrpc_hr01_001 S 00000000ffffffff 0 3431 2 0x00000080 [18310.535925] ffff880823ed3dc0 0000000000000046 ffff8808297faf10 ffff880823ed3fd8 [18310.544935] ffff880823ed3fd8 ffff880823ed3fd8 ffff8808297faf10 ffff88082abac240 [18310.553934] 0000000000000000 ffff8808297faf10 ffff88082abac238 00000000ffffffff [18310.562944] Call Trace: [18310.566362] [] schedule+0x29/0x70 [18310.572610] [] ptlrpc_hr_main+0x675/0x8f0 [ptlrpc] [18310.580485] [] ? sched_clock_cpu+0x85/0xc0 [18310.587581] [] ? wake_up_state+0x20/0x20 [18310.594512] [] ? ptlrpc_svcpt_stop_threads+0x590/0x590 [ptlrpc] [18310.603653] [] kthread+0xcf/0xe0 [18310.609773] [] ? kthread_create_on_node+0x140/0x140 [18310.617745] [] ret_from_fork+0x58/0x90 [18310.624446] [] ? kthread_create_on_node+0x140/0x140 [18310.632416] ptlrpc_hr01_002 S 00000000ffffffff 0 3432 2 0x00000080 [18310.641002] ffff88082af4fdc0 0000000000000046 ffff8808297fbec0 ffff88082af4ffd8 [18310.649996] ffff88082af4ffd8 ffff88082af4ffd8 ffff8808297fbec0 ffff88082abac278 [18310.658986] 0000000000000000 ffff8808297fbec0 ffff88082abac270 00000000ffffffff [18310.667994] Call Trace: [18310.671400] [] schedule+0x29/0x70 [18310.677658] [] ptlrpc_hr_main+0x675/0x8f0 [ptlrpc] [18310.685537] [] ? sched_clock_cpu+0x85/0xc0 [18310.692644] [] ? wake_up_state+0x20/0x20 [18310.699568] [] ? ptlrpc_svcpt_stop_threads+0x590/0x590 [ptlrpc] [18310.708709] [] kthread+0xcf/0xe0 [18310.714843] [] ? kthread_create_on_node+0x140/0x140 [18310.722828] [] ret_from_fork+0x58/0x90 [18310.729532] [] ? kthread_create_on_node+0x140/0x140 [18310.737514] ptlrpc_hr01_003 S 00000000ffffffff 0 3433 2 0x00000080 [18310.746098] ffff88082aefbdc0 0000000000000046 ffff8808297fce70 ffff88082aefbfd8 [18310.755101] ffff88082aefbfd8 ffff88082aefbfd8 ffff8808297fce70 ffff88082abac2b0 [18310.764097] 0000000000000000 ffff8808297fce70 ffff88082abac2a8 00000000ffffffff [18310.773097] Call Trace: [18310.776502] [] schedule+0x29/0x70 [18310.782755] [] ptlrpc_hr_main+0x675/0x8f0 [ptlrpc] [18310.790636] [] ? sched_clock_cpu+0x85/0xc0 [18310.797745] [] ? wake_up_state+0x20/0x20 [18310.804665] [] ? ptlrpc_svcpt_stop_threads+0x590/0x590 [ptlrpc] [18310.813813] [] kthread+0xcf/0xe0 [18310.819948] [] ? kthread_create_on_node+0x140/0x140 [18310.827932] [] ret_from_fork+0x58/0x90 [18310.834636] [] ? kthread_create_on_node+0x140/0x140 [18310.842606] ptlrpc_hr01_004 S 00000000ffffffff 0 3434 2 0x00000080 [18310.851181] ffff88082a547dc0 0000000000000046 ffff8808297fde20 ffff88082a547fd8 [18310.860164] ffff88082a547fd8 ffff88082a547fd8 ffff8808297fde20 ffff88082abac2e8 [18310.869161] 0000000000000000 ffff8808297fde20 ffff88082abac2e0 00000000ffffffff [18310.878167] Call Trace: [18310.881573] [] schedule+0x29/0x70 [18310.887824] [] ptlrpc_hr_main+0x675/0x8f0 [ptlrpc] [18310.895698] [] ? sched_clock_cpu+0x85/0xc0 [18310.902798] [] ? wake_up_state+0x20/0x20 [18310.909727] [] ? ptlrpc_svcpt_stop_threads+0x590/0x590 [ptlrpc] [18310.918861] [] kthread+0xcf/0xe0 [18310.925005] [] ? kthread_create_on_node+0x140/0x140 [18310.932983] [] ret_from_fork+0x58/0x90 [18310.939694] [] ? kthread_create_on_node+0x140/0x140 [18310.947672] ptlrpc_hr01_005 S 00000000ffffffff 0 3435 2 0x00000080 [18310.956249] ffff880824727dc0 0000000000000046 ffff8808297fedd0 ffff880824727fd8 [18310.965239] ffff880824727fd8 ffff880824727fd8 ffff8808297fedd0 ffff88082abac320 [18310.974224] 0000000000000000 ffff8808297fedd0 ffff88082abac318 00000000ffffffff [18310.983218] Call Trace: [18310.986628] [] schedule+0x29/0x70 [18310.992869] [] ptlrpc_hr_main+0x675/0x8f0 [ptlrpc] [18311.000741] [] ? sched_clock_cpu+0x85/0xc0 [18311.007850] [] ? wake_up_state+0x20/0x20 [18311.014774] [] ? ptlrpc_svcpt_stop_threads+0x590/0x590 [ptlrpc] [18311.023919] [] kthread+0xcf/0xe0 [18311.030042] [] ? kthread_create_on_node+0x140/0x140 [18311.038012] [] ret_from_fork+0x58/0x90 [18311.044725] [] ? kthread_create_on_node+0x140/0x140 [18311.052696] ptlrpc_hr01_006 S 00000000ffffffff 0 3436 2 0x00000080 [18311.061272] ffff88083f893dc0 0000000000000046 ffff880824290000 ffff88083f893fd8 [18311.070278] ffff88083f893fd8 ffff88083f893fd8 ffff880824290000 ffff88082abac358 [18311.079256] 0000000000000000 ffff880824290000 ffff88082abac350 00000000ffffffff [18311.088245] Call Trace: [18311.091663] [] schedule+0x29/0x70 [18311.097918] [] ptlrpc_hr_main+0x675/0x8f0 [ptlrpc] [18311.105791] [] ? sched_clock_cpu+0x85/0xc0 [18311.112885] [] ? wake_up_state+0x20/0x20 [18311.119812] [] ? ptlrpc_svcpt_stop_threads+0x590/0x590 [ptlrpc] [18311.128958] [] kthread+0xcf/0xe0 [18311.135071] [] ? kthread_create_on_node+0x140/0x140 [18311.143040] [] ret_from_fork+0x58/0x90 [18311.149753] [] ? kthread_create_on_node+0x140/0x140 [18311.157730] ptlrpc_hr01_007 S 00000000ffffffff 0 3437 2 0x00000080 [18311.166306] ffff88081e7a3dc0 0000000000000046 ffff880824290fb0 ffff88081e7a3fd8 [18311.175304] ffff88081e7a3fd8 ffff88081e7a3fd8 ffff880824290fb0 ffff88082abac390 [18311.184300] 0000000000000000 ffff880824290fb0 ffff88082abac388 00000000ffffffff [18311.193297] Call Trace: [18311.196720] [] schedule+0x29/0x70 [18311.202975] [] ptlrpc_hr_main+0x675/0x8f0 [ptlrpc] [18311.210847] [] ? sched_clock_cpu+0x85/0xc0 [18311.217964] [] ? wake_up_state+0x20/0x20 [18311.224888] [] ? ptlrpc_svcpt_stop_threads+0x590/0x590 [ptlrpc] [18311.234023] [] kthread+0xcf/0xe0 [18311.240149] [] ? kthread_create_on_node+0x140/0x140 [18311.248119] [] ret_from_fork+0x58/0x90 [18311.254824] [] ? kthread_create_on_node+0x140/0x140 [18311.262799] kiblnd_connd S 000000000000f888 0 3447 2 0x00000080 [18311.271392] ffff88082a27bd60 0000000000000046 ffff880824291f60 ffff88082a27bfd8 [18311.280385] ffff88082a27bfd8 ffff88082a27bfd8 ffff880824291f60 ffff8804b9740000 [18311.289384] ffff88082a27bd90 000000010112d378 ffff8804b9740000 000000000000f888 [18311.298382] Call Trace: [18311.301799] [] schedule+0x29/0x70 [18311.308032] [] schedule_timeout+0x174/0x2c0 [18311.315219] [] ? internal_add_timer+0x70/0x70 [18311.322620] [] kiblnd_connd+0x60c/0x900 [ko2iblnd] [18311.330489] [] ? wake_up_state+0x20/0x20 [18311.337394] [] ? kiblnd_check_conns+0x840/0x840 [ko2iblnd] [18311.346041] [] kthread+0xcf/0xe0 [18311.352161] [] ? kthread_create_on_node+0x140/0x140 [18311.360122] [] ret_from_fork+0x58/0x90 [18311.366827] [] ? kthread_create_on_node+0x140/0x140 [18311.374787] kiblnd_sd_00_00 S ffff88040fb03ee0 0 3448 2 0x00000080 [18311.383363] ffff88082af9fd98 0000000000000046 ffff880824292f10 ffff88082af9ffd8 [18311.392363] ffff88082af9ffd8 ffff88082af9ffd8 ffff880824292f10 ffff88040fb03ec0 [18311.401370] 0000000000000246 ffff88040fb03ec8 ffff880824292f10 ffff88040fb03ee0 [18311.410348] Call Trace: [18311.413760] [] schedule+0x29/0x70 [18311.419973] [] kiblnd_scheduler+0x32c/0x1130 [ko2iblnd] [18311.428324] [] ? dequeue_task_fair+0x41e/0x660 [18311.435805] [] ? sched_clock_cpu+0x85/0xc0 [18311.442896] [] ? wake_up_state+0x20/0x20 [18311.449795] [] ? kiblnd_cq_event+0x90/0x90 [ko2iblnd] [18311.457960] [] kthread+0xcf/0xe0 [18311.464078] [] ? kthread_create_on_node+0x140/0x140 [18311.472037] [] ret_from_fork+0x58/0x90 [18311.478738] [] ? kthread_create_on_node+0x140/0x140 [18311.486689] kiblnd_sd_00_01 S ffff88040fb03ee0 0 3449 2 0x00000080 [18311.495251] ffff88081e7e3d98 0000000000000046 ffff880824293ec0 ffff88081e7e3fd8 [18311.504237] ffff88081e7e3fd8 ffff88081e7e3fd8 ffff880824293ec0 ffff88040fb03ec0 [18311.513221] 0000000000000246 ffff88040fb03ec8 ffff880824293ec0 ffff88040fb03ee0 [18311.522195] Call Trace: [18311.525583] [] schedule+0x29/0x70 [18311.531802] [] kiblnd_scheduler+0x32c/0x1130 [ko2iblnd] [18311.540149] [] ? dequeue_task_fair+0x41e/0x660 [18311.547616] [] ? wake_up_state+0x20/0x20 [18311.554509] [] ? kiblnd_cq_event+0x90/0x90 [ko2iblnd] [18311.562663] [] kthread+0xcf/0xe0 [18311.568760] [] ? kthread_create_on_node+0x140/0x140 [18311.576712] [] ret_from_fork+0x58/0x90 [18311.583407] [] ? kthread_create_on_node+0x140/0x140 [18311.591350] kiblnd_sd_00_02 S ffff88040fb03ee0 0 3450 2 0x00000080 [18311.599924] ffff880826207d98 0000000000000046 ffff880824294e70 ffff880826207fd8 [18311.608907] ffff880826207fd8 ffff880826207fd8 ffff880824294e70 ffff88040fb03ec0 [18311.617899] 0000000000000292 ffff88040fb03ec8 ffff880824294e70 ffff88040fb03ee0 [18311.626887] Call Trace: [18311.630267] [] schedule+0x29/0x70 [18311.636471] [] kiblnd_scheduler+0x32c/0x1130 [ko2iblnd] [18311.644825] [] ? dequeue_task_fair+0x41e/0x660 [18311.652289] [] ? sched_clock_cpu+0x85/0xc0 [18311.659354] [] ? wake_up_state+0x20/0x20 [18311.666228] [] ? kiblnd_cq_event+0x90/0x90 [ko2iblnd] [18311.674368] [] kthread+0xcf/0xe0 [18311.680466] [] ? kthread_create_on_node+0x140/0x140 [18311.688410] [] ret_from_fork+0x58/0x90 [18311.695102] [] ? kthread_create_on_node+0x140/0x140 [18311.703047] kiblnd_sd_00_03 S ffff88040fb03ee0 0 3451 2 0x00000080 [18311.711614] ffff88082bff3d98 0000000000000046 ffff880824295e20 ffff88082bff3fd8 [18311.720592] ffff88082bff3fd8 ffff88082bff3fd8 ffff880824295e20 ffff88040fb03ec0 [18311.729585] 0000000000000292 ffff88040fb03ec8 ffff880824295e20 ffff88040fb03ee0 [18311.738561] Call Trace: [18311.741945] [] schedule+0x29/0x70 [18311.748142] [] kiblnd_scheduler+0x32c/0x1130 [ko2iblnd] [18311.756486] [] ? dequeue_task_fair+0x41e/0x660 [18311.763948] [] ? sched_clock_cpu+0x85/0xc0 [18311.771019] [] ? wake_up_state+0x20/0x20 [18311.777898] [] ? kiblnd_cq_event+0x90/0x90 [ko2iblnd] [18311.786042] [] kthread+0xcf/0xe0 [18311.792139] [] ? kthread_create_on_node+0x140/0x140 [18311.800080] [] ret_from_fork+0x58/0x90 [18311.806773] [] ? kthread_create_on_node+0x140/0x140 [18311.814725] kiblnd_sd_01_00 S ffff88083f90b860 0 3452 2 0x00000080 [18311.823275] ffff88083f76bd98 0000000000000046 ffff880824296dd0 ffff88083f76bfd8 [18311.832249] ffff88083f76bfd8 ffff88083f76bfd8 ffff880824296dd0 ffff88083f90b840 [18311.841225] 0000000000000246 ffff88083f90b848 ffff880824296dd0 ffff88083f90b860 [18311.850194] Call Trace: [18311.853568] [] schedule+0x29/0x70 [18311.859772] [] kiblnd_scheduler+0x32c/0x1130 [ko2iblnd] [18311.868108] [] ? dequeue_task_fair+0x41e/0x660 [18311.875571] [] ? sched_clock_cpu+0x85/0xc0 [18311.882642] [] ? wake_up_state+0x20/0x20 [18311.889517] [] ? kiblnd_cq_event+0x90/0x90 [ko2iblnd] [18311.897658] [] kthread+0xcf/0xe0 [18311.903756] [] ? kthread_create_on_node+0x140/0x140 [18311.911707] [] ret_from_fork+0x58/0x90 [18311.918388] [] ? kthread_create_on_node+0x140/0x140 [18311.926323] kiblnd_sd_01_01 S ffff88083f90b860 0 3453 2 0x00000080 [18311.934902] ffff880829c8bd98 0000000000000046 ffff880827730000 ffff880829c8bfd8 [18311.943879] ffff880829c8bfd8 ffff880829c8bfd8 ffff880827730000 ffff88083f90b840 [18311.952857] 0000000000000246 ffff88083f90b848 ffff880827730000 ffff88083f90b860 [18311.961821] Call Trace: [18311.965201] [] schedule+0x29/0x70 [18311.971404] [] kiblnd_scheduler+0x32c/0x1130 [ko2iblnd] [18311.979737] [] ? dequeue_task_fair+0x41e/0x660 [18311.987198] [] ? sched_clock_cpu+0x85/0xc0 [18311.994258] [] ? wake_up_state+0x20/0x20 [18312.001122] [] ? kiblnd_cq_event+0x90/0x90 [ko2iblnd] [18312.009257] [] kthread+0xcf/0xe0 [18312.015355] [] ? kthread_create_on_node+0x140/0x140 [18312.023298] [] ret_from_fork+0x58/0x90 [18312.029991] [] ? kthread_create_on_node+0x140/0x140 [18312.037948] kiblnd_sd_01_02 S ffff88083f90b860 0 3454 2 0x00000080 [18312.046517] ffff8808290cfd98 0000000000000046 ffff880827730fb0 ffff8808290cffd8 [18312.055503] ffff8808290cffd8 ffff8808290cffd8 ffff880827730fb0 ffff88083f90b840 [18312.064474] 0000000000000292 ffff88083f90b848 ffff880827730fb0 ffff88083f90b860 [18312.073419] Call Trace: [18312.076792] [] schedule+0x29/0x70 [18312.082993] [] kiblnd_scheduler+0x32c/0x1130 [ko2iblnd] [18312.091325] [] ? sched_clock+0x9/0x10 [18312.097917] [] ? sched_clock_cpu+0x85/0xc0 [18312.104998] [] ? wake_up_state+0x20/0x20 [18312.111863] [] ? kiblnd_cq_event+0x90/0x90 [ko2iblnd] [18312.119998] [] kthread+0xcf/0xe0 [18312.126089] [] ? kthread_create_on_node+0x140/0x140 [18312.134032] [] ret_from_fork+0x58/0x90 [18312.140706] [] ? kthread_create_on_node+0x140/0x140 [18312.148651] kiblnd_sd_01_03 S ffff88083f90b860 0 3455 2 0x00000080 [18312.157197] ffff880824303d98 0000000000000046 ffff880827731f60 ffff880824303fd8 [18312.166176] ffff880824303fd8 ffff880824303fd8 ffff880827731f60 ffff88083f90b840 [18312.175146] 0000000000000246 ffff88083f90b848 ffff880827731f60 ffff88083f90b860 [18312.184098] Call Trace: [18312.187477] [] schedule+0x29/0x70 [18312.193676] [] kiblnd_scheduler+0x32c/0x1130 [ko2iblnd] [18312.202018] [] ? dequeue_task_fair+0x41e/0x660 [18312.209476] [] ? sched_clock_cpu+0x85/0xc0 [18312.216539] [] ? wake_up_state+0x20/0x20 [18312.223411] [] ? kiblnd_cq_event+0x90/0x90 [ko2iblnd] [18312.231546] [] kthread+0xcf/0xe0 [18312.237650] [] ? kthread_create_on_node+0x140/0x140 [18312.245599] [] ret_from_fork+0x58/0x90 [18312.252267] [] ? kthread_create_on_node+0x140/0x140 [18312.260199] router_checker S 000000000000f888 0 3456 2 0x00000080 [18312.268766] ffff880826f03d20 0000000000000046 ffff880827732f10 ffff880826f03fd8 [18312.277739] ffff880826f03fd8 ffff880826f03fd8 ffff880827732f10 ffff8801795fc000 [18312.286724] ffff880826f03d50 000000010112d6b4 ffff8801795fc000 000000000000f888 [18312.295701] Call Trace: [18312.299074] [] schedule+0x29/0x70 [18312.305272] [] schedule_timeout+0x174/0x2c0 [18312.312430] [] ? internal_add_timer+0x70/0x70 [18312.319785] [] ? prepare_to_wait+0x56/0x90 [18312.326882] [] lnet_router_checker+0x527/0x890 [lnet] [18312.335016] [] ? wake_up_atomic_t+0x30/0x30 [18312.342179] [] ? lnet_notify_locked+0x240/0x240 [lnet] [18312.350401] [] kthread+0xcf/0xe0 [18312.356491] [] ? kthread_create_on_node+0x140/0x140 [18312.364431] [] ret_from_fork+0x58/0x90 [18312.371103] [] ? kthread_create_on_node+0x140/0x140 [18312.379046] ptlrpcd_rcv S 000000000000f888 0 3457 2 0x00000080 [18312.387601] ffff880826f07cf0 0000000000000046 ffff880827733ec0 ffff880826f07fd8 [18312.396567] ffff880826f07fd8 ffff880826f07fd8 ffff880827733ec0 ffff880179d4c000 [18312.405529] ffff880826f07d20 000000010112d783 ffff880179d4c000 000000000000f888 [18312.414494] Call Trace: [18312.417880] [] schedule+0x29/0x70 [18312.424069] [] schedule_timeout+0x174/0x2c0 [18312.431228] [] ? internal_add_timer+0x70/0x70 [18312.438616] [] ptlrpcd+0x298/0x560 [ptlrpc] [18312.445777] [] ? wake_up_state+0x20/0x20 [18312.452680] [] ? ptlrpcd_check+0x5d0/0x5d0 [ptlrpc] [18312.460616] [] kthread+0xcf/0xe0 [18312.466717] [] ? kthread_create_on_node+0x140/0x140 [18312.474659] [] ret_from_fork+0x58/0x90 [18312.481331] [] ? kthread_create_on_node+0x140/0x140 [18312.489264] ptlrpcd_00_00 S 000000000000f888 0 3458 2 0x00000080 [18312.497819] ffff880826f0bcf0 0000000000000046 ffff880827734e70 ffff880826f0bfd8 [18312.506812] ffff880826f0bfd8 ffff880826f0bfd8 ffff880827734e70 ffff8801795fc000 [18312.515768] ffff880826f0bd20 000000010112d7be ffff8801795fc000 000000000000f888 [18312.524749] Call Trace: [18312.528127] [] schedule+0x29/0x70 [18312.534317] [] schedule_timeout+0x174/0x2c0 [18312.541484] [] ? internal_add_timer+0x70/0x70 [18312.548867] [] ptlrpcd+0x298/0x560 [ptlrpc] [18312.556026] [] ? wake_up_state+0x20/0x20 [18312.562920] [] ? ptlrpcd_check+0x5d0/0x5d0 [ptlrpc] [18312.570857] [] kthread+0xcf/0xe0 [18312.576947] [] ? kthread_create_on_node+0x140/0x140 [18312.584882] [] ret_from_fork+0x58/0x90 [18312.591545] [] ? kthread_create_on_node+0x140/0x140 [18312.599480] ptlrpcd_00_01 S 000000000000f888 0 3459 2 0x00000080 [18312.608030] ffff880826f0fcf0 0000000000000046 ffff880827735e20 ffff880826f0ffd8 [18312.616995] ffff880826f0ffd8 ffff880826f0ffd8 ffff880827735e20 ffff880179068000 [18312.625957] ffff880826f0fd20 000000010112d7bf ffff880179068000 000000000000f888 [18312.634931] Call Trace: [18312.638298] [] schedule+0x29/0x70 [18312.644490] [] schedule_timeout+0x174/0x2c0 [18312.651646] [] ? internal_add_timer+0x70/0x70 [18312.659025] [] ptlrpcd+0x298/0x560 [ptlrpc] [18312.666178] [] ? wake_up_state+0x20/0x20 [18312.673062] [] ? ptlrpcd_check+0x5d0/0x5d0 [ptlrpc] [18312.680998] [] kthread+0xcf/0xe0 [18312.687080] [] ? kthread_create_on_node+0x140/0x140 [18312.695008] [] ret_from_fork+0x58/0x90 [18312.701680] [] ? kthread_create_on_node+0x140/0x140 [18312.709607] ptlrpcd_00_02 S 000000000000f888 0 3460 2 0x00000080 [18312.718144] ffff880825a53cf0 0000000000000046 ffff880827736dd0 ffff880825a53fd8 [18312.727109] ffff880825a53fd8 ffff880825a53fd8 ffff880827736dd0 ffff8801796e4000 [18312.736074] ffff880825a53d20 000000010112d7bd ffff8801796e4000 000000000000f888 [18312.745031] Call Trace: [18312.748397] [] schedule+0x29/0x70 [18312.754590] [] schedule_timeout+0x174/0x2c0 [18312.761743] [] ? internal_add_timer+0x70/0x70 [18312.769100] [] ptlrpcd+0x298/0x560 [ptlrpc] [18312.776248] [] ? wake_up_state+0x20/0x20 [18312.783121] [] ? ptlrpcd_check+0x5d0/0x5d0 [ptlrpc] [18312.791058] [] kthread+0xcf/0xe0 [18312.797139] [] ? kthread_create_on_node+0x140/0x140 [18312.805061] [] ret_from_fork+0x58/0x90 [18312.811722] [] ? kthread_create_on_node+0x140/0x140 [18312.819658] ptlrpcd_00_03 S 000000000000f888 0 3461 2 0x00000080 [18312.828197] ffff880825a57cf0 0000000000000046 ffff880825a58000 ffff880825a57fd8 [18312.837148] ffff880825a57fd8 ffff880825a57fd8 ffff880825a58000 ffff8801796b8000 [18312.846107] ffff880825a57d20 000000010112d7bc ffff8801796b8000 000000000000f888 [18312.855054] Call Trace: [18312.858422] [] schedule+0x29/0x70 [18312.864605] [] schedule_timeout+0x174/0x2c0 [18312.871748] [] ? internal_add_timer+0x70/0x70 [18312.879120] [] ptlrpcd+0x298/0x560 [ptlrpc] [18312.886261] [] ? wake_up_state+0x20/0x20 [18312.893150] [] ? ptlrpcd_check+0x5d0/0x5d0 [ptlrpc] [18312.901062] [] kthread+0xcf/0xe0 [18312.907134] [] ? kthread_create_on_node+0x140/0x140 [18312.915064] [] ret_from_fork+0x58/0x90 [18312.921730] [] ? kthread_create_on_node+0x140/0x140 [18312.929664] ptlrpcd_00_04 S 000000000000f888 0 3462 2 0x00000080 [18312.938217] ffff880824793cf0 0000000000000046 ffff880825a58fb0 ffff880824793fd8 [18312.947183] ffff880824793fd8 ffff880824793fd8 ffff880825a58fb0 ffff880179674000 [18312.956146] ffff880824793d20 000000010112d809 ffff880179674000 000000000000f888 [18312.965104] Call Trace: [18312.968472] [] schedule+0x29/0x70 [18312.974674] [] schedule_timeout+0x174/0x2c0 [18312.981825] [] ? internal_add_timer+0x70/0x70 [18312.989188] [] ptlrpcd+0x298/0x560 [ptlrpc] [18312.996336] [] ? wake_up_state+0x20/0x20 [18313.003218] [] ? ptlrpcd_check+0x5d0/0x5d0 [ptlrpc] [18313.011146] [] kthread+0xcf/0xe0 [18313.017223] [] ? kthread_create_on_node+0x140/0x140 [18313.025150] [] ret_from_fork+0x58/0x90 [18313.031818] [] ? kthread_create_on_node+0x140/0x140 [18313.039749] ptlrpcd_00_05 S 000000000000f888 0 3463 2 0x00000080 [18313.048287] ffff880824797cf0 0000000000000046 ffff880825a59f60 ffff880824797fd8 [18313.057251] ffff880824797fd8 ffff880824797fd8 ffff880825a59f60 ffff880179644000 [18313.066218] ffff880824797d20 000000010112d848 ffff880179644000 000000000000f888 [18313.075174] Call Trace: [18313.078557] [] schedule+0x29/0x70 [18313.084752] [] schedule_timeout+0x174/0x2c0 [18313.091907] [] ? internal_add_timer+0x70/0x70 [18313.099274] [] ptlrpcd+0x298/0x560 [ptlrpc] [18313.106421] [] ? wake_up_state+0x20/0x20 [18313.113302] [] ? ptlrpcd_check+0x5d0/0x5d0 [ptlrpc] [18313.121232] [] kthread+0xcf/0xe0 [18313.127306] [] ? kthread_create_on_node+0x140/0x140 [18313.135236] [] ret_from_fork+0x58/0x90 [18313.141892] [] ? kthread_create_on_node+0x140/0x140 [18313.149829] ptlrpcd_00_06 S 000000000000f888 0 3464 2 0x00000080 [18313.158359] ffff88082479bcf0 0000000000000046 ffff880825a5af10 ffff88082479bfd8 [18313.167325] ffff88082479bfd8 ffff88082479bfd8 ffff880825a5af10 ffff8801795fc000 [18313.176271] ffff88082479bd20 000000010112d859 ffff8801795fc000 000000000000f888 [18313.185221] Call Trace: [18313.188591] [] schedule+0x29/0x70 [18313.194777] [] schedule_timeout+0x174/0x2c0 [18313.201931] [] ? internal_add_timer+0x70/0x70 [18313.209290] [] ptlrpcd+0x298/0x560 [ptlrpc] [18313.216437] [] ? wake_up_state+0x20/0x20 [18313.223318] [] ? ptlrpcd_check+0x5d0/0x5d0 [ptlrpc] [18313.231245] [] kthread+0xcf/0xe0 [18313.237323] [] ? kthread_create_on_node+0x140/0x140 [18313.245252] [] ret_from_fork+0x58/0x90 [18313.251916] [] ? kthread_create_on_node+0x140/0x140 [18313.259845] ptlrpcd_00_07 S 000000000000f888 0 3465 2 0x00000080 [18313.268383] ffff88082479fcf0 0000000000000046 ffff880825a5bec0 ffff88082479ffd8 [18313.277338] ffff88082479ffd8 ffff88082479ffd8 ffff880825a5bec0 ffff88017971c000 [18313.286269] ffff88082479fd20 000000010112dba5 ffff88017971c000 000000000000f888 [18313.295230] Call Trace: [18313.298596] [] schedule+0x29/0x70 [18313.304795] [] schedule_timeout+0x174/0x2c0 [18313.311929] [] ? internal_add_timer+0x70/0x70 [18313.319292] [] ptlrpcd+0x298/0x560 [ptlrpc] [18313.326440] [] ? wake_up_state+0x20/0x20 [18313.333321] [] ? ptlrpcd_check+0x5d0/0x5d0 [ptlrpc] [18313.341243] [] kthread+0xcf/0xe0 [18313.347320] [] ? kthread_create_on_node+0x140/0x140 [18313.355246] [] ret_from_fork+0x58/0x90 [18313.361923] [] ? kthread_create_on_node+0x140/0x140 [18313.369852] ptlrpcd_00_08 S 000000000000f888 0 3466 2 0x00000080 [18313.378388] ffff88081bab3cf0 0000000000000046 ffff880825a5ce70 ffff88081bab3fd8 [18313.387348] ffff88081bab3fd8 ffff88081bab3fd8 ffff880825a5ce70 ffff880179130000 [18313.396293] ffff88081bab3d20 000000010112dba5 ffff880179130000 000000000000f888 [18313.405220] Call Trace: [18313.408591] [] schedule+0x29/0x70 [18313.414776] [] schedule_timeout+0x174/0x2c0 [18313.421932] [] ? internal_add_timer+0x70/0x70 [18313.429297] [] ptlrpcd+0x298/0x560 [ptlrpc] [18313.436452] [] ? wake_up_state+0x20/0x20 [18313.443335] [] ? ptlrpcd_check+0x5d0/0x5d0 [ptlrpc] [18313.451261] [] kthread+0xcf/0xe0 [18313.457338] [] ? finish_task_switch+0x53/0x180 [18313.464787] [] ? kthread_create_on_node+0x140/0x140 [18313.472720] [] ret_from_fork+0x58/0x90 [18313.479383] [] ? kthread_create_on_node+0x140/0x140 [18313.487309] ptlrpcd_00_09 S 000000000000f888 0 3467 2 0x00000080 [18313.495871] ffff88081bab7cf0 0000000000000046 ffff880825a5de20 ffff88081bab7fd8 [18313.504842] ffff88081bab7fd8 ffff88081bab7fd8 ffff880825a5de20 ffff880179644000 [18313.513802] ffff88081bab7d20 000000010112dba4 ffff880179644000 000000000000f888 [18313.522754] Call Trace: [18313.526119] [] schedule+0x29/0x70 [18313.532296] [] schedule_timeout+0x174/0x2c0 [18313.539442] [] ? internal_add_timer+0x70/0x70 [18313.546811] [] ptlrpcd+0x298/0x560 [ptlrpc] [18313.553960] [] ? wake_up_state+0x20/0x20 [18313.560837] [] ? ptlrpcd_check+0x5d0/0x5d0 [ptlrpc] [18313.568772] [] kthread+0xcf/0xe0 [18313.574856] [] ? kthread_create_on_node+0x140/0x140 [18313.582776] [] ret_from_fork+0x58/0x90 [18313.589442] [] ? kthread_create_on_node+0x140/0x140 [18313.597365] ptlrpcd_00_10 S 000000000000f888 0 3468 2 0x00000080 [18313.605909] ffff88081babbcf0 0000000000000046 ffff880825a5edd0 ffff88081babbfd8 [18313.614877] ffff88081babbfd8 ffff88081babbfd8 ffff880825a5edd0 ffff880179674000 [18313.623840] ffff88081babbd20 000000010112dba4 ffff880179674000 000000000000f888 [18313.632805] Call Trace: [18313.636178] [] schedule+0x29/0x70 [18313.642357] [] schedule_timeout+0x174/0x2c0 [18313.649502] [] ? internal_add_timer+0x70/0x70 [18313.656879] [] ptlrpcd+0x298/0x560 [ptlrpc] [18313.664036] [] ? wake_up_state+0x20/0x20 [18313.670918] [] ? ptlrpcd_check+0x5d0/0x5d0 [ptlrpc] [18313.678847] [] kthread+0xcf/0xe0 [18313.684929] [] ? kthread_create_on_node+0x140/0x140 [18313.692856] [] ret_from_fork+0x58/0x90 [18313.699528] [] ? kthread_create_on_node+0x140/0x140 [18313.707455] ptlrpcd_00_11 S 000000000000f888 0 3469 2 0x00000080 [18313.715996] ffff88081babfcf0 0000000000000046 ffff880825b10000 ffff88081babffd8 [18313.724952] ffff88081babffd8 ffff88081babffd8 ffff880825b10000 ffff880179674000 [18313.733901] ffff88081babfd20 000000010112dba5 ffff880179674000 000000000000f888 [18313.742863] Call Trace: [18313.746228] [] schedule+0x29/0x70 [18313.752404] [] schedule_timeout+0x174/0x2c0 [18313.759551] [] ? internal_add_timer+0x70/0x70 [18313.766926] [] ptlrpcd+0x298/0x560 [ptlrpc] [18313.774081] [] ? wake_up_state+0x20/0x20 [18313.780971] [] ? ptlrpcd_check+0x5d0/0x5d0 [ptlrpc] [18313.788907] [] kthread+0xcf/0xe0 [18313.794997] [] ? kthread_create_on_node+0x140/0x140 [18313.802929] [] ret_from_fork+0x58/0x90 [18313.809592] [] ? kthread_create_on_node+0x140/0x140 [18313.817522] ptlrpcd_00_12 S 000000000000f888 0 3470 2 0x00000080 [18313.826093] ffff880825b1bcf0 0000000000000046 ffff880825b10fb0 ffff880825b1bfd8 [18313.835064] ffff880825b1bfd8 ffff880825b1bfd8 ffff880825b10fb0 ffff8801796b8000 [18313.844027] ffff880825b1bd20 000000010112dba5 ffff8801796b8000 000000000000f888 [18313.852991] Call Trace: [18313.856354] [] schedule+0x29/0x70 [18313.862547] [] schedule_timeout+0x174/0x2c0 [18313.869685] [] ? internal_add_timer+0x70/0x70 [18313.877064] [] ptlrpcd+0x298/0x560 [ptlrpc] [18313.884218] [] ? wake_up_state+0x20/0x20 [18313.891109] [] ? ptlrpcd_check+0x5d0/0x5d0 [ptlrpc] [18313.899042] [] kthread+0xcf/0xe0 [18313.905123] [] ? kthread_create_on_node+0x140/0x140 [18313.913057] [] ret_from_fork+0x58/0x90 [18313.919737] [] ? kthread_create_on_node+0x140/0x140 [18313.927673] ptlrpcd_00_13 S 000000000000f888 0 3471 2 0x00000080 [18313.936214] ffff880825b1fcf0 0000000000000046 ffff880825b11f60 ffff880825b1ffd8 [18313.945173] ffff880825b1ffd8 ffff880825b1ffd8 ffff880825b11f60 ffff880179d4c000 [18313.954133] ffff880825b1fd20 000000010112dba4 ffff880179d4c000 000000000000f888 [18313.963090] Call Trace: [18313.966452] [] schedule+0x29/0x70 [18313.972638] [] schedule_timeout+0x174/0x2c0 [18313.979779] [] ? internal_add_timer+0x70/0x70 [18313.987136] [] ptlrpcd+0x298/0x560 [ptlrpc] [18313.994283] [] ? wake_up_state+0x20/0x20 [18314.001164] [] ? ptlrpcd_check+0x5d0/0x5d0 [ptlrpc] [18314.009087] [] kthread+0xcf/0xe0 [18314.015161] [] ? kthread_create_on_node+0x140/0x140 [18314.023090] [] ret_from_fork+0x58/0x90 [18314.029759] [] ? kthread_create_on_node+0x140/0x140 [18314.037697] ptlrpcd_00_14 S 000000000000f888 0 3472 2 0x00000080 [18314.046242] ffff880825bd3cf0 0000000000000046 ffff880825b12f10 ffff880825bd3fd8 [18314.055185] ffff880825bd3fd8 ffff880825bd3fd8 ffff880825b12f10 ffff8801795fc000 [18314.064139] ffff880825bd3d20 000000010112dba4 ffff8801795fc000 000000000000f888 [18314.073090] Call Trace: [18314.076459] [] schedule+0x29/0x70 [18314.082646] [] schedule_timeout+0x174/0x2c0 [18314.089792] [] ? internal_add_timer+0x70/0x70 [18314.097163] [] ptlrpcd+0x298/0x560 [ptlrpc] [18314.104305] [] ? wake_up_state+0x20/0x20 [18314.111196] [] ? ptlrpcd_check+0x5d0/0x5d0 [ptlrpc] [18314.119125] [] kthread+0xcf/0xe0 [18314.125208] [] ? kthread_create_on_node+0x140/0x140 [18314.133136] [] ret_from_fork+0x58/0x90 [18314.139802] [] ? kthread_create_on_node+0x140/0x140 [18314.147729] ptlrpcd_00_15 S 000000000000f888 0 3473 2 0x00000080 [18314.156267] ffff880825bd7cf0 0000000000000046 ffff880825b13ec0 ffff880825bd7fd8 [18314.165229] ffff880825bd7fd8 ffff880825bd7fd8 ffff880825b13ec0 ffff880179644000 [18314.174176] ffff880825bd7d20 000000010112dba6 ffff880179644000 000000000000f888 [18314.183139] Call Trace: [18314.186518] [] schedule+0x29/0x70 [18314.192709] [] schedule_timeout+0x174/0x2c0 [18314.199864] [] ? internal_add_timer+0x70/0x70 [18314.207233] [] ptlrpcd+0x298/0x560 [ptlrpc] [18314.214383] [] ? wake_up_state+0x20/0x20 [18314.221266] [] ? ptlrpcd_check+0x5d0/0x5d0 [ptlrpc] [18314.229193] [] kthread+0xcf/0xe0 [18314.235270] [] ? kthread_create_on_node+0x140/0x140 [18314.243200] [] ret_from_fork+0x58/0x90 [18314.249868] [] ? kthread_create_on_node+0x140/0x140 [18314.257799] ptlrpcd_01_00 S 000000000000f888 0 3474 2 0x00000080 [18314.266338] ffff880825bdbcf0 0000000000000046 ffff880825b14e70 ffff880825bdbfd8 [18314.275296] ffff880825bdbfd8 ffff880825bdbfd8 ffff880825b14e70 ffff8804b97a4000 [18314.284254] ffff880825bdbd20 000000010112dd50 ffff8804b97a4000 000000000000f888 [18314.293211] Call Trace: [18314.296586] [] schedule+0x29/0x70 [18314.302787] [] schedule_timeout+0x174/0x2c0 [18314.309944] [] ? internal_add_timer+0x70/0x70 [18314.317310] [] ptlrpcd+0x298/0x560 [ptlrpc] [18314.324453] [] ? wake_up_state+0x20/0x20 [18314.331331] [] ? ptlrpcd_check+0x5d0/0x5d0 [ptlrpc] [18314.339264] [] kthread+0xcf/0xe0 [18314.345349] [] ? kthread_create_on_node+0x140/0x140 [18314.353279] [] ret_from_fork+0x58/0x90 [18314.359944] [] ? kthread_create_on_node+0x140/0x140 [18314.367880] ptlrpcd_01_01 S 000000000000f888 0 3475 2 0x00000080 [18314.376422] ffff880825bdfcf0 0000000000000046 ffff880825b15e20 ffff880825bdffd8 [18314.385381] ffff880825bdffd8 ffff880825bdffd8 ffff880825b15e20 ffff8804b96b8000 [18314.394329] ffff880825bdfd20 000000010112dd50 ffff8804b96b8000 000000000000f888 [18314.403278] Call Trace: [18314.406659] [] schedule+0x29/0x70 [18314.412847] [] schedule_timeout+0x174/0x2c0 [18314.420001] [] ? internal_add_timer+0x70/0x70 [18314.427362] [] ptlrpcd+0x298/0x560 [ptlrpc] [18314.434510] [] ? wake_up_state+0x20/0x20 [18314.441393] [] ? ptlrpcd_check+0x5d0/0x5d0 [ptlrpc] [18314.449322] [] kthread+0xcf/0xe0 [18314.455404] [] ? kthread_create_on_node+0x140/0x140 [18314.463330] [] ret_from_fork+0x58/0x90 [18314.470006] [] ? kthread_create_on_node+0x140/0x140 [18314.477943] ptlrpcd_01_02 S 000000000000f888 0 3476 2 0x00000080 [18314.486505] ffff88082a463cf0 0000000000000046 ffff880825b16dd0 ffff88082a463fd8 [18314.495456] ffff88082a463fd8 ffff88082a463fd8 ffff880825b16dd0 ffff8804b97a4000 [18314.504417] ffff88082a463d20 000000010112dd90 ffff8804b97a4000 000000000000f888 [18314.513360] Call Trace: [18314.516740] [] schedule+0x29/0x70 [18314.522929] [] schedule_timeout+0x174/0x2c0 [18314.530089] [] ? internal_add_timer+0x70/0x70 [18314.537455] [] ptlrpcd+0x298/0x560 [ptlrpc] [18314.544615] [] ? wake_up_state+0x20/0x20 [18314.551499] [] ? ptlrpcd_check+0x5d0/0x5d0 [ptlrpc] [18314.559419] [] kthread+0xcf/0xe0 [18314.565510] [] ? kthread_create_on_node+0x140/0x140 [18314.573432] [] ret_from_fork+0x58/0x90 [18314.580111] [] ? kthread_create_on_node+0x140/0x140 [18314.588052] ptlrpcd_01_03 S 000000000000f888 0 3477 2 0x00000080 [18314.596610] ffff88082a467cf0 0000000000000046 ffff88082a468000 ffff88082a467fd8 [18314.605586] ffff88082a467fd8 ffff88082a467fd8 ffff88082a468000 ffff8804b90a4000 [18314.614544] ffff88082a467d20 000000010112df60 ffff8804b90a4000 000000000000f888 [18314.623498] Call Trace: [18314.626878] [] schedule+0x29/0x70 [18314.633072] [] schedule_timeout+0x174/0x2c0 [18314.640211] [] ? internal_add_timer+0x70/0x70 [18314.647580] [] ptlrpcd+0x298/0x560 [ptlrpc] [18314.654740] [] ? wake_up_state+0x20/0x20 [18314.661620] [] ? ptlrpcd_check+0x5d0/0x5d0 [ptlrpc] [18314.669546] [] kthread+0xcf/0xe0 [18314.675629] [] ? kthread_create_on_node+0x140/0x140 [18314.683554] [] ret_from_fork+0x58/0x90 [18314.690222] [] ? kthread_create_on_node+0x140/0x140 [18314.698147] ptlrpcd_01_04 S 000000000000f888 0 3478 2 0x00000080 [18314.706695] ffff8804b93ebcf0 0000000000000046 ffff88082a468fb0 ffff8804b93ebfd8 [18314.715664] ffff8804b93ebfd8 ffff8804b93ebfd8 ffff88082a468fb0 ffff8804b9688000 [18314.724612] ffff8804b93ebd20 000000010112e090 ffff8804b9688000 000000000000f888 [18314.733567] Call Trace: [18314.736931] [] schedule+0x29/0x70 [18314.743114] [] schedule_timeout+0x174/0x2c0 [18314.750254] [] ? internal_add_timer+0x70/0x70 [18314.757617] [] ptlrpcd+0x298/0x560 [ptlrpc] [18314.764766] [] ? wake_up_state+0x20/0x20 [18314.771653] [] ? ptlrpcd_check+0x5d0/0x5d0 [ptlrpc] [18314.779580] [] kthread+0xcf/0xe0 [18314.785659] [] ? kthread_create_on_node+0x140/0x140 [18314.793582] [] ret_from_fork+0x58/0x90 [18314.800249] [] ? kthread_create_on_node+0x140/0x140 [18314.808169] ptlrpcd_01_05 S 000000000000f888 0 3479 2 0x00000080 [18314.816740] ffff8804b93efcf0 0000000000000046 ffff88082a469f60 ffff8804b93effd8 [18314.825693] ffff8804b93effd8 ffff8804b93effd8 ffff88082a469f60 ffff8804b9770000 [18314.834656] ffff8804b93efd20 000000010112e090 ffff8804b9770000 000000000000f888 [18314.843617] Call Trace: [18314.846995] [] schedule+0x29/0x70 [18314.853176] [] schedule_timeout+0x174/0x2c0 [18314.860319] [] ? internal_add_timer+0x70/0x70 [18314.867688] [] ptlrpcd+0x298/0x560 [ptlrpc] [18314.874840] [] ? wake_up_state+0x20/0x20 [18314.881716] [] ? ptlrpcd_check+0x5d0/0x5d0 [ptlrpc] [18314.889641] [] kthread+0xcf/0xe0 [18314.895723] [] ? kthread_create_on_node+0x140/0x140 [18314.903648] [] ret_from_fork+0x58/0x90 [18314.910309] [] ? kthread_create_on_node+0x140/0x140 [18314.918217] ptlrpcd_01_06 S 000000000000f888 0 3480 2 0x00000080 [18314.926773] ffff88082672bcf0 0000000000000046 ffff88082a46af10 ffff88082672bfd8 [18314.935747] ffff88082672bfd8 ffff88082672bfd8 ffff88082a46af10 ffff8804b97a4000 [18314.944699] ffff88082672bd20 000000010112e038 ffff8804b97a4000 000000000000f888 [18314.953652] Call Trace: [18314.957019] [] schedule+0x29/0x70 [18314.963209] [] schedule_timeout+0x174/0x2c0 [18314.970350] [] ? internal_add_timer+0x70/0x70 [18314.977712] [] ptlrpcd+0x298/0x560 [ptlrpc] [18314.984867] [] ? wake_up_state+0x20/0x20 [18314.991748] [] ? ptlrpcd_check+0x5d0/0x5d0 [ptlrpc] [18314.999683] [] kthread+0xcf/0xe0 [18315.005769] [] ? kthread_create_on_node+0x140/0x140 [18315.013698] [] ret_from_fork+0x58/0x90 [18315.020363] [] ? kthread_create_on_node+0x140/0x140 [18315.028289] ptlrpcd_01_07 S 000000000000f888 0 3481 2 0x00000080 [18315.036846] ffff88082672fcf0 0000000000000046 ffff88082a46bec0 ffff88082672ffd8 [18315.045808] ffff88082672ffd8 ffff88082672ffd8 ffff88082a46bec0 ffff8804b9770000 [18315.054774] ffff88082672fd20 000000010112e048 ffff8804b9770000 000000000000f888 [18315.063738] Call Trace: [18315.067118] [] schedule+0x29/0x70 [18315.073297] [] schedule_timeout+0x174/0x2c0 [18315.080439] [] ? internal_add_timer+0x70/0x70 [18315.087803] [] ptlrpcd+0x298/0x560 [ptlrpc] [18315.094952] [] ? wake_up_state+0x20/0x20 [18315.101842] [] ? ptlrpcd_check+0x5d0/0x5d0 [ptlrpc] [18315.109775] [] kthread+0xcf/0xe0 [18315.115864] [] ? kthread_create_on_node+0x140/0x140 [18315.123809] [] ret_from_fork+0x58/0x90 [18315.130466] [] ? kthread_create_on_node+0x140/0x140 [18315.138390] ptlrpcd_01_08 S 000000000000f888 0 3482 2 0x00000080 [18315.146948] ffff880827723cf0 0000000000000046 ffff88082a46ce70 ffff880827723fd8 [18315.155924] ffff880827723fd8 ffff880827723fd8 ffff88082a46ce70 ffff8804b97d8000 [18315.164886] ffff880827723d20 000000010112e1c8 ffff8804b97d8000 000000000000f888 [18315.173839] Call Trace: [18315.177205] [] schedule+0x29/0x70 [18315.183388] [] schedule_timeout+0x174/0x2c0 [18315.190541] [] ? internal_add_timer+0x70/0x70 [18315.197917] [] ptlrpcd+0x298/0x560 [ptlrpc] [18315.205073] [] ? wake_up_state+0x20/0x20 [18315.211951] [] ? ptlrpcd_check+0x5d0/0x5d0 [ptlrpc] [18315.219886] [] kthread+0xcf/0xe0 [18315.225969] [] ? kthread_create_on_node+0x140/0x140 [18315.233897] [] ret_from_fork+0x58/0x90 [18315.240572] [] ? kthread_create_on_node+0x140/0x140 [18315.248504] ptlrpcd_01_09 S 000000000000f888 0 3483 2 0x00000080 [18315.257056] ffff880827727cf0 0000000000000046 ffff88082a46de20 ffff880827727fd8 [18315.266018] ffff880827727fd8 ffff880827727fd8 ffff88082a46de20 ffff8804b91b0000 [18315.274986] ffff880827727d20 000000010112e1c8 ffff8804b91b0000 000000000000f888 [18315.283947] Call Trace: [18315.287327] [] schedule+0x29/0x70 [18315.293517] [] schedule_timeout+0x174/0x2c0 [18315.300662] [] ? internal_add_timer+0x70/0x70 [18315.308035] [] ptlrpcd+0x298/0x560 [ptlrpc] [18315.315185] [] ? wake_up_state+0x20/0x20 [18315.322081] [] ? ptlrpcd_check+0x5d0/0x5d0 [ptlrpc] [18315.330010] [] kthread+0xcf/0xe0 [18315.336096] [] ? kthread_create_on_node+0x140/0x140 [18315.344033] [] ret_from_fork+0x58/0x90 [18315.350696] [] ? kthread_create_on_node+0x140/0x140 [18315.358621] ptlrpcd_01_10 S 000000000000f888 0 3484 2 0x00000080 [18315.367164] ffff88082772bcf0 0000000000000046 ffff88082a46edd0 ffff88082772bfd8 [18315.376127] ffff88082772bfd8 ffff88082772bfd8 ffff88082a46edd0 ffff8804b97a4000 [18315.385083] ffff88082772bd20 000000010112e3c8 ffff8804b97a4000 000000000000f888 [18315.394037] Call Trace: [18315.397406] [] schedule+0x29/0x70 [18315.403597] [] schedule_timeout+0x174/0x2c0 [18315.410751] [] ? internal_add_timer+0x70/0x70 [18315.418115] [] ptlrpcd+0x298/0x560 [ptlrpc] [18315.425269] [] ? wake_up_state+0x20/0x20 [18315.432149] [] ? ptlrpcd_check+0x5d0/0x5d0 [ptlrpc] [18315.440088] [] kthread+0xcf/0xe0 [18315.446163] [] ? kthread_create_on_node+0x140/0x140 [18315.454110] [] ret_from_fork+0x58/0x90 [18315.460771] [] ? kthread_create_on_node+0x140/0x140 [18315.468692] ptlrpcd_01_11 S 000000000000f888 0 3485 2 0x00000080 [18315.477229] ffff88082772fcf0 0000000000000046 ffff880826e20000 ffff88082772ffd8 [18315.486203] ffff88082772ffd8 ffff88082772ffd8 ffff880826e20000 ffff8804b9068000 [18315.495159] ffff88082772fd20 000000010112e3c8 ffff8804b9068000 000000000000f888 [18315.504112] Call Trace: [18315.507477] [] schedule+0x29/0x70 [18315.513667] [] schedule_timeout+0x174/0x2c0 [18315.520815] [] ? internal_add_timer+0x70/0x70 [18315.528178] [] ptlrpcd+0x298/0x560 [ptlrpc] [18315.535324] [] ? wake_up_state+0x20/0x20 [18315.542201] [] ? ptlrpcd_check+0x5d0/0x5d0 [ptlrpc] [18315.550129] [] kthread+0xcf/0xe0 [18315.556217] [] ? kthread_create_on_node+0x140/0x140 [18315.564147] [] ret_from_fork+0x58/0x90 [18315.570810] [] ? kthread_create_on_node+0x140/0x140 [18315.578743] ptlrpcd_01_12 S 000000000000f888 0 3486 2 0x00000080 [18315.587294] ffff880826e2bcf0 0000000000000046 ffff880826e20fb0 ffff880826e2bfd8 [18315.596259] ffff880826e2bfd8 ffff880826e2bfd8 ffff880826e20fb0 ffff8804b97d8000 [18315.605205] ffff880826e2bd20 000000010112e3c8 ffff8804b97d8000 000000000000f888 [18315.614153] Call Trace: [18315.617523] [] schedule+0x29/0x70 [18315.623709] [] schedule_timeout+0x174/0x2c0 [18315.630850] [] ? internal_add_timer+0x70/0x70 [18315.638205] [] ptlrpcd+0x298/0x560 [ptlrpc] [18315.645345] [] ? wake_up_state+0x20/0x20 [18315.652218] [] ? ptlrpcd_check+0x5d0/0x5d0 [ptlrpc] [18315.660148] [] kthread+0xcf/0xe0 [18315.666228] [] ? kthread_create_on_node+0x140/0x140 [18315.674152] [] ret_from_fork+0x58/0x90 [18315.680823] [] ? kthread_create_on_node+0x140/0x140 [18315.688754] ptlrpcd_01_13 S 000000000000f888 0 3487 2 0x00000080 [18315.697296] ffff880826e2fcf0 0000000000000046 ffff880826e21f60 ffff880826e2ffd8 [18315.706253] ffff880826e2ffd8 ffff880826e2ffd8 ffff880826e21f60 ffff8804b96ec000 [18315.715209] ffff880826e2fd20 000000010112e4f8 ffff8804b96ec000 000000000000f888 [18315.724161] Call Trace: [18315.727531] [] schedule+0x29/0x70 [18315.733714] [] schedule_timeout+0x174/0x2c0 [18315.740866] [] ? internal_add_timer+0x70/0x70 [18315.748229] [] ptlrpcd+0x298/0x560 [ptlrpc] [18315.755375] [] ? wake_up_state+0x20/0x20 [18315.762247] [] ? ptlrpcd_check+0x5d0/0x5d0 [ptlrpc] [18315.770186] [] kthread+0xcf/0xe0 [18315.776259] [] ? kthread_create_on_node+0x140/0x140 [18315.784194] [] ret_from_fork+0x58/0x90 [18315.790859] [] ? kthread_create_on_node+0x140/0x140 [18315.798793] ptlrpcd_01_14 S 000000000000f888 0 3488 2 0x00000080 [18315.807329] ffff880827b63cf0 0000000000000046 ffff880826e22f10 ffff880827b63fd8 [18315.816291] ffff880827b63fd8 ffff880827b63fd8 ffff880826e22f10 ffff8804b90d0000 [18315.825249] ffff880827b63d20 000000010112e510 ffff8804b90d0000 000000000000f888 [18315.834201] Call Trace: [18315.837570] [] schedule+0x29/0x70 [18315.843774] [] schedule_timeout+0x174/0x2c0 [18315.850929] [] ? internal_add_timer+0x70/0x70 [18315.858290] [] ptlrpcd+0x298/0x560 [ptlrpc] [18315.865437] [] ? wake_up_state+0x20/0x20 [18315.872315] [] ? ptlrpcd_check+0x5d0/0x5d0 [ptlrpc] [18315.880240] [] kthread+0xcf/0xe0 [18315.886320] [] ? kthread_create_on_node+0x140/0x140 [18315.894241] [] ret_from_fork+0x58/0x90 [18315.900897] [] ? kthread_create_on_node+0x140/0x140 [18315.908825] ptlrpcd_01_15 S 000000000000f888 0 3489 2 0x00000080 [18315.917353] ffff880827b67cf0 0000000000000046 ffff880826e23ec0 ffff880827b67fd8 [18315.926324] ffff880827b67fd8 ffff880827b67fd8 ffff880826e23ec0 ffff8804b9740000 [18315.935278] ffff880827b67d20 000000010112e475 ffff8804b9740000 000000000000f888 [18315.944227] Call Trace: [18315.947605] [] schedule+0x29/0x70 [18315.953797] [] schedule_timeout+0x174/0x2c0 [18315.960947] [] ? internal_add_timer+0x70/0x70 [18315.968308] [] ptlrpcd+0x298/0x560 [ptlrpc] [18315.975456] [] ? wake_up_state+0x20/0x20 [18315.982339] [] ? ptlrpcd_check+0x5d0/0x5d0 [ptlrpc] [18315.990271] [] kthread+0xcf/0xe0 [18315.996354] [] ? kthread_create_on_node+0x140/0x140 [18316.004282] [] ret_from_fork+0x58/0x90 [18316.010959] [] ? kthread_create_on_node+0x140/0x140 [18316.018897] ll_ping S 000000000000f888 0 3490 2 0x00000080 [18316.027450] ffff880827b6bd20 0000000000000046 ffff880826e24e70 ffff880827b6bfd8 [18316.036393] ffff880827b6bfd8 ffff880827b6bfd8 ffff880826e24e70 ffff8801796e4000 [18316.045351] ffff880827b6bd50 0000000101132dac ffff8801796e4000 000000000000f888 [18316.054311] Call Trace: [18316.057679] [] schedule+0x29/0x70 [18316.063875] [] schedule_timeout+0x174/0x2c0 [18316.071024] [] ? internal_add_timer+0x70/0x70 [18316.078389] [] ptlrpc_pinger_main+0x92c/0xbb0 [ptlrpc] [18316.086618] [] ? wake_up_state+0x20/0x20 [18316.093503] [] ? ptlrpc_obd_ping+0x130/0x130 [ptlrpc] [18316.101629] [] kthread+0xcf/0xe0 [18316.107728] [] ? kthread_create_on_node+0x140/0x140 [18316.115674] [] ret_from_fork+0x58/0x90 [18316.122340] [] ? kthread_create_on_node+0x140/0x140 [18316.130273] sptlrpc_gc S 000000000000f888 0 3491 2 0x00000080 [18316.138848] ffff880827b6fd70 0000000000000046 ffff880826e25e20 ffff880827b6ffd8 [18316.147835] ffff880827b6ffd8 ffff880827b6ffd8 ffff880826e25e20 ffff8804b9100000 [18316.156820] ffff880827b6fda0 00000001012ae3bc ffff8804b9100000 000000000000f888 [18316.165786] Call Trace: [18316.169161] [] schedule+0x29/0x70 [18316.175345] [] schedule_timeout+0x174/0x2c0 [18316.182490] [] ? internal_add_timer+0x70/0x70 [18316.189850] [] ? cfs_block_sigsinv+0x71/0xa0 [libcfs] [18316.198014] [] sec_gc_main+0x2e5/0x420 [ptlrpc] [18316.205573] [] ? wake_up_state+0x20/0x20 [18316.212468] [] ? sptlrpc_gc_add_ctx+0x130/0x130 [ptlrpc] [18316.220902] [] kthread+0xcf/0xe0 [18316.227008] [] ? kthread_create_on_node+0x140/0x140 [18316.234958] [] ret_from_fork+0x58/0x90 [18316.241636] [] ? kthread_create_on_node+0x140/0x140 [18316.249586] kmmpd-dm-5 S 000000000000f888 0 3492 2 0x00000080 [18316.258169] ffff880826b67d60 0000000000000046 ffff880826e26dd0 ffff880826b67fd8 [18316.267135] ffff880826b67fd8 ffff880826b67fd8 ffff880826e26dd0 ffff880179130000 [18316.276117] ffff880826b67d90 000000010112e6d4 ffff880179130000 000000000000f888 [18316.285103] Call Trace: [18316.288479] [] schedule+0x29/0x70 [18316.294669] [] schedule_timeout+0x174/0x2c0 [18316.301837] [] ? internal_add_timer+0x70/0x70 [18316.309209] [] ? write_mmp_block+0x132/0x170 [ldiskfs] [18316.317440] [] schedule_timeout_interruptible+0x1e/0x20 [18316.325789] [] kmmpd+0x216/0x430 [ldiskfs] [18316.332862] [] ? __schedule+0x3b0/0x990 [18316.339649] [] ? __dump_mmp_msg+0x70/0x70 [ldiskfs] [18316.347611] [] kthread+0xcf/0xe0 [18316.353710] [] ? kthread_create_on_node+0x140/0x140 [18316.361653] [] ret_from_fork+0x58/0x90 [18316.368338] [] ? kthread_create_on_node+0x140/0x140 [18316.376275] jbd2/dm-5-8 S ffff88042b897800 0 3493 2 0x00000080 [18316.384856] ffff88082bcc7e30 0000000000000046 ffff88082d210000 ffff88082bcc7fd8 [18316.393843] ffff88082bcc7fd8 ffff88082bcc7fd8 ffff88082d210000 ffff88042b897828 [18316.402822] ffff88042b897be8 ffff88042b8978a8 ffff88082bcc7e80 ffff88042b897800 [18316.411796] Call Trace: [18316.415183] [] schedule+0x29/0x70 [18316.421383] [] kjournald2+0x203/0x260 [jbd2] [18316.428649] [] ? wake_up_atomic_t+0x30/0x30 [18316.435831] [] ? commit_timeout+0x10/0x10 [jbd2] [18316.443480] [] kthread+0xcf/0xe0 [18316.449575] [] ? kthread_create_on_node+0x140/0x140 [18316.457503] [] ret_from_fork+0x58/0x90 [18316.464192] [] ? kthread_create_on_node+0x140/0x140 [18316.472138] ldiskfs-rsv-con S ffff88082ab2e800 0 3494 2 0x00000080 [18316.480706] ffff88082c943e08 0000000000000046 ffff88082d210fb0 ffff88082c943fd8 [18316.489675] ffff88082c943fd8 ffff88082c943fd8 ffff88082d210fb0 ffff88082d210fb0 [18316.498637] ffff88082ab2e800 ffff88082aba9e80 ffff88082ab2e830 ffff88082ab2e800 [18316.507612] Call Trace: [18316.511001] [] schedule+0x29/0x70 [18316.517189] [] rescuer_thread+0x359/0x460 [18316.524172] [] ? __schedule+0x3b0/0x990 [18316.530943] [] ? manage_workers.isra.24+0x2d0/0x2d0 [18316.538888] [] kthread+0xcf/0xe0 [18316.544976] [] ? kthread_create_on_node+0x140/0x140 [18316.552915] [] ret_from_fork+0x58/0x90 [18316.559589] [] ? kthread_create_on_node+0x140/0x140 [18316.567520] ldlm_cb00_000 S ffff88042d674c00 0 3498 2 0x00000080 [18316.576068] ffff8808253dfd38 0000000000000046 ffff88082d211f60 ffff8808253dffd8 [18316.585053] ffff8808253dffd8 ffff8808253dffd8 ffff88082d211f60 0000000000000000 [18316.594033] ffff88082d211f60 ffff8804036c0080 ffff88042d674c68 ffff88042d674c00 [18316.603011] Call Trace: [18316.606390] [] schedule+0x29/0x70 [18316.612612] [] ptlrpc_wait_event+0x325/0x340 [ptlrpc] [18316.620753] [] ? wake_up_state+0x20/0x20 [18316.627654] [] ptlrpc_main+0x80b/0x1dd0 [ptlrpc] [18316.635302] [] ? __switch_to+0xd9/0x4c0 [18316.642076] [] ? __schedule+0x3b0/0x990 [18316.648882] [] ? ptlrpc_register_service+0xe60/0xe60 [ptlrpc] [18316.657807] [] kthread+0xcf/0xe0 [18316.663910] [] ? kthread_create_on_node+0x140/0x140 [18316.671858] [] ret_from_fork+0x58/0x90 [18316.678527] [] ? kthread_create_on_node+0x140/0x140 [18316.686469] lc_watchdogd S ffff88082d212f10 0 3499 2 0x00000080 [18316.695048] ffff8808253e3e18 0000000000000046 ffff88082d212f10 ffff8808253e3fd8 [18316.704037] ffff8808253e3fd8 ffff8808253e3fd8 ffff88082d212f10 ffffffffa0bbaed0 [18316.713023] 0000000000000000 ffffffffa0b9e910 0000000000000000 ffff88082d212f10 [18316.722025] Call Trace: [18316.725422] [] ? lc_watchdog_add+0x790/0x790 [libcfs] [18316.733585] [] schedule+0x29/0x70 [18316.739802] [] lcw_dispatch_main+0x799/0x990 [libcfs] [18316.747959] [] ? wake_up_atomic_t+0x30/0x30 [18316.755166] [] ? lc_watchdog_add+0x790/0x790 [libcfs] [18316.763320] [] kthread+0xcf/0xe0 [18316.769433] [] ? kthread_create_on_node+0x140/0x140 [18316.777387] [] ret_from_fork+0x58/0x90 [18316.784097] [] ? kthread_create_on_node+0x140/0x140 [18316.792069] ldlm_cb00_001 S ffff88042d674c00 0 3500 2 0x00000080 [18316.800654] ffff88082bf6fd38 0000000000000046 ffff88082d213ec0 ffff88082bf6ffd8 [18316.809645] ffff88082bf6ffd8 ffff88082bf6ffd8 ffff88082d213ec0 0000000000000000 [18316.818646] ffff88082d213ec0 ffff8804036c0580 ffff88042d674c68 ffff88042d674c00 [18316.827633] Call Trace: [18316.831037] [] schedule+0x29/0x70 [18316.837280] [] ptlrpc_wait_event+0x325/0x340 [ptlrpc] [18316.845446] [] ? wake_up_state+0x20/0x20 [18316.852380] [] ptlrpc_main+0x80b/0x1dd0 [ptlrpc] [18316.860059] [] ? __switch_to+0xd9/0x4c0 [18316.866898] [] ? ptlrpc_register_service+0xe60/0xe60 [ptlrpc] [18316.875844] [] kthread+0xcf/0xe0 [18316.881986] [] ? kthread_create_on_node+0x140/0x140 [18316.889975] [] ret_from_fork+0x58/0x90 [18316.896694] [] ? kthread_create_on_node+0x140/0x140 [18316.904688] ldlm_cb01_000 S ffff880827238000 0 3501 2 0x00000080 [18316.913280] ffff88082bed3d38 0000000000000046 ffff88082d214e70 ffff88082bed3fd8 [18316.922304] ffff88082bed3fd8 ffff88082bed3fd8 ffff88082d214e70 0000000000000000 [18316.931322] ffff88082d214e70 ffff88082ca28b80 ffff880827238068 ffff880827238000 [18316.940324] Call Trace: [18316.943748] [] schedule+0x29/0x70 [18316.950022] [] ptlrpc_wait_event+0x325/0x340 [ptlrpc] [18316.958214] [] ? wake_up_state+0x20/0x20 [18316.965159] [] ptlrpc_main+0x80b/0x1dd0 [ptlrpc] [18316.972875] [] ? ptlrpc_register_service+0xe60/0xe60 [ptlrpc] [18316.981845] [] kthread+0xcf/0xe0 [18316.987994] [] ? kthread_create_on_node+0x140/0x140 [18316.995980] [] ret_from_fork+0x58/0x90 [18317.002708] [] ? kthread_create_on_node+0x140/0x140 [18317.010695] ldlm_cb01_001 S ffff880827238000 0 3502 2 0x00000080 [18317.019289] ffff88082aaf3d38 0000000000000046 ffff88082d215e20 ffff88082aaf3fd8 [18317.028317] ffff88082aaf3fd8 ffff88082aaf3fd8 ffff88082d215e20 0000000000000000 [18317.037345] ffff88082d215e20 ffff88082ca28980 ffff880827238068 ffff880827238000 [18317.046372] Call Trace: [18317.049805] [] schedule+0x29/0x70 [18317.056082] [] ptlrpc_wait_event+0x325/0x340 [ptlrpc] [18317.064268] [] ? wake_up_state+0x20/0x20 [18317.071228] [] ptlrpc_main+0x80b/0x1dd0 [ptlrpc] [18317.078945] [] ? ptlrpc_register_service+0xe60/0xe60 [ptlrpc] [18317.087908] [] kthread+0xcf/0xe0 [18317.094055] [] ? kthread_create_on_node+0x140/0x140 [18317.102042] [] ret_from_fork+0x58/0x90 [18317.108756] [] ? kthread_create_on_node+0x140/0x140 [18317.116744] ldlm_cn00_000 S ffff88042d677c00 0 3503 2 0x00000080 [18317.125349] ffff88082c7dfd38 0000000000000046 ffff88082d216dd0 ffff88082c7dffd8 [18317.134371] ffff88082c7dffd8 ffff88082c7dffd8 ffff88082d216dd0 0000000000000000 [18317.143397] ffff88082d216dd0 ffff8804036c0600 ffff88042d677c68 ffff88042d677c00 [18317.152423] Call Trace: [18317.155857] [] schedule+0x29/0x70 [18317.162115] [] ptlrpc_wait_event+0x325/0x340 [ptlrpc] [18317.170295] [] ? wake_up_state+0x20/0x20 [18317.177237] [] ptlrpc_main+0x80b/0x1dd0 [ptlrpc] [18317.184950] [] ? ptlrpc_register_service+0xe60/0xe60 [ptlrpc] [18317.193901] [] kthread+0xcf/0xe0 [18317.200050] [] ? kthread_create_on_node+0x140/0x140 [18317.208042] [] ret_from_fork+0x58/0x90 [18317.214764] [] ? kthread_create_on_node+0x140/0x140 [18317.222758] ldlm_cn00_001 S ffff88042d677c00 0 3504 2 0x00000080 [18317.231348] ffff88083f6ebd38 0000000000000046 ffff880826af8000 ffff88083f6ebfd8 [18317.240370] ffff88083f6ebfd8 ffff88083f6ebfd8 ffff880826af8000 0000000000000000 [18317.249387] ffff880826af8000 ffff8804036c0000 ffff88042d677c68 ffff88042d677c00 [18317.258397] Call Trace: [18317.261837] [] schedule+0x29/0x70 [18317.268114] [] ptlrpc_wait_event+0x325/0x340 [ptlrpc] [18317.276291] [] ? wake_up_state+0x20/0x20 [18317.283246] [] ptlrpc_main+0x80b/0x1dd0 [ptlrpc] [18317.290968] [] ? ptlrpc_register_service+0xe60/0xe60 [ptlrpc] [18317.299933] [] kthread+0xcf/0xe0 [18317.306087] [] ? kthread_create_on_node+0x140/0x140 [18317.314078] [] ret_from_fork+0x58/0x90 [18317.320806] [] ? kthread_create_on_node+0x140/0x140 [18317.328794] ldlm_cn00_002 S ffff88042d677c00 0 3505 2 0x00000080 [18317.337379] ffff880826a83d38 0000000000000046 ffff880826af8fb0 ffff880826a83fd8 [18317.346401] ffff880826a83fd8 ffff880826a83fd8 ffff880826af8fb0 0000000000000000 [18317.355418] ffff880826af8fb0 ffff8804036c0880 ffff88042d677c68 ffff88042d677c00 [18317.364433] Call Trace: [18317.367875] [] schedule+0x29/0x70 [18317.374151] [] ptlrpc_wait_event+0x325/0x340 [ptlrpc] [18317.382339] [] ? wake_up_state+0x20/0x20 [18317.389287] [] ptlrpc_main+0x80b/0x1dd0 [ptlrpc] [18317.397009] [] ? ptlrpc_register_service+0xe60/0xe60 [ptlrpc] [18317.405974] [] kthread+0xcf/0xe0 [18317.412130] [] ? kthread_create_on_node+0x140/0x140 [18317.420112] [] ret_from_fork+0x58/0x90 [18317.426828] [] ? kthread_create_on_node+0x140/0x140 [18317.434823] ldlm_cn01_000 S ffff88082723fc00 0 3506 2 0x00000080 [18317.443421] ffff880826a87d38 0000000000000046 ffff880826af9f60 ffff880826a87fd8 [18317.452440] ffff880826a87fd8 ffff880826a87fd8 ffff880826af9f60 0000000000000000 [18317.461461] ffff880826af9f60 ffff88082ca28800 ffff88082723fc68 ffff88082723fc00 [18317.470471] Call Trace: [18317.473893] [] schedule+0x29/0x70 [18317.480174] [] ptlrpc_wait_event+0x325/0x340 [ptlrpc] [18317.488353] [] ? wake_up_state+0x20/0x20 [18317.495292] [] ptlrpc_main+0x80b/0x1dd0 [ptlrpc] [18317.503014] [] ? ptlrpc_register_service+0xe60/0xe60 [ptlrpc] [18317.511979] [] kthread+0xcf/0xe0 [18317.518137] [] ? kthread_create_on_node+0x140/0x140 [18317.526145] [] ret_from_fork+0x58/0x90 [18317.532856] [] ? kthread_create_on_node+0x140/0x140 [18317.540839] ldlm_cn01_001 S ffff88082723fc00 0 3507 2 0x00000080 [18317.549434] ffff880826a8bd38 0000000000000046 ffff880826afaf10 ffff880826a8bfd8 [18317.558450] ffff880826a8bfd8 ffff880826a8bfd8 ffff880826afaf10 0000000000000000 [18317.567464] ffff880826afaf10 ffff88082ca29200 ffff88082723fc68 ffff88082723fc00 [18317.576469] Call Trace: [18317.579890] [] schedule+0x29/0x70 [18317.586153] [] ptlrpc_wait_event+0x325/0x340 [ptlrpc] [18317.594321] [] ? wake_up_state+0x20/0x20 [18317.601277] [] ptlrpc_main+0x80b/0x1dd0 [ptlrpc] [18317.609003] [] ? ptlrpc_register_service+0xe60/0xe60 [ptlrpc] [18317.617953] [] kthread+0xcf/0xe0 [18317.624100] [] ? finish_task_switch+0x53/0x180 [18317.631600] [] ? kthread_create_on_node+0x140/0x140 [18317.639579] [] ret_from_fork+0x58/0x90 [18317.646298] [] ? kthread_create_on_node+0x140/0x140 [18317.654279] ldlm_cn01_002 S ffff88082723fc00 0 3508 2 0x00000080 [18317.662883] ffff880826a8fd38 0000000000000046 ffff880826afbec0 ffff880826a8ffd8 [18317.671909] ffff880826a8ffd8 ffff880826a8ffd8 ffff880826afbec0 0000000000000000 [18317.680931] ffff880826afbec0 ffff88082ca29480 ffff88082723fc68 ffff88082723fc00 [18317.689952] Call Trace: [18317.693369] [] schedule+0x29/0x70 [18317.699630] [] ptlrpc_wait_event+0x325/0x340 [ptlrpc] [18317.707809] [] ? wake_up_state+0x20/0x20 [18317.714758] [] ptlrpc_main+0x80b/0x1dd0 [ptlrpc] [18317.722467] [] ? ptlrpc_register_service+0xe60/0xe60 [ptlrpc] [18317.731420] [] kthread+0xcf/0xe0 [18317.737554] [] ? kthread_create_on_node+0x140/0x140 [18317.745533] [] ret_from_fork+0x58/0x90 [18317.752249] [] ? kthread_create_on_node+0x140/0x140 [18317.760228] ldlm_bl_01 S ffff88082ca291a8 0 3509 2 0x00000080 [18317.768823] ffff880826a93e08 0000000000000046 ffff880826afce70 ffff880826a93fd8 [18317.777824] ffff880826a93fd8 ffff880826a93fd8 ffff880826afce70 ffff88082ca29180 [18317.786846] ffff880826afce70 ffff88082ca291e4 0000000000000000 ffff88082ca291a8 [18317.795872] Call Trace: [18317.799284] [] schedule+0x29/0x70 [18317.805531] [] ldlm_bl_thread_main+0x42d/0x700 [ptlrpc] [18317.813901] [] ? wake_up_state+0x20/0x20 [18317.820827] [] ? ldlm_handle_bl_callback+0x410/0x410 [ptlrpc] [18317.829786] [] kthread+0xcf/0xe0 [18317.835915] [] ? kthread_create_on_node+0x140/0x140 [18317.843894] [] ret_from_fork+0x58/0x90 [18317.850609] [] ? kthread_create_on_node+0x140/0x140 [18317.858587] ldlm_bl_02 S ffff88082ca291a8 0 3510 2 0x00000080 [18317.867197] ffff880826a97e08 0000000000000046 ffff880826afde20 ffff880826a97fd8 [18317.876225] ffff880826a97fd8 ffff880826a97fd8 ffff880826afde20 ffff88082ca29180 [18317.885241] ffff880826afde20 ffff88082ca291e4 0000000000000000 ffff88082ca291a8 [18317.894242] Call Trace: [18317.897663] [] schedule+0x29/0x70 [18317.903923] [] ldlm_bl_thread_main+0x42d/0x700 [ptlrpc] [18317.912284] [] ? wake_up_state+0x20/0x20 [18317.919216] [] ? ldlm_handle_bl_callback+0x410/0x410 [ptlrpc] [18317.928194] [] kthread+0xcf/0xe0 [18317.934318] [] ? kthread_create_on_node+0x140/0x140 [18317.942294] [] ret_from_fork+0x58/0x90 [18317.949013] [] ? kthread_create_on_node+0x140/0x140 [18317.956998] ldlm_elt S 0000000000000000 0 3511 2 0x00000080 [18317.965598] ffff880826a9be00 0000000000000046 ffff880826afedd0 ffff880826a9bfd8 [18317.974616] ffff880826a9bfd8 ffff880826a9bfd8 ffff880826afedd0 ffff880826afedd0 [18317.983619] 0000000000000000 ffffffffa0ed4a00 ffff880826afedd0 0000000000000000 [18317.992625] Call Trace: [18317.996071] [] ? ldlm_bl_to_thread+0x490/0x510 [ptlrpc] [18318.004451] [] schedule+0x29/0x70 [18318.010715] [] expired_lock_main+0x1c2/0x910 [ptlrpc] [18318.018900] [] ? wake_up_state+0x20/0x20 [18318.025825] [] ? ldlm_bl_to_thread+0x510/0x510 [ptlrpc] [18318.034207] [] kthread+0xcf/0xe0 [18318.040342] [] ? kthread_create_on_node+0x140/0x140 [18318.048323] [] ret_from_fork+0x58/0x90 [18318.055040] [] ? kthread_create_on_node+0x140/0x140 [18318.063031] ldlm_poold S 000000000000f888 0 3512 2 0x00000080 [18318.071641] ffff880826a9fd70 0000000000000046 ffff880827f20000 ffff880826a9ffd8 [18318.080662] ffff880826a9ffd8 ffff880826a9ffd8 ffff880827f20000 ffff8804b9688000 [18318.089672] ffff880826a9fda0 000000010112eec0 ffff8804b9688000 000000000000f888 [18318.098673] Call Trace: [18318.102096] [] schedule+0x29/0x70 [18318.108334] [] schedule_timeout+0x174/0x2c0 [18318.115538] [] ? internal_add_timer+0x70/0x70 [18318.122943] [] ? cfs_block_sigsinv+0x71/0xa0 [libcfs] [18318.131130] [] ldlm_pools_thread_main+0x23b/0x330 [ptlrpc] [18318.139797] [] ? wake_up_state+0x20/0x20 [18318.146730] [] ? ldlm_pools_recalc+0x3d0/0x3d0 [ptlrpc] [18318.155106] [] kthread+0xcf/0xe0 [18318.161247] [] ? kthread_create_on_node+0x140/0x140 [18318.169236] [] ret_from_fork+0x58/0x90 [18318.175948] [] ? kthread_create_on_node+0x140/0x140 [18318.183929] mgs_params_noti S ffff880827f20fb0 0 3513 2 0x00000080 [18318.192498] ffff880827f3bdd8 0000000000000046 ffff880827f20fb0 ffff880827f3bfd8 [18318.201495] ffff880827f3bfd8 ffff880827f3bfd8 ffff880827f20fb0 ffff880827f20fb0 [18318.210495] 0000000200000000 ffff88082abae328 ffff88082abae200 ffff880827f20fb0 [18318.219490] Call Trace: [18318.222903] [] schedule+0x29/0x70 [18318.229135] [] mgs_ir_notify+0x1c0/0x2b0 [mgs] [18318.236619] [] ? wake_up_state+0x20/0x20 [18318.243524] [] ? lprocfs_ir_set_state+0x170/0x170 [mgs] [18318.251877] [] kthread+0xcf/0xe0 [18318.257990] [] ? kthread_create_on_node+0x140/0x140 [18318.265946] [] ret_from_fork+0x58/0x90 [18318.272654] [] ? kthread_create_on_node+0x140/0x140 [18318.280613] ll_mgs_0000 S ffff8803fce6fc00 0 3514 2 0x00000080 [18318.289200] ffff880825af3d38 0000000000000046 ffff880827f21f60 ffff880825af3fd8 [18318.298209] ffff880825af3fd8 ffff880825af3fd8 ffff880827f21f60 0000000000000000 [18318.307221] ffff880827f21f60 ffff8804036c0680 ffff8803fce6fc68 ffff8803fce6fc00 [18318.316223] Call Trace: [18318.319642] [] schedule+0x29/0x70 [18318.319669] [] [ 0.000000] Initializing cgroup subsys cpuset [ 0.000000] Initializing cgroup subsys cpu [ 0.000000] Initializing cgroup subsys cpuacct