[ 147.203822] Lustre: soaked-MDT0000: Connection restored to 880972f7-970d-b37a-1a61-2887fd2e2d91 (at 192.168.1.142@o2ib100) [ 147.203825] Lustre: Skipped 6 previous similar messages [ 156.268838] Lustre: soaked-MDT0000: Connection restored to 192.168.1.110@o2ib10 (at 192.168.1.110@o2ib10) [ 156.279660] Lustre: Skipped 24 previous similar messages [ 173.233671] Lustre: MGS: Connection restored to 9665679f-f79b-1bf9-6fd6-abc25509729a (at 192.168.1.131@o2ib100) [ 173.245048] Lustre: Skipped 28 previous similar messages [ 190.236882] Lustre: soaked-MDT0000: Recovery over after 0:43, of 32 clients 32 recovered and 0 were evicted. [ 218.357311] Lustre: MGS: Connection restored to 192.168.1.105@o2ib10 (at 192.168.1.105@o2ib10) [ 218.368494] Lustre: Skipped 12 previous similar messages [ 806.665594] Lustre: soaked-MDT0000: Client 91eacee7-b83c-19ec-9326-47c5ebda930d (at 192.168.1.116@o2ib100) reconnecting [ 806.678906] Lustre: soaked-MDT0000: Connection restored to 91eacee7-b83c-19ec-9326-47c5ebda930d (at 192.168.1.116@o2ib100) [ 806.692353] Lustre: Skipped 9 previous similar messages [ 876.738766] Lustre: soaked-MDT0000: Client 60b5ead5-442a-039e-5d32-f9759a392209 (at 192.168.1.128@o2ib100) reconnecting [ 1020.276376] Lustre: soaked-MDT0000: Client e211cd09-6b0c-e43b-059c-f657101042a0 (at 192.168.1.117@o2ib100) reconnecting [ 1020.291053] Lustre: soaked-MDT0000: Connection restored to e211cd09-6b0c-e43b-059c-f657101042a0 (at 192.168.1.117@o2ib100) [ 1020.305862] Lustre: Skipped 1 previous similar message [ 1216.697961] Lustre: soaked-MDT0000: Client e211cd09-6b0c-e43b-059c-f657101042a0 (at 192.168.1.117@o2ib100) reconnecting [ 1239.749007] Lustre: soaked-MDT0000: Client 31859d53-53e2-75a2-da2d-75f6bdca7375 (at 192.168.1.118@o2ib100) reconnecting [ 1300.750770] Lustre: soaked-MDT0000: Client 6f6036ce-2e3a-7e95-1c07-11e5d5a627fe (at 192.168.1.124@o2ib100) reconnecting [ 1300.765260] Lustre: soaked-MDT0000: Connection restored to 5a0e6555-4c2d-39c9-90c4-a332f62c8353 (at 192.168.1.124@o2ib100) [ 1300.779749] Lustre: Skipped 2 previous similar messages [ 1385.754795] Lustre: soaked-MDT0000: Client d8128f9f-f4c7-b311-df64-40bd3b032886 (at 192.168.1.127@o2ib100) reconnecting [ 1469.707717] Lustre: soaked-MDT0000: Client e211cd09-6b0c-e43b-059c-f657101042a0 (at 192.168.1.117@o2ib100) reconnecting [ 1542.764014] Lustre: soaked-MDT0000: Client e211cd09-6b0c-e43b-059c-f657101042a0 (at 192.168.1.117@o2ib100) reconnecting [ 1742.697552] Lustre: soaked-MDT0000: Client e211cd09-6b0c-e43b-059c-f657101042a0 (at 192.168.1.117@o2ib100) reconnecting [ 1742.712158] Lustre: Skipped 2 previous similar messages [ 1818.700200] Lustre: soaked-MDT0000: Connection restored to e211cd09-6b0c-e43b-059c-f657101042a0 (at 192.168.1.117@o2ib100) [ 1818.714597] Lustre: Skipped 7 previous similar messages [ 1960.700811] Lustre: 3698:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1494355306/real 1494355306] req@ffff8808053c2d00 x1566943337184128/t0(0) o104->soaked-MDT0000@192.168.1.123@o2ib100:15/16 lens 296/224 e 0 to 1 dl 1494355313 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [ 2035.771561] Lustre: soaked-MDT0000: Client 54aab9cb-1b41-13ac-e1b9-f63d1e78e154 (at 192.168.1.137@o2ib100) reconnecting [ 2035.785133] Lustre: Skipped 4 previous similar messages [ 2540.719596] Lustre: soaked-MDT0000: Connection restored to e211cd09-6b0c-e43b-059c-f657101042a0 (at 192.168.1.117@o2ib100) [ 2540.735126] Lustre: Skipped 5 previous similar messages [ 2588.807156] Lustre: soaked-MDT0000: Client e81e92b7-aede-6d47-4552-8475e879b939 (at 192.168.1.134@o2ib100) reconnecting [ 2588.822296] Lustre: Skipped 3 previous similar messages [ 3168.746656] Lustre: soaked-MDT0000: Connection restored to e211cd09-6b0c-e43b-059c-f657101042a0 (at 192.168.1.117@o2ib100) [ 3168.761959] Lustre: Skipped 8 previous similar messages [ 3271.274472] Lustre: soaked-MDT0000: Client a60139c3-b36a-37dd-9360-e306d02df1da (at 192.168.1.131@o2ib100) reconnecting [ 3271.289131] Lustre: Skipped 8 previous similar messages [ 4071.250600] Lustre: soaked-MDT0000: Client e211cd09-6b0c-e43b-059c-f657101042a0 (at 192.168.1.117@o2ib100) reconnecting [ 4071.265568] Lustre: Skipped 3 previous similar messages [ 4071.274511] Lustre: soaked-MDT0000: Connection restored to e211cd09-6b0c-e43b-059c-f657101042a0 (at 192.168.1.117@o2ib100) [ 4071.289924] Lustre: Skipped 4 previous similar messages [ 4107.234610] Lustre: 3702:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1494357452/real 1494357452] req@ffff8807c13c2a00 x1566943434075904/t0(0) o104->soaked-MDT0000@192.168.1.133@o2ib100:15/16 lens 296/224 e 0 to 1 dl 1494357459 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [ 4203.237982] Lustre: 3690:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1494357548/real 1494357548] req@ffff8807e682d700 x1566943439034896/t0(0) o104->soaked-MDT0000@192.168.1.138@o2ib100:15/16 lens 296/224 e 0 to 1 dl 1494357555 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [ 4731.411273] Lustre: soaked-MDT0000: Client dc7bce1e-8cc2-256d-8a24-aa9176826892 (at 192.168.1.125@o2ib100) reconnecting [ 4731.426572] Lustre: Skipped 2 previous similar messages [ 4731.435634] Lustre: soaked-MDT0000: Connection restored to eee4fa14-7423-c90d-6b47-20743657a666 (at 192.168.1.125@o2ib100) [ 4731.451200] Lustre: Skipped 2 previous similar messages [ 4852.341464] Lustre: MGS: Received new LWP connection from 192.168.1.117@o2ib100, removing former export from same NID [ 5679.798406] Lustre: MGS: Received new LWP connection from 192.168.1.125@o2ib100, removing former export from same NID [ 5679.813587] Lustre: MGS: Connection restored to eee4fa14-7423-c90d-6b47-20743657a666 (at 192.168.1.125@o2ib100) [ 5679.827819] Lustre: Skipped 5 previous similar messages [ 5815.372459] Lustre: soaked-MDT0000: Client f75498d6-bc9c-d0bf-fdad-54dda94a52b2 (at 192.168.1.123@o2ib100) reconnecting [ 5815.387540] Lustre: Skipped 4 previous similar messages [ 6707.068214] Lustre: 3465:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1494360052/real 0] req@ffff88009c88a700 x1566943440356288/t0(0) o13->soaked-OST0009-osc-MDT0000@192.168.1.105@o2ib10:7/4 lens 224/368 e 0 to 1 dl 1494360059 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 [ 6707.109009] Lustre: soaked-OST0009-osc-MDT0000: Connection to soaked-OST0009 (at 192.168.1.105@o2ib10) was lost; in progress operations using this service will wait for recovery to complete [ 6708.204232] Lustre: 3468:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1494360053/real 0] req@ffff8803f36fda00 x1566943440356384/t0(0) o13->soaked-OST0003-osc-MDT0000@192.168.1.105@o2ib10:7/4 lens 224/368 e 0 to 1 dl 1494360060 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 [ 6708.245384] Lustre: soaked-OST0003-osc-MDT0000: Connection to soaked-OST0003 (at 192.168.1.105@o2ib10) was lost; in progress operations using this service will wait for recovery to complete [ 6709.700307] Lustre: 3459:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1494360055/real 0] req@ffff8803d9a4b300 x1566943440356528/t0(0) o13->soaked-OST000f-osc-MDT0000@192.168.1.105@o2ib10:7/4 lens 224/368 e 0 to 1 dl 1494360062 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 [ 6709.742155] Lustre: 3459:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 6709.756469] Lustre: soaked-OST000f-osc-MDT0000: Connection to soaked-OST000f (at 192.168.1.105@o2ib10) was lost; in progress operations using this service will wait for recovery to complete [ 6709.781932] Lustre: Skipped 1 previous similar message [ 6712.790413] Lustre: 3457:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1494360059/real 0] req@ffff8803f36f8000 x1566943440356832/t0(0) o8->soaked-OST0009-osc-MDT0000@192.168.1.105@o2ib10:28/4 lens 520/544 e 0 to 1 dl 1494360065 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [ 6770.792424] Lustre: 3457:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1494360112/real 0] req@ffff880360faec00 x1566943440361664/t0(0) o8->soaked-OST000f-osc-MDT0000@192.168.1.105@o2ib10:28/4 lens 520/544 e 0 to 1 dl 1494360123 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [ 6770.834398] Lustre: 3457:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 6792.429530] Lustre: soaked-MDT0000: Client 6985bb7d-8a06-d23c-3360-00e6c4a7e369 (at 192.168.1.143@o2ib100) reconnecting [ 6792.445246] Lustre: Skipped 1 previous similar message [ 6792.454636] Lustre: soaked-MDT0000: Connection restored to 7ea3a484-c027-1559-8163-dc85944c882c (at 192.168.1.143@o2ib100) [ 6792.470642] Lustre: Skipped 2 previous similar messages [ 6797.703361] LNet: 3447:0:(o2iblnd_cb.c:3198:kiblnd_check_conns()) Timed out tx for 192.168.1.105@o2ib10: 2 seconds [ 6850.795225] Lustre: 3457:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1494360187/real 0] req@ffff880360faef00 x1566943440366816/t0(0) o8->soaked-OST0015-osc-MDT0000@192.168.1.105@o2ib10:28/4 lens 520/544 e 0 to 1 dl 1494360203 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [ 6850.837297] Lustre: 3457:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 6873.706004] LNet: 3447:0:(o2iblnd_cb.c:3198:kiblnd_check_conns()) Timed out tx for 192.168.1.105@o2ib10: 3 seconds [ 6873.721194] LNet: 3447:0:(o2iblnd_cb.c:3198:kiblnd_check_conns()) Skipped 1 previous similar message [ 6875.796087] Lustre: 3457:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1494360212/real 0] req@ffff880360faa700 x1566943440368512/t0(0) o8->soaked-OST0003-osc-MDT0000@192.168.1.105@o2ib10:28/4 lens 520/544 e 0 to 1 dl 1494360228 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [ 6875.837956] Lustre: 3457:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 6924.482528] Lustre: MGS: haven't heard from client 6d59f9e1-ce44-7010-80ac-34a863a225a8 (at 192.168.1.105@o2ib10) in 230 seconds. I think it's dead, and I am evicting it. exp ffff8803ef392c00, cur 1494360276 expire 1494360126 last 1494360046 [ 6930.797983] Lustre: 3457:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1494360262/real 0] req@ffff880360fabf00 x1566943440371952/t0(0) o8->soaked-OST000f-osc-MDT0000@192.168.1.105@o2ib10:28/4 lens 520/544 e 0 to 1 dl 1494360283 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [ 6930.840157] Lustre: 3457:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 6950.903056] Lustre: soaked-MDT0000: Received LWP connection from 192.168.1.104@o2ib10, removing former export from 192.168.1.105@o2ib10 [ 6956.871962] Lustre: soaked-MDT0000: Received LWP connection from 192.168.1.104@o2ib10, removing former export from 192.168.1.105@o2ib10 [ 6961.975059] Lustre: soaked-MDT0000: Received LWP connection from 192.168.1.104@o2ib10, removing former export from 192.168.1.105@o2ib10 [ 6967.630431] Lustre: soaked-MDT0000: Received LWP connection from 192.168.1.104@o2ib10, removing former export from 192.168.1.105@o2ib10 [ 7015.073630] LustreError: 11-0: soaked-OST0009-osc-MDT0000: operation ost_create to node 192.168.1.104@o2ib10 failed: rc = -19 [ 7015.089895] Lustre: soaked-OST0009-osc-MDT0000: Connection to soaked-OST0009 (at 192.168.1.104@o2ib10) was lost; in progress operations using this service will wait for recovery to complete [ 7015.115197] LustreError: 3577:0:(osp_precreate.c:602:osp_precreate_send()) soaked-OST0009-osc-MDT0000: can't precreate: rc = -19 [ 7015.134131] LustreError: 3577:0:(osp_precreate.c:1257:osp_precreate_thread()) soaked-OST0009-osc-MDT0000: cannot precreate objects: rc = -19 [ 7065.802724] Lustre: 3457:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1494360412/real 0] req@ffff8803f037d700 x1566943440507120/t0(0) o8->soaked-OST0009-osc-MDT0000@192.168.1.105@o2ib10:28/4 lens 520/544 e 0 to 1 dl 1494360418 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [ 7065.844563] Lustre: 3457:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 7104.026447] LustreError: 11-0: soaked-OST0015-osc-MDT0000: operation ost_statfs to node 192.168.1.104@o2ib10 failed: rc = -107 [ 7104.046785] Lustre: soaked-OST0015-osc-MDT0000: Connection to soaked-OST0015 (at 192.168.1.104@o2ib10) was lost; in progress operations using this service will wait for recovery to complete [ 7113.338839] LustreError: 11-0: soaked-OST0003-osc-MDT0000: operation ost_statfs to node 192.168.1.104@o2ib10 failed: rc = -107 [ 7113.359126] Lustre: soaked-OST0003-osc-MDT0000: Connection to soaked-OST0003 (at 192.168.1.104@o2ib10) was lost; in progress operations using this service will wait for recovery to complete [ 7149.548919] LustreError: 11-0: soaked-OST000f-osc-MDT0000: operation ost_statfs to node 192.168.1.104@o2ib10 failed: rc = -107 [ 7149.565481] Lustre: soaked-OST000f-osc-MDT0000: Connection to soaked-OST000f (at 192.168.1.104@o2ib10) was lost; in progress operations using this service will wait for recovery to complete [ 7166.616620] LustreError: 5550:0:(lfsck_lib.c:2680:lfsck_load_one_trace_file()) soaked-MDT0000-osd: unlink lfsck sub trace file lfsck_namespace_00: rc = 0 [ 7408.971576] NMI watchdog: BUG: soft lockup - CPU#6 stuck for 23s! [OI_scrub:5551] [ 7408.981437] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 7409.118481] CPU: 6 PID: 5551 Comm: OI_scrub Tainted: P OE ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 7409.132130] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 7409.149309] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 7409.159160] RIP: 0010:[] [] osd_inode_iteration+0x489/0xcc0 [osd_ldiskfs] [ 7409.171858] RSP: 0018:ffff880703603d18 EFLAGS: 00000293 [ 7409.179266] RAX: 0000000000000004 RBX: 0000000023f30a01 RCX: 0000000000000000 [ 7409.188702] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 7409.198119] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [ 7409.207516] R10: 0000000023f30a01 R11: ffffea000c8fcc00 R12: 0000000023f30a01 [ 7409.216888] R13: ffff880703603d08 R14: 0000000023f30a01 R15: ffff880703603d08 [ 7409.226244] FS: 0000000000000000(0000) GS:ffff88042e180000(0000) knlGS:0000000000000000 [ 7409.236663] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7409.244443] CR2: 00007f64d55202e0 CR3: 00000000019be000 CR4: 00000000000407e0 [ 7409.253766] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 7409.263065] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 7409.272335] Stack: [ 7409.275859] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [ 7409.285442] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [ 7409.294988] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [ 7409.304513] Call Trace: [ 7409.308465] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 7409.317858] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 7409.326938] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 7409.335799] [] ? wake_up_state+0x20/0x20 [ 7409.343188] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 7409.352696] [] kthread+0xcf/0xe0 [ 7409.359268] [] ? kthread_create_on_node+0x140/0x140 [ 7409.367669] [] ret_from_fork+0x58/0x90 [ 7409.374792] [] ? kthread_create_on_node+0x140/0x140 [ 7409.383151] Code: 00 e8 7c eb 97 ff e9 0f fc ff ff 0f 1f 80 00 00 00 00 45 89 e9 4c 8d 85 67 ff ff ff 48 8b 4d a8 48 8d 55 88 48 8b b5 38 ff ff ff <48> 8b bd 48 ff ff ff 48 8b 85 28 ff ff ff ff d0 85 c0 41 89 c5 [ 7449.101981] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [ 7449.111641] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 7449.244273] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 7449.257593] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 7449.273896] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 7449.283300] RIP: 0010:[] [] osd_inode_iteration+0x299/0xcc0 [osd_ldiskfs] [ 7449.295582] RSP: 0018:ffff880703603d18 EFLAGS: 00000287 [ 7449.302543] RAX: ffff88042bd78800 RBX: ffff880703603d57 RCX: ffff880703603d98 [ 7449.311540] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff88042bd79800 [ 7449.320527] RBP: ffff880703603df0 R08: 0000000000000001 R09: 0000000000000004 [ 7449.329498] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [ 7449.338460] R13: ffffea00027e3c80 R14: 000000009f8f2001 R15: ffff880703603d08 [ 7449.347419] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 7449.357453] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7449.364869] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 7449.373845] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 7449.382821] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 7449.391781] Stack: [ 7449.394999] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [ 7449.404292] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [ 7449.413581] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [ 7449.422887] Call Trace: [ 7449.426631] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 7449.435796] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 7449.444675] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 7449.453365] [] ? wake_up_state+0x20/0x20 [ 7449.460583] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 7449.469952] [] kthread+0xcf/0xe0 [ 7449.476389] [] ? kthread_create_on_node+0x140/0x140 [ 7449.484672] [] ret_from_fork+0x58/0x90 [ 7449.491690] [] ? kthread_create_on_node+0x140/0x140 [ 7449.499965] Code: 8b 45 88 48 8b 80 a8 03 00 00 48 3b 50 20 0f 83 57 01 00 00 44 0f b6 85 67 ff ff ff 48 8d 4d a8 48 8d 55 88 48 8b b5 38 ff ff ff <48> 8b bd 48 ff ff ff 48 8b 85 30 ff ff ff ff d0 83 f8 03 41 89 [ 7477.102955] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [ 7477.112593] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 7477.244786] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 7477.258109] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 7477.274414] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 7477.283822] RIP: 0010:[] [] osd_scrub_exec+0x1c/0x5b0 [osd_ldiskfs] [ 7477.295506] RSP: 0018:ffff880703603c78 EFLAGS: 00000296 [ 7477.302474] RAX: ffffffffa121b990 RBX: ffff880703603c37 RCX: 0000000000000000 [ 7477.311475] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 7477.320458] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [ 7477.329426] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [ 7477.338386] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [ 7477.347339] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 7477.357380] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7477.364792] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 7477.373764] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 7477.382733] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 7477.391685] Stack: [ 7477.394896] ffff880703603d08 ffff880703603d57 ffff880703603df0 000000009f8f2001 [ 7477.404190] ffff880703603d57 000000009f8f2001 0000000000000004 ffff880703603d57 [ 7477.413486] ffff88042bd78800 ffff88042bd78800 00000000000072cf ffff880419dc3a00 [ 7477.422765] Call Trace: [ 7477.426502] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [ 7477.435674] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 7477.444848] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 7477.453722] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 7477.462408] [] ? wake_up_state+0x20/0x20 [ 7477.469617] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 7477.478970] [] kthread+0xcf/0xe0 [ 7477.485408] [] ? kthread_create_on_node+0x140/0x140 [ 7477.493700] [] ret_from_fork+0x58/0x90 [ 7477.500712] [] ? kthread_create_on_node+0x140/0x140 [ 7477.508990] Code: 89 04 24 31 c0 e8 a5 03 98 ff e9 c7 fd ff ff 66 66 66 66 90 55 48 89 e5 41 57 49 89 f7 41 56 41 55 41 54 45 31 e4 53 48 83 ec 68 <48> 8b 9e 08 10 00 00 4c 89 45 90 65 48 8b 04 25 28 00 00 00 48 [ 7483.585176] INFO: rcu_sched self-detected stall on CPU { 24} (t=60000 jiffies g=907112 c=907111 q=32519) [ 7483.597097] Task dump for CPU 24: [ 7483.601835] OI_scrub R running task 0 5551 2 0x00000088 [ 7483.610750] INFO: rcu_sched detected stalls on CPUs/tasks: { [ 7483.610750] ffff88083fde6dd0 00000000fb6db880 ffff88082da03db0 ffffffff810c47e8 [ 7483.627368] 0000000000000018 ffffffff81a1e7c0 ffff88082da03dc8 ffffffff810c8079 [ 7483.636644] 0000000000000009 ffff88082da03df8 ffffffff811378e0 ffff88082da101c0 [ 7483.645897] Call Trace: [ 7483.649561] [] sched_show_task+0xa8/0x110 [ 7483.657516] [] dump_cpu_task+0x39/0x70 [ 7483.664455] [] rcu_dump_cpu_stacks+0x90/0xd0 [ 7483.671954] [] rcu_check_callbacks+0x442/0x720 [ 7483.679627] [] ? tick_sched_handle.isra.13+0x60/0x60 [ 7483.687875] [] update_process_times+0x47/0x80 [ 7483.695408] [] tick_sched_handle.isra.13+0x25/0x60 [ 7483.703412] [] tick_sched_timer+0x41/0x70 [ 7483.710514] [] __hrtimer_run_queues+0xd2/0x260 [ 7483.718081] [] hrtimer_interrupt+0xb0/0x1e0 [ 7483.725353] [] ? call_softirq+0x1c/0x30 [ 7483.732218] [] local_apic_timer_interrupt+0x37/0x60 [ 7483.740259] [] smp_apic_timer_interrupt+0x3f/0x60 [ 7483.748085] [] apic_timer_interrupt+0x6d/0x80 [ 7483.755511] [] ? osd_inode_iteration+0x4ac/0xcc0 [osd_ldiskfs] [ 7483.765316] [] ? osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [ 7483.774416] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 7483.783334] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 7483.791961] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 7483.800373] [] ? wake_up_state+0x20/0x20 [ 7483.807324] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 7483.816412] [] kthread+0xcf/0xe0 [ 7483.822603] [] ? kthread_create_on_node+0x140/0x140 [ 7483.830628] [] ret_from_fork+0x58/0x90 [ 7483.837381] [] ? kthread_create_on_node+0x140/0x140 [ 7483.845401] 24} (detected by 9, t=60262 jiffies, g=907112, c=907111, q=32606) [ 7483.854251] Task dump for CPU 24: [ 7483.858683] OI_scrub R running task 0 5551 2 0x00000088 [ 7483.867293] ffff880703603640 ffffffff8168bdc0 ffff88083fde6dd0 ffff880703603fd8 [ 7483.876334] ffff880703603fd8 ffff880703603fd8 ffff88083fde6dd0 ffff88082da16c40 [ 7483.885368] 0000000000000000 7fffffffffffffff ffff88043ffac170 ffffffff8168a440 [ 7483.894398] Call Trace: [ 7483.897867] [] ? __schedule+0x3b0/0x990 [ 7483.904734] [] ? bit_wait+0x50/0x50 [ 7483.911197] [] schedule+0x29/0x70 [ 7483.917460] [] schedule_timeout+0x239/0x2c0 [ 7483.924705] [] ? ktime_get_ts64+0x4c/0xf0 [ 7483.931744] [] ? delayacct_end+0x8f/0xb0 [ 7483.938663] [] ? __delayacct_blkio_end+0x34/0x60 [ 7483.946341] [] io_schedule_timeout+0xe8/0x130 [ 7483.953708] [] ? __wait_on_bit+0x7e/0x90 [ 7483.960586] [] ? bit_wait+0x50/0x50 [ 7483.966974] [] ? iam_lfix_split+0x140/0x140 [osd_ldiskfs] [ 7483.975489] [] ? out_of_line_wait_on_bit+0x81/0xb0 [ 7483.983318] [] ? wake_bit_function+0x40/0x40 [ 7483.990568] [] ? __wait_on_buffer+0x2a/0x30 [ 7483.997734] [] ? ldiskfs_bread+0x7c/0xc0 [ldiskfs] [ 7484.005577] [] ? ldiskfs_es_lookup_extent+0x2a/0x180 [ldiskfs] [ 7484.014572] [] ? ldiskfs_es_lookup_extent+0x2a/0x180 [ldiskfs] [ 7484.023547] [] ? mark_page_accessed+0x59/0xa0 [ 7484.030866] [] ? __find_get_block+0xbc/0x120 [ 7484.038089] [] ? __getblk+0x2d/0x300 [ 7484.044541] [] ? ldiskfs_getblk+0xa6/0x200 [ldiskfs] [ 7484.052550] [] ? ldiskfs_bread+0x27/0xc0 [ldiskfs] [ 7484.060360] [] ? iam_lfix_lookup+0x19d/0x3b0 [osd_ldiskfs] [ 7484.068996] [] ? lprocfs_counter_sub+0xc1/0x130 [obdclass] [ 7484.077593] [] ? __brelse+0x3d/0x50 [ 7484.083960] [] ? __wake_up+0x44/0x50 [ 7484.090422] [] ? jbd2_journal_stop+0x1ea/0x3d0 [jbd2] [ 7484.098538] [] ? osd_oi_lookup+0xa4/0x160 [osd_ldiskfs] [ 7484.106846] [] ? __slab_free+0x81/0x2f0 [ 7484.113602] [] ? __ldiskfs_journal_stop+0x3c/0xb0 [ldiskfs] [ 7484.122299] [] ? osd_scrub_refresh_mapping+0x1b5/0x440 [osd_ldiskfs] [ 7484.131845] [] ? osd_scrub_check_update+0x4d1/0x10c0 [osd_ldiskfs] [ 7484.141208] [] ? kfree+0x103/0x140 [ 7484.147472] [] ? osd_scrub_check_update+0x4d1/0x10c0 [osd_ldiskfs] [ 7484.156840] [] ? osd_scrub_exec+0x79/0x5b0 [osd_ldiskfs] [ 7484.165246] [] ? osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [ 7484.174236] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 7484.183014] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 7484.191503] [] ? osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 7484.199983] [] ? wake_up_state+0x20/0x20 [ 7484.206821] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 7484.215791] [] ? kthread+0xcf/0xe0 [ 7484.222021] [] ? kthread_create_on_node+0x140/0x140 [ 7484.229908] [] ? ret_from_fork+0x58/0x90 [ 7484.236718] [] ? kthread_create_on_node+0x140/0x140 [ 7509.104065] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [ 7509.113369] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 7509.242314] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 7509.255409] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 7509.271320] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 7509.280561] RIP: 0010:[] [] osd_scrub_exec+0x8d/0x5b0 [osd_ldiskfs] [ 7509.292090] RSP: 0018:ffff880703603ce8 EFLAGS: 00000292 [ 7509.298925] RAX: 0000000000000000 RBX: ffff880703603c78 RCX: 0000000000000000 [ 7509.307803] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 7509.316693] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [ 7509.325591] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000010 [ 7509.334485] R13: ffffffffa121b9c4 R14: ffffffffffffff10 R15: ffff8803162f6000 [ 7509.343378] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 7509.353346] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7509.360693] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 7509.369618] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 7509.378521] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 7509.387406] Stack: [ 7509.390565] ffff8800b2a3746c 0000000000000004 00000000ffffffff 0000000000000001 [ 7509.399800] ffff880703603df0 ffffffffa121d1e9 ffffffffa121b990 ffffffffa1217a20 [ 7509.409057] ffff8800b2a36000 00000000810d354f ffff8803162f6000 ffff8800b2a37468 [ 7509.418311] Call Trace: [ 7509.421999] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [ 7509.431126] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 7509.440249] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 7509.449073] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 7509.457701] [] ? wake_up_state+0x20/0x20 [ 7509.464870] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 7509.474188] [] kthread+0xcf/0xe0 [ 7509.480577] [] ? kthread_create_on_node+0x140/0x140 [ 7509.488817] [] ret_from_fork+0x58/0x90 [ 7509.495804] [] ? kthread_create_on_node+0x140/0x140 [ 7509.504051] Code: 85 c0 41 89 c1 0f 84 a3 01 00 00 41 80 a7 74 14 00 00 fe 48 8b 4d d0 65 48 33 0c 25 28 00 00 00 0f 85 b1 03 00 00 48 83 c4 68 5b <41> 5c 41 5d 41 5e 41 5f 5d c3 66 0f 1f 84 00 00 00 00 00 4c 8d [ 7537.105047] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [ 7537.114663] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 7537.247015] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 7537.260328] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 7537.276595] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 7537.286000] RIP: 0010:[] [] osd_scrub_exec+0x8d/0x5b0 [osd_ldiskfs] [ 7537.297674] RSP: 0018:ffff880703603ce8 EFLAGS: 00000292 [ 7537.304628] RAX: 0000000000000000 RBX: ffff880703603c78 RCX: 0000000000000000 [ 7537.313620] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 7537.322600] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [ 7537.331561] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000010 [ 7537.340513] R13: ffffffffa121ba87 R14: ffffffffffffff10 R15: ffff8803162f6000 [ 7537.349467] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 7537.359492] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7537.366905] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 7537.375875] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 7537.384843] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 7537.393790] Stack: [ 7537.397004] ffff8800b2a3746c 0000000000000004 00000000ffffffff 0000000000000001 [ 7537.406306] ffff880703603df0 ffffffffa121d1e9 ffffffffa121b990 ffffffffa1217a20 [ 7537.415609] ffff8800b2a36000 00000000810d354f ffff8803162f6000 ffff8800b2a37468 [ 7537.424905] Call Trace: [ 7537.428632] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [ 7537.437797] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 7537.446967] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 7537.455835] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 7537.464502] [] ? wake_up_state+0x20/0x20 [ 7537.471716] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 7537.481075] [] kthread+0xcf/0xe0 [ 7537.487515] [] ? kthread_create_on_node+0x140/0x140 [ 7537.495819] [] ret_from_fork+0x58/0x90 [ 7537.502832] [] ? kthread_create_on_node+0x140/0x140 [ 7537.511119] Code: 85 c0 41 89 c1 0f 84 a3 01 00 00 41 80 a7 74 14 00 00 fe 48 8b 4d d0 65 48 33 0c 25 28 00 00 00 0f 85 b1 03 00 00 48 83 c4 68 5b <41> 5c 41 5d 41 5e 41 5f 5d c3 66 0f 1f 84 00 00 00 00 00 4c 8d [ 7565.106022] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [ 7565.115664] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 7565.247902] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 7565.261189] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 7565.277449] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 7565.286832] RIP: 0010:[] [] osd_scrub_next+0x5/0x3c0 [osd_ldiskfs] [ 7565.298396] RSP: 0018:ffff880703603d10 EFLAGS: 00000287 [ 7565.305342] RAX: ffffffffa1217a20 RBX: ffff880703603d57 RCX: ffff880703603d98 [ 7565.314325] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 7565.323293] RBP: ffff880703603df0 R08: 0000000000000001 R09: 0000000000000004 [ 7565.332256] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [ 7565.341209] R13: ffffea00027e3c80 R14: 000000009f8f2001 R15: ffff880703603d08 [ 7565.350164] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 7565.360200] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7565.367615] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 7565.376596] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 7565.385560] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 7565.394512] Stack: [ 7565.397729] ffffffffa121cff9 ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 [ 7565.407010] 00000000810d354f ffff8803162f6000 ffff8800b2a37468 0000000020000000 [ 7565.416320] 010000000000000c 0000000000000000 0000000000000000 ffff8800b2a36000 [ 7565.425624] Call Trace: [ 7565.429352] [] ? osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [ 7565.438720] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 7565.447892] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 7565.456767] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 7565.465434] [] ? wake_up_state+0x20/0x20 [ 7565.472655] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 7565.482018] [] kthread+0xcf/0xe0 [ 7565.488453] [] ? kthread_create_on_node+0x140/0x140 [ 7565.496748] [] ret_from_fork+0x58/0x90 [ 7565.503768] [] ? kthread_create_on_node+0x140/0x140 [ 7565.512041] Code: 48 c1 e0 03 48 8d 4c 02 10 48 01 c2 e8 25 fc ff ff 48 83 c4 08 5b 41 5c 41 5d 41 5e 5d c3 0f 1f 84 00 00 00 00 00 66 66 66 66 90 <55> 48 89 e5 41 57 49 89 f7 41 56 41 55 49 89 fd 41 54 49 89 d4 [ 7593.106996] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [ 7593.116636] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 7593.248385] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 7593.261703] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 7593.277982] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 7593.287378] RIP: 0010:[] [] osd_inode_iteration+0x251/0xcc0 [osd_ldiskfs] [ 7593.299615] RSP: 0018:ffff880703603d18 EFLAGS: 00000286 [ 7593.306576] RAX: ffff8800b2a37468 RBX: 000000009f8f2001 RCX: 0000000000000000 [ 7593.315559] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 7593.324539] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [ 7593.333514] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffff880703603d57 [ 7593.342481] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [ 7593.351449] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 7593.361494] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7593.368919] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 7593.377902] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 7593.386867] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 7593.395820] Stack: [ 7593.399038] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [ 7593.408348] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [ 7593.417638] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [ 7593.426928] Call Trace: [ 7593.430651] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 7593.439824] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 7593.448702] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 7593.457373] [] ? wake_up_state+0x20/0x20 [ 7593.464586] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 7593.473935] [] kthread+0xcf/0xe0 [ 7593.480371] [] ? kthread_create_on_node+0x140/0x140 [ 7593.488652] [] ret_from_fork+0x58/0x90 [ 7593.495668] [] ? kthread_create_on_node+0x140/0x140 [ 7593.503944] Code: b0 00 00 00 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 00 8b 75 98 48 8b 7d 88 e8 dc 2d f5 ff 48 85 c0 48 89 45 90 0f 84 b3 08 00 00 <44> 8b 7d a0 48 8b 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 e8 77 [ 7621.107972] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [ 7621.117616] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 7621.250206] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 7621.263532] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 7621.279855] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 7621.289275] RIP: 0010:[] [] osd_inode_iteration+0x473/0xcc0 [osd_ldiskfs] [ 7621.301566] RSP: 0018:ffff880703603d18 EFLAGS: 00000293 [ 7621.308545] RAX: 0000000000000004 RBX: 000000009f8f2001 RCX: ffff880703603d98 [ 7621.317539] RDX: ffff880703603d78 RSI: 0000000000000000 RDI: ffff8803162f6000 [ 7621.326542] RBP: ffff880703603df0 R08: 0000000000000001 R09: 0000000000000004 [ 7621.335515] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [ 7621.344468] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [ 7621.353425] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 7621.363457] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7621.370871] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 7621.379849] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 7621.388817] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 7621.397769] Stack: [ 7621.400987] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [ 7621.410288] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [ 7621.419586] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [ 7621.428890] Call Trace: [ 7621.432623] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 7621.441778] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 7621.450655] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 7621.459344] [] ? wake_up_state+0x20/0x20 [ 7621.466561] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 7621.475914] [] kthread+0xcf/0xe0 [ 7621.482350] [] ? kthread_create_on_node+0x140/0x140 [ 7621.490652] [] ret_from_fork+0x58/0x90 [ 7621.497669] [] ? kthread_create_on_node+0x140/0x140 [ 7621.505952] Code: 00 00 48 c7 05 9b 86 02 00 00 00 00 00 c7 05 89 86 02 00 01 00 00 00 e8 7c eb 97 ff e9 0f fc ff ff 0f 1f 80 00 00 00 00 45 89 e9 <4c> 8d 85 67 ff ff ff 48 8b 4d a8 48 8d 55 88 48 8b b5 38 ff ff [ 7649.108955] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [ 7649.118578] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 7649.250555] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 7649.263860] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 7649.280148] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 7649.289550] RIP: 0010:[] [] osd_scrub_exec+0x5/0x5b0 [osd_ldiskfs] [ 7649.301104] RSP: 0018:ffff880703603d10 EFLAGS: 00000293 [ 7649.308063] RAX: ffffffffa121b990 RBX: 000000009f8f2001 RCX: 0000000000000000 [ 7649.317046] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 7649.326028] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [ 7649.334988] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffff880703603d57 [ 7649.343936] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [ 7649.352885] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 7649.362911] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7649.370319] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 7649.379293] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 7649.388258] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 7649.397215] Stack: [ 7649.400438] ffffffffa121d1e9 ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 [ 7649.409730] 00000000810d354f ffff8803162f6000 ffff8800b2a37468 0000000020000000 [ 7649.419029] 010000000000000c 0000000000000000 0000000000000000 ffff8800b2a36000 [ 7649.428334] Call Trace: [ 7649.432069] [] ? osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [ 7649.441434] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 7649.450606] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 7649.459492] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 7649.468169] [] ? wake_up_state+0x20/0x20 [ 7649.475390] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 7649.484744] [] kthread+0xcf/0xe0 [ 7649.491187] [] ? kthread_create_on_node+0x140/0x140 [ 7649.499478] [] ret_from_fork+0x58/0x90 [ 7649.506495] [] ? kthread_create_on_node+0x140/0x140 [ 7649.514770] Code: 05 85 a5 02 00 00 00 02 00 89 54 24 08 48 8b 40 40 ba 07 00 00 00 48 89 04 24 31 c0 e8 a5 03 98 ff e9 c7 fd ff ff 66 66 66 66 90 <55> 48 89 e5 41 57 49 89 f7 41 56 41 55 41 54 45 31 e4 53 48 83 [ 7663.615454] INFO: rcu_sched self-detected stall on CPU { 24} (t=240024 jiffies g=907112 c=907111 q=317538) [ 7663.627634] Task dump for CPU 24: [ 7663.632386] OI_scrub R running task 0 5551 2 0x00000088 [ 7663.641323] ffff88083fde6dd0 00000000fb6db880 ffff88082da03db0 ffffffff810c47e8 [ 7663.650653] 0000000000000018 ffffffff81a1e7c0 ffff88082da03dc8 ffffffff810c8079 [ 7663.659955] 0000000000000009 ffff88082da03df8 ffffffff811378e0 ffff88082da101c0 [ 7663.669240] Call Trace: [ 7663.672935] [] sched_show_task+0xa8/0x110 [ 7663.680934] [] dump_cpu_task+0x39/0x70 [ 7663.687912] [] rcu_dump_cpu_stacks+0x90/0xd0 [ 7663.695447] [] rcu_check_callbacks+0x442/0x720 [ 7663.703157] [] ? tick_sched_handle.isra.13+0x60/0x60 [ 7663.711421] [] update_process_times+0x47/0x80 [ 7663.718983] [] tick_sched_handle.isra.13+0x25/0x60 [ 7663.727007] [] tick_sched_timer+0x41/0x70 [ 7663.734156] [] __hrtimer_run_queues+0xd2/0x260 [ 7663.741759] [] hrtimer_interrupt+0xb0/0x1e0 [ 7663.749052] [] ? call_softirq+0x1c/0x30 [ 7663.755938] [] local_apic_timer_interrupt+0x37/0x60 [ 7663.763976] [] smp_apic_timer_interrupt+0x3f/0x60 [ 7663.771824] [] apic_timer_interrupt+0x6d/0x80 [ 7663.779260] [] ? osd_scrub_next+0x149/0x3c0 [osd_ldiskfs] [ 7663.788604] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [ 7663.797522] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 7663.806424] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 7663.815043] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 7663.823465] [] ? wake_up_state+0x20/0x20 [ 7663.830419] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 7663.839515] [] kthread+0xcf/0xe0 [ 7663.845691] [] ? kthread_create_on_node+0x140/0x140 [ 7663.853721] [] ret_from_fork+0x58/0x90 [ 7663.860482] [] ? kthread_create_on_node+0x140/0x140 [ 7689.110349] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [ 7689.119772] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 7689.249982] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 7689.263158] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 7689.279206] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 7689.288508] RIP: 0010:[] [] osd_inode_iteration+0x2be/0xcc0 [osd_ldiskfs] [ 7689.300660] RSP: 0018:ffff880703603d18 EFLAGS: 00000297 [ 7689.307516] RAX: 0000000000000004 RBX: 000000009f8f2001 RCX: ffff880703603d98 [ 7689.316415] RDX: ffff880703603d78 RSI: 0000000000000000 RDI: ffff8803162f6000 [ 7689.325307] RBP: ffff880703603df0 R08: 0000000000000001 R09: 0000000000000004 [ 7689.334197] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffff880703603d57 [ 7689.343091] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [ 7689.351979] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 7689.361941] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7689.369286] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 7689.378188] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 7689.387095] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 7689.395988] Stack: [ 7689.399152] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [ 7689.408394] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [ 7689.417632] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [ 7689.426898] Call Trace: [ 7689.430592] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 7689.439729] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 7689.448569] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 7689.457213] [] ? wake_up_state+0x20/0x20 [ 7689.464402] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 7689.473722] [] kthread+0xcf/0xe0 [ 7689.480118] [] ? kthread_create_on_node+0x140/0x140 [ 7689.488366] [] ret_from_fork+0x58/0x90 [ 7689.495347] [] ? kthread_create_on_node+0x140/0x140 [ 7689.503597] Code: 8b b5 38 ff ff ff 48 8b bd 48 ff ff ff 48 8b 85 30 ff ff ff ff d0 83 f8 03 41 89 c5 0f 84 4b 02 00 00 0f 8e 15 01 00 00 83 f8 05 <66> 90 0f 84 8a 02 00 00 83 f8 06 0f 85 a1 01 00 00 48 8b 7d 90 [ 7717.111318] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [ 7717.120935] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 7717.253424] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 7717.266748] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 7717.283101] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 7717.292534] RIP: 0010:[] [] ldiskfs_itable_unused_count+0x6/0x30 [ldiskfs] [ 7717.304894] RSP: 0018:ffff880703603d08 EFLAGS: 00000286 [ 7717.311879] RAX: ffff8800b2a37468 RBX: ffff880703603d57 RCX: 0000000000000000 [ 7717.320879] RDX: ffff880703603d78 RSI: ffff880419dc3a00 RDI: ffff88042bd79800 [ 7717.329874] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [ 7717.338848] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [ 7717.347809] R13: 0000000000000018 R14: ffff880703603c78 R15: 0000000000000296 [ 7717.356756] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 7717.366776] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7717.374181] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 7717.383155] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 7717.392119] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 7717.401065] Stack: [ 7717.404272] ffff880703603df0 ffffffffa121cfb9 ffffffffa121b990 ffffffffa1217a20 [ 7717.413548] ffff8800b2a36000 00000000810d354f ffff8803162f6000 ffff8800b2a37468 [ 7717.422845] 0000000020000000 010000000000000c 0000000000000000 0000000000000000 [ 7717.432137] Call Trace: [ 7717.435883] [] ? osd_inode_iteration+0x269/0xcc0 [osd_ldiskfs] [ 7717.445239] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 7717.454414] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 7717.463280] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 7717.471962] [] ? wake_up_state+0x20/0x20 [ 7717.479173] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 7717.488536] [] kthread+0xcf/0xe0 [ 7717.494963] [] ? kthread_create_on_node+0x140/0x140 [ 7717.503238] [] ret_from_fork+0x58/0x90 [ 7717.510245] [] ? kthread_create_on_node+0x140/0x140 [ 7717.518512] Code: 00 00 31 c0 0f b7 56 10 48 89 e5 48 83 39 3f 76 07 0f b7 46 30 c1 e0 10 09 d0 5d c3 66 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 <48> 8b 8f a8 03 00 00 31 c0 0f b7 56 1c 48 89 e5 48 83 39 3f 76 [ 7745.112287] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [ 7745.121947] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 7745.254065] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 7745.267351] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 7745.283624] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 7745.293014] RIP: 0010:[] [] ldiskfs_itable_unused_count+0x25/0x30 [ldiskfs] [ 7745.305452] RSP: 0018:ffff880703603d08 EFLAGS: 00000202 [ 7745.312393] RAX: 00000000000072ce RBX: 0000000000000001 RCX: ffff88042bd78800 [ 7745.321391] RDX: 00000000000072ce RSI: ffff880419dc3a00 RDI: ffff88042bd79800 [ 7745.330368] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [ 7745.339313] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [ 7745.348262] R13: 0000000000000018 R14: ffff880703603c78 R15: 0000000000000296 [ 7745.357211] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 7745.367235] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7745.374639] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 7745.383605] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 7745.392568] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 7745.401514] Stack: [ 7745.404718] ffff880703603df0 ffffffffa121cfb9 ffffffffa121b990 ffffffffa1217a20 [ 7745.414005] ffff8800b2a36000 00000000810d354f ffff8803162f6000 ffff8800b2a37468 [ 7745.423313] 0000000020000000 010000000000000c 0000000000000000 0000000000000000 [ 7745.432596] Call Trace: [ 7745.436323] [] osd_inode_iteration+0x269/0xcc0 [osd_ldiskfs] [ 7745.445503] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 7745.454666] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 7745.463526] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 7745.472198] [] ? wake_up_state+0x20/0x20 [ 7745.479398] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 7745.488764] [] kthread+0xcf/0xe0 [ 7745.495186] [] ? kthread_create_on_node+0x140/0x140 [ 7745.503460] [] ret_from_fork+0x58/0x90 [ 7745.510467] [] ? kthread_create_on_node+0x140/0x140 [ 7745.518736] Code: 84 00 00 00 00 00 66 66 66 66 90 55 48 8b 8f a8 03 00 00 31 c0 0f b7 56 1c 48 89 e5 48 83 39 3f 76 07 0f b7 46 32 c1 e0 10 09 d0 <5d> c3 66 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 89 16 48 8b [ 7773.113267] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [ 7773.122905] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 7773.255132] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 7773.268412] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 7773.284667] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 7773.294051] RIP: 0010:[] [] osd_scrub_exec+0xf9/0x5b0 [osd_ldiskfs] [ 7773.305698] RSP: 0018:ffff880703603c78 EFLAGS: 00000246 [ 7773.312640] RAX: 0000000000000000 RBX: ffff880703603c37 RCX: 0000000000000000 [ 7773.321605] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 7773.330566] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [ 7773.339514] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [ 7773.348459] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [ 7773.357409] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 7773.367435] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7773.374842] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 7773.383816] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 7773.392779] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 7773.401735] Stack: [ 7773.404951] 0000000000000004 0000000000000001 ffff880703603df0 ffff880703603d98 [ 7773.414264] ffff880703603d57 000000009f8f2001 0000000000000004 ffff880703603d57 [ 7773.423561] 0000000000000004 ffff880703603d98 ffff880703603d78 0000000000000000 [ 7773.432850] Call Trace: [ 7773.436576] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [ 7773.445743] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 7773.454903] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 7773.463771] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 7773.472458] [] ? wake_up_state+0x20/0x20 [ 7773.479664] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 7773.489025] [] kthread+0xcf/0xe0 [ 7773.495453] [] ? kthread_create_on_node+0x140/0x140 [ 7773.503753] [] ret_from_fork+0x58/0x90 [ 7773.510765] [] ? kthread_create_on_node+0x140/0x140 [ 7773.519034] Code: 88 48 85 db 74 23 f6 83 18 0a 00 00 08 74 1a 4d 85 e4 74 15 41 8b 87 6c 14 00 00 41 39 84 24 0c 0a 00 00 0f 82 d9 00 00 00 31 c0 <41> f6 87 74 14 00 00 04 0f 85 6e ff ff ff 41 83 f9 02 0f 84 64 [ 7801.114237] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [ 7801.123938] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 7801.256247] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 7801.269533] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 7801.285798] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 7801.295184] RIP: 0010:[] [] osd_scrub_exec+0x1c/0x5b0 [osd_ldiskfs] [ 7801.306838] RSP: 0018:ffff880703603c78 EFLAGS: 00000296 [ 7801.313785] RAX: ffffffffa121b990 RBX: ffff880703603c37 RCX: 0000000000000000 [ 7801.322770] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 7801.331741] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [ 7801.340702] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [ 7801.349659] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [ 7801.358617] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 7801.368652] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7801.376065] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 7801.385029] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 7801.393991] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 7801.402940] Stack: [ 7801.406156] ffff880703603d08 0000000000000001 ffff880703603df0 000000009f8f2001 [ 7801.415463] ffff880703603d57 000000009f8f2001 0000000000000004 ffff880703603d57 [ 7801.424760] ffff88042bd78800 ffff88042bd78800 00000000000072cf ffff880419dc3a00 [ 7801.434050] Call Trace: [ 7801.437771] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [ 7801.446947] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 7801.456108] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 7801.464971] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 7801.473643] [] ? wake_up_state+0x20/0x20 [ 7801.480851] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 7801.490206] [] kthread+0xcf/0xe0 [ 7801.496636] [] ? kthread_create_on_node+0x140/0x140 [ 7801.504929] [] ret_from_fork+0x58/0x90 [ 7801.511939] [] ? kthread_create_on_node+0x140/0x140 [ 7801.520215] Code: 89 04 24 31 c0 e8 a5 03 98 ff e9 c7 fd ff ff 66 66 66 66 90 55 48 89 e5 41 57 49 89 f7 41 56 41 55 41 54 45 31 e4 53 48 83 ec 68 <48> 8b 9e 08 10 00 00 4c 89 45 90 65 48 8b 04 25 28 00 00 00 48 [ 7829.115219] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [ 7829.124851] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 7829.257016] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 7829.270298] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 7829.286560] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 7829.295930] RIP: 0010:[] [] ldiskfs_itable_unused_count+0x23/0x30 [ldiskfs] [ 7829.308364] RSP: 0018:ffff880703603d08 EFLAGS: 00000297 [ 7829.315308] RAX: 0000000000000000 RBX: 0000000000000001 RCX: ffff88042bd78800 [ 7829.324289] RDX: 00000000000072ce RSI: ffff880419dc3a00 RDI: ffff88042bd79800 [ 7829.333248] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [ 7829.342204] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [ 7829.351156] R13: 0000000000000018 R14: ffff880703603c78 R15: 0000000000000246 [ 7829.360105] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 7829.370132] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7829.377534] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 7829.386503] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 7829.395453] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 7829.404403] Stack: [ 7829.407607] ffff880703603df0 ffffffffa121cfb9 ffffffffa121b990 ffffffffa1217a20 [ 7829.416904] ffff8800b2a36000 00000000810d354f ffff8803162f6000 ffff8800b2a37468 [ 7829.426195] 0000000020000000 010000000000000c 0000000000000000 0000000000000000 [ 7829.435482] Call Trace: [ 7829.439215] [] osd_inode_iteration+0x269/0xcc0 [osd_ldiskfs] [ 7829.448378] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 7829.457538] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 7829.466399] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 7829.475074] [] ? wake_up_state+0x20/0x20 [ 7829.482282] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 7829.491637] [] kthread+0xcf/0xe0 [ 7829.498066] [] ? kthread_create_on_node+0x140/0x140 [ 7829.506354] [] ret_from_fork+0x58/0x90 [ 7829.513362] [] ? kthread_create_on_node+0x140/0x140 [ 7829.521629] Code: 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 8b 8f a8 03 00 00 31 c0 0f b7 56 1c 48 89 e5 48 83 39 3f 76 07 0f b7 46 32 c1 e0 10 <09> d0 5d c3 66 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 89 16 [ 7843.624724] INFO: rcu_sched self-detected stall on CPU { 24} (t=420027 jiffies g=907112 c=907111 q=736665) [ 7843.636869] Task dump for CPU 24: [ 7843.641610] OI_scrub R running task 0 5551 2 0x00000088 [ 7843.650505] ffff88083fde6dd0 00000000fb6db880 ffff88082da03db0 ffffffff810c47e8 [ 7843.659825] 0000000000000018 ffffffff81a1e7c0 ffff88082da03dc8 ffffffff810c8079 [ 7843.669131] 0000000000000009 ffff88082da03df8 ffffffff811378e0 ffff88082da101c0 [ 7843.678406] Call Trace: [ 7843.682093] [] sched_show_task+0xa8/0x110 [ 7843.690070] [] dump_cpu_task+0x39/0x70 [ 7843.697024] [] rcu_dump_cpu_stacks+0x90/0xd0 [ 7843.704546] [] rcu_check_callbacks+0x442/0x720 [ 7843.712237] [] ? tick_sched_handle.isra.13+0x60/0x60 [ 7843.720488] [] update_process_times+0x47/0x80 [ 7843.728035] [] tick_sched_handle.isra.13+0x25/0x60 [ 7843.736038] [] tick_sched_timer+0x41/0x70 [ 7843.743154] [] __hrtimer_run_queues+0xd2/0x260 [ 7843.750729] [] hrtimer_interrupt+0xb0/0x1e0 [ 7843.758002] [] ? call_softirq+0x1c/0x30 [ 7843.764881] [] local_apic_timer_interrupt+0x37/0x60 [ 7843.772900] [] smp_apic_timer_interrupt+0x3f/0x60 [ 7843.780721] [] apic_timer_interrupt+0x6d/0x80 [ 7843.788142] [] ? osd_inode_iteration+0x49b/0xcc0 [osd_ldiskfs] [ 7843.797973] [] ? osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [ 7843.807065] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 7843.815959] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 7843.824566] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 7843.832979] [] ? wake_up_state+0x20/0x20 [ 7843.839935] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 7843.849048] [] kthread+0xcf/0xe0 [ 7843.855225] [] ? kthread_create_on_node+0x140/0x140 [ 7843.863269] [] ret_from_fork+0x58/0x90 [ 7843.870032] [] ? kthread_create_on_node+0x140/0x140 [ 7869.116620] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [ 7869.126049] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 7869.256223] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 7869.269396] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 7869.285484] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 7869.294797] RIP: 0010:[] [] osd_scrub_exec+0xd4/0x5b0 [osd_ldiskfs] [ 7869.306380] RSP: 0018:ffff880703603c78 EFLAGS: 00000286 [ 7869.313247] RAX: ffff8803162f1008 RBX: ffff880703603c37 RCX: 0000000000000000 [ 7869.322158] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 7869.331063] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [ 7869.339966] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [ 7869.348863] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [ 7869.357763] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 7869.367738] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7869.375092] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 7869.384003] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 7869.392922] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 7869.401820] Stack: [ 7869.405003] 0000000000000004 0000000000000001 0000000000000004 ffff880703603d98 [ 7869.414269] ffff880703603d57 0000000000000000 ffff8803162f6000 ffffffffffffff10 [ 7869.423526] ffffffffa1217b80 0000000000000010 0000000000000282 ffff880703603ce0 [ 7869.432774] Call Trace: [ 7869.436482] [] ? osd_scrub_next+0x160/0x3c0 [osd_ldiskfs] [ 7869.445332] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [ 7869.454476] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 7869.463614] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 7869.472457] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 7869.481104] [] ? wake_up_state+0x20/0x20 [ 7869.488277] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 7869.497603] [] kthread+0xcf/0xe0 [ 7869.504005] [] ? kthread_create_on_node+0x140/0x140 [ 7869.512255] [] ret_from_fork+0x58/0x90 [ 7869.519240] [] ? kthread_create_on_node+0x140/0x140 [ 7869.527491] Code: 4d 88 4c 89 ef e8 1d fb 46 e0 41 83 87 68 14 00 00 01 49 83 87 88 11 00 00 01 4c 89 ef e8 f5 9f e9 df 44 8b 4d 88 48 85 db 74 23 83 18 0a 00 00 08 74 1a 4d 85 e4 74 15 41 8b 87 6c 14 00 00 [ 7888.786761] perf: interrupt took too long (2552 > 2500), lowering kernel.perf_event_max_sample_rate to 78000 [ 7913.118138] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [ 7913.127746] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 7913.259299] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 7913.272570] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 7913.288783] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 7913.298151] RIP: 0010:[] [] osd_inode_iteration+0x49b/0xcc0 [osd_ldiskfs] [ 7913.310371] RSP: 0018:ffff880703603d18 EFLAGS: 00000246 [ 7913.317309] RAX: 0000000000000000 RBX: 000000009f8f2001 RCX: 0000000000000000 [ 7913.326285] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 7913.335257] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [ 7913.344228] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffff880703603d57 [ 7913.353209] R13: ffff880703603d08 R14: 000000009f8f2001 R15: ffffea00027e3c80 [ 7913.362181] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 7913.372233] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7913.379655] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 7913.388634] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 7913.397619] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 7913.406591] Stack: [ 7913.409834] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [ 7913.419151] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [ 7913.428463] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [ 7913.437783] Call Trace: [ 7913.441527] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 7913.450708] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 7913.459594] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 7913.468289] [] ? wake_up_state+0x20/0x20 [ 7913.475524] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 7913.484895] [] kthread+0xcf/0xe0 [ 7913.491358] [] ? kthread_create_on_node+0x140/0x140 [ 7913.499671] [] ret_from_fork+0x58/0x90 [ 7913.506723] [] ? kthread_create_on_node+0x140/0x140 [ 7913.515022] Code: 45 89 e9 4c 8d 85 67 ff ff ff 48 8b 4d a8 48 8d 55 88 48 8b b5 38 ff ff ff 48 8b bd 48 ff ff ff 48 8b 85 28 ff ff ff ff d0 85 c0 <41> 89 c5 75 1c 8b 85 58 ff ff ff 41 3b 04 24 72 10 48 8b 85 50 [ 7941.119120] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [ 7941.128790] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 7941.261388] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 7941.274706] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 7941.291031] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 7941.300454] RIP: 0010:[] [] osd_scrub_exec+0x30/0x5b0 [osd_ldiskfs] [ 7941.312154] RSP: 0018:ffff880703603c78 EFLAGS: 00000296 [ 7941.319131] RAX: 00000000fb6db880 RBX: ffff880703603c37 RCX: 0000000000000000 [ 7941.328130] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 7941.337109] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [ 7941.346080] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [ 7941.355044] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [ 7941.363995] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 7941.374031] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7941.381447] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 7941.390422] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 7941.399400] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 7941.408369] Stack: [ 7941.411588] ffff880703603df0 000000009f8f2001 ffffea00027e3c80 000000009f8f2001 [ 7941.420880] ffff880703603d57 ffff880703603d57 ffff8800b2a37468 0000000000000000 [ 7941.430169] ffff880703603d78 ffff880419dc3a00 ffff88042bd79800 ffffffffffffff10 [ 7941.439465] Call Trace: [ 7941.443198] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [ 7941.452369] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 7941.461526] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 7941.470398] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 7941.479076] [] ? wake_up_state+0x20/0x20 [ 7941.486284] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 7941.495639] [] kthread+0xcf/0xe0 [ 7941.502069] [] ? kthread_create_on_node+0x140/0x140 [ 7941.510361] [] ret_from_fork+0x58/0x90 [ 7941.517353] [] ? kthread_create_on_node+0x140/0x140 [ 7941.525623] Code: 55 48 89 e5 41 57 49 89 f7 41 56 41 55 41 54 45 31 e4 53 48 83 ec 68 48 8b 9e 08 10 00 00 4c 89 45 90 65 48 8b 04 25 28 00 00 00 <48> 89 45 d0 31 c0 48 8d 43 08 48 85 db 4c 0f 45 e0 41 83 f9 04 [ 7969.120090] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [ 7969.129754] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 7969.261862] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 7969.275159] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 7969.291410] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 7969.300802] RIP: 0010:[] [] ldiskfs_itable_unused_count+0x23/0x30 [ldiskfs] [ 7969.313246] RSP: 0018:ffff880703603d08 EFLAGS: 00000297 [ 7969.320196] RAX: 0000000000000000 RBX: 000000009f8f2001 RCX: ffff88042bd78800 [ 7969.329171] RDX: 00000000000072ce RSI: ffff880419dc3a00 RDI: ffff88042bd79800 [ 7969.338135] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [ 7969.347081] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [ 7969.356025] R13: 0000000000000018 R14: ffff880703603c78 R15: 0000000000000246 [ 7969.364974] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 7969.375000] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7969.382402] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 7969.391364] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 7969.400320] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 7969.409266] Stack: [ 7969.412470] ffff880703603df0 ffffffffa121cfb9 ffffffffa121b990 ffffffffa1217a20 [ 7969.421746] ffff8800b2a36000 00000000810d354f ffff8803162f6000 ffff8800b2a37468 [ 7969.431025] 0000000020000000 010000000000000c 0000000000000000 0000000000000000 [ 7969.440317] Call Trace: [ 7969.444049] [] osd_inode_iteration+0x269/0xcc0 [osd_ldiskfs] [ 7969.453216] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 7969.462380] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 7969.471244] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 7969.479914] [] ? wake_up_state+0x20/0x20 [ 7969.487122] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 7969.496477] [] kthread+0xcf/0xe0 [ 7969.502915] [] ? kthread_create_on_node+0x140/0x140 [ 7969.511194] [] ret_from_fork+0x58/0x90 [ 7969.518206] [] ? kthread_create_on_node+0x140/0x140 [ 7969.526484] Code: 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 8b 8f a8 03 00 00 31 c0 0f b7 56 1c 48 89 e5 48 83 39 3f 76 07 0f b7 46 32 c1 e0 10 <09> d0 5d c3 66 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 89 16 [ 7997.121077] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [ 7997.130716] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 7997.262935] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 7997.276226] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 7997.292496] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 7997.301883] RIP: 0010:[] [] osd_scrub_exec+0x1c/0x5b0 [osd_ldiskfs] [ 7997.313547] RSP: 0018:ffff880703603c78 EFLAGS: 00000296 [ 7997.320483] RAX: ffffffffa121b990 RBX: ffff880703603c37 RCX: 0000000000000000 [ 7997.329454] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 7997.338418] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [ 7997.347370] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [ 7997.356319] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [ 7997.365272] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 7997.375303] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7997.382713] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 7997.391689] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 7997.400652] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 7997.409611] Stack: [ 7997.412828] ffff880703603d08 000000009f8f2001 ffff880703603df0 000000009f8f2001 [ 7997.422139] ffff880703603d57 000000009f8f2001 0000000000000004 ffff880703603d57 [ 7997.431433] ffff8800b2a37468 0000000000000000 ffff880703603d78 ffff8800b2a36000 [ 7997.440727] Call Trace: [ 7997.444457] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [ 7997.453626] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 7997.462781] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 7997.471647] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 7997.480318] [] ? wake_up_state+0x20/0x20 [ 7997.487523] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 7997.496879] [] kthread+0xcf/0xe0 [ 7997.503318] [] ? kthread_create_on_node+0x140/0x140 [ 7997.511605] [] ret_from_fork+0x58/0x90 [ 7997.518619] [] ? kthread_create_on_node+0x140/0x140 [ 7997.526891] Code: 89 04 24 31 c0 e8 a5 03 98 ff e9 c7 fd ff ff 66 66 66 66 90 55 48 89 e5 41 57 49 89 f7 41 56 41 55 41 54 45 31 e4 53 48 83 ec 68 <48> 8b 9e 08 10 00 00 4c 89 45 90 65 48 8b 04 25 28 00 00 00 48 [ 8023.633993] INFO: rcu_sched self-detected stall on CPU { 24} (t=600030 jiffies g=907112 c=907111 q=1108460) [ 8023.646258] Task dump for CPU 24: [ 8023.651005] OI_scrub R running task 0 5551 2 0x00000088 [ 8023.659926] ffff88083fde6dd0 00000000fb6db880 ffff88082da03db0 ffffffff810c47e8 [ 8023.669234] 0000000000000018 ffffffff81a1e7c0 ffff88082da03dc8 ffffffff810c8079 [ 8023.678525] 0000000000000009 ffff88082da03df8 ffffffff811378e0 ffff88082da101c0 [ 8023.687789] Call Trace: [ 8023.691478] [] sched_show_task+0xa8/0x110 [ 8023.699452] [] dump_cpu_task+0x39/0x70 [ 8023.706417] [] rcu_dump_cpu_stacks+0x90/0xd0 [ 8023.713942] [] rcu_check_callbacks+0x442/0x720 [ 8023.721633] [] ? tick_sched_handle.isra.13+0x60/0x60 [ 8023.729891] [] update_process_times+0x47/0x80 [ 8023.737443] [] tick_sched_handle.isra.13+0x25/0x60 [ 8023.745457] [] tick_sched_timer+0x41/0x70 [ 8023.752577] [] __hrtimer_run_queues+0xd2/0x260 [ 8023.760163] [] hrtimer_interrupt+0xb0/0x1e0 [ 8023.767445] [] ? call_softirq+0x1c/0x30 [ 8023.774323] [] local_apic_timer_interrupt+0x37/0x60 [ 8023.782349] [] smp_apic_timer_interrupt+0x3f/0x60 [ 8023.790177] [] apic_timer_interrupt+0x6d/0x80 [ 8023.797608] [] ? osd_inode_iteration+0x251/0xcc0 [osd_ldiskfs] [ 8023.807445] [] ? osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [ 8023.816543] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 8023.825451] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 8023.834068] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 8023.842480] [] ? wake_up_state+0x20/0x20 [ 8023.849433] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 8023.858526] [] kthread+0xcf/0xe0 [ 8023.864699] [] ? kthread_create_on_node+0x140/0x140 [ 8023.872740] [] ret_from_fork+0x58/0x90 [ 8023.879503] [] ? kthread_create_on_node+0x140/0x140 [ 8049.122881] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [ 8049.132267] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 8049.262514] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 8049.275697] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 8049.291782] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 8049.301086] RIP: 0010:[] [] osd_inode_iteration+0x4a6/0xcc0 [osd_ldiskfs] [ 8049.313241] RSP: 0018:ffff880703603d18 EFLAGS: 00000246 [ 8049.320113] RAX: 0000000020000000 RBX: 000000009f8f2001 RCX: 0000000000000000 [ 8049.329026] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 8049.337939] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [ 8049.346837] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [ 8049.355745] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [ 8049.364650] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 8049.374629] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8049.381983] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 8049.390898] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8049.399819] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 8049.408727] Stack: [ 8049.411904] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [ 8049.421167] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [ 8049.430440] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [ 8049.439683] Call Trace: [ 8049.443369] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 8049.452515] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 8049.461364] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 8049.470030] [] ? wake_up_state+0x20/0x20 [ 8049.477234] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 8049.486571] [] kthread+0xcf/0xe0 [ 8049.492991] [] ? kthread_create_on_node+0x140/0x140 [ 8049.501251] [] ret_from_fork+0x58/0x90 [ 8049.508242] [] ? kthread_create_on_node+0x140/0x140 [ 8049.516501] Code: 8b 4d a8 48 8d 55 88 48 8b b5 38 ff ff ff 48 8b bd 48 ff ff ff 48 8b 85 28 ff ff ff ff d0 85 c0 41 89 c5 75 1c 8b 85 58 ff ff ff <41> 3b 04 24 72 10 48 8b 85 50 ff ff ff 44 3b 30 0f 87 95 fd ff [ 8077.123851] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [ 8077.133459] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 8077.265336] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 8077.278659] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 8077.294975] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 8077.304377] RIP: 0010:[] [] osd_inode_iteration+0x251/0xcc0 [osd_ldiskfs] [ 8077.316632] RSP: 0018:ffff880703603d18 EFLAGS: 00000286 [ 8077.323601] RAX: ffff8800b2a37468 RBX: 000000009f8f2001 RCX: 0000000000000000 [ 8077.332597] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 8077.341584] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [ 8077.350554] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [ 8077.359514] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [ 8077.368469] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 8077.378489] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8077.385899] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 8077.394869] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8077.403834] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 8077.412792] Stack: [ 8077.416010] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [ 8077.425305] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [ 8077.434596] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [ 8077.443892] Call Trace: [ 8077.447602] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 8077.456762] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 8077.465637] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 8077.474315] [] ? wake_up_state+0x20/0x20 [ 8077.481537] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 8077.490884] [] kthread+0xcf/0xe0 [ 8077.497317] [] ? kthread_create_on_node+0x140/0x140 [ 8077.505602] [] ret_from_fork+0x58/0x90 [ 8077.512624] [] ? kthread_create_on_node+0x140/0x140 [ 8077.520909] Code: b0 00 00 00 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 00 8b 75 98 48 8b 7d 88 e8 dc 2d f5 ff 48 85 c0 48 89 45 90 0f 84 b3 08 00 00 <44> 8b 7d a0 48 8b 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 e8 77 [ 8098.494062] perf: interrupt took too long (3223 > 3190), lowering kernel.perf_event_max_sample_rate to 62000 [ 8125.125524] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [ 8125.135792] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 8125.268115] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 8125.281439] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 8125.297751] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 8125.307192] RIP: 0010:[] [] osd_scrub_next+0x3d/0x3c0 [osd_ldiskfs] [ 8125.318888] RSP: 0018:ffff880703603c70 EFLAGS: 00000246 [ 8125.325856] RAX: 0000000000000000 RBX: ffff880703603c37 RCX: ffff880703603d98 [ 8125.334860] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 8125.343854] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [ 8125.352832] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [ 8125.361800] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [ 8125.370776] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 8125.380831] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8125.388267] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 8125.397270] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8125.406258] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 8125.415236] Stack: [ 8125.418468] 000000009f8f2001 0000000000000004 ffff880703603d57 ffff880703603df0 [ 8125.427784] 0000000000000000 ffff880703603d57 000000009f8f2001 0000000000000004 [ 8125.437107] ffff880703603d57 ffff8800b2a37468 0000000000000000 ffff880703603d78 [ 8125.446420] Call Trace: [ 8125.450181] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [ 8125.459371] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 8125.468559] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 8125.477448] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 8125.486150] [] ? wake_up_state+0x20/0x20 [ 8125.493401] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 8125.502783] [] kthread+0xcf/0xe0 [ 8125.509234] [] ? kthread_create_on_node+0x140/0x140 [ 8125.517554] [] ret_from_fork+0x58/0x90 [ 8125.524591] [] ? kthread_create_on_node+0x140/0x140 [ 8125.532893] Code: 49 89 fd 41 54 49 89 d4 53 48 89 cb 48 83 ec 70 65 48 8b 04 25 28 00 00 00 48 89 45 d0 31 c0 48 8b 05 f8 59 9c ff 48 85 c0 75 7b <41> f6 87 58 10 00 00 08 0f 84 62 03 00 00 4d 8d b7 e0 10 00 00 [ 8153.126499] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [ 8153.136264] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 8153.268064] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 8153.281369] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 8153.297634] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 8153.307031] RIP: 0010:[] [] osd_inode_iteration+0x489/0xcc0 [osd_ldiskfs] [ 8153.319285] RSP: 0018:ffff880703603d18 EFLAGS: 00000293 [ 8153.326240] RAX: 0000000000000004 RBX: 000000009f8f2001 RCX: 0000000000000000 [ 8153.335674] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 8153.344653] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [ 8153.353629] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [ 8153.362589] R13: ffff880703603d08 R14: 000000009f8f2001 R15: ffff880703603d08 [ 8153.371569] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 8153.381612] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8153.389025] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 8153.398000] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8153.406970] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 8153.415932] Stack: [ 8153.419151] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [ 8153.428437] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [ 8153.438223] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [ 8153.447535] Call Trace: [ 8153.451257] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 8153.460445] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 8153.469329] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 8153.478003] [] ? wake_up_state+0x20/0x20 [ 8153.485218] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 8153.494578] [] kthread+0xcf/0xe0 [ 8153.501010] [] ? kthread_create_on_node+0x140/0x140 [ 8153.509315] [] ret_from_fork+0x58/0x90 [ 8153.516342] [] ? kthread_create_on_node+0x140/0x140 [ 8153.524626] Code: 00 e8 7c eb 97 ff e9 0f fc ff ff 0f 1f 80 00 00 00 00 45 89 e9 4c 8d 85 67 ff ff ff 48 8b 4d a8 48 8d 55 88 48 8b b5 38 ff ff ff <48> 8b bd 48 ff ff ff 48 8b 85 28 ff ff ff ff d0 85 c0 41 89 c5 [ 8181.127481] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [ 8181.137147] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 8181.269565] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 8181.282882] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 8181.299193] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 8181.308618] RIP: 0010:[] [] ldiskfs_itable_unused_count+0x23/0x30 [ldiskfs] [ 8181.321095] RSP: 0018:ffff880703603d08 EFLAGS: 00000297 [ 8181.328069] RAX: 0000000000000000 RBX: ffff880703603d57 RCX: ffff88042bd78800 [ 8181.337067] RDX: 00000000000072ce RSI: ffff880419dc3a00 RDI: ffff88042bd79800 [ 8181.346039] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [ 8181.355013] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [ 8181.363975] R13: 0000000000000018 R14: ffff880703603c78 R15: 0000000000000246 [ 8181.372924] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 8181.382952] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8181.390353] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 8181.399321] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8181.408282] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 8181.417236] Stack: [ 8181.420449] ffff880703603df0 ffffffffa121cfb9 ffffffffa121b990 ffffffffa1217a20 [ 8181.429733] ffff8800b2a36000 00000000810d354f ffff8803162f6000 ffff8800b2a37468 [ 8181.439010] 0000000020000000 010000000000000c 0000000000000000 0000000000000000 [ 8181.448294] Call Trace: [ 8181.452052] [] osd_inode_iteration+0x269/0xcc0 [osd_ldiskfs] [ 8181.461219] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 8181.470374] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 8181.479245] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 8181.487926] [] ? wake_up_state+0x20/0x20 [ 8181.495131] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 8181.504482] [] kthread+0xcf/0xe0 [ 8181.510896] [] ? kthread_create_on_node+0x140/0x140 [ 8181.519178] [] ret_from_fork+0x58/0x90 [ 8181.526186] [] ? kthread_create_on_node+0x140/0x140 [ 8181.534454] Code: 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 8b 8f a8 03 00 00 31 c0 0f b7 56 1c 48 89 e5 48 83 39 3f 76 07 0f b7 46 32 c1 e0 10 <09> d0 5d c3 66 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 89 16 [ 8203.643258] INFO: rcu_sched self-detected stall on CPU { 24} (t=780033 jiffies g=907112 c=907111 q=1271641) [ 8203.655527] Task dump for CPU 24: [ 8203.660271] OI_scrub R running task 0 5551 2 0x00000088 [ 8203.669205] ffff88083fde6dd0 00000000fb6db880 ffff88082da03db0 ffffffff810c47e8 [ 8203.678539] 0000000000000018 ffffffff81a1e7c0 ffff88082da03dc8 ffffffff810c8079 [ 8203.687824] 0000000000000009 ffff88082da03df8 ffffffff811378e0 ffff88082da101c0 [ 8203.697102] Call Trace: [ 8203.700774] [] sched_show_task+0xa8/0x110 [ 8203.708760] [] dump_cpu_task+0x39/0x70 [ 8203.715715] [] rcu_dump_cpu_stacks+0x90/0xd0 [ 8203.723229] [] rcu_check_callbacks+0x442/0x720 [ 8203.730917] [] ? tick_sched_handle.isra.13+0x60/0x60 [ 8203.739167] [] update_process_times+0x47/0x80 [ 8203.746742] [] tick_sched_handle.isra.13+0x25/0x60 [ 8203.754764] [] tick_sched_timer+0x41/0x70 [ 8203.761890] [] __hrtimer_run_queues+0xd2/0x260 [ 8203.769481] [] hrtimer_interrupt+0xb0/0x1e0 [ 8203.776750] [] ? call_softirq+0x1c/0x30 [ 8203.783629] [] local_apic_timer_interrupt+0x37/0x60 [ 8203.791651] [] smp_apic_timer_interrupt+0x3f/0x60 [ 8203.799484] [] apic_timer_interrupt+0x6d/0x80 [ 8203.806914] [] ? osd_scrub_next+0x2b/0x3c0 [osd_ldiskfs] [ 8203.816164] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [ 8203.825059] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 8203.833950] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 8203.842558] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 8203.850980] [] ? wake_up_state+0x20/0x20 [ 8203.857929] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 8203.867035] [] kthread+0xcf/0xe0 [ 8203.873205] [] ? kthread_create_on_node+0x140/0x140 [ 8203.881233] [] ret_from_fork+0x58/0x90 [ 8203.887979] [] ? kthread_create_on_node+0x140/0x140 [ 8229.129152] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [ 8229.138572] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 8229.268671] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 8229.281835] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 8229.297898] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 8229.307193] RIP: 0010:[] [] osd_inode_iteration+0x4ac/0xcc0 [osd_ldiskfs] [ 8229.319328] RSP: 0018:ffff880703603d18 EFLAGS: 00000216 [ 8229.326188] RAX: 0000000020000000 RBX: 000000009f8f2001 RCX: 0000000000000000 [ 8229.335085] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 8229.343987] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [ 8229.352880] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000001 [ 8229.361779] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [ 8229.370669] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 8229.380626] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8229.387968] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 8229.396870] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8229.405778] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 8229.414673] Stack: [ 8229.417837] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [ 8229.427082] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [ 8229.436319] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [ 8229.445556] Call Trace: [ 8229.449253] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 8229.458386] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 8229.467228] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 8229.475889] [] ? wake_up_state+0x20/0x20 [ 8229.483079] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 8229.492414] [] kthread+0xcf/0xe0 [ 8229.498813] [] ? kthread_create_on_node+0x140/0x140 [ 8229.507081] [] ret_from_fork+0x58/0x90 [ 8229.514072] [] ? kthread_create_on_node+0x140/0x140 [ 8229.522325] Code: 88 48 8b b5 38 ff ff ff 48 8b bd 48 ff ff ff 48 8b 85 28 ff ff ff ff d0 85 c0 41 89 c5 75 1c 8b 85 58 ff ff ff 41 3b 04 24 72 10 <48> 8b 85 50 ff ff ff 44 3b 30 0f 87 95 fd ff ff 45 31 ff e9 a2 [ 8257.130127] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [ 8257.139734] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 8257.271672] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 8257.284980] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 8257.301298] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 8257.310697] RIP: 0010:[] [] osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 8257.322765] RSP: 0018:ffff880703603d10 EFLAGS: 00000293 [ 8257.329727] RAX: ffffffffa121b990 RBX: 000000009f8f2001 RCX: 0000000000000000 [ 8257.338709] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 8257.347699] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [ 8257.356665] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffff880703603d57 [ 8257.365622] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [ 8257.374576] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 8257.384603] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8257.392014] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 8257.400984] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8257.409954] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 8257.418910] Stack: [ 8257.422129] ffffffffa121d1e9 ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 [ 8257.431420] 00000000810d354f ffff8803162f6000 ffff8800b2a37468 0000000020000000 [ 8257.440721] 010000000000000c 0000000000000000 0000000000000000 ffff8800b2a36000 [ 8257.450022] Call Trace: [ 8257.453748] [] ? osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [ 8257.463119] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 8257.472294] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 8257.481177] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 8257.489877] [] ? wake_up_state+0x20/0x20 [ 8257.497096] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 8257.506453] [] kthread+0xcf/0xe0 [ 8257.512895] [] ? kthread_create_on_node+0x140/0x140 [ 8257.521190] [] ret_from_fork+0x58/0x90 [ 8257.528218] [] ? kthread_create_on_node+0x140/0x140 [ 8257.536499] Code: b5 07 00 00 c7 05 85 a5 02 00 00 00 02 00 89 54 24 08 48 8b 40 40 ba 07 00 00 00 48 89 04 24 31 c0 e8 a5 03 98 ff e9 c7 fd ff ff <66> 66 66 66 90 55 48 89 e5 41 57 49 89 f7 41 56 41 55 41 54 45 [ 8285.131096] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [ 8285.140754] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 8285.272515] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 8285.285810] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 8285.302089] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 8285.311482] RIP: 0010:[] [] osd_scrub_next+0x2b/0x3c0 [osd_ldiskfs] [ 8285.323144] RSP: 0018:ffff880703603c70 EFLAGS: 00000282 [ 8285.330091] RAX: 00000000fb6db880 RBX: ffff880703603c37 RCX: ffff880703603d98 [ 8285.339082] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 8285.348055] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [ 8285.357011] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [ 8285.365975] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [ 8285.374940] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 8285.384982] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8285.392402] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 8285.401380] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8285.410348] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 8285.419303] Stack: [ 8285.422508] 000000009f8f2001 ffff880703603d08 000000009f8f2001 ffff880703603df0 [ 8285.431812] 000000009f8f2001 ffff880703603d57 000000009f8f2001 0000000000000004 [ 8285.441117] ffff880703603d57 0000000000000000 0000000000000000 ffff880703603d78 [ 8285.450392] Call Trace: [ 8285.454107] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [ 8285.463273] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 8285.472439] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 8285.481307] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 8285.489977] [] ? wake_up_state+0x20/0x20 [ 8285.497186] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 8285.506538] [] kthread+0xcf/0xe0 [ 8285.512969] [] ? kthread_create_on_node+0x140/0x140 [ 8285.521252] [] ret_from_fork+0x58/0x90 [ 8285.528270] [] ? kthread_create_on_node+0x140/0x140 [ 8285.536537] Code: 66 66 66 66 90 55 48 89 e5 41 57 49 89 f7 41 56 41 55 49 89 fd 41 54 49 89 d4 53 48 89 cb 48 83 ec 70 65 48 8b 04 25 28 00 00 00 <48> 89 45 d0 31 c0 48 8b 05 f8 59 9c ff 48 85 c0 75 7b 41 f6 87 [ 8313.132072] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [ 8313.141708] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 8313.273288] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 8313.286569] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 8313.302808] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 8313.312195] RIP: 0010:[] [] osd_scrub_next+0x160/0x3c0 [osd_ldiskfs] [ 8313.323920] RSP: 0018:ffff880703603ce0 EFLAGS: 00000282 [ 8313.330861] RAX: 0000000000000004 RBX: ffff880703603c78 RCX: ffff880703603d98 [ 8313.339832] RDX: ffff880703603d78 RSI: 0000000000000000 RDI: ffff8803162f6000 [ 8313.348792] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [ 8313.357738] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000010 [ 8313.366678] R13: ffffffffa121ba87 R14: ffffffffffffff10 R15: ffff8803162f6000 [ 8313.375619] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 8313.385639] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8313.393033] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 8313.401989] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8313.410933] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 8313.419864] Stack: [ 8313.423059] ffff880419dc3a00 ffff8800b2a3746c 0000000000000000 00000000ffffffff [ 8313.432342] 0000000000000001 ffff880703603df0 ffffffffa121cff9 ffffffffa121b990 [ 8313.441634] ffffffffa1217a20 ffff8800b2a36000 00000000810d354f ffff8803162f6000 [ 8313.450925] Call Trace: [ 8313.454636] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [ 8313.463801] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 8313.472961] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 8313.481821] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 8313.490478] [] ? wake_up_state+0x20/0x20 [ 8313.497684] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 8313.507026] [] kthread+0xcf/0xe0 [ 8313.513453] [] ? kthread_create_on_node+0x140/0x140 [ 8313.521734] [] ret_from_fork+0x58/0x90 [ 8313.528749] [] ? kthread_create_on_node+0x140/0x140 [ 8313.537015] Code: 10 00 00 02 00 00 00 48 89 df e8 fc 69 47 e0 b8 05 00 00 00 48 8b 75 d0 65 48 33 34 25 28 00 00 00 0f 85 4c 02 00 00 48 83 c4 70 <5b> 41 5c 41 5d 41 5e 41 5f 5d c3 48 8b 05 be 58 9c ff 48 85 c0 [ 8341.133049] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [ 8341.142675] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 8341.274370] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 8341.287660] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 8341.303922] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 8341.313306] RIP: 0010:[] [] osd_scrub_exec+0x30/0x5b0 [osd_ldiskfs] [ 8341.324941] RSP: 0018:ffff880703603c78 EFLAGS: 00000296 [ 8341.331881] RAX: 00000000fb6db880 RBX: ffff880703603c37 RCX: 0000000000000000 [ 8341.340857] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 8341.349821] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [ 8341.358772] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [ 8341.367723] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [ 8341.376673] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 8341.386710] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8341.394128] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 8341.403100] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8341.412060] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 8341.421008] Stack: [ 8341.424221] ffff880703603d08 ffff880703603d57 ffff880703603df0 000000009f8f2001 [ 8341.433506] ffff880703603d57 000000009f8f2001 0000000000000004 ffff880703603d57 [ 8341.442808] ffff88042bd78800 ffff88042bd78800 00000000000072cf ffff880419dc3a00 [ 8341.452106] Call Trace: [ 8341.455825] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [ 8341.464997] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 8341.474155] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 8341.483022] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 8341.491684] [] ? wake_up_state+0x20/0x20 [ 8341.498890] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 8341.508230] [] kthread+0xcf/0xe0 [ 8341.514664] [] ? kthread_create_on_node+0x140/0x140 [ 8341.522944] [] ret_from_fork+0x58/0x90 [ 8341.529952] [] ? kthread_create_on_node+0x140/0x140 [ 8341.538223] Code: 55 48 89 e5 41 57 49 89 f7 41 56 41 55 41 54 45 31 e4 53 48 83 ec 68 48 8b 9e 08 10 00 00 4c 89 45 90 65 48 8b 04 25 28 00 00 00 <48> 89 45 d0 31 c0 48 8d 43 08 48 85 db 4c 0f 45 e0 41 83 f9 04 [ 8369.134029] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [ 8369.143658] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 8369.275277] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 8369.288566] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 8369.304826] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 8369.314206] RIP: 0010:[] [] osd_inode_iteration+0x2c0/0xcc0 [osd_ldiskfs] [ 8369.326427] RSP: 0018:ffff880703603d18 EFLAGS: 00000297 [ 8369.333367] RAX: 0000000000000004 RBX: 000000009f8f2001 RCX: ffff880703603d98 [ 8369.342344] RDX: ffff880703603d78 RSI: 0000000000000000 RDI: ffff8803162f6000 [ 8369.351308] RBP: ffff880703603df0 R08: 0000000000000001 R09: 0000000000000004 [ 8369.360247] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000001 [ 8369.369191] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [ 8369.378140] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 8369.388160] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8369.395563] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 8369.404519] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8369.413465] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 8369.422398] Stack: [ 8369.425597] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [ 8369.434878] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [ 8369.444171] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [ 8369.453463] Call Trace: [ 8369.457161] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 8369.466321] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 8369.475183] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 8369.483840] [] ? wake_up_state+0x20/0x20 [ 8369.491036] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 8369.500381] [] kthread+0xcf/0xe0 [ 8369.506806] [] ? kthread_create_on_node+0x140/0x140 [ 8369.515074] [] ret_from_fork+0x58/0x90 [ 8369.522087] [] ? kthread_create_on_node+0x140/0x140 [ 8369.530354] Code: 38 ff ff ff 48 8b bd 48 ff ff ff 48 8b 85 30 ff ff ff ff d0 83 f8 03 41 89 c5 0f 84 4b 02 00 00 0f 8e 15 01 00 00 83 f8 05 66 90 <0f> 84 8a 02 00 00 83 f8 06 0f 85 a1 01 00 00 48 8b 7d 90 48 85 [ 8383.652544] INFO: rcu_sched self-detected stall on CPU { 24} (t=960036 jiffies g=907112 c=907111 q=1306848) [ 8383.664808] Task dump for CPU 24: [ 8383.669559] OI_scrub R running task 0 5551 2 0x00000088 [ 8383.678489] ffff88083fde6dd0 00000000fb6db880 ffff88082da03db0 ffffffff810c47e8 [ 8383.687814] 0000000000000018 ffffffff81a1e7c0 ffff88082da03dc8 ffffffff810c8079 [ 8383.697134] 0000000000000009 ffff88082da03df8 ffffffff811378e0 ffff88082da101c0 [ 8383.706441] Call Trace: [ 8383.710137] [] sched_show_task+0xa8/0x110 [ 8383.718137] [] dump_cpu_task+0x39/0x70 [ 8383.725105] [] rcu_dump_cpu_stacks+0x90/0xd0 [ 8383.732643] [] rcu_check_callbacks+0x442/0x720 [ 8383.740347] [] ? tick_sched_handle.isra.13+0x60/0x60 [ 8383.748618] [] update_process_times+0x47/0x80 [ 8383.756185] [] tick_sched_handle.isra.13+0x25/0x60 [ 8383.764209] [] tick_sched_timer+0x41/0x70 [ 8383.771340] [] __hrtimer_run_queues+0xd2/0x260 [ 8383.778940] [] hrtimer_interrupt+0xb0/0x1e0 [ 8383.786231] [] ? call_softirq+0x1c/0x30 [ 8383.793122] [] local_apic_timer_interrupt+0x37/0x60 [ 8383.801155] [] smp_apic_timer_interrupt+0x3f/0x60 [ 8383.808996] [] apic_timer_interrupt+0x6d/0x80 [ 8383.816429] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 8383.825750] [] ? osd_scrub_next+0x1e/0x3c0 [osd_ldiskfs] [ 8383.834272] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [ 8383.843176] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 8383.852080] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 8383.860695] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 8383.869103] [] ? wake_up_state+0x20/0x20 [ 8383.876054] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 8383.885138] [] kthread+0xcf/0xe0 [ 8383.891317] [] ? kthread_create_on_node+0x140/0x140 [ 8383.899340] [] ret_from_fork+0x58/0x90 [ 8383.906095] [] ? kthread_create_on_node+0x140/0x140 [ 8409.135416] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [ 8409.144794] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 8409.274556] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 8409.287727] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 8409.303769] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 8409.313060] RIP: 0010:[] [] osd_scrub_exec+0x96/0x5b0 [osd_ldiskfs] [ 8409.324621] RSP: 0018:ffff880703603d10 EFLAGS: 00000292 [ 8409.331490] RAX: 0000000000000000 RBX: 000000009f8f2001 RCX: 0000000000000000 [ 8409.340395] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 8409.349292] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [ 8409.358194] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000001 [ 8409.367089] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [ 8409.375967] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 8409.385933] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8409.393278] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 8409.402192] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8409.411097] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 8409.419988] Stack: [ 8409.423155] ffffffffa121d1e9 ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 [ 8409.432404] 00000000810d354f ffff8803162f6000 ffff8800b2a37468 0000000020000000 [ 8409.441649] 010000000000000c 0000000000000000 0000000000000000 ffff8800b2a36000 [ 8409.450901] Call Trace: [ 8409.454593] [] ? osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [ 8409.463919] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 8409.473049] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 8409.481890] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 8409.490538] [] ? wake_up_state+0x20/0x20 [ 8409.497714] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 8409.507046] [] kthread+0xcf/0xe0 [ 8409.513453] [] ? kthread_create_on_node+0x140/0x140 [ 8409.521700] [] ret_from_fork+0x58/0x90 [ 8409.528688] [] ? kthread_create_on_node+0x140/0x140 [ 8409.536937] Code: 00 00 41 80 a7 74 14 00 00 fe 48 8b 4d d0 65 48 33 0c 25 28 00 00 00 0f 85 b1 03 00 00 48 83 c4 68 5b 41 5c 41 5d 41 5e 41 5f 5d 66 0f 1f 84 00 00 00 00 00 4c 8d ae f0 10 00 00 44 89 4d 88 [ 8437.136398] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [ 8437.145991] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 8437.278124] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 8437.291409] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 8437.307676] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 8437.317060] RIP: 0010:[] [] osd_inode_iteration+0x251/0xcc0 [osd_ldiskfs] [ 8437.329319] RSP: 0018:ffff880703603d18 EFLAGS: 00000286 [ 8437.336270] RAX: ffff8800b2a37468 RBX: 000000009f8f2001 RCX: 0000000000000000 [ 8437.345259] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 8437.354236] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [ 8437.363191] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [ 8437.372151] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [ 8437.381107] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 8437.391143] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8437.398553] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 8437.407523] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8437.416502] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 8437.425446] Stack: [ 8437.428659] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [ 8437.437927] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [ 8437.447236] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [ 8437.456535] Call Trace: [ 8437.460272] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 8437.469440] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 8437.478320] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 8437.486998] [] ? wake_up_state+0x20/0x20 [ 8437.494212] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 8437.503577] [] kthread+0xcf/0xe0 [ 8437.510015] [] ? kthread_create_on_node+0x140/0x140 [ 8437.518320] [] ret_from_fork+0x58/0x90 [ 8437.525340] [] ? kthread_create_on_node+0x140/0x140 [ 8437.533624] Code: b0 00 00 00 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 00 8b 75 98 48 8b 7d 88 e8 dc 2d f5 ff 48 85 c0 48 89 45 90 0f 84 b3 08 00 00 <44> 8b 7d a0 48 8b 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 e8 77 [ 8460.520491] Lustre: MGS: Received new LWP connection from 192.168.1.124@o2ib100, removing former export from same NID [ 8460.533571] Lustre: MGS: Connection restored to 5a0e6555-4c2d-39c9-90c4-a332f62c8353 (at 192.168.1.124@o2ib100) [ 8460.545926] Lustre: Skipped 17 previous similar messages [ 8465.137373] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [ 8465.147021] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 8465.279251] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 8465.292541] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 8465.308810] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 8465.318211] RIP: 0010:[] [] osd_inode_iteration+0x4ac/0xcc0 [osd_ldiskfs] [ 8465.330469] RSP: 0018:ffff880703603d18 EFLAGS: 00000216 [ 8465.337436] RAX: 0000000020000000 RBX: ffff880703603d57 RCX: 0000000000000000 [ 8465.346429] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 8465.355425] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [ 8465.364417] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [ 8465.373406] R13: ffffea00027e3c80 R14: 000000009f8f2001 R15: 0000000000000018 [ 8465.382403] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 8465.392482] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8465.399926] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 8465.408925] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8465.417907] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 8465.426893] Stack: [ 8465.430158] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [ 8465.439494] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [ 8465.448822] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [ 8465.458147] Call Trace: [ 8465.461913] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 8465.471117] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 8465.480024] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 8465.488740] [] ? wake_up_state+0x20/0x20 [ 8465.495993] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 8465.505394] [] kthread+0xcf/0xe0 [ 8465.511868] [] ? kthread_create_on_node+0x140/0x140 [ 8465.520205] [] ret_from_fork+0x58/0x90 [ 8465.527257] [] ? kthread_create_on_node+0x140/0x140 [ 8465.535567] Code: 88 48 8b b5 38 ff ff ff 48 8b bd 48 ff ff ff 48 8b 85 28 ff ff ff ff d0 85 c0 41 89 c5 75 1c 8b 85 58 ff ff ff 41 3b 04 24 72 10 <48> 8b 85 50 ff ff ff 44 3b 30 0f 87 95 fd ff ff 45 31 ff e9 a2 [ 8493.138348] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [ 8493.148023] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 8493.280438] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 8493.293742] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 8493.310037] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 8493.319449] RIP: 0010:[] [] osd_inode_iteration+0x282/0xcc0 [osd_ldiskfs] [ 8493.331709] RSP: 0018:ffff880703603d18 EFLAGS: 00000287 [ 8493.338673] RAX: ffff88042bd78800 RBX: 000000009f8f2001 RCX: ffff88042bd78800 [ 8493.347664] RDX: 00000000000072cf RSI: ffff880419dc3a00 RDI: ffff88042bd79800 [ 8493.356647] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [ 8493.365616] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [ 8493.374574] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [ 8493.383528] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 8493.393557] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8493.400968] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 8493.409947] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8493.418935] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 8493.427912] Stack: [ 8493.431133] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [ 8493.440459] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [ 8493.449765] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [ 8493.459073] Call Trace: [ 8493.462835] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 8493.472036] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 8493.480919] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 8493.489610] [] ? wake_up_state+0x20/0x20 [ 8493.496832] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 8493.506200] [] kthread+0xcf/0xe0 [ 8493.512637] [] ? kthread_create_on_node+0x140/0x140 [ 8493.520925] [] ret_from_fork+0x58/0x90 [ 8493.527946] [] ? kthread_create_on_node+0x140/0x140 [ 8493.536233] Code: 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 e8 77 5b f4 ff 42 8d 14 38 48 8b 45 88 48 8b 80 a8 03 00 00 48 3b 50 20 0f 83 57 01 00 00 <44> 0f b6 85 67 ff ff ff 48 8d 4d a8 48 8d 55 88 48 8b b5 38 ff [ 8521.139318] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [ 8521.148963] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 8521.280892] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 8521.294204] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 8521.310529] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 8521.319945] RIP: 0010:[] [] osd_scrub_exec+0x79/0x5b0 [osd_ldiskfs] [ 8521.331617] RSP: 0018:ffff880703603c78 EFLAGS: 00000202 [ 8521.338595] RAX: 0000000000000000 RBX: ffff880703603c37 RCX: 00000000fb6db880 [ 8521.347592] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 8521.356581] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [ 8521.365561] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [ 8521.374513] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [ 8521.383472] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 8521.393509] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8521.400925] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 8521.409900] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8521.418870] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 8521.427827] Stack: [ 8521.431048] ffffea00027e3c80 000000009f8f2001 ffff880703603df0 0000000000000001 [ 8521.440342] ffff880703603d57 000000009f8f2001 0000000000000004 0000000000000001 [ 8521.449630] ffffffffa1217a20 ffff880703603d98 ffff880703603d78 ffff8800b2a36000 [ 8521.458928] Call Trace: [ 8521.462656] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 8521.471545] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [ 8521.480742] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 8521.489927] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 8521.498791] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 8521.507459] [] ? wake_up_state+0x20/0x20 [ 8521.514694] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 8521.524057] [] kthread+0xcf/0xe0 [ 8521.530488] [] ? kthread_create_on_node+0x140/0x140 [ 8521.538778] [] ret_from_fork+0x58/0x90 [ 8521.545792] [] ? kthread_create_on_node+0x140/0x140 [ 8521.554060] Code: 07 74 4f 41 83 f9 02 74 78 48 89 ca 44 89 c9 e8 0e c6 ff ff 85 c0 41 89 c1 0f 84 a3 01 00 00 41 80 a7 74 14 00 00 fe 48 8b 4d d0 <65> 48 33 0c 25 28 00 00 00 0f 85 b1 03 00 00 48 83 c4 68 5b 41 [ 8549.140299] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [ 8549.149924] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 8549.281524] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 8549.294789] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 8549.311005] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 8549.320370] RIP: 0010:[] [] ldiskfs_itable_unused_count+0x6/0x30 [ldiskfs] [ 8549.332691] RSP: 0018:ffff880703603d08 EFLAGS: 00000286 [ 8549.339637] RAX: ffff8800b2a37468 RBX: 000000009f8f2001 RCX: 0000000000000000 [ 8549.348605] RDX: ffff880703603d78 RSI: ffff880419dc3a00 RDI: ffff88042bd79800 [ 8549.357570] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [ 8549.366529] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [ 8549.375488] R13: 0000000000000018 R14: ffff880703603c78 R15: 0000000000000286 [ 8549.384440] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 8549.394468] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8549.401867] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 8549.410825] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8549.419776] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 8549.428715] Stack: [ 8549.431927] ffff880703603df0 ffffffffa121cfb9 ffffffffa121b990 ffffffffa1217a20 [ 8549.441230] ffff8800b2a36000 00000000810d354f ffff8803162f6000 ffff8800b2a37468 [ 8549.450513] 0000000020000000 010000000000000c 0000000000000000 0000000000000000 [ 8549.459807] Call Trace: [ 8549.463532] [] ? osd_inode_iteration+0x269/0xcc0 [osd_ldiskfs] [ 8549.472896] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 8549.482060] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 8549.490927] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 8549.499584] [] ? wake_up_state+0x20/0x20 [ 8549.506800] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 8549.516151] [] kthread+0xcf/0xe0 [ 8549.522580] [] ? kthread_create_on_node+0x140/0x140 [ 8549.530863] [] ret_from_fork+0x58/0x90 [ 8549.537874] [] ? kthread_create_on_node+0x140/0x140 [ 8549.546151] Code: 00 00 31 c0 0f b7 56 10 48 89 e5 48 83 39 3f 76 07 0f b7 46 30 c1 e0 10 09 d0 5d c3 66 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 <48> 8b 8f a8 03 00 00 31 c0 0f b7 56 1c 48 89 e5 48 83 39 3f 76 [ 8563.661807] INFO: rcu_sched self-detected stall on CPU { 24} (t=1140039 jiffies g=907112 c=907111 q=1391933) [ 8563.674140] Task dump for CPU 24: [ 8563.678865] OI_scrub R running task 0 5551 2 0x00000088 [ 8563.687782] ffff88083fde6dd0 00000000fb6db880 ffff88082da03db0 ffffffff810c47e8 [ 8563.697105] 0000000000000018 ffffffff81a1e7c0 ffff88082da03dc8 ffffffff810c8079 [ 8563.706407] 0000000000000009 ffff88082da03df8 ffffffff811378e0 ffff88082da101c0 [ 8563.715688] Call Trace: [ 8563.719368] [] sched_show_task+0xa8/0x110 [ 8563.727339] [] dump_cpu_task+0x39/0x70 [ 8563.734293] [] rcu_dump_cpu_stacks+0x90/0xd0 [ 8563.741813] [] rcu_check_callbacks+0x442/0x720 [ 8563.749507] [] ? tick_sched_handle.isra.13+0x60/0x60 [ 8563.757757] [] update_process_times+0x47/0x80 [ 8563.765305] [] tick_sched_handle.isra.13+0x25/0x60 [ 8563.773307] [] tick_sched_timer+0x41/0x70 [ 8563.780430] [] __hrtimer_run_queues+0xd2/0x260 [ 8563.788014] [] hrtimer_interrupt+0xb0/0x1e0 [ 8563.795284] [] ? call_softirq+0x1c/0x30 [ 8563.802157] [] local_apic_timer_interrupt+0x37/0x60 [ 8563.810179] [] smp_apic_timer_interrupt+0x3f/0x60 [ 8563.817996] [] apic_timer_interrupt+0x6d/0x80 [ 8563.825414] [] ? osd_scrub_next+0x2b/0x3c0 [osd_ldiskfs] [ 8563.834646] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [ 8563.843544] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 8563.852445] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 8563.861051] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 8563.869461] [] ? wake_up_state+0x20/0x20 [ 8563.876414] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 8563.885499] [] kthread+0xcf/0xe0 [ 8563.891664] [] ? kthread_create_on_node+0x140/0x140 [ 8563.899682] [] ret_from_fork+0x58/0x90 [ 8563.906430] [] ? kthread_create_on_node+0x140/0x140 [ 8589.141692] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [ 8589.151085] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 8589.280566] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 8589.293731] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 8589.309793] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 8589.319081] RIP: 0010:[] [] osd_scrub_next+0x3d/0x3c0 [osd_ldiskfs] [ 8589.330636] RSP: 0018:ffff880703603c70 EFLAGS: 00000246 [ 8589.337486] RAX: 0000000000000000 RBX: ffff880703603c37 RCX: ffff880703603d98 [ 8589.346389] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 8589.355279] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [ 8589.364167] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [ 8589.373057] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [ 8589.381943] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 8589.391904] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8589.399240] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 8589.408136] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8589.417036] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 8589.425927] Stack: [ 8589.429087] 000000009f8f2001 ffff880703603d08 0000000000000001 ffff880703603df0 [ 8589.438309] 000000009f8f2001 ffff880703603d57 000000009f8f2001 0000000000000004 [ 8589.447558] ffff880703603d57 ffff88042bd78800 ffff88042bd78800 00000000000072cf [ 8589.456798] Call Trace: [ 8589.460461] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [ 8589.469588] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 8589.478709] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 8589.487538] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 8589.496174] [] ? wake_up_state+0x20/0x20 [ 8589.503345] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 8589.512645] [] kthread+0xcf/0xe0 [ 8589.519037] [] ? kthread_create_on_node+0x140/0x140 [ 8589.527280] [] ret_from_fork+0x58/0x90 [ 8589.534251] [] ? kthread_create_on_node+0x140/0x140 [ 8589.542486] Code: 49 89 fd 41 54 49 89 d4 53 48 89 cb 48 83 ec 70 65 48 8b 04 25 28 00 00 00 48 89 45 d0 31 c0 48 8b 05 f8 59 9c ff 48 85 c0 75 7b <41> f6 87 58 10 00 00 08 0f 84 62 03 00 00 4d 8d b7 e0 10 00 00 [ 8617.142669] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [ 8617.152268] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 8617.283823] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 8617.297101] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 8617.313343] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 8617.322721] RIP: 0010:[] [] ldiskfs_itable_unused_count+0x16/0x30 [ldiskfs] [ 8617.335139] RSP: 0018:ffff880703603d08 EFLAGS: 00000246 [ 8617.342074] RAX: 0000000000000000 RBX: 0000000000000001 RCX: ffff88042bd78800 [ 8617.351049] RDX: 00000000000072ce RSI: ffff880419dc3a00 RDI: ffff88042bd79800 [ 8617.360020] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [ 8617.368968] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [ 8617.377912] R13: 0000000000000018 R14: ffff880703603c78 R15: 0000000000000286 [ 8617.386856] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 8617.396872] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8617.404266] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 8617.413217] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8617.422159] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 8617.431097] Stack: [ 8617.434291] ffff880703603df0 ffffffffa121cfb9 ffffffffa121b990 ffffffffa1217a20 [ 8617.443573] ffff8800b2a36000 00000000810d354f ffff8803162f6000 ffff8800b2a37468 [ 8617.452859] 0000000020000000 010000000000000c 0000000000000000 0000000000000000 [ 8617.462135] Call Trace: [ 8617.465853] [] osd_inode_iteration+0x269/0xcc0 [osd_ldiskfs] [ 8617.475022] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 8617.484177] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 8617.493039] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 8617.501691] [] ? wake_up_state+0x20/0x20 [ 8617.508898] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 8617.518241] [] kthread+0xcf/0xe0 [ 8617.524664] [] ? kthread_create_on_node+0x140/0x140 [ 8617.532939] [] ret_from_fork+0x58/0x90 [ 8617.539944] [] ? kthread_create_on_node+0x140/0x140 [ 8617.548206] Code: 07 0f b7 46 30 c1 e0 10 09 d0 5d c3 66 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 8b 8f a8 03 00 00 31 c0 0f b7 56 1c 48 89 e5 <48> 83 39 3f 76 07 0f b7 46 32 c1 e0 10 09 d0 5d c3 66 0f 1f 84 [ 8645.143643] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [ 8645.153277] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 8645.284989] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 8645.298261] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 8645.314518] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 8645.323898] RIP: 0010:[] [] osd_inode_iteration+0x282/0xcc0 [osd_ldiskfs] [ 8645.336119] RSP: 0018:ffff880703603d18 EFLAGS: 00000287 [ 8645.343054] RAX: ffff88042bd78800 RBX: ffff880703603d57 RCX: ffff88042bd78800 [ 8645.352028] RDX: 00000000000072cf RSI: ffff880419dc3a00 RDI: ffff88042bd79800 [ 8645.360988] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [ 8645.369934] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [ 8645.378882] R13: ffffea00027e3c80 R14: 000000009f8f2001 R15: ffff880703603d08 [ 8645.387832] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 8645.397862] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8645.405265] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 8645.414231] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8645.423190] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 8645.432145] Stack: [ 8645.435357] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [ 8645.444652] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [ 8645.453970] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [ 8645.463269] Call Trace: [ 8645.466989] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 8645.476157] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 8645.485030] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 8645.493711] [] ? wake_up_state+0x20/0x20 [ 8645.500918] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 8645.510282] [] kthread+0xcf/0xe0 [ 8645.516724] [] ? kthread_create_on_node+0x140/0x140 [ 8645.525015] [] ret_from_fork+0x58/0x90 [ 8645.532028] [] ? kthread_create_on_node+0x140/0x140 [ 8645.540303] Code: 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 e8 77 5b f4 ff 42 8d 14 38 48 8b 45 88 48 8b 80 a8 03 00 00 48 3b 50 20 0f 83 57 01 00 00 <44> 0f b6 85 67 ff ff ff 48 8d 4d a8 48 8d 55 88 48 8b b5 38 ff [ 8673.144612] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [ 8673.154237] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 8673.286111] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 8673.299421] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 8673.315735] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 8673.325134] RIP: 0010:[] [] osd_scrub_exec+0x96/0x5b0 [osd_ldiskfs] [ 8673.336804] RSP: 0018:ffff880703603d10 EFLAGS: 00000292 [ 8673.343773] RAX: 0000000000000000 RBX: ffff880703603d98 RCX: 0000000000000000 [ 8673.352763] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 8673.361735] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [ 8673.370691] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000001 [ 8673.379650] R13: 0000000000000004 R14: 000000009f8f2001 R15: ffffea00027e3c80 [ 8673.388597] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 8673.398617] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8673.406019] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 8673.414981] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8673.423937] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 8673.432884] Stack: [ 8673.436090] ffffffffa121d1e9 ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 [ 8673.445364] 00000000810d354f ffff8803162f6000 ffff8800b2a37468 0000000020000000 [ 8673.454641] 010000000000000c 0000000000000000 0000000000000000 ffff8800b2a36000 [ 8673.463952] Call Trace: [ 8673.467671] [] ? osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [ 8673.477020] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 8673.486189] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 8673.495059] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 8673.503730] [] ? wake_up_state+0x20/0x20 [ 8673.510927] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 8673.520270] [] kthread+0xcf/0xe0 [ 8673.526704] [] ? kthread_create_on_node+0x140/0x140 [ 8673.534975] [] ret_from_fork+0x58/0x90 [ 8673.541982] [] ? kthread_create_on_node+0x140/0x140 [ 8673.550240] Code: 00 00 41 80 a7 74 14 00 00 fe 48 8b 4d d0 65 48 33 0c 25 28 00 00 00 0f 85 b1 03 00 00 48 83 c4 68 5b 41 5c 41 5d 41 5e 41 5f 5d 66 0f 1f 84 00 00 00 00 00 4c 8d ae f0 10 00 00 44 89 4d 88 [ 8701.145595] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [ 8701.155222] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 8701.286903] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 8701.300188] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 8701.316428] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 8701.325810] RIP: 0010:[] [] osd_inode_iteration+0x4ac/0xcc0 [osd_ldiskfs] [ 8701.338043] RSP: 0018:ffff880703603d18 EFLAGS: 00000216 [ 8701.344982] RAX: 0000000020000000 RBX: 000000009f8f2001 RCX: 0000000000000000 [ 8701.353960] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 8701.362926] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [ 8701.371881] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [ 8701.380823] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [ 8701.389773] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 8701.399800] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8701.407216] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 8701.416194] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8701.425157] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 8701.434106] Stack: [ 8701.437328] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [ 8701.446615] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [ 8701.455930] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [ 8701.465223] Call Trace: [ 8701.468952] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 8701.478127] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 8701.487003] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 8701.495687] [] ? wake_up_state+0x20/0x20 [ 8701.502898] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 8701.512259] [] kthread+0xcf/0xe0 [ 8701.518692] [] ? kthread_create_on_node+0x140/0x140 [ 8701.526975] [] ret_from_fork+0x58/0x90 [ 8701.533979] [] ? kthread_create_on_node+0x140/0x140 [ 8701.542258] Code: 88 48 8b b5 38 ff ff ff 48 8b bd 48 ff ff ff 48 8b 85 28 ff ff ff ff d0 85 c0 41 89 c5 75 1c 8b 85 58 ff ff ff 41 3b 04 24 72 10 <48> 8b 85 50 ff ff ff 44 3b 30 0f 87 95 fd ff ff 45 31 ff e9 a2 [ 8729.146563] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [ 8729.156187] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 8729.287992] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 8729.301303] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 8729.317614] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 8729.327018] RIP: 0010:[] [] osd_scrub_exec+0xf9/0x5b0 [osd_ldiskfs] [ 8729.338675] RSP: 0018:ffff880703603c78 EFLAGS: 00000246 [ 8729.345634] RAX: 0000000000000000 RBX: ffff880703603c37 RCX: 0000000000000000 [ 8729.354627] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 8729.363604] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [ 8729.372563] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [ 8729.381513] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [ 8729.390458] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 8729.400481] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8729.407883] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 8729.416847] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8729.425799] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 8729.434745] Stack: [ 8729.437956] ffff880703603d08 000000009f8f2001 ffffea00027e3c80 000000009f8f2001 [ 8729.447233] ffff880703603d57 ffff880703603d57 ffffffffa121b990 0000000000000000 [ 8729.456519] ffff880703603d78 ffff8800b2a36000 ffff8803162f6000 ffffffffffffff10 [ 8729.465812] Call Trace: [ 8729.469535] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 8729.478694] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [ 8729.487848] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 8729.496999] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 8729.505861] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 8729.514510] [] ? wake_up_state+0x20/0x20 [ 8729.521705] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 8729.531051] [] kthread+0xcf/0xe0 [ 8729.537474] [] ? kthread_create_on_node+0x140/0x140 [ 8729.545742] [] ret_from_fork+0x58/0x90 [ 8729.552733] [] ? kthread_create_on_node+0x140/0x140 [ 8729.560992] Code: 88 48 85 db 74 23 f6 83 18 0a 00 00 08 74 1a 4d 85 e4 74 15 41 8b 87 6c 14 00 00 41 39 84 24 0c 0a 00 00 0f 82 d9 00 00 00 31 c0 <41> f6 87 74 14 00 00 04 0f 85 6e ff ff ff 41 83 f9 02 0f 84 64 [ 8743.671076] INFO: rcu_sched self-detected stall on CPU { 24} (t=1320042 jiffies g=907112 c=907111 q=1404478) [ 8743.683386] Task dump for CPU 24: [ 8743.688111] OI_scrub R running task 0 5551 2 0x00000088 [ 8743.697007] ffff88083fde6dd0 00000000fb6db880 ffff88082da03db0 ffffffff810c47e8 [ 8743.706310] 0000000000000018 ffffffff81a1e7c0 ffff88082da03dc8 ffffffff810c8079 [ 8743.715594] 0000000000000009 ffff88082da03df8 ffffffff811378e0 ffff88082da101c0 [ 8743.724850] Call Trace: [ 8743.728513] [] sched_show_task+0xa8/0x110 [ 8743.736459] [] dump_cpu_task+0x39/0x70 [ 8743.743391] [] rcu_dump_cpu_stacks+0x90/0xd0 [ 8743.750889] [] rcu_check_callbacks+0x442/0x720 [ 8743.758557] [] ? tick_sched_handle.isra.13+0x60/0x60 [ 8743.766788] [] update_process_times+0x47/0x80 [ 8743.774324] [] tick_sched_handle.isra.13+0x25/0x60 [ 8743.782310] [] tick_sched_timer+0x41/0x70 [ 8743.789407] [] __hrtimer_run_queues+0xd2/0x260 [ 8743.796975] [] hrtimer_interrupt+0xb0/0x1e0 [ 8743.804230] [] ? call_softirq+0x1c/0x30 [ 8743.811089] [] local_apic_timer_interrupt+0x37/0x60 [ 8743.819096] [] smp_apic_timer_interrupt+0x3f/0x60 [ 8743.826909] [] apic_timer_interrupt+0x6d/0x80 [ 8743.834316] [] ? osd_inode_iteration+0x255/0xcc0 [osd_ldiskfs] [ 8743.844118] [] ? osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [ 8743.853210] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 8743.862107] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 8743.870713] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 8743.879108] [] ? wake_up_state+0x20/0x20 [ 8743.886050] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 8743.895132] [] kthread+0xcf/0xe0 [ 8743.901308] [] ? kthread_create_on_node+0x140/0x140 [ 8743.909326] [] ret_from_fork+0x58/0x90 [ 8743.916071] [] ? kthread_create_on_node+0x140/0x140 [ 8769.147963] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [ 8769.157338] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 8769.286866] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 8769.300048] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 8769.316119] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 8769.325423] RIP: 0010:[] [] osd_inode_iteration+0x269/0xcc0 [osd_ldiskfs] [ 8769.337564] RSP: 0018:ffff880703603d18 EFLAGS: 00000202 [ 8769.344429] RAX: 00000000000072ce RBX: ffff880703603d57 RCX: ffff88042bd78800 [ 8769.353339] RDX: 00000000000072ce RSI: ffff880419dc3a00 RDI: ffff88042bd79800 [ 8769.362234] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [ 8769.371131] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [ 8769.380037] R13: ffffea00027e3c80 R14: 000000009f8f2001 R15: 0000000000000018 [ 8769.388934] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 8769.398897] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8769.406245] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 8769.415147] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8769.424053] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 8769.432948] Stack: [ 8769.436113] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [ 8769.445348] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [ 8769.454602] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [ 8769.463854] Call Trace: [ 8769.467528] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 8769.476663] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 8769.485505] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 8769.494149] [] ? wake_up_state+0x20/0x20 [ 8769.501340] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 8769.510680] [] kthread+0xcf/0xe0 [ 8769.517078] [] ? kthread_create_on_node+0x140/0x140 [ 8769.525321] [] ret_from_fork+0x58/0x90 [ 8769.532301] [] ? kthread_create_on_node+0x140/0x140 [ 8769.540547] Code: 88 e8 dc 2d f5 ff 48 85 c0 48 89 45 90 0f 84 b3 08 00 00 44 8b 7d a0 48 8b 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 e8 77 5b f4 ff <42> 8d 14 38 48 8b 45 88 48 8b 80 a8 03 00 00 48 3b 50 20 0f 83 [ 8797.148932] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [ 8797.158526] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 8797.290259] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 8797.303571] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 8797.319869] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 8797.329272] RIP: 0010:[] [] osd_scrub_exec+0x5/0x5b0 [osd_ldiskfs] [ 8797.340838] RSP: 0018:ffff880703603d10 EFLAGS: 00000293 [ 8797.347796] RAX: ffffffffa121b990 RBX: ffff880703603d98 RCX: 0000000000000000 [ 8797.356777] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 8797.365740] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [ 8797.374701] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000001 [ 8797.383645] R13: 0000000000000004 R14: 000000009f8f2001 R15: 0000000000000018 [ 8797.392581] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 8797.402603] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8797.410000] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 8797.418953] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8797.427902] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 8797.436845] Stack: [ 8797.440047] ffffffffa121d1e9 ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 [ 8797.449306] 00000000810d354f ffff8803162f6000 ffff8800b2a37468 0000000020000000 [ 8797.458587] 010000000000000c 0000000000000000 0000000000000000 ffff8800b2a36000 [ 8797.467876] Call Trace: [ 8797.471589] [] ? osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [ 8797.480943] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 8797.490106] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 8797.498962] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 8797.507620] [] ? wake_up_state+0x20/0x20 [ 8797.514820] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 8797.524158] [] kthread+0xcf/0xe0 [ 8797.530579] [] ? kthread_create_on_node+0x140/0x140 [ 8797.538854] [] ret_from_fork+0x58/0x90 [ 8797.545862] [] ? kthread_create_on_node+0x140/0x140 [ 8797.554121] Code: 05 85 a5 02 00 00 00 02 00 89 54 24 08 48 8b 40 40 ba 07 00 00 00 48 89 04 24 31 c0 e8 a5 03 98 ff e9 c7 fd ff ff 66 66 66 66 90 <55> 48 89 e5 41 57 49 89 f7 41 56 41 55 41 54 45 31 e4 53 48 83 [ 8825.149909] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [ 8825.159522] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 8825.291092] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 8825.304374] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 8825.320626] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 8825.330005] RIP: 0010:[] [] osd_scrub_next+0x169/0x3c0 [osd_ldiskfs] [ 8825.341741] RSP: 0018:ffff880703603d08 EFLAGS: 00000282 [ 8825.348674] RAX: 0000000000000004 RBX: 000000009f8f2001 RCX: ffff880703603d98 [ 8825.357646] RDX: ffff880703603d78 RSI: 0000000000000000 RDI: ffff8803162f6000 [ 8825.366605] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [ 8825.375549] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [ 8825.384488] R13: 0000000000000018 R14: ffff880703603c78 R15: 0000000000000202 [ 8825.393426] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 8825.403438] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8825.410831] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 8825.419789] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8825.428737] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 8825.437674] Stack: [ 8825.440875] ffff880703603df0 ffffffffa121cff9 ffffffffa121b990 ffffffffa1217a20 [ 8825.450176] ffff8800b2a36000 00000000810d354f ffff8803162f6000 ffff8800b2a37468 [ 8825.459471] 0000000020000000 010000000000000c 0000000000000000 0000000000000000 [ 8825.468742] Call Trace: [ 8825.472454] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [ 8825.481620] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 8825.490767] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 8825.499624] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 8825.508281] [] ? wake_up_state+0x20/0x20 [ 8825.515478] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 8825.524818] [] kthread+0xcf/0xe0 [ 8825.531236] [] ? kthread_create_on_node+0x140/0x140 [ 8825.539500] [] ret_from_fork+0x58/0x90 [ 8825.546503] [] ? kthread_create_on_node+0x140/0x140 [ 8825.554766] Code: df e8 fc 69 47 e0 b8 05 00 00 00 48 8b 75 d0 65 48 33 34 25 28 00 00 00 0f 85 4c 02 00 00 48 83 c4 70 5b 41 5c 41 5d 41 5e 41 5f <5d> c3 48 8b 05 be 58 9c ff 48 85 c0 0f 84 c2 fe ff ff 0f 1f 44 [ 8853.150884] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [ 8853.160498] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 8853.292053] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 8853.305334] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 8853.321569] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 8853.330944] RIP: 0010:[] [] osd_inode_iteration+0x269/0xcc0 [osd_ldiskfs] [ 8853.343160] RSP: 0018:ffff880703603d18 EFLAGS: 00000202 [ 8853.350092] RAX: 00000000000072ce RBX: ffff880703603d57 RCX: ffff88042bd78800 [ 8853.359062] RDX: 00000000000072ce RSI: ffff880419dc3a00 RDI: ffff88042bd79800 [ 8853.368020] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [ 8853.376959] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [ 8853.385902] R13: ffffea00027e3c80 R14: 000000009f8f2001 R15: ffff880703603d08 [ 8853.394841] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 8853.404861] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8853.412261] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 8853.421223] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8853.430177] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 8853.439119] Stack: [ 8853.442327] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [ 8853.451607] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [ 8853.460906] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [ 8853.470218] Call Trace: [ 8853.473937] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 8853.483101] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 8853.491973] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 8853.500639] [] ? wake_up_state+0x20/0x20 [ 8853.507844] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 8853.517186] [] kthread+0xcf/0xe0 [ 8853.523616] [] ? kthread_create_on_node+0x140/0x140 [ 8853.531897] [] ret_from_fork+0x58/0x90 [ 8853.538916] [] ? kthread_create_on_node+0x140/0x140 [ 8853.547193] Code: 88 e8 dc 2d f5 ff 48 85 c0 48 89 45 90 0f 84 b3 08 00 00 44 8b 7d a0 48 8b 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 e8 77 5b f4 ff <42> 8d 14 38 48 8b 45 88 48 8b 80 a8 03 00 00 48 3b 50 20 0f 83 [ 8881.151858] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [ 8881.161505] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 8881.293327] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 8881.306638] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 8881.322938] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 8881.332345] RIP: 0010:[] [] osd_inode_iteration+0x4ac/0xcc0 [osd_ldiskfs] [ 8881.344587] RSP: 0018:ffff880703603d18 EFLAGS: 00000216 [ 8881.351549] RAX: 0000000020000000 RBX: ffff880703603d98 RCX: 0000000000000000 [ 8881.360529] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 8881.369506] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [ 8881.378470] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000001 [ 8881.387420] R13: 0000000000000004 R14: 000000009f8f2001 R15: ffffea00027e3c80 [ 8881.396366] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 8881.406391] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8881.413794] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 8881.422757] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8881.431715] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 8881.440657] Stack: [ 8881.443862] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [ 8881.453137] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [ 8881.462426] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [ 8881.471719] Call Trace: [ 8881.475436] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 8881.484612] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 8881.493482] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 8881.502152] [] ? wake_up_state+0x20/0x20 [ 8881.509366] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 8881.518713] [] kthread+0xcf/0xe0 [ 8881.525141] [] ? kthread_create_on_node+0x140/0x140 [ 8881.533412] [] ret_from_fork+0x58/0x90 [ 8881.540419] [] ? kthread_create_on_node+0x140/0x140 [ 8881.548686] Code: 88 48 8b b5 38 ff ff ff 48 8b bd 48 ff ff ff 48 8b 85 28 ff ff ff ff d0 85 c0 41 89 c5 75 1c 8b 85 58 ff ff ff 41 3b 04 24 72 10 <48> 8b 85 50 ff ff ff 44 3b 30 0f 87 95 fd ff ff 45 31 ff e9 a2 [ 8909.152833] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [ 8909.162446] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 8909.294151] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 8909.307457] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 8909.323746] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 8909.333143] RIP: 0010:[] [] osd_inode_iteration+0x47a/0xcc0 [osd_ldiskfs] [ 8909.345374] RSP: 0018:ffff880703603d18 EFLAGS: 00000293 [ 8909.352330] RAX: 0000000000000004 RBX: ffff880703603d57 RCX: ffff880703603d98 [ 8909.361312] RDX: ffff880703603d78 RSI: 0000000000000000 RDI: ffff8803162f6000 [ 8909.370286] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [ 8909.379245] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [ 8909.388192] R13: ffffea00027e3c80 R14: 000000009f8f2001 R15: ffff880703603d08 [ 8909.397138] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 8909.407158] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8909.414560] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 8909.423523] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8909.432485] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 8909.441434] Stack: [ 8909.444648] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [ 8909.453938] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [ 8909.463221] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [ 8909.472514] Call Trace: [ 8909.476245] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 8909.485418] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 8909.494284] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 8909.502971] [] ? wake_up_state+0x20/0x20 [ 8909.510207] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 8909.519553] [] kthread+0xcf/0xe0 [ 8909.525991] [] ? kthread_create_on_node+0x140/0x140 [ 8909.534274] [] ret_from_fork+0x58/0x90 [ 8909.541294] [] ? kthread_create_on_node+0x140/0x140 [ 8909.549567] Code: 02 00 00 00 00 00 c7 05 89 86 02 00 01 00 00 00 e8 7c eb 97 ff e9 0f fc ff ff 0f 1f 80 00 00 00 00 45 89 e9 4c 8d 85 67 ff ff ff <48> 8b 4d a8 48 8d 55 88 48 8b b5 38 ff ff ff 48 8b bd 48 ff ff [ 8923.680346] INFO: rcu_sched self-detected stall on CPU { 24} (t=1500045 jiffies g=907112 c=907111 q=1415359) [ 8923.692648] Task dump for CPU 24: [ 8923.697398] OI_scrub R running task 0 5551 2 0x00000088 [ 8923.706319] ffff88083fde6dd0 00000000fb6db880 ffff88082da03db0 ffffffff810c47e8 [ 8923.715643] 0000000000000018 ffffffff81a1e7c0 ffff88082da03dc8 ffffffff810c8079 [ 8923.724960] 0000000000000009 ffff88082da03df8 ffffffff811378e0 ffff88082da101c0 [ 8923.734272] Call Trace: [ 8923.737975] [] sched_show_task+0xa8/0x110 [ 8923.745963] [] dump_cpu_task+0x39/0x70 [ 8923.752940] [] rcu_dump_cpu_stacks+0x90/0xd0 [ 8923.760477] [] rcu_check_callbacks+0x442/0x720 [ 8923.768185] [] ? tick_sched_handle.isra.13+0x60/0x60 [ 8923.776457] [] update_process_times+0x47/0x80 [ 8923.784027] [] tick_sched_handle.isra.13+0x25/0x60 [ 8923.792069] [] tick_sched_timer+0x41/0x70 [ 8923.799212] [] __hrtimer_run_queues+0xd2/0x260 [ 8923.806816] [] hrtimer_interrupt+0xb0/0x1e0 [ 8923.814118] [] ? call_softirq+0x1c/0x30 [ 8923.821000] [] local_apic_timer_interrupt+0x37/0x60 [ 8923.829040] [] smp_apic_timer_interrupt+0x3f/0x60 [ 8923.836871] [] apic_timer_interrupt+0x6d/0x80 [ 8923.844289] [] ? osd_inode_iteration+0x490/0xcc0 [osd_ldiskfs] [ 8923.854091] [] ? osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [ 8923.863197] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 8923.872109] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 8923.880732] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 8923.889144] [] ? wake_up_state+0x20/0x20 [ 8923.896102] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 8923.905182] [] kthread+0xcf/0xe0 [ 8923.911349] [] ? kthread_create_on_node+0x140/0x140 [ 8923.919370] [] ret_from_fork+0x58/0x90 [ 8923.926118] [] ? kthread_create_on_node+0x140/0x140 [ 8949.154233] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [ 8949.163628] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 8949.293143] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 8949.306317] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 8949.322378] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 8949.331682] RIP: 0010:[] [] osd_scrub_exec+0xe9/0x5b0 [osd_ldiskfs] [ 8949.343237] RSP: 0018:ffff880703603c78 EFLAGS: 00000282 [ 8949.350095] RAX: 0000000000280001 RBX: ffff880703603c37 RCX: 0000000000000000 [ 8949.359001] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 8949.367898] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [ 8949.376794] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [ 8949.385691] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [ 8949.394583] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 8949.404550] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8949.411894] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 8949.420801] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8949.429708] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 8949.438606] Stack: [ 8949.441774] ffff880703603d08 0000000000000001 ffff880703603df0 000000009f8f2001 [ 8949.451011] ffff880703603d57 000000009f8f2001 0000000000000004 0000000000000001 [ 8949.460258] 0000000000000004 ffff880703603d98 ffff880703603d78 0000000000000000 [ 8949.469507] Call Trace: [ 8949.473189] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [ 8949.482324] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 8949.491454] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 8949.500296] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 8949.508951] [] ? wake_up_state+0x20/0x20 [ 8949.516137] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 8949.525455] [] kthread+0xcf/0xe0 [ 8949.531851] [] ? kthread_create_on_node+0x140/0x140 [ 8949.540091] [] ret_from_fork+0x58/0x90 [ 8949.547072] [] ? kthread_create_on_node+0x140/0x140 [ 8949.555317] Code: 88 11 00 00 01 4c 89 ef e8 f5 9f e9 df 44 8b 4d 88 48 85 db 74 23 f6 83 18 0a 00 00 08 74 1a 4d 85 e4 74 15 41 8b 87 6c 14 00 00 <41> 39 84 24 0c 0a 00 00 0f 82 d9 00 00 00 31 c0 41 f6 87 74 14 [ 8977.155202] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [ 8977.164799] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 8977.296298] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 8977.309581] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 8977.325830] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 8977.335211] RIP: 0010:[] [] osd_inode_iteration+0x489/0xcc0 [osd_ldiskfs] [ 8977.347427] RSP: 0018:ffff880703603d18 EFLAGS: 00000293 [ 8977.354366] RAX: 0000000000000004 RBX: 000000009f8f2001 RCX: 0000000000000000 [ 8977.363341] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 8977.372312] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [ 8977.381263] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [ 8977.390207] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [ 8977.399153] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 8977.409170] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8977.416573] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 8977.425536] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 8977.434487] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 8977.443424] Stack: [ 8977.446627] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [ 8977.455903] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [ 8977.465192] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [ 8977.474469] Call Trace: [ 8977.478181] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 8977.487345] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 8977.496223] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 8977.504891] [] ? wake_up_state+0x20/0x20 [ 8977.512098] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 8977.521442] [] kthread+0xcf/0xe0 [ 8977.527874] [] ? kthread_create_on_node+0x140/0x140 [ 8977.536151] [] ret_from_fork+0x58/0x90 [ 8977.543165] [] ? kthread_create_on_node+0x140/0x140 [ 8977.551438] Code: 00 e8 7c eb 97 ff e9 0f fc ff ff 0f 1f 80 00 00 00 00 45 89 e9 4c 8d 85 67 ff ff ff 48 8b 4d a8 48 8d 55 88 48 8b b5 38 ff ff ff <48> 8b bd 48 ff ff ff 48 8b 85 28 ff ff ff ff d0 85 c0 41 89 c5 [ 9005.156184] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [ 9005.165817] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 9005.297689] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 9005.310999] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 9005.327290] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 9005.336695] RIP: 0010:[] [] osd_inode_iteration+0x282/0xcc0 [osd_ldiskfs] [ 9005.348944] RSP: 0018:ffff880703603d18 EFLAGS: 00000287 [ 9005.355915] RAX: ffff88042bd78800 RBX: 000000009f8f2001 RCX: ffff88042bd78800 [ 9005.364912] RDX: 00000000000072cf RSI: ffff880419dc3a00 RDI: ffff88042bd79800 [ 9005.373892] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [ 9005.382860] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [ 9005.391815] R13: ffff880703603df0 R14: 000000009f8f2001 R15: ffff880703603d08 [ 9005.400761] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 9005.410799] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9005.418211] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 9005.427183] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 9005.436150] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 9005.445103] Stack: [ 9005.448322] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [ 9005.457616] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [ 9005.466906] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [ 9005.476214] Call Trace: [ 9005.479933] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 9005.489096] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 9005.497975] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 9005.506655] [] ? wake_up_state+0x20/0x20 [ 9005.513876] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 9005.523233] [] kthread+0xcf/0xe0 [ 9005.529676] [] ? kthread_create_on_node+0x140/0x140 [ 9005.537957] [] ret_from_fork+0x58/0x90 [ 9005.544974] [] ? kthread_create_on_node+0x140/0x140 [ 9005.553252] Code: 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 e8 77 5b f4 ff 42 8d 14 38 48 8b 45 88 48 8b 80 a8 03 00 00 48 3b 50 20 0f 83 57 01 00 00 <44> 0f b6 85 67 ff ff ff 48 8d 4d a8 48 8d 55 88 48 8b b5 38 ff [ 9033.157155] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [ 9033.166791] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 9033.298619] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 9033.311935] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 9033.328240] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 9033.337655] RIP: 0010:[] [] osd_inode_iteration+0x2ac/0xcc0 [osd_ldiskfs] [ 9033.349907] RSP: 0018:ffff880703603d18 EFLAGS: 00000202 [ 9033.356876] RAX: 0000000000000004 RBX: ffff880703603d98 RCX: ffff880703603d98 [ 9033.365873] RDX: ffff880703603d78 RSI: 0000000000000000 RDI: ffff8803162f6000 [ 9033.374863] RBP: ffff880703603df0 R08: 0000000000000001 R09: 0000000000000004 [ 9033.383834] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000001 [ 9033.392792] R13: 0000000000000004 R14: 000000009f8f2001 R15: 0000000000000018 [ 9033.401742] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 9033.411766] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9033.419169] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 9033.428130] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 9033.437085] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 9033.446026] Stack: [ 9033.449230] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [ 9033.458515] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [ 9033.467792] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [ 9033.477095] Call Trace: [ 9033.480809] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 9033.489966] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 9033.498821] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 9033.507486] [] ? wake_up_state+0x20/0x20 [ 9033.514694] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 9033.524036] [] kthread+0xcf/0xe0 [ 9033.530458] [] ? kthread_create_on_node+0x140/0x140 [ 9033.538727] [] ret_from_fork+0x58/0x90 [ 9033.545732] [] ? kthread_create_on_node+0x140/0x140 [ 9033.553995] Code: 00 44 0f b6 85 67 ff ff ff 48 8d 4d a8 48 8d 55 88 48 8b b5 38 ff ff ff 48 8b bd 48 ff ff ff 48 8b 85 30 ff ff ff ff d0 83 f8 03 <41> 89 c5 0f 84 4b 02 00 00 0f 8e 15 01 00 00 83 f8 05 66 90 0f [ 9061.158128] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [ 9061.167747] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 9061.299416] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 9061.312725] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 9061.329001] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 9061.338398] RIP: 0010:[] [] osd_scrub_exec+0x95/0x5b0 [osd_ldiskfs] [ 9061.350044] RSP: 0018:ffff880703603d08 EFLAGS: 00000292 [ 9061.356996] RAX: 0000000000000000 RBX: 000000009f8f2001 RCX: 0000000000000000 [ 9061.365981] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 9061.374952] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [ 9061.383907] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [ 9061.392848] R13: 0000000000000018 R14: ffff880703603c78 R15: 0000000000000286 [ 9061.401783] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 9061.411793] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9061.419184] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 9061.428134] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 9061.437085] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 9061.446026] Stack: [ 9061.449231] ffff880703603df0 ffffffffa121d1e9 ffffffffa121b990 ffffffffa1217a20 [ 9061.458512] ffff8800b2a36000 00000000810d354f ffff8803162f6000 ffff8800b2a37468 [ 9061.467778] 0000000020000000 010000000000000c 0000000000000000 0000000000000000 [ 9061.477063] Call Trace: [ 9061.480781] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [ 9061.489942] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 9061.499100] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 9061.507967] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 9061.516635] [] ? wake_up_state+0x20/0x20 [ 9061.523841] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 9061.533184] [] kthread+0xcf/0xe0 [ 9061.539612] [] ? kthread_create_on_node+0x140/0x140 [ 9061.547888] [] ret_from_fork+0x58/0x90 [ 9061.554897] [] ? kthread_create_on_node+0x140/0x140 [ 9061.563166] Code: 01 00 00 41 80 a7 74 14 00 00 fe 48 8b 4d d0 65 48 33 0c 25 28 00 00 00 0f 85 b1 03 00 00 48 83 c4 68 5b 41 5c 41 5d 41 5e 41 5f <5d> c3 66 0f 1f 84 00 00 00 00 00 4c 8d ae f0 10 00 00 44 89 4d [ 9089.159104] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [ 9089.168717] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 9089.300264] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 9089.313535] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 9089.329767] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 9089.339145] RIP: 0010:[] [] osd_scrub_exec+0xf7/0x5b0 [osd_ldiskfs] [ 9089.350765] RSP: 0018:ffff880703603c78 EFLAGS: 00000246 [ 9089.357695] RAX: 0000000000280001 RBX: ffff880703603c37 RCX: 0000000000000000 [ 9089.366664] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 9089.375626] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [ 9089.384568] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [ 9089.393509] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [ 9089.402449] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 9089.412467] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9089.419865] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 9089.428825] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 9089.437779] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 9089.446719] Stack: [ 9089.449925] ffff880703603d08 ffff880703603d57 ffffea00027e3c80 000000009f8f2001 [ 9089.459209] ffff880703603d57 ffff880703603d57 0000000000000000 ffff88042bd78800 [ 9089.468501] 00000000000072ce ffff880419dc3a00 ffff88042bd79800 ffffffffffffff10 [ 9089.477793] Call Trace: [ 9089.481506] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [ 9089.490668] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 9089.499829] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 9089.508684] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 9089.517343] [] ? wake_up_state+0x20/0x20 [ 9089.524544] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 9089.533889] [] kthread+0xcf/0xe0 [ 9089.540314] [] ? kthread_create_on_node+0x140/0x140 [ 9089.548585] [] ret_from_fork+0x58/0x90 [ 9089.555589] [] ? kthread_create_on_node+0x140/0x140 [ 9089.563852] Code: 8b 4d 88 48 85 db 74 23 f6 83 18 0a 00 00 08 74 1a 4d 85 e4 74 15 41 8b 87 6c 14 00 00 41 39 84 24 0c 0a 00 00 0f 82 d9 00 00 00 <31> c0 41 f6 87 74 14 00 00 04 0f 85 6e ff ff ff 41 83 f9 02 0f [ 9103.689617] INFO: rcu_sched self-detected stall on CPU { 24} (t=1680048 jiffies g=907112 c=907111 q=1422050) [ 9103.701962] Task dump for CPU 24: [ 9103.706694] OI_scrub R running task 0 5551 2 0x00000088 [ 9103.715606] ffff88083fde6dd0 00000000fb6db880 ffff88082da03db0 ffffffff810c47e8 [ 9103.724918] 0000000000000018 ffffffff81a1e7c0 ffff88082da03dc8 ffffffff810c8079 [ 9103.734203] 0000000000000009 ffff88082da03df8 ffffffff811378e0 ffff88082da101c0 [ 9103.743478] Call Trace: [ 9103.747162] [] sched_show_task+0xa8/0x110 [ 9103.755119] [] dump_cpu_task+0x39/0x70 [ 9103.762075] [] rcu_dump_cpu_stacks+0x90/0xd0 [ 9103.769590] [] rcu_check_callbacks+0x442/0x720 [ 9103.777270] [] ? tick_sched_handle.isra.13+0x60/0x60 [ 9103.785512] [] update_process_times+0x47/0x80 [ 9103.793058] [] tick_sched_handle.isra.13+0x25/0x60 [ 9103.801067] [] tick_sched_timer+0x41/0x70 [ 9103.808178] [] __hrtimer_run_queues+0xd2/0x260 [ 9103.815762] [] hrtimer_interrupt+0xb0/0x1e0 [ 9103.823028] [] ? call_softirq+0x1c/0x30 [ 9103.829885] [] local_apic_timer_interrupt+0x37/0x60 [ 9103.837904] [] smp_apic_timer_interrupt+0x3f/0x60 [ 9103.845718] [] apic_timer_interrupt+0x6d/0x80 [ 9103.853144] [] ? osd_inode_iteration+0x282/0xcc0 [osd_ldiskfs] [ 9103.862950] [] ? osd_inode_iteration+0x269/0xcc0 [osd_ldiskfs] [ 9103.872047] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 9103.880953] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 9103.889559] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 9103.897960] [] ? wake_up_state+0x20/0x20 [ 9103.904904] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 9103.913982] [] kthread+0xcf/0xe0 [ 9103.920155] [] ? kthread_create_on_node+0x140/0x140 [ 9103.928174] [] ret_from_fork+0x58/0x90 [ 9103.934933] [] ? kthread_create_on_node+0x140/0x140 [ 9129.160503] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [ 9129.169899] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 9129.299494] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 9129.312676] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 9129.328757] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 9129.338059] RIP: 0010:[] [] osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 9129.350006] RSP: 0018:ffff880703603d10 EFLAGS: 00000293 [ 9129.356868] RAX: ffffffffa121b990 RBX: 0000000000000000 RCX: 0000000000000000 [ 9129.365772] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 9129.374670] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [ 9129.383576] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffff880703603d57 [ 9129.392481] R13: 0000000000000004 R14: 000000009f8f2001 R15: ffffea00027e3c80 [ 9129.401375] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 9129.411331] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9129.418672] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 9129.427582] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 9129.436483] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 9129.445377] Stack: [ 9129.448541] ffffffffa121d1e9 ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 [ 9129.457783] 00000000810d354f ffff8803162f6000 ffff8800b2a37468 0000000020000000 [ 9129.467031] 010000000000000c 0000000000000000 0000000000000000 ffff8800b2a36000 [ 9129.476287] Call Trace: [ 9129.479978] [] ? osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [ 9129.489300] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 9129.498436] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 9129.507281] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 9129.515927] [] ? wake_up_state+0x20/0x20 [ 9129.523115] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 9129.532435] [] kthread+0xcf/0xe0 [ 9129.538834] [] ? kthread_create_on_node+0x140/0x140 [ 9129.547078] [] ret_from_fork+0x58/0x90 [ 9129.554067] [] ? kthread_create_on_node+0x140/0x140 [ 9129.562309] Code: b5 07 00 00 c7 05 85 a5 02 00 00 00 02 00 89 54 24 08 48 8b 40 40 ba 07 00 00 00 48 89 04 24 31 c0 e8 a5 03 98 ff e9 c7 fd ff ff <66> 66 66 66 90 55 48 89 e5 41 57 49 89 f7 41 56 41 55 41 54 45 [ 9157.161480] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [ 9157.171073] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 9157.302756] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 9157.316045] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 9157.332314] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 9157.341700] RIP: 0010:[] [] osd_scrub_exec+0xf7/0x5b0 [osd_ldiskfs] [ 9157.353344] RSP: 0018:ffff880703603c78 EFLAGS: 00000246 [ 9157.360286] RAX: 0000000000280001 RBX: ffff880703603c37 RCX: 0000000000000000 [ 9157.369265] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 9157.378230] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [ 9157.387190] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [ 9157.396140] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [ 9157.405090] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 9157.415120] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9157.422527] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 9157.431497] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 9157.440456] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 9157.449405] Stack: [ 9157.452620] ffffea00027e3c80 000000009f8f2001 ffff880703603df0 ffff880703603d57 [ 9157.461909] ffff880703603d57 000000009f8f2001 0000000000000004 ffff880703603d57 [ 9157.471190] ffffffffa121b990 0000000000000000 ffff880703603d78 ffff8800b2a36000 [ 9157.480472] Call Trace: [ 9157.484196] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 9157.493353] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [ 9157.502510] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 9157.511657] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 9157.520507] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 9157.529164] [] ? wake_up_state+0x20/0x20 [ 9157.536360] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 9157.545694] [] kthread+0xcf/0xe0 [ 9157.552118] [] ? kthread_create_on_node+0x140/0x140 [ 9157.560389] [] ret_from_fork+0x58/0x90 [ 9157.567387] [] ? kthread_create_on_node+0x140/0x140 [ 9157.575639] Code: 8b 4d 88 48 85 db 74 23 f6 83 18 0a 00 00 08 74 1a 4d 85 e4 74 15 41 8b 87 6c 14 00 00 41 39 84 24 0c 0a 00 00 0f 82 d9 00 00 00 <31> c0 41 f6 87 74 14 00 00 04 0f 85 6e ff ff ff 41 83 f9 02 0f [ 9185.162448] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [ 9185.172054] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 9185.303362] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 9185.316621] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 9185.332812] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 9185.342169] RIP: 0010:[] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [ 9185.354358] RSP: 0018:ffff880703603d18 EFLAGS: 00000292 [ 9185.361282] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000 [ 9185.370250] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 9185.379198] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [ 9185.388138] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffff880703603d57 [ 9185.397080] R13: 0000000000000004 R14: 000000009f8f2001 R15: 0000000000000018 [ 9185.406023] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 9185.416036] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9185.423428] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 9185.432373] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 9185.441311] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 9185.450236] Stack: [ 9185.453434] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [ 9185.462722] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [ 9185.471984] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [ 9185.481261] Call Trace: [ 9185.484968] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 9185.494126] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 9185.502983] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 9185.511635] [] ? wake_up_state+0x20/0x20 [ 9185.518839] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 9185.528182] [] kthread+0xcf/0xe0 [ 9185.534618] [] ? kthread_create_on_node+0x140/0x140 [ 9185.542892] [] ret_from_fork+0x58/0x90 [ 9185.549903] [] ? kthread_create_on_node+0x140/0x140 [ 9185.558180] Code: 00 00 45 89 e9 4c 8d 85 67 ff ff ff 48 8b 4d a8 48 8d 55 88 48 8b b5 38 ff ff ff 48 8b bd 48 ff ff ff 48 8b 85 28 ff ff ff ff d0 <85> c0 41 89 c5 75 1c 8b 85 58 ff ff ff 41 3b 04 24 72 10 48 8b [ 9213.163423] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [ 9213.173046] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 9213.304823] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 9213.318127] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 9213.334409] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 9213.343816] RIP: 0010:[] [] osd_inode_iteration+0x264/0xcc0 [osd_ldiskfs] [ 9213.356057] RSP: 0018:ffff880703603d18 EFLAGS: 00000286 [ 9213.363016] RAX: ffff8800b2a37468 RBX: 000000009f8f2001 RCX: 0000000000000000 [ 9213.372003] RDX: ffff880703603d78 RSI: ffff880419dc3a00 RDI: ffff88042bd79800 [ 9213.380981] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [ 9213.389943] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [ 9213.398894] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [ 9213.407842] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 9213.417863] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9213.425266] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 9213.434228] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 9213.443188] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 9213.452136] Stack: [ 9213.455346] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [ 9213.464628] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [ 9213.473908] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [ 9213.483205] Call Trace: [ 9213.486925] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 9213.496089] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 9213.504959] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 9213.513632] [] ? wake_up_state+0x20/0x20 [ 9213.520835] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 9213.530177] [] kthread+0xcf/0xe0 [ 9213.536606] [] ? kthread_create_on_node+0x140/0x140 [ 9213.544879] [] ret_from_fork+0x58/0x90 [ 9213.551896] [] ? kthread_create_on_node+0x140/0x140 [ 9213.560174] Code: 75 98 48 8b 7d 88 e8 dc 2d f5 ff 48 85 c0 48 89 45 90 0f 84 b3 08 00 00 44 8b 7d a0 48 8b 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 77 5b f4 ff 42 8d 14 38 48 8b 45 88 48 8b 80 a8 03 00 00 48 [ 9241.164399] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [ 9241.174039] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 9241.305878] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 9241.319183] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 9241.335470] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 9241.344882] RIP: 0010:[] [] osd_scrub_exec+0x1c/0x5b0 [osd_ldiskfs] [ 9241.356549] RSP: 0018:ffff880703603c78 EFLAGS: 00000296 [ 9241.363517] RAX: ffffffffa121b990 RBX: ffff880703603c37 RCX: 0000000000000000 [ 9241.372502] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 9241.381478] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [ 9241.390444] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [ 9241.399393] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [ 9241.408338] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 9241.418362] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9241.425772] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 9241.434736] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 9241.443696] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 9241.452646] Stack: [ 9241.455855] ffffea00027e3c80 000000009f8f2001 ffff880703603df0 0000000000000001 [ 9241.465134] ffff880703603d57 000000009f8f2001 0000000000000004 ffff880703603d57 [ 9241.474404] ffff8800b2a37468 0000000000000000 ffff880703603d78 ffff8800b2a36000 [ 9241.483700] Call Trace: [ 9241.487407] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [ 9241.496569] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 9241.505726] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 9241.514590] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 9241.523258] [] ? wake_up_state+0x20/0x20 [ 9241.530459] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 9241.539788] [] kthread+0xcf/0xe0 [ 9241.546217] [] ? kthread_create_on_node+0x140/0x140 [ 9241.554488] [] ret_from_fork+0x58/0x90 [ 9241.561505] [] ? kthread_create_on_node+0x140/0x140 [ 9241.569766] Code: 89 04 24 31 c0 e8 a5 03 98 ff e9 c7 fd ff ff 66 66 66 66 90 55 48 89 e5 41 57 49 89 f7 41 56 41 55 41 54 45 31 e4 53 48 83 ec 68 <48> 8b 9e 08 10 00 00 4c 89 45 90 65 48 8b 04 25 28 00 00 00 48 [ 9269.165380] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [ 9269.175004] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 9269.306578] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 9269.319860] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 9269.336107] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 9269.345492] RIP: 0010:[] [] osd_scrub_exec+0xf7/0x5b0 [osd_ldiskfs] [ 9269.357121] RSP: 0018:ffff880703603c78 EFLAGS: 00000246 [ 9269.364056] RAX: 0000000000280001 RBX: ffff880703603c37 RCX: 0000000000000000 [ 9269.373029] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 9269.381991] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [ 9269.390940] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [ 9269.399886] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [ 9269.408829] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 9269.418855] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9269.426258] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 9269.435212] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 9269.444164] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 9269.453102] Stack: [ 9269.456298] 0000000000000004 0000000000000001 ffff880703603df0 ffff880703603d98 [ 9269.465577] ffff880703603d57 000000009f8f2001 0000000000000004 ffff880703603d57 [ 9269.474862] 0000000020000000 0000000000000000 ffff880703603d78 ffff8800b2a36000 [ 9269.484141] Call Trace: [ 9269.487854] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [ 9269.497015] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 9269.506173] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 9269.515041] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 9269.523702] [] ? wake_up_state+0x20/0x20 [ 9269.530909] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 9269.540257] [] kthread+0xcf/0xe0 [ 9269.546683] [] ? kthread_create_on_node+0x140/0x140 [ 9269.554956] [] ret_from_fork+0x58/0x90 [ 9269.561971] [] ? kthread_create_on_node+0x140/0x140 [ 9269.570244] Code: 8b 4d 88 48 85 db 74 23 f6 83 18 0a 00 00 08 74 1a 4d 85 e4 74 15 41 8b 87 6c 14 00 00 41 39 84 24 0c 0a 00 00 0f 82 d9 00 00 00 <31> c0 41 f6 87 74 14 00 00 04 0f 85 6e ff ff ff 41 83 f9 02 0f [ 9283.698887] INFO: rcu_sched self-detected stall on CPU { 24} (t=1860051 jiffies g=907112 c=907111 q=1432823) [ 9283.711229] Task dump for CPU 24: [ 9283.715957] OI_scrub R running task 0 5551 2 0x00000088 [ 9283.724867] ffff88083fde6dd0 00000000fb6db880 ffff88082da03db0 ffffffff810c47e8 [ 9283.734190] 0000000000000018 ffffffff81a1e7c0 ffff88082da03dc8 ffffffff810c8079 [ 9283.743475] 0000000000000009 ffff88082da03df8 ffffffff811378e0 ffff88082da101c0 [ 9283.752736] Call Trace: [ 9283.756414] [] sched_show_task+0xa8/0x110 [ 9283.764381] [] dump_cpu_task+0x39/0x70 [ 9283.771332] [] rcu_dump_cpu_stacks+0x90/0xd0 [ 9283.778840] [] rcu_check_callbacks+0x442/0x720 [ 9283.786527] [] ? tick_sched_handle.isra.13+0x60/0x60 [ 9283.794772] [] update_process_times+0x47/0x80 [ 9283.802316] [] tick_sched_handle.isra.13+0x25/0x60 [ 9283.810325] [] tick_sched_timer+0x41/0x70 [ 9283.817438] [] __hrtimer_run_queues+0xd2/0x260 [ 9283.825023] [] hrtimer_interrupt+0xb0/0x1e0 [ 9283.832291] [] ? call_softirq+0x1c/0x30 [ 9283.839162] [] local_apic_timer_interrupt+0x37/0x60 [ 9283.847189] [] smp_apic_timer_interrupt+0x3f/0x60 [ 9283.855005] [] apic_timer_interrupt+0x6d/0x80 [ 9283.862441] [] ? osd_scrub_next+0x161/0x3c0 [osd_ldiskfs] [ 9283.871756] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [ 9283.880655] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 9283.889550] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 9283.898161] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 9283.906559] [] ? wake_up_state+0x20/0x20 [ 9283.913507] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 9283.922585] [] kthread+0xcf/0xe0 [ 9283.928750] [] ? kthread_create_on_node+0x140/0x140 [ 9283.936761] [] ret_from_fork+0x58/0x90 [ 9283.943514] [] ? kthread_create_on_node+0x140/0x140 [ 9309.166768] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [ 9309.176157] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 9309.305604] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 9309.318778] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 9309.334840] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 9309.344137] RIP: 0010:[] [] osd_inode_iteration+0x4a6/0xcc0 [osd_ldiskfs] [ 9309.356277] RSP: 0018:ffff880703603d18 EFLAGS: 00000246 [ 9309.363136] RAX: 0000000020000000 RBX: 000000009f8f2001 RCX: 0000000000000000 [ 9309.372040] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 9309.380937] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [ 9309.389833] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [ 9309.398728] R13: ffff880703603d08 R14: 000000009f8f2001 R15: ffff880703603d08 [ 9309.407623] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 9309.417598] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9309.424941] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 9309.433839] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 9309.442745] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 9309.451645] Stack: [ 9309.454813] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [ 9309.464061] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [ 9309.473317] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [ 9309.482573] Call Trace: [ 9309.486271] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 9309.495404] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 9309.504247] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 9309.512891] [] ? wake_up_state+0x20/0x20 [ 9309.520082] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 9309.529408] [] kthread+0xcf/0xe0 [ 9309.535808] [] ? kthread_create_on_node+0x140/0x140 [ 9309.544048] [] ret_from_fork+0x58/0x90 [ 9309.551034] [] ? kthread_create_on_node+0x140/0x140 [ 9309.559286] Code: 8b 4d a8 48 8d 55 88 48 8b b5 38 ff ff ff 48 8b bd 48 ff ff ff 48 8b 85 28 ff ff ff ff d0 85 c0 41 89 c5 75 1c 8b 85 58 ff ff ff <41> 3b 04 24 72 10 48 8b 85 50 ff ff ff 44 3b 30 0f 87 95 fd ff [ 9337.167749] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [ 9337.177347] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 9337.309158] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 9337.322466] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 9337.338766] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 9337.348171] RIP: 0010:[] [] ldiskfs_used_dirs_count+0x30/0x30 [ldiskfs] [ 9337.360244] RSP: 0018:ffff880703603d10 EFLAGS: 00000286 [ 9337.367215] RAX: ffff8800b2a37468 RBX: 0000000000000000 RCX: 0000000000000000 [ 9337.376209] RDX: ffff880703603d78 RSI: ffff880419dc3a00 RDI: ffff88042bd79800 [ 9337.385190] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [ 9337.394168] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffff880703603d57 [ 9337.403122] R13: 0000000000000004 R14: 000000009f8f2001 R15: ffffea00027e3c80 [ 9337.412070] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 9337.422093] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9337.429496] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 9337.438459] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 9337.447423] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 9337.456364] Stack: [ 9337.459570] ffffffffa121cfb9 ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 [ 9337.468851] 00000000810d354f ffff8803162f6000 ffff8800b2a37468 0000000020000000 [ 9337.478136] 010000000000000c 0000000000000000 0000000000000000 ffff8800b2a36000 [ 9337.487416] Call Trace: [ 9337.491143] [] ? osd_inode_iteration+0x269/0xcc0 [osd_ldiskfs] [ 9337.500500] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 9337.509664] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 9337.518536] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 9337.527200] [] ? wake_up_state+0x20/0x20 [ 9337.534398] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 9337.543743] [] kthread+0xcf/0xe0 [ 9337.550179] [] ? kthread_create_on_node+0x140/0x140 [ 9337.558454] [] ret_from_fork+0x58/0x90 [ 9337.565469] [] ? kthread_create_on_node+0x140/0x140 [ 9337.573742] Code: 55 48 8b 8f a8 03 00 00 31 c0 0f b7 56 10 48 89 e5 48 83 39 3f 76 07 0f b7 46 30 c1 e0 10 09 d0 5d c3 66 0f 1f 84 00 00 00 00 00 <66> 66 66 66 90 55 48 8b 8f a8 03 00 00 31 c0 0f b7 56 1c 48 89 [ 9365.168719] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [ 9365.178354] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 9365.310090] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 9365.323382] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 9365.339638] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 9365.349026] RIP: 0010:[] [] osd_inode_iteration+0x282/0xcc0 [osd_ldiskfs] [ 9365.361258] RSP: 0018:ffff880703603d18 EFLAGS: 00000287 [ 9365.368212] RAX: ffff88042bd78800 RBX: ffff880703603d98 RCX: ffff88042bd78800 [ 9365.377196] RDX: 00000000000072cf RSI: ffff880419dc3a00 RDI: ffff88042bd79800 [ 9365.386165] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [ 9365.395131] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000001 [ 9365.404091] R13: 0000000000000004 R14: 000000009f8f2001 R15: 0000000000000018 [ 9365.413040] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 9365.423064] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9365.430473] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 9365.439434] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 9365.448389] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 9365.457336] Stack: [ 9365.460543] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [ 9365.469835] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [ 9365.479124] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [ 9365.488406] Call Trace: [ 9365.492122] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 9365.501284] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 9365.510153] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 9365.518822] [] ? wake_up_state+0x20/0x20 [ 9365.526025] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 9365.535364] [] kthread+0xcf/0xe0 [ 9365.541795] [] ? kthread_create_on_node+0x140/0x140 [ 9365.550074] [] ret_from_fork+0x58/0x90 [ 9365.557087] [] ? kthread_create_on_node+0x140/0x140 [ 9365.565361] Code: 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 e8 77 5b f4 ff 42 8d 14 38 48 8b 45 88 48 8b 80 a8 03 00 00 48 3b 50 20 0f 83 57 01 00 00 <44> 0f b6 85 67 ff ff ff 48 8d 4d a8 48 8d 55 88 48 8b b5 38 ff [ 9393.169701] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [ 9393.179310] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 9393.311038] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 9393.324340] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 9393.340623] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 9393.350019] RIP: 0010:[] [] osd_inode_iteration+0x251/0xcc0 [osd_ldiskfs] [ 9393.362258] RSP: 0018:ffff880703603d18 EFLAGS: 00000286 [ 9393.369215] RAX: ffff8800b2a37468 RBX: 000000009f8f2001 RCX: 0000000000000000 [ 9393.378200] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 9393.387173] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [ 9393.396133] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [ 9393.405086] R13: ffff880703603df0 R14: 000000009f8f2001 R15: ffff880703603d08 [ 9393.414031] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 9393.424056] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9393.431459] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 9393.440422] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 9393.449377] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 9393.458329] Stack: [ 9393.461544] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [ 9393.470839] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [ 9393.480126] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [ 9393.489420] Call Trace: [ 9393.493146] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 9393.502317] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 9393.511195] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 9393.519874] [] ? wake_up_state+0x20/0x20 [ 9393.527093] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 9393.536441] [] kthread+0xcf/0xe0 [ 9393.542875] [] ? kthread_create_on_node+0x140/0x140 [ 9393.551145] [] ret_from_fork+0x58/0x90 [ 9393.558168] [] ? kthread_create_on_node+0x140/0x140 [ 9393.566448] Code: b0 00 00 00 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 00 8b 75 98 48 8b 7d 88 e8 dc 2d f5 ff 48 85 c0 48 89 45 90 0f 84 b3 08 00 00 <44> 8b 7d a0 48 8b 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 e8 77 [ 9421.170669] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [ 9421.180298] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 9421.312162] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 9421.325472] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 9421.341791] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 9421.351197] RIP: 0010:[] [] osd_scrub_exec+0x12/0x5b0 [osd_ldiskfs] [ 9421.362860] RSP: 0018:ffff880703603cf0 EFLAGS: 00000293 [ 9421.369822] RAX: ffffffffa121b990 RBX: 000000009f8f2001 RCX: 0000000000000000 [ 9421.378802] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 9421.387788] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [ 9421.396756] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffff880703603c70 [ 9421.405712] R13: 0000000000000202 R14: 0000000000000010 R15: ffffffffa1217b69 [ 9421.414652] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 9421.424673] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9421.432079] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 9421.441043] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 9421.450000] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 9421.458942] Stack: [ 9421.462149] 0000000000000004 00000000ffffffff 0000000000000001 ffff880703603df0 [ 9421.471430] ffffffffa121d1e9 ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 [ 9421.480730] 00000000810d354f ffff8803162f6000 ffff8800b2a37468 0000000020000000 [ 9421.490031] Call Trace: [ 9421.493759] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [ 9421.502940] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 9421.512100] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 9421.520966] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 9421.529627] [] ? wake_up_state+0x20/0x20 [ 9421.536838] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 9421.546186] [] kthread+0xcf/0xe0 [ 9421.552618] [] ? kthread_create_on_node+0x140/0x140 [ 9421.560880] [] ret_from_fork+0x58/0x90 [ 9421.567887] [] ? kthread_create_on_node+0x140/0x140 [ 9421.576156] Code: 48 8b 40 40 ba 07 00 00 00 48 89 04 24 31 c0 e8 a5 03 98 ff e9 c7 fd ff ff 66 66 66 66 90 55 48 89 e5 41 57 49 89 f7 41 56 41 55 <41> 54 45 31 e4 53 48 83 ec 68 48 8b 9e 08 10 00 00 4c 89 45 90 [ 9449.171645] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [ 9449.181279] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 9449.312901] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 9449.326186] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 9449.342430] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 9449.351819] RIP: 0010:[] [] osd_inode_iteration+0x489/0xcc0 [osd_ldiskfs] [ 9449.364051] RSP: 0018:ffff880703603d18 EFLAGS: 00000293 [ 9449.370990] RAX: 0000000000000004 RBX: ffff880703603d98 RCX: 0000000000000000 [ 9449.379966] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 9449.388933] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [ 9449.397884] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000001 [ 9449.406832] R13: 0000000000000004 R14: 000000009f8f2001 R15: ffffea00027e3c80 [ 9449.415780] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 9449.425810] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9449.433209] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 9449.442171] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 9449.451123] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 9449.460064] Stack: [ 9449.463270] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [ 9449.472566] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [ 9449.481857] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [ 9449.491143] Call Trace: [ 9449.494856] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 9449.504016] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 9449.512882] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 9449.521548] [] ? wake_up_state+0x20/0x20 [ 9449.528758] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 9449.538098] [] kthread+0xcf/0xe0 [ 9449.544532] [] ? kthread_create_on_node+0x140/0x140 [ 9449.552811] [] ret_from_fork+0x58/0x90 [ 9449.559820] [] ? kthread_create_on_node+0x140/0x140 [ 9449.568088] Code: 00 e8 7c eb 97 ff e9 0f fc ff ff 0f 1f 80 00 00 00 00 45 89 e9 4c 8d 85 67 ff ff ff 48 8b 4d a8 48 8d 55 88 48 8b b5 38 ff ff ff <48> 8b bd 48 ff ff ff 48 8b 85 28 ff ff ff ff d0 85 c0 41 89 c5 [ 9463.708158] INFO: rcu_sched self-detected stall on CPU { 24} (t=2040054 jiffies g=907112 c=907111 q=1439501) [ 9463.720487] Task dump for CPU 24: [ 9463.725229] OI_scrub R running task 0 5551 2 0x00000088 [ 9463.734143] ffff88083fde6dd0 00000000fb6db880 ffff88082da03db0 ffffffff810c47e8 [ 9463.743467] 0000000000000018 ffffffff81a1e7c0 ffff88082da03dc8 ffffffff810c8079 [ 9463.752764] 0000000000000009 ffff88082da03df8 ffffffff811378e0 ffff88082da101c0 [ 9463.762057] Call Trace: [ 9463.765753] [] sched_show_task+0xa8/0x110 [ 9463.773734] [] dump_cpu_task+0x39/0x70 [ 9463.780701] [] rcu_dump_cpu_stacks+0x90/0xd0 [ 9463.788230] [] rcu_check_callbacks+0x442/0x720 [ 9463.795930] [] ? tick_sched_handle.isra.13+0x60/0x60 [ 9463.804195] [] update_process_times+0x47/0x80 [ 9463.811765] [] tick_sched_handle.isra.13+0x25/0x60 [ 9463.819792] [] tick_sched_timer+0x41/0x70 [ 9463.826928] [] __hrtimer_run_queues+0xd2/0x260 [ 9463.834529] [] hrtimer_interrupt+0xb0/0x1e0 [ 9463.841813] [] ? call_softirq+0x1c/0x30 [ 9463.848687] [] local_apic_timer_interrupt+0x37/0x60 [ 9463.856718] [] smp_apic_timer_interrupt+0x3f/0x60 [ 9463.864541] [] apic_timer_interrupt+0x6d/0x80 [ 9463.871964] [] ? osd_scrub_next+0x167/0x3c0 [osd_ldiskfs] [ 9463.881276] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [ 9463.890171] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 9463.899063] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 9463.907665] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 9463.916073] [] ? wake_up_state+0x20/0x20 [ 9463.923023] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 9463.932105] [] kthread+0xcf/0xe0 [ 9463.938274] [] ? kthread_create_on_node+0x140/0x140 [ 9463.946283] [] ret_from_fork+0x58/0x90 [ 9463.953039] [] ? kthread_create_on_node+0x140/0x140 [ 9489.173038] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [ 9489.182437] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 9489.311963] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 9489.325131] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 9489.341186] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 9489.350478] RIP: 0010:[] [] osd_scrub_exec+0xe9/0x5b0 [osd_ldiskfs] [ 9489.362023] RSP: 0018:ffff880703603c78 EFLAGS: 00000282 [ 9489.368876] RAX: 0000000000280001 RBX: ffff880703603c37 RCX: 0000000000000000 [ 9489.377771] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 9489.386661] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [ 9489.395551] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [ 9489.404441] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [ 9489.413328] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 9489.423288] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9489.430624] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 9489.439523] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 9489.448421] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 9489.457312] Stack: [ 9489.460473] ffff880703603d08 000000009f8f2001 ffff880703603df0 000000009f8f2001 [ 9489.469714] ffff880703603d57 000000009f8f2001 0000000000000004 ffff880703603d57 [ 9489.478947] 0000000000000004 0000000000000000 ffff880703603d78 ffff8800b2a36000 [ 9489.488192] Call Trace: [ 9489.491874] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [ 9489.501001] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 9489.510124] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 9489.518957] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 9489.527598] [] ? wake_up_state+0x20/0x20 [ 9489.534770] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 9489.544089] [] kthread+0xcf/0xe0 [ 9489.550484] [] ? kthread_create_on_node+0x140/0x140 [ 9489.558730] [] ret_from_fork+0x58/0x90 [ 9489.565719] [] ? kthread_create_on_node+0x140/0x140 [ 9489.573952] Code: 88 11 00 00 01 4c 89 ef e8 f5 9f e9 df 44 8b 4d 88 48 85 db 74 23 f6 83 18 0a 00 00 08 74 1a 4d 85 e4 74 15 41 8b 87 6c 14 00 00 <41> 39 84 24 0c 0a 00 00 0f 82 d9 00 00 00 31 c0 41 f6 87 74 14 [ 9517.174013] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [ 9517.183614] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 9517.315222] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 9517.328511] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 9517.344762] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 9517.354144] RIP: 0010:[] [] osd_scrub_exec+0x30/0x5b0 [osd_ldiskfs] [ 9517.365788] RSP: 0018:ffff880703603c78 EFLAGS: 00000296 [ 9517.372722] RAX: 00000000fb6db880 RBX: ffff880703603c37 RCX: 0000000000000000 [ 9517.381694] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 9517.390653] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [ 9517.399602] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [ 9517.408545] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [ 9517.417490] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 9517.427514] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9517.434918] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 9517.443871] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 9517.452821] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 9517.461759] Stack: [ 9517.464953] ffff880703603d08 0000000000000001 ffffea00027e3c80 000000009f8f2001 [ 9517.474241] ffff880703603d57 0000000000000001 ffffffffa1217a20 ffff880703603d98 [ 9517.483533] ffff880703603d78 ffff8800b2a36000 ffff8803162f6000 ffffffffffffff10 [ 9517.492818] Call Trace: [ 9517.496526] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 9517.505392] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [ 9517.514549] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 9517.523687] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 9517.532546] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 9517.541208] [] ? wake_up_state+0x20/0x20 [ 9517.548415] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 9517.557753] [] kthread+0xcf/0xe0 [ 9517.564184] [] ? kthread_create_on_node+0x140/0x140 [ 9517.572459] [] ret_from_fork+0x58/0x90 [ 9517.579466] [] ? kthread_create_on_node+0x140/0x140 [ 9517.587736] Code: 55 48 89 e5 41 57 49 89 f7 41 56 41 55 41 54 45 31 e4 53 48 83 ec 68 48 8b 9e 08 10 00 00 4c 89 45 90 65 48 8b 04 25 28 00 00 00 <48> 89 45 d0 31 c0 48 8d 43 08 48 85 db 4c 0f 45 e0 41 83 f9 04 [ 9545.174996] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [ 9545.184603] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 9545.315987] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 9545.329249] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 9545.345457] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 9545.354820] RIP: 0010:[] [] osd_scrub_exec+0x30/0x5b0 [osd_ldiskfs] [ 9545.366439] RSP: 0018:ffff880703603c78 EFLAGS: 00000296 [ 9545.373365] RAX: 00000000fb6db880 RBX: ffff880703603c37 RCX: 0000000000000000 [ 9545.382323] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 9545.391274] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [ 9545.400224] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [ 9545.409168] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [ 9545.418122] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 9545.428150] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9545.435550] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 9545.444502] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 9545.453448] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 9545.462381] Stack: [ 9545.465585] 0000000000000004 0000000000000001 ffff880703603df0 ffff880703603d98 [ 9545.474866] ffff880703603d57 000000009f8f2001 0000000000000004 ffff880703603d57 [ 9545.484154] ffff8800b2a37468 0000000000000000 ffff880703603d78 ffff880419dc3a00 [ 9545.493428] Call Trace: [ 9545.497131] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [ 9545.506291] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 9545.515440] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 9545.524284] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 9545.532937] [] ? wake_up_state+0x20/0x20 [ 9545.540140] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 9545.549480] [] kthread+0xcf/0xe0 [ 9545.555908] [] ? kthread_create_on_node+0x140/0x140 [ 9545.564183] [] ret_from_fork+0x58/0x90 [ 9545.571187] [] ? kthread_create_on_node+0x140/0x140 [ 9545.579450] Code: 55 48 89 e5 41 57 49 89 f7 41 56 41 55 41 54 45 31 e4 53 48 83 ec 68 48 8b 9e 08 10 00 00 4c 89 45 90 65 48 8b 04 25 28 00 00 00 <48> 89 45 d0 31 c0 48 8d 43 08 48 85 db 4c 0f 45 e0 41 83 f9 04 [ 9573.175964] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [ 9573.185585] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 9573.317166] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 9573.330449] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 9573.346686] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 9573.356067] RIP: 0010:[] [] osd_inode_iteration+0x28a/0xcc0 [osd_ldiskfs] [ 9573.368292] RSP: 0018:ffff880703603d18 EFLAGS: 00000287 [ 9573.375230] RAX: ffff88042bd78800 RBX: ffff880703603d98 RCX: ffff88042bd78800 [ 9573.384206] RDX: 00000000000072cf RSI: ffff880419dc3a00 RDI: ffff88042bd79800 [ 9573.393165] RBP: ffff880703603df0 R08: 0000000000000001 R09: 0000000000000004 [ 9573.402117] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000001 [ 9573.411065] R13: 0000000000000004 R14: 000000009f8f2001 R15: ffffea00027e3c80 [ 9573.420012] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 9573.430025] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9573.437416] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 9573.446374] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 9573.455324] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 9573.464259] Stack: [ 9573.467462] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [ 9573.476746] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [ 9573.486030] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [ 9573.495318] Call Trace: [ 9573.499039] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 9573.508199] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 9573.517067] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 9573.525731] [] ? wake_up_state+0x20/0x20 [ 9573.532935] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 9573.542285] [] kthread+0xcf/0xe0 [ 9573.548706] [] ? kthread_create_on_node+0x140/0x140 [ 9573.556986] [] ret_from_fork+0x58/0x90 [ 9573.564003] [] ? kthread_create_on_node+0x140/0x140 [ 9573.572271] Code: a8 00 00 00 00 e8 77 5b f4 ff 42 8d 14 38 48 8b 45 88 48 8b 80 a8 03 00 00 48 3b 50 20 0f 83 57 01 00 00 44 0f b6 85 67 ff ff ff <48> 8d 4d a8 48 8d 55 88 48 8b b5 38 ff ff ff 48 8b bd 48 ff ff [ 9601.176946] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [ 9601.186575] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 9601.318391] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 9601.331703] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 9601.348002] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 9601.357405] RIP: 0010:[] [] osd_scrub_next+0x12/0x3c0 [osd_ldiskfs] [ 9601.369070] RSP: 0018:ffff880703603cf0 EFLAGS: 00000287 [ 9601.376027] RAX: ffffffffa1217a20 RBX: 000000009f8f2001 RCX: ffff880703603d98 [ 9601.385013] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 9601.393998] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [ 9601.402967] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffff880703603c70 [ 9601.411920] R13: 0000000000000282 R14: 0000000000000010 R15: ffffffffa1217a4b [ 9601.420869] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 9601.430893] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9601.438291] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 9601.447249] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 9601.456206] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 9601.465151] Stack: [ 9601.468361] 0000000000000000 00000000ffffffff 0000000000000001 ffff880703603df0 [ 9601.477648] ffffffffa121cff9 ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 [ 9601.486929] 00000000810d354f ffff8803162f6000 ffff8800b2a37468 0000000020000000 [ 9601.496227] Call Trace: [ 9601.499951] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [ 9601.509126] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 9601.518290] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 9601.527162] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 9601.535828] [] ? wake_up_state+0x20/0x20 [ 9601.543034] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 9601.552377] [] kthread+0xcf/0xe0 [ 9601.558812] [] ? kthread_create_on_node+0x140/0x140 [ 9601.567094] [] ret_from_fork+0x58/0x90 [ 9601.574102] [] ? kthread_create_on_node+0x140/0x140 [ 9601.582377] Code: 25 fc ff ff 48 83 c4 08 5b 41 5c 41 5d 41 5e 5d c3 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 89 e5 41 57 49 89 f7 41 56 41 55 <49> 89 fd 41 54 49 89 d4 53 48 89 cb 48 83 ec 70 65 48 8b 04 25 [ 9629.177917] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [ 9629.187543] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 9629.319182] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 9629.332470] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 9629.348727] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 9629.358119] RIP: 0010:[] [] osd_scrub_exec+0x79/0x5b0 [osd_ldiskfs] [ 9629.369762] RSP: 0018:ffff880703603c78 EFLAGS: 00000202 [ 9629.376701] RAX: 0000000000000000 RBX: ffff880703603c37 RCX: 00000000fb6db880 [ 9629.385677] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 9629.394643] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [ 9629.403594] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [ 9629.412544] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [ 9629.421496] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 9629.431525] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9629.438933] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 9629.447900] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 9629.456858] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 9629.465805] Stack: [ 9629.469017] ffff880703603d08 0000000000000001 ffffea00027e3c80 000000009f8f2001 [ 9629.478305] ffff880703603d57 ffff880703603d57 0000000000000000 ffff88042bd78800 [ 9629.487597] 00000000000072ce ffff880419dc3a00 ffff88042bd79800 ffffffffffffff10 [ 9629.496903] Call Trace: [ 9629.500624] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [ 9629.509792] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 9629.518953] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 9629.527818] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 9629.536485] [] ? wake_up_state+0x20/0x20 [ 9629.543699] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 9629.553059] [] kthread+0xcf/0xe0 [ 9629.559494] [] ? kthread_create_on_node+0x140/0x140 [ 9629.567778] [] ret_from_fork+0x58/0x90 [ 9629.574789] [] ? kthread_create_on_node+0x140/0x140 [ 9629.583065] Code: 07 74 4f 41 83 f9 02 74 78 48 89 ca 44 89 c9 e8 0e c6 ff ff 85 c0 41 89 c1 0f 84 a3 01 00 00 41 80 a7 74 14 00 00 fe 48 8b 4d d0 <65> 48 33 0c 25 28 00 00 00 0f 85 b1 03 00 00 48 83 c4 68 5b 41 [ 9643.717428] INFO: rcu_sched self-detected stall on CPU { 24} (t=2220057 jiffies g=907112 c=907111 q=1450326) [ 9643.729781] Task dump for CPU 24: [ 9643.734514] OI_scrub R running task 0 5551 2 0x00000088 [ 9643.743428] ffff88083fde6dd0 00000000fb6db880 ffff88082da03db0 ffffffff810c47e8 [ 9643.752747] 0000000000000018 ffffffff81a1e7c0 ffff88082da03dc8 ffffffff810c8079 [ 9643.762046] 0000000000000009 ffff88082da03df8 ffffffff811378e0 ffff88082da101c0 [ 9643.771332] Call Trace: [ 9643.775013] [] sched_show_task+0xa8/0x110 [ 9643.782976] [] dump_cpu_task+0x39/0x70 [ 9643.789931] [] rcu_dump_cpu_stacks+0x90/0xd0 [ 9643.797436] [] rcu_check_callbacks+0x442/0x720 [ 9643.805121] [] ? tick_sched_handle.isra.13+0x60/0x60 [ 9643.813388] [] update_process_times+0x47/0x80 [ 9643.820937] [] tick_sched_handle.isra.13+0x25/0x60 [ 9643.828953] [] tick_sched_timer+0x41/0x70 [ 9643.836078] [] __hrtimer_run_queues+0xd2/0x260 [ 9643.843668] [] hrtimer_interrupt+0xb0/0x1e0 [ 9643.850942] [] ? call_softirq+0x1c/0x30 [ 9643.857830] [] local_apic_timer_interrupt+0x37/0x60 [ 9643.865855] [] smp_apic_timer_interrupt+0x3f/0x60 [ 9643.873669] [] apic_timer_interrupt+0x6d/0x80 [ 9643.881101] [] ? osd_inode_iteration+0x49e/0xcc0 [osd_ldiskfs] [ 9643.890909] [] ? osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [ 9643.900018] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 9643.908927] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 9643.917555] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 9643.925974] [] ? wake_up_state+0x20/0x20 [ 9643.932932] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 9643.942013] [] kthread+0xcf/0xe0 [ 9643.948183] [] ? kthread_create_on_node+0x140/0x140 [ 9643.956211] [] ret_from_fork+0x58/0x90 [ 9643.962966] [] ? kthread_create_on_node+0x140/0x140 [ 9669.179310] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [ 9669.188695] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 9669.318246] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 9669.331422] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 9669.347486] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 9669.356784] RIP: 0010:[] [] osd_inode_iteration+0x2ac/0xcc0 [osd_ldiskfs] [ 9669.368926] RSP: 0018:ffff880703603d18 EFLAGS: 00000202 [ 9669.375779] RAX: 0000000000000004 RBX: 000000009f8f2001 RCX: ffff880703603d98 [ 9669.384687] RDX: ffff880703603d78 RSI: 0000000000000000 RDI: ffff8803162f6000 [ 9669.393587] RBP: ffff880703603df0 R08: 0000000000000001 R09: 0000000000000004 [ 9669.402482] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [ 9669.411385] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [ 9669.420284] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 9669.430251] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9669.437594] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 9669.446498] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 9669.455402] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 9669.464297] Stack: [ 9669.467464] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [ 9669.476713] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [ 9669.485962] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [ 9669.495219] Call Trace: [ 9669.498901] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 9669.508042] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 9669.516875] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 9669.525522] [] ? wake_up_state+0x20/0x20 [ 9669.532719] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 9669.542038] [] kthread+0xcf/0xe0 [ 9669.548435] [] ? kthread_create_on_node+0x140/0x140 [ 9669.556690] [] ret_from_fork+0x58/0x90 [ 9669.563660] [] ? kthread_create_on_node+0x140/0x140 [ 9669.571906] Code: 00 44 0f b6 85 67 ff ff ff 48 8d 4d a8 48 8d 55 88 48 8b b5 38 ff ff ff 48 8b bd 48 ff ff ff 48 8b 85 30 ff ff ff ff d0 83 f8 03 <41> 89 c5 0f 84 4b 02 00 00 0f 8e 15 01 00 00 83 f8 05 66 90 0f [ 9697.180284] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [ 9697.189868] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 9697.321625] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 9697.334929] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 9697.351224] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 9697.360625] RIP: 0010:[] [] osd_scrub_exec+0x30/0x5b0 [osd_ldiskfs] [ 9697.372284] RSP: 0018:ffff880703603c78 EFLAGS: 00000296 [ 9697.379243] RAX: 00000000fb6db880 RBX: ffff880703603c37 RCX: 0000000000000000 [ 9697.388227] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 9697.397202] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [ 9697.406162] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [ 9697.415113] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [ 9697.424059] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 9697.434085] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9697.441494] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 9697.450459] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 9697.459437] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 9697.468386] Stack: [ 9697.471600] ffffea00027e3c80 000000009f8f2001 ffff880703603df0 0000000000000001 [ 9697.480891] ffff880703603d57 000000009f8f2001 0000000000000004 ffff880703603d57 [ 9697.490184] ffff88042bd78800 ffff88042bd78800 00000000000072cf ffff880419dc3a00 [ 9697.499483] Call Trace: [ 9697.503211] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [ 9697.512383] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 9697.521544] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 9697.530414] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 9697.539082] [] ? wake_up_state+0x20/0x20 [ 9697.546292] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 9697.555638] [] kthread+0xcf/0xe0 [ 9697.562064] [] ? kthread_create_on_node+0x140/0x140 [ 9697.570334] [] ret_from_fork+0x58/0x90 [ 9697.577338] [] ? kthread_create_on_node+0x140/0x140 [ 9697.585609] Code: 55 48 89 e5 41 57 49 89 f7 41 56 41 55 41 54 45 31 e4 53 48 83 ec 68 48 8b 9e 08 10 00 00 4c 89 45 90 65 48 8b 04 25 28 00 00 00 <48> 89 45 d0 31 c0 48 8d 43 08 48 85 db 4c 0f 45 e0 41 83 f9 04 [ 9725.181259] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [ 9725.190894] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 9725.322493] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 9725.335777] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 9725.352022] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 9725.361402] RIP: 0010:[] [] ldiskfs_itable_unused_count+0x25/0x30 [ldiskfs] [ 9725.373830] RSP: 0018:ffff880703603d08 EFLAGS: 00000202 [ 9725.380768] RAX: 00000000000072ce RBX: 0000000000000001 RCX: ffff88042bd78800 [ 9725.389750] RDX: 00000000000072ce RSI: ffff880419dc3a00 RDI: ffff88042bd79800 [ 9725.398706] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [ 9725.407653] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [ 9725.416596] R13: 0000000000000018 R14: ffff880703603c78 R15: 0000000000000296 [ 9725.425536] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 9725.435550] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9725.442943] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 9725.451894] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 9725.460835] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 9725.469765] Stack: [ 9725.472958] ffff880703603df0 ffffffffa121cfb9 ffffffffa121b990 ffffffffa1217a20 [ 9725.482231] ffff8800b2a36000 00000000810d354f ffff8803162f6000 ffff8800b2a37468 [ 9725.491522] 0000000020000000 010000000000000c 0000000000000000 0000000000000000 [ 9725.500803] Call Trace: [ 9725.504519] [] osd_inode_iteration+0x269/0xcc0 [osd_ldiskfs] [ 9725.513664] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 9725.522815] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 9725.531670] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 9725.540330] [] ? wake_up_state+0x20/0x20 [ 9725.547541] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 9725.556875] [] kthread+0xcf/0xe0 [ 9725.563295] [] ? kthread_create_on_node+0x140/0x140 [ 9725.571566] [] ret_from_fork+0x58/0x90 [ 9725.578566] [] ? kthread_create_on_node+0x140/0x140 [ 9725.586824] Code: 84 00 00 00 00 00 66 66 66 66 90 55 48 8b 8f a8 03 00 00 31 c0 0f b7 56 1c 48 89 e5 48 83 39 3f 76 07 0f b7 46 32 c1 e0 10 09 d0 <5d> c3 66 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 89 16 48 8b [ 9753.182241] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [ 9753.191850] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 9753.323400] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 9753.336679] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 9753.352917] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 9753.362294] RIP: 0010:[] [] osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 9753.374026] RSP: 0018:ffff880703603d10 EFLAGS: 00000287 [ 9753.380959] RAX: ffffffffa1217a20 RBX: 000000009f8f2001 RCX: ffff880703603d98 [ 9753.389927] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 9753.398881] RBP: ffff880703603df0 R08: 0000000000000001 R09: 0000000000000004 [ 9753.407823] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [ 9753.416763] R13: ffff880703603df0 R14: 000000009f8f2001 R15: ffff880703603d08 [ 9753.425710] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 9753.435732] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9753.443125] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 9753.452089] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 9753.461037] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 9753.469981] Stack: [ 9753.473191] ffffffffa121cff9 ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 [ 9753.482503] 00000000810d354f ffff8803162f6000 ffff8800b2a37468 0000000020000000 [ 9753.491798] 010000000000000c 0000000000000000 0000000000000000 ffff8800b2a36000 [ 9753.501094] Call Trace: [ 9753.504814] [] ? osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [ 9753.514176] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 9753.523366] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 9753.532236] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 9753.540901] [] ? wake_up_state+0x20/0x20 [ 9753.548106] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 9753.557457] [] kthread+0xcf/0xe0 [ 9753.563901] [] ? kthread_create_on_node+0x140/0x140 [ 9753.572183] [] ret_from_fork+0x58/0x90 [ 9753.579206] [] ? kthread_create_on_node+0x140/0x140 [ 9753.587487] Code: f7 48 8d 04 80 48 c1 e0 03 48 8d 4c 02 10 48 01 c2 e8 25 fc ff ff 48 83 c4 08 5b 41 5c 41 5d 41 5e 5d c3 0f 1f 84 00 00 00 00 00 <66> 66 66 66 90 55 48 89 e5 41 57 49 89 f7 41 56 41 55 49 89 fd [ 9781.183216] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [ 9781.192849] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 9781.324586] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 9781.337881] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 9781.354151] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 9781.363542] RIP: 0010:[] [] osd_scrub_exec+0x96/0x5b0 [osd_ldiskfs] [ 9781.375200] RSP: 0018:ffff880703603d10 EFLAGS: 00000292 [ 9781.382146] RAX: 0000000000000000 RBX: 000000009f8f2001 RCX: 0000000000000000 [ 9781.391129] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 9781.400109] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [ 9781.409069] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [ 9781.418023] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [ 9781.426979] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 9781.437014] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9781.444426] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 9781.453394] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 9781.462356] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 9781.471302] Stack: [ 9781.474515] ffffffffa121d1e9 ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 [ 9781.483800] 00000000810d354f ffff8803162f6000 ffff8800b2a37468 0000000020000000 [ 9781.493102] 010000000000000c 0000000000000000 0000000000000000 ffff8800b2a36000 [ 9781.502395] Call Trace: [ 9781.506116] [] ? osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [ 9781.515479] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 9781.524648] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 9781.533518] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 9781.542188] [] ? wake_up_state+0x20/0x20 [ 9781.549399] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 9781.558747] [] kthread+0xcf/0xe0 [ 9781.565180] [] ? kthread_create_on_node+0x140/0x140 [ 9781.573460] [] ret_from_fork+0x58/0x90 [ 9781.580473] [] ? kthread_create_on_node+0x140/0x140 [ 9781.588751] Code: 00 00 41 80 a7 74 14 00 00 fe 48 8b 4d d0 65 48 33 0c 25 28 00 00 00 0f 85 b1 03 00 00 48 83 c4 68 5b 41 5c 41 5d 41 5e 41 5f 5d 66 0f 1f 84 00 00 00 00 00 4c 8d ae f0 10 00 00 44 89 4d 88 [ 9809.184192] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [ 9809.193813] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 9809.325456] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 9809.338745] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 9809.354999] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 9809.364387] RIP: 0010:[] [] osd_inode_iteration+0x269/0xcc0 [osd_ldiskfs] [ 9809.376612] RSP: 0018:ffff880703603d18 EFLAGS: 00000202 [ 9809.383556] RAX: 00000000000072ce RBX: ffff880703603d98 RCX: ffff88042bd78800 [ 9809.392539] RDX: 00000000000072ce RSI: ffff880419dc3a00 RDI: ffff88042bd79800 [ 9809.401503] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [ 9809.410460] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000001 [ 9809.419408] R13: 0000000000000004 R14: 000000009f8f2001 R15: ffffea00027e3c80 [ 9809.428358] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 9809.438382] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9809.445784] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 9809.454745] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 9809.463697] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 9809.472637] Stack: [ 9809.475840] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [ 9809.485129] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [ 9809.494428] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [ 9809.503712] Call Trace: [ 9809.507423] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 9809.516583] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 9809.525448] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 9809.534106] [] ? wake_up_state+0x20/0x20 [ 9809.541313] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 9809.550654] [] kthread+0xcf/0xe0 [ 9809.557076] [] ? kthread_create_on_node+0x140/0x140 [ 9809.565344] [] ret_from_fork+0x58/0x90 [ 9809.572353] [] ? kthread_create_on_node+0x140/0x140 [ 9809.580621] Code: 88 e8 dc 2d f5 ff 48 85 c0 48 89 45 90 0f 84 b3 08 00 00 44 8b 7d a0 48 8b 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 e8 77 5b f4 ff <42> 8d 14 38 48 8b 45 88 48 8b 80 a8 03 00 00 48 3b 50 20 0f 83 [ 9823.726699] INFO: rcu_sched self-detected stall on CPU { 24} (t=2400060 jiffies g=907112 c=907111 q=1457053) [ 9823.739027] Task dump for CPU 24: [ 9823.743757] OI_scrub R running task 0 5551 2 0x00000088 [ 9823.752674] ffff88083fde6dd0 00000000fb6db880 ffff88082da03db0 ffffffff810c47e8 [ 9823.761990] 0000000000000018 ffffffff81a1e7c0 ffff88082da03dc8 ffffffff810c8079 [ 9823.771306] 0000000000000009 ffff88082da03df8 ffffffff811378e0 ffff88082da101c0 [ 9823.780601] Call Trace: [ 9823.784282] [] sched_show_task+0xa8/0x110 [ 9823.792254] [] dump_cpu_task+0x39/0x70 [ 9823.799216] [] rcu_dump_cpu_stacks+0x90/0xd0 [ 9823.806752] [] rcu_check_callbacks+0x442/0x720 [ 9823.814439] [] ? tick_sched_handle.isra.13+0x60/0x60 [ 9823.822707] [] update_process_times+0x47/0x80 [ 9823.830272] [] tick_sched_handle.isra.13+0x25/0x60 [ 9823.838298] [] tick_sched_timer+0x41/0x70 [ 9823.845440] [] __hrtimer_run_queues+0xd2/0x260 [ 9823.853032] [] hrtimer_interrupt+0xb0/0x1e0 [ 9823.860305] [] ? call_softirq+0x1c/0x30 [ 9823.867186] [] local_apic_timer_interrupt+0x37/0x60 [ 9823.875219] [] smp_apic_timer_interrupt+0x3f/0x60 [ 9823.883040] [] apic_timer_interrupt+0x6d/0x80 [ 9823.890463] [] ? osd_inode_iteration+0x28a/0xcc0 [osd_ldiskfs] [ 9823.900255] [] ? osd_inode_iteration+0x269/0xcc0 [osd_ldiskfs] [ 9823.909350] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 9823.918249] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 9823.926866] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 9823.935278] [] ? wake_up_state+0x20/0x20 [ 9823.942238] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 9823.951329] [] kthread+0xcf/0xe0 [ 9823.957505] [] ? kthread_create_on_node+0x140/0x140 [ 9823.965520] [] ret_from_fork+0x58/0x90 [ 9823.972287] [] ? kthread_create_on_node+0x140/0x140 [ 9849.185579] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [ 9849.194976] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 9849.324562] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 9849.337728] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 9849.353804] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 9849.363106] RIP: 0010:[] [] osd_inode_iteration+0x282/0xcc0 [osd_ldiskfs] [ 9849.375249] RSP: 0018:ffff880703603d18 EFLAGS: 00000287 [ 9849.382103] RAX: ffff88042bd78800 RBX: 000000009f8f2001 RCX: ffff88042bd78800 [ 9849.391013] RDX: 00000000000072cf RSI: ffff880419dc3a00 RDI: ffff88042bd79800 [ 9849.399913] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [ 9849.408822] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [ 9849.417725] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [ 9849.426613] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 9849.436581] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9849.443928] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 9849.452838] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 9849.461747] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 9849.470649] Stack: [ 9849.473821] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [ 9849.483072] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [ 9849.492318] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [ 9849.501565] Call Trace: [ 9849.505250] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 9849.514390] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 9849.523238] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 9849.531887] [] ? wake_up_state+0x20/0x20 [ 9849.539081] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 9849.548415] [] kthread+0xcf/0xe0 [ 9849.554825] [] ? kthread_create_on_node+0x140/0x140 [ 9849.563080] [] ret_from_fork+0x58/0x90 [ 9849.570070] [] ? kthread_create_on_node+0x140/0x140 [ 9849.578316] Code: 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 e8 77 5b f4 ff 42 8d 14 38 48 8b 45 88 48 8b 80 a8 03 00 00 48 3b 50 20 0f 83 57 01 00 00 <44> 0f b6 85 67 ff ff ff 48 8d 4d a8 48 8d 55 88 48 8b b5 38 ff [ 9877.186554] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [ 9877.196155] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 9877.328006] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 9877.341327] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 9877.357624] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 9877.367039] RIP: 0010:[] [] osd_inode_iteration+0x251/0xcc0 [osd_ldiskfs] [ 9877.379281] RSP: 0018:ffff880703603d18 EFLAGS: 00000286 [ 9877.386247] RAX: ffff8800b2a37468 RBX: 000000009f8f2001 RCX: 0000000000000000 [ 9877.395239] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 9877.404223] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [ 9877.413192] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffff880703603d57 [ 9877.422151] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [ 9877.431107] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 9877.441139] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9877.448548] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 9877.457519] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 9877.466486] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 9877.475437] Stack: [ 9877.478655] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [ 9877.487951] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [ 9877.497234] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [ 9877.506545] Call Trace: [ 9877.510271] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 9877.519442] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 9877.528320] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 9877.537000] [] ? wake_up_state+0x20/0x20 [ 9877.544218] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 9877.553557] [] kthread+0xcf/0xe0 [ 9877.559991] [] ? kthread_create_on_node+0x140/0x140 [ 9877.568277] [] ret_from_fork+0x58/0x90 [ 9877.575300] [] ? kthread_create_on_node+0x140/0x140 [ 9877.583582] Code: b0 00 00 00 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 00 8b 75 98 48 8b 7d 88 e8 dc 2d f5 ff 48 85 c0 48 89 45 90 0f 84 b3 08 00 00 <44> 8b 7d a0 48 8b 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 e8 77 [ 9905.187530] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [ 9905.197161] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 9905.328966] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 9905.342276] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 9905.358580] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 9905.367987] RIP: 0010:[] [] osd_scrub_exec+0xf7/0x5b0 [osd_ldiskfs] [ 9905.379659] RSP: 0018:ffff880703603c78 EFLAGS: 00000246 [ 9905.386613] RAX: 0000000000280001 RBX: ffff880703603c37 RCX: 0000000000000000 [ 9905.395603] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 9905.404592] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [ 9905.413564] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [ 9905.422512] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [ 9905.431462] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 9905.441498] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9905.448907] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 9905.457882] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 9905.466845] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 9905.475783] Stack: [ 9905.478986] ffff880703603df0 000000009f8f2001 ffffea00027e3c80 000000009f8f2001 [ 9905.488266] ffff880703603d57 ffff880703603d57 ffff8800b2a37468 0000000000000000 [ 9905.497544] ffff880703603d78 ffff880419dc3a00 ffff88042bd79800 ffffffffffffff10 [ 9905.506850] Call Trace: [ 9905.510569] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [ 9905.519732] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 9905.528890] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 9905.537756] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 9905.546416] [] ? wake_up_state+0x20/0x20 [ 9905.553616] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 9905.562956] [] kthread+0xcf/0xe0 [ 9905.569378] [] ? kthread_create_on_node+0x140/0x140 [ 9905.577645] [] ret_from_fork+0x58/0x90 [ 9905.584662] [] ? kthread_create_on_node+0x140/0x140 [ 9905.592938] Code: 8b 4d 88 48 85 db 74 23 f6 83 18 0a 00 00 08 74 1a 4d 85 e4 74 15 41 8b 87 6c 14 00 00 41 39 84 24 0c 0a 00 00 0f 82 d9 00 00 00 <31> c0 41 f6 87 74 14 00 00 04 0f 85 6e ff ff ff 41 83 f9 02 0f [ 9933.188513] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [ 9933.198131] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 9933.329673] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 9933.342953] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 9933.359187] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 9933.368564] RIP: 0010:[] [] osd_inode_iteration+0x2ac/0xcc0 [osd_ldiskfs] [ 9933.380783] RSP: 0018:ffff880703603d18 EFLAGS: 00000202 [ 9933.387711] RAX: 0000000000000004 RBX: 0000000000000000 RCX: ffff880703603d98 [ 9933.396675] RDX: ffff880703603d78 RSI: 0000000000000000 RDI: ffff8803162f6000 [ 9933.405622] RBP: ffff880703603df0 R08: 0000000000000001 R09: 0000000000000004 [ 9933.414572] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffff880703603d57 [ 9933.423510] R13: 0000000000000004 R14: 000000009f8f2001 R15: 0000000000000018 [ 9933.432451] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 9933.442466] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9933.449857] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 9933.458810] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 9933.467766] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 9933.476695] Stack: [ 9933.479880] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [ 9933.489146] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [ 9933.498436] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [ 9933.507707] Call Trace: [ 9933.511399] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 9933.520543] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 9933.529407] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 9933.538065] [] ? wake_up_state+0x20/0x20 [ 9933.545262] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 9933.554598] [] kthread+0xcf/0xe0 [ 9933.561020] [] ? kthread_create_on_node+0x140/0x140 [ 9933.569289] [] ret_from_fork+0x58/0x90 [ 9933.576294] [] ? kthread_create_on_node+0x140/0x140 [ 9933.584559] Code: 00 44 0f b6 85 67 ff ff ff 48 8d 4d a8 48 8d 55 88 48 8b b5 38 ff ff ff 48 8b bd 48 ff ff ff 48 8b 85 30 ff ff ff ff d0 83 f8 03 <41> 89 c5 0f 84 4b 02 00 00 0f 8e 15 01 00 00 83 f8 05 66 90 0f [ 9961.189480] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [ 9961.199108] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 9961.330870] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 9961.344170] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 9961.360436] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 9961.369833] RIP: 0010:[] [] osd_scrub_next+0x149/0x3c0 [osd_ldiskfs] [ 9961.381597] RSP: 0018:ffff880703603c70 EFLAGS: 00000202 [ 9961.388550] RAX: 0000000000000004 RBX: ffff880703603c37 RCX: ffff880703603d98 [ 9961.397531] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 9961.406502] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [ 9961.415458] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [ 9961.424412] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [ 9961.433357] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 9961.443381] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9961.450789] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 9961.459752] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 9961.468712] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 9961.477660] Stack: [ 9961.480874] 000000009f8f2001 ffff880703603d08 ffff880703603d57 ffffea00027e3c80 [ 9961.490166] 000000009f8f2001 ffff880703603d57 ffff880703603d57 0000000000000000 [ 9961.499462] ffff88042bd78800 00000000000072ce ffff880419dc3a00 ffff88042bd79800 [ 9961.508757] Call Trace: [ 9961.512481] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [ 9961.521644] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 9961.530803] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 9961.539670] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 9961.548322] [] ? wake_up_state+0x20/0x20 [ 9961.555527] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 9961.564864] [] kthread+0xcf/0xe0 [ 9961.571292] [] ? kthread_create_on_node+0x140/0x140 [ 9961.579562] [] ret_from_fork+0x58/0x90 [ 9961.586572] [] ? kthread_create_on_node+0x140/0x140 [ 9961.594839] Code: 45 90 74 49 49 8d 9f 10 11 00 00 48 89 df e8 df 6a 47 e0 41 c7 87 58 10 00 00 02 00 00 00 48 89 df e8 fc 69 47 e0 b8 05 00 00 00 <48> 8b 75 d0 65 48 33 34 25 28 00 00 00 0f 85 4c 02 00 00 48 83 [ 9989.190462] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [ 9989.200087] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [ 9989.331717] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [ 9989.345006] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 9989.361257] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [ 9989.370630] RIP: 0010:[] [] osd_inode_iteration+0x4a6/0xcc0 [osd_ldiskfs] [ 9989.382859] RSP: 0018:ffff880703603d18 EFLAGS: 00000246 [ 9989.389797] RAX: 0000000020000000 RBX: 000000009f8f2001 RCX: 0000000000000000 [ 9989.398772] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [ 9989.407736] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [ 9989.416694] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffff880703603d57 [ 9989.425642] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [ 9989.434605] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [ 9989.444627] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9989.452024] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [ 9989.460989] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 9989.469943] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 9989.478885] Stack: [ 9989.482094] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [ 9989.491389] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [ 9989.500686] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [ 9989.509981] Call Trace: [ 9989.513695] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [ 9989.522865] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [ 9989.531741] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [ 9989.540412] [] ? wake_up_state+0x20/0x20 [ 9989.547624] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [ 9989.556972] [] kthread+0xcf/0xe0 [ 9989.563410] [] ? kthread_create_on_node+0x140/0x140 [ 9989.571697] [] ret_from_fork+0x58/0x90 [ 9989.578723] [] ? kthread_create_on_node+0x140/0x140 [ 9989.587003] Code: 8b 4d a8 48 8d 55 88 48 8b b5 38 ff ff ff 48 8b bd 48 ff ff ff 48 8b 85 28 ff ff ff ff d0 85 c0 41 89 c5 75 1c 8b 85 58 ff ff ff <41> 3b 04 24 72 10 48 8b 85 50 ff ff ff 44 3b 30 0f 87 95 fd ff [10003.735970] INFO: rcu_sched self-detected stall on CPU { 24} (t=2580063 jiffies g=907112 c=907111 q=1468877) [10003.748283] Task dump for CPU 24: [10003.753037] OI_scrub R running task 0 5551 2 0x00000088 [10003.761982] ffff88083fde6dd0 00000000fb6db880 ffff88082da03db0 ffffffff810c47e8 [10003.771305] 0000000000000018 ffffffff81a1e7c0 ffff88082da03dc8 ffffffff810c8079 [10003.780622] 0000000000000009 ffff88082da03df8 ffffffff811378e0 ffff88082da101c0 [10003.789913] Call Trace: [10003.793615] [] sched_show_task+0xa8/0x110 [10003.801615] [] dump_cpu_task+0x39/0x70 [10003.808591] [] rcu_dump_cpu_stacks+0x90/0xd0 [10003.816135] [] rcu_check_callbacks+0x442/0x720 [10003.823844] [] ? tick_sched_handle.isra.13+0x60/0x60 [10003.832114] [] update_process_times+0x47/0x80 [10003.839685] [] tick_sched_handle.isra.13+0x25/0x60 [10003.847727] [] tick_sched_timer+0x41/0x70 [10003.854867] [] __hrtimer_run_queues+0xd2/0x260 [10003.862469] [] hrtimer_interrupt+0xb0/0x1e0 [10003.869766] [] ? call_softirq+0x1c/0x30 [10003.876656] [] local_apic_timer_interrupt+0x37/0x60 [10003.884695] [] smp_apic_timer_interrupt+0x3f/0x60 [10003.892525] [] apic_timer_interrupt+0x6d/0x80 [10003.899952] [] ? osd_scrub_exec+0x3d/0x5b0 [osd_ldiskfs] [10003.909194] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [10003.918095] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [10003.927010] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [10003.935613] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [10003.944024] [] ? wake_up_state+0x20/0x20 [10003.950982] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [10003.960065] [] kthread+0xcf/0xe0 [10003.966222] [] ? kthread_create_on_node+0x140/0x140 [10003.974229] [] ret_from_fork+0x58/0x90 [10003.980985] [] ? kthread_create_on_node+0x140/0x140 [10029.191857] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [10029.201226] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [10029.331105] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [10029.344274] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [10029.360332] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [10029.369630] RIP: 0010:[] [] osd_inode_iteration+0x251/0xcc0 [osd_ldiskfs] [10029.381768] RSP: 0018:ffff880703603d18 EFLAGS: 00000286 [10029.388629] RAX: ffff8800b2a37468 RBX: 000000009f8f2001 RCX: 0000000000000000 [10029.397527] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [10029.406434] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [10029.415339] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [10029.424237] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [10029.433137] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [10029.443101] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10029.450443] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [10029.459340] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [10029.468256] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [10029.477160] Stack: [10029.480327] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [10029.489577] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [10029.498829] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [10029.508078] Call Trace: [10029.511767] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [10029.520894] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [10029.529737] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [10029.538390] [] ? wake_up_state+0x20/0x20 [10029.545582] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [10029.554912] [] kthread+0xcf/0xe0 [10029.561371] [] ? kthread_create_on_node+0x140/0x140 [10029.569624] [] ret_from_fork+0x58/0x90 [10029.576605] [] ? kthread_create_on_node+0x140/0x140 [10029.584845] Code: b0 00 00 00 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 00 8b 75 98 48 8b 7d 88 e8 dc 2d f5 ff 48 85 c0 48 89 45 90 0f 84 b3 08 00 00 <44> 8b 7d a0 48 8b 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 e8 77 [10057.192832] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [10057.202451] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [10057.334725] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [10057.348033] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [10057.364338] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [10057.373740] RIP: 0010:[] [] osd_inode_iteration+0x264/0xcc0 [osd_ldiskfs] [10057.386015] RSP: 0018:ffff880703603d18 EFLAGS: 00000286 [10057.392987] RAX: ffff8800b2a37468 RBX: ffff880703603d98 RCX: 0000000000000000 [10057.401984] RDX: ffff880703603d78 RSI: ffff880419dc3a00 RDI: ffff88042bd79800 [10057.410980] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [10057.419952] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000001 [10057.428908] R13: 0000000000000004 R14: 000000009f8f2001 R15: ffffea00027e3c80 [10057.437865] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [10057.447890] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10057.455297] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [10057.464262] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [10057.473232] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [10057.482217] Stack: [10057.485435] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [10057.494720] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [10057.504014] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [10057.513292] Call Trace: [10057.517016] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [10057.526184] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [10057.535059] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [10057.543751] [] ? wake_up_state+0x20/0x20 [10057.550970] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [10057.560322] [] kthread+0xcf/0xe0 [10057.566754] [] ? kthread_create_on_node+0x140/0x140 [10057.575044] [] ret_from_fork+0x58/0x90 [10057.582059] [] ? kthread_create_on_node+0x140/0x140 [10057.590339] Code: 75 98 48 8b 7d 88 e8 dc 2d f5 ff 48 85 c0 48 89 45 90 0f 84 b3 08 00 00 44 8b 7d a0 48 8b 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 77 5b f4 ff 42 8d 14 38 48 8b 45 88 48 8b 80 a8 03 00 00 48 [10085.193806] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [10085.203473] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [10085.335776] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [10085.349077] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [10085.365372] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [10085.374777] RIP: 0010:[] [] osd_scrub_next+0x2b/0x3c0 [osd_ldiskfs] [10085.386448] RSP: 0018:ffff880703603c70 EFLAGS: 00000282 [10085.393402] RAX: 00000000fb6db880 RBX: ffff880703603c37 RCX: ffff880703603d98 [10085.402390] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [10085.411376] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [10085.420352] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [10085.429304] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [10085.438254] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [10085.448280] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10085.455689] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [10085.464665] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [10085.473631] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [10085.482585] Stack: [10085.485810] 000000009f8f2001 ffff880703603d08 0000000000000001 ffff880703603df0 [10085.495116] 000000009f8f2001 ffff880703603d57 000000009f8f2001 0000000000000004 [10085.504409] ffff880703603d57 ffff88042bd78800 ffff88042bd78800 00000000000072cf [10085.513712] Call Trace: [10085.517452] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [10085.526619] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [10085.535784] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [10085.544657] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [10085.553330] [] ? wake_up_state+0x20/0x20 [10085.560537] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [10085.569900] [] kthread+0xcf/0xe0 [10085.576322] [] ? kthread_create_on_node+0x140/0x140 [10085.584604] [] ret_from_fork+0x58/0x90 [10085.591614] [] ? kthread_create_on_node+0x140/0x140 [10085.599880] Code: 66 66 66 66 90 55 48 89 e5 41 57 49 89 f7 41 56 41 55 49 89 fd 41 54 49 89 d4 53 48 89 cb 48 83 ec 70 65 48 8b 04 25 28 00 00 00 <48> 89 45 d0 31 c0 48 8b 05 f8 59 9c ff 48 85 c0 75 7b 41 f6 87 [10113.194783] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [10113.204427] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [10113.336591] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [10113.349872] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [10113.366120] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [10113.375505] RIP: 0010:[] [] osd_inode_iteration+0x264/0xcc0 [osd_ldiskfs] [10113.387743] RSP: 0018:ffff880703603d18 EFLAGS: 00000286 [10113.394686] RAX: ffff8800b2a37468 RBX: 000000009f8f2001 RCX: 0000000000000000 [10113.403667] RDX: ffff880703603d78 RSI: ffff880419dc3a00 RDI: ffff88042bd79800 [10113.412629] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [10113.421596] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [10113.430543] R13: ffff880703603d08 R14: 000000009f8f2001 R15: ffff880703603d08 [10113.439498] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [10113.449531] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10113.456941] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [10113.465912] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [10113.474877] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [10113.483824] Stack: [10113.487043] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [10113.496338] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [10113.505642] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [10113.514941] Call Trace: [10113.518666] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [10113.527840] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [10113.536723] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [10113.545401] [] ? wake_up_state+0x20/0x20 [10113.552616] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [10113.561991] [] kthread+0xcf/0xe0 [10113.568434] [] ? kthread_create_on_node+0x140/0x140 [10113.576734] [] ret_from_fork+0x58/0x90 [10113.583757] [] ? kthread_create_on_node+0x140/0x140 [10113.592048] Code: 75 98 48 8b 7d 88 e8 dc 2d f5 ff 48 85 c0 48 89 45 90 0f 84 b3 08 00 00 44 8b 7d a0 48 8b 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 77 5b f4 ff 42 8d 14 38 48 8b 45 88 48 8b 80 a8 03 00 00 48 [10141.195759] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [10141.205390] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [10141.337809] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [10141.351142] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [10141.367453] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [10141.376868] RIP: 0010:[] [] osd_scrub_next+0x167/0x3c0 [osd_ldiskfs] [10141.388651] RSP: 0018:ffff880703603d00 EFLAGS: 00000282 [10141.395630] RAX: 0000000000000004 RBX: 0000000000000001 RCX: ffff880703603d98 [10141.404626] RDX: ffff880703603d78 RSI: 0000000000000000 RDI: ffff8803162f6000 [10141.413611] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [10141.422580] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [10141.431535] R13: 0000000000000018 R14: ffff880703603c78 R15: 0000000000000286 [10141.440485] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [10141.450506] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10141.457908] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [10141.466870] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [10141.475833] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [10141.484767] Stack: [10141.487974] 0000000000000001 ffff880703603df0 ffffffffa121cff9 ffffffffa121b990 [10141.497272] ffffffffa1217a20 ffff8800b2a36000 00000000810d354f ffff8803162f6000 [10141.506555] ffff8800b2a37468 0000000020000000 010000000000000c 0000000000000000 [10141.515848] Call Trace: [10141.519581] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [10141.528742] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [10141.537899] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [10141.546764] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [10141.555433] [] ? wake_up_state+0x20/0x20 [10141.562635] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [10141.571982] [] kthread+0xcf/0xe0 [10141.578404] [] ? kthread_create_on_node+0x140/0x140 [10141.586684] [] ret_from_fork+0x58/0x90 [10141.593686] [] ? kthread_create_on_node+0x140/0x140 [10141.601953] Code: 48 89 df e8 fc 69 47 e0 b8 05 00 00 00 48 8b 75 d0 65 48 33 34 25 28 00 00 00 0f 85 4c 02 00 00 48 83 c4 70 5b 41 5c 41 5d 41 5e <41> 5f 5d c3 48 8b 05 be 58 9c ff 48 85 c0 0f 84 c2 fe ff ff 0f [10169.196727] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [10169.206353] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [10169.338361] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [10169.351646] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [10169.367892] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [10169.377273] RIP: 0010:[] [] osd_scrub_next+0x12/0x3c0 [osd_ldiskfs] [10169.388944] RSP: 0018:ffff880703603cf0 EFLAGS: 00000287 [10169.395892] RAX: ffffffffa1217a20 RBX: 000000009f8f2001 RCX: ffff880703603d98 [10169.404871] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [10169.413840] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [10169.422789] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffff880703603c78 [10169.431735] R13: 0000000000000286 R14: 0000000000000010 R15: ffffffffa121ba64 [10169.440685] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [10169.450711] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10169.458116] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [10169.467084] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [10169.476051] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [10169.484998] Stack: [10169.488213] 0000000000000000 00000000ffffffff 0000000000000001 ffff880703603df0 [10169.497503] ffffffffa121cff9 ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 [10169.506805] 00000000810d354f ffff8803162f6000 ffff8800b2a37468 0000000020000000 [10169.516107] Call Trace: [10169.519820] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [10169.528995] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [10169.538154] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [10169.547024] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [10169.555700] [] ? wake_up_state+0x20/0x20 [10169.562915] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [10169.572281] [] kthread+0xcf/0xe0 [10169.578720] [] ? kthread_create_on_node+0x140/0x140 [10169.587025] [] ret_from_fork+0x58/0x90 [10169.594042] [] ? kthread_create_on_node+0x140/0x140 [10169.602318] Code: 25 fc ff ff 48 83 c4 08 5b 41 5c 41 5d 41 5e 5d c3 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 89 e5 41 57 49 89 f7 41 56 41 55 <49> 89 fd 41 54 49 89 d4 53 48 89 cb 48 83 ec 70 65 48 8b 04 25 [10183.745240] INFO: rcu_sched self-detected stall on CPU { 24} (t=2760066 jiffies g=907112 c=907111 q=1553403) [10183.757576] Task dump for CPU 24: [10183.762313] OI_scrub R running task 0 5551 2 0x00000088 [10183.771234] ffff88083fde6dd0 00000000fb6db880 ffff88082da03db0 ffffffff810c47e8 [10183.780564] 0000000000000018 ffffffff81a1e7c0 ffff88082da03dc8 ffffffff810c8079 [10183.789870] 0000000000000009 ffff88082da03df8 ffffffff811378e0 ffff88082da101c0 [10183.799149] Call Trace: [10183.802831] [] sched_show_task+0xa8/0x110 [10183.810815] [] dump_cpu_task+0x39/0x70 [10183.817776] [] rcu_dump_cpu_stacks+0x90/0xd0 [10183.825317] [] rcu_check_callbacks+0x442/0x720 [10183.833021] [] ? tick_sched_handle.isra.13+0x60/0x60 [10183.841290] [] update_process_times+0x47/0x80 [10183.848843] [] tick_sched_handle.isra.13+0x25/0x60 [10183.856868] [] tick_sched_timer+0x41/0x70 [10183.864012] [] __hrtimer_run_queues+0xd2/0x260 [10183.871616] [] hrtimer_interrupt+0xb0/0x1e0 [10183.878918] [] ? call_softirq+0x1c/0x30 [10183.885806] [] local_apic_timer_interrupt+0x37/0x60 [10183.893840] [] smp_apic_timer_interrupt+0x3f/0x60 [10183.901660] [] apic_timer_interrupt+0x6d/0x80 [10183.909097] [] ? osd_scrub_next+0x350/0x3c0 [osd_ldiskfs] [10183.918432] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [10183.927337] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [10183.936251] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [10183.945152] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [10183.953757] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [10183.962159] [] ? wake_up_state+0x20/0x20 [10183.969109] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [10183.978206] [] kthread+0xcf/0xe0 [10183.984384] [] ? kthread_create_on_node+0x140/0x140 [10183.992401] [] ret_from_fork+0x58/0x90 [10183.999147] [] ? kthread_create_on_node+0x140/0x140 [10202.504635] Lustre: soaked-MDT0000: Client e211cd09-6b0c-e43b-059c-f657101042a0 (at 192.168.1.117@o2ib100) reconnecting [10202.517650] Lustre: soaked-MDT0000: Connection restored to e211cd09-6b0c-e43b-059c-f657101042a0 (at 192.168.1.117@o2ib100) [10209.198127] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [10209.207553] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [10209.337628] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [10209.350826] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [10209.366902] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [10209.376200] RIP: 0010:[] [] osd_inode_iteration+0x4a6/0xcc0 [osd_ldiskfs] [10209.388366] RSP: 0018:ffff880703603d18 EFLAGS: 00000246 [10209.395260] RAX: 0000000020000000 RBX: 0000000000000001 RCX: 0000000000000000 [10209.404200] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [10209.413133] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [10209.422063] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [10209.430997] R13: ffffea00027e3c80 R14: 000000009f8f2001 R15: 0000000000000018 [10209.439922] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [10209.449927] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10209.457304] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [10209.466250] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [10209.475206] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [10209.484146] Stack: [10209.487363] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [10209.496644] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [10209.505931] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [10209.515222] Call Trace: [10209.518942] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [10209.528106] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [10209.536975] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [10209.545639] [] ? wake_up_state+0x20/0x20 [10209.552855] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [10209.562223] [] kthread+0xcf/0xe0 [10209.568664] [] ? kthread_create_on_node+0x140/0x140 [10209.576950] [] ret_from_fork+0x58/0x90 [10209.583979] [] ? kthread_create_on_node+0x140/0x140 [10209.592271] Code: 8b 4d a8 48 8d 55 88 48 8b b5 38 ff ff ff 48 8b bd 48 ff ff ff 48 8b 85 28 ff ff ff ff d0 85 c0 41 89 c5 75 1c 8b 85 58 ff ff ff <41> 3b 04 24 72 10 48 8b 85 50 ff ff ff 44 3b 30 0f 87 95 fd ff [10237.199102] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [10237.208785] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [10237.340666] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [10237.353970] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [10237.370258] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [10237.379649] RIP: 0010:[] [] osd_scrub_exec+0xf7/0x5b0 [osd_ldiskfs] [10237.391333] RSP: 0018:ffff880703603c78 EFLAGS: 00000246 [10237.398293] RAX: 0000000000280001 RBX: ffff880703603c37 RCX: 0000000000000000 [10237.407274] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [10237.416256] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [10237.425215] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [10237.434169] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [10237.443126] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [10237.453155] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10237.460567] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [10237.469532] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [10237.478489] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [10237.487434] Stack: [10237.490651] ffff880703603d08 ffff880703603d57 ffff880703603df0 000000009f8f2001 [10237.499944] ffff880703603d57 000000009f8f2001 0000000000000004 ffff880703603d57 [10237.509234] ffff8800b2a37468 0000000000000000 ffff880703603d78 ffff8800b2a36000 [10237.518525] Call Trace: [10237.522253] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [10237.531424] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [10237.540584] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [10237.549459] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [10237.558115] [] ? wake_up_state+0x20/0x20 [10237.565318] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [10237.574662] [] kthread+0xcf/0xe0 [10237.581086] [] ? kthread_create_on_node+0x140/0x140 [10237.589365] [] ret_from_fork+0x58/0x90 [10237.596378] [] ? kthread_create_on_node+0x140/0x140 [10237.604657] Code: 8b 4d 88 48 85 db 74 23 f6 83 18 0a 00 00 08 74 1a 4d 85 e4 74 15 41 8b 87 6c 14 00 00 41 39 84 24 0c 0a 00 00 0f 82 d9 00 00 00 <31> c0 41 f6 87 74 14 00 00 04 0f 85 6e ff ff ff 41 83 f9 02 0f [10265.200078] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [10265.209776] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [10265.341785] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [10265.355076] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [10265.371313] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [10265.380696] RIP: 0010:[] [] osd_scrub_exec+0x12/0x5b0 [osd_ldiskfs] [10265.392344] RSP: 0018:ffff880703603cf0 EFLAGS: 00000293 [10265.399282] RAX: ffffffffa121b990 RBX: 000000009f8f2001 RCX: 0000000000000000 [10265.408258] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [10265.417222] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [10265.426171] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffff880703603c78 [10265.435120] R13: ffff880703603d08 R14: 0000000000000010 R15: ffffffffa121ba79 [10265.444067] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [10265.454095] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10265.461501] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [10265.470467] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [10265.479429] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [10265.488374] Stack: [10265.491595] 0000000000000004 00000000ffffffff 0000000000000001 ffff880703603df0 [10265.500887] ffffffffa121d1e9 ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 [10265.510192] 00000000810d354f ffff8803162f6000 ffff8800b2a37468 0000000020000000 [10265.519482] Call Trace: [10265.523210] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [10265.532378] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [10265.541541] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [10265.550415] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [10265.559093] [] ? wake_up_state+0x20/0x20 [10265.566308] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [10265.575660] [] kthread+0xcf/0xe0 [10265.582093] [] ? kthread_create_on_node+0x140/0x140 [10265.590375] [] ret_from_fork+0x58/0x90 [10265.597389] [] ? kthread_create_on_node+0x140/0x140 [10265.605661] Code: 48 8b 40 40 ba 07 00 00 00 48 89 04 24 31 c0 e8 a5 03 98 ff e9 c7 fd ff ff 66 66 66 66 90 55 48 89 e5 41 57 49 89 f7 41 56 41 55 <41> 54 45 31 e4 53 48 83 ec 68 48 8b 9e 08 10 00 00 4c 89 45 90 [10293.201053] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [10293.210698] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [10293.342808] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [10293.356110] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [10293.372398] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [10293.381793] RIP: 0010:[] [] osd_scrub_next+0x9/0x3c0 [osd_ldiskfs] [10293.393366] RSP: 0018:ffff880703603d08 EFLAGS: 00000287 [10293.400315] RAX: ffffffffa1217a20 RBX: 000000009f8f2001 RCX: ffff880703603d98 [10293.409296] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [10293.418258] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [10293.427229] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [10293.436192] R13: ffff880703603d08 R14: ffff880703603c78 R15: 0000000000000246 [10293.445152] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [10293.455196] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10293.462610] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [10293.471583] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [10293.480553] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [10293.489502] Stack: [10293.492725] ffff880703603df0 ffffffffa121cff9 ffffffffa121b990 ffffffffa1217a20 [10293.502029] ffff8800b2a36000 00000000810d354f ffff8803162f6000 ffff8800b2a37468 [10293.511333] 0000000020000000 010000000000000c 0000000000000000 0000000000000000 [10293.520633] Call Trace: [10293.524368] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [10293.533544] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [10293.542698] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [10293.551557] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [10293.560236] [] ? wake_up_state+0x20/0x20 [10293.567441] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [10293.576792] [] kthread+0xcf/0xe0 [10293.583226] [] ? kthread_create_on_node+0x140/0x140 [10293.591500] [] ret_from_fork+0x58/0x90 [10293.598514] [] ? kthread_create_on_node+0x140/0x140 [10293.606793] Code: 48 8d 4c 02 10 48 01 c2 e8 25 fc ff ff 48 83 c4 08 5b 41 5c 41 5d 41 5e 5d c3 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 89 e5 <41> 57 49 89 f7 41 56 41 55 49 89 fd 41 54 49 89 d4 53 48 89 cb [10316.493459] Lustre: soaked-MDT0000: Client e211cd09-6b0c-e43b-059c-f657101042a0 (at 192.168.1.117@o2ib100) reconnecting [10316.508215] Lustre: soaked-MDT0000: Connection restored to e211cd09-6b0c-e43b-059c-f657101042a0 (at 192.168.1.117@o2ib100) [10321.202024] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [10321.211646] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [10321.343645] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [10321.356930] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [10321.373175] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [10321.382558] RIP: 0010:[] [] ldiskfs_itable_unused_count+0x16/0x30 [ldiskfs] [10321.395018] RSP: 0018:ffff880703603d08 EFLAGS: 00000246 [10321.401980] RAX: 0000000000000000 RBX: 0000000000000001 RCX: ffff88042bd78800 [10321.410973] RDX: 00000000000072ce RSI: ffff880419dc3a00 RDI: ffff88042bd79800 [10321.419957] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [10321.428944] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [10321.437936] R13: 0000000000000018 R14: ffff880703603c78 R15: 0000000000000296 [10321.446927] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [10321.456991] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10321.464427] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [10321.473412] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [10321.482389] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [10321.491362] Stack: [10321.494620] ffff880703603df0 ffffffffa121cfb9 ffffffffa121b990 ffffffffa1217a20 [10321.503939] ffff8800b2a36000 00000000810d354f ffff8803162f6000 ffff8800b2a37468 [10321.513260] 0000000020000000 010000000000000c 0000000000000000 0000000000000000 [10321.522581] Call Trace: [10321.526355] [] osd_inode_iteration+0x269/0xcc0 [osd_ldiskfs] [10321.535556] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [10321.544725] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [10321.553628] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [10321.562332] [] ? wake_up_state+0x20/0x20 [10321.569578] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [10321.578955] [] kthread+0xcf/0xe0 [10321.585469] [] ? kthread_create_on_node+0x140/0x140 [10321.593789] [] ret_from_fork+0x58/0x90 [10321.600835] [] ? kthread_create_on_node+0x140/0x140 [10321.609143] Code: 07 0f b7 46 30 c1 e0 10 09 d0 5d c3 66 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 8b 8f a8 03 00 00 31 c0 0f b7 56 1c 48 89 e5 <48> 83 39 3f 76 07 0f b7 46 32 c1 e0 10 09 d0 5d c3 66 0f 1f 84 [10349.202997] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [10349.212708] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [10349.344462] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [10349.357748] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [10349.373986] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [10349.383366] RIP: 0010:[] [] osd_scrub_exec+0xe9/0x5b0 [osd_ldiskfs] [10349.395016] RSP: 0018:ffff880703603c78 EFLAGS: 00000282 [10349.401962] RAX: 0000000000280001 RBX: ffff880703603c37 RCX: 0000000000000000 [10349.410935] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [10349.419890] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [10349.428839] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [10349.437787] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [10349.446739] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [10349.456768] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10349.464175] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [10349.473159] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [10349.482122] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [10349.491079] Stack: [10349.494296] 0000000000000004 ffff880703603d57 ffff880703603df0 0000000000000000 [10349.503599] ffff880703603d57 000000009f8f2001 0000000000000004 ffff880703603d57 [10349.512896] ffff8800b2a37468 0000000000000000 ffff880703603d78 ffff880419dc3a00 [10349.522200] Call Trace: [10349.525921] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [10349.535109] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [10349.544281] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [10349.553152] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [10349.561838] [] ? wake_up_state+0x20/0x20 [10349.569054] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [10349.578416] [] kthread+0xcf/0xe0 [10349.584854] [] ? kthread_create_on_node+0x140/0x140 [10349.593155] [] ret_from_fork+0x58/0x90 [10349.600166] [] ? kthread_create_on_node+0x140/0x140 [10349.608443] Code: 88 11 00 00 01 4c 89 ef e8 f5 9f e9 df 44 8b 4d 88 48 85 db 74 23 f6 83 18 0a 00 00 08 74 1a 4d 85 e4 74 15 41 8b 87 6c 14 00 00 <41> 39 84 24 0c 0a 00 00 0f 82 d9 00 00 00 31 c0 41 f6 87 74 14 [10363.754505] INFO: rcu_sched self-detected stall on CPU { 24} (t=2940069 jiffies g=907112 c=907111 q=2019436) [10363.766914] Task dump for CPU 24: [10363.771645] OI_scrub R running task 0 5551 2 0x00000088 [10363.780554] ffff88083fde6dd0 00000000fb6db880 ffff88082da03db0 ffffffff810c47e8 [10363.789878] 0000000000000018 ffffffff81a1e7c0 ffff88082da03dc8 ffffffff810c8079 [10363.799189] 0000000000000009 ffff88082da03df8 ffffffff811378e0 ffff88082da101c0 [10363.808479] Call Trace: [10363.812161] [] sched_show_task+0xa8/0x110 [10363.820135] [] dump_cpu_task+0x39/0x70 [10363.827101] [] rcu_dump_cpu_stacks+0x90/0xd0 [10363.834630] [] rcu_check_callbacks+0x442/0x720 [10363.842328] [] ? tick_sched_handle.isra.13+0x60/0x60 [10363.850582] [] update_process_times+0x47/0x80 [10363.858130] [] tick_sched_handle.isra.13+0x25/0x60 [10363.866145] [] tick_sched_timer+0x41/0x70 [10363.873255] [] __hrtimer_run_queues+0xd2/0x260 [10363.880840] [] hrtimer_interrupt+0xb0/0x1e0 [10363.888114] [] ? call_softirq+0x1c/0x30 [10363.894987] [] local_apic_timer_interrupt+0x37/0x60 [10363.903021] [] smp_apic_timer_interrupt+0x3f/0x60 [10363.910845] [] apic_timer_interrupt+0x6d/0x80 [10363.918272] [] ? osd_inode_iteration+0x473/0xcc0 [osd_ldiskfs] [10363.928104] [] ? osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [10363.937207] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [10363.946116] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [10363.954738] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [10363.963148] [] ? wake_up_state+0x20/0x20 [10363.970098] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [10363.979191] [] kthread+0xcf/0xe0 [10363.985362] [] ? kthread_create_on_node+0x140/0x140 [10363.993385] [] ret_from_fork+0x58/0x90 [10364.000141] [] ? kthread_create_on_node+0x140/0x140 [10389.204398] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [10389.213823] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [10389.344008] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [10389.357209] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [10389.373294] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [10389.382606] RIP: 0010:[] [] osd_scrub_exec+0xf9/0x5b0 [osd_ldiskfs] [10389.394168] RSP: 0018:ffff880703603c78 EFLAGS: 00000246 [10389.401034] RAX: 0000000000000000 RBX: ffff880703603c37 RCX: 0000000000000000 [10389.409943] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [10389.418849] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [10389.427752] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [10389.436662] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [10389.445565] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [10389.455541] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10389.462893] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [10389.471798] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [10389.480710] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [10389.489616] Stack: [10389.492799] ffff880703603d08 ffff880703603d57 ffff880703603df0 000000009f8f2001 [10389.502049] ffff880703603d57 000000009f8f2001 0000000000000004 ffff880703603d57 [10389.511320] 0000000000000000 0000000000000000 ffff880703603d78 ffff8800b2a36000 [10389.520587] Call Trace: [10389.524291] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [10389.533436] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [10389.542575] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [10389.551421] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [10389.560079] [] ? wake_up_state+0x20/0x20 [10389.567271] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [10389.576596] [] kthread+0xcf/0xe0 [10389.582996] [] ? kthread_create_on_node+0x140/0x140 [10389.591258] [] ret_from_fork+0x58/0x90 [10389.598242] [] ? kthread_create_on_node+0x140/0x140 [10389.606490] Code: 88 48 85 db 74 23 f6 83 18 0a 00 00 08 74 1a 4d 85 e4 74 15 41 8b 87 6c 14 00 00 41 39 84 24 0c 0a 00 00 0f 82 d9 00 00 00 31 c0 <41> f6 87 74 14 00 00 04 0f 85 6e ff ff ff 41 83 f9 02 0f 84 64 [10417.205373] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [10417.215003] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [10417.347255] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [10417.360550] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [10417.376843] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [10417.386237] RIP: 0010:[] [] osd_scrub_exec+0xe9/0x5b0 [osd_ldiskfs] [10417.397910] RSP: 0018:ffff880703603c78 EFLAGS: 00000282 [10417.404869] RAX: 0000000000280001 RBX: ffff880703603c37 RCX: 0000000000000000 [10417.413853] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [10417.422831] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [10417.431804] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [10417.440761] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [10417.449719] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [10417.459754] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10417.467168] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [10417.476146] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [10417.485108] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [10417.494063] Stack: [10417.497276] ffff880703603d08 000000009f8f2001 ffff880703603df0 000000009f8f2001 [10417.506550] ffff880703603d57 000000009f8f2001 0000000000000004 ffff880703603d57 [10417.515837] 0000000000000000 0000000000000000 ffff880703603d78 ffff8800b2a36000 [10417.525133] Call Trace: [10417.528862] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [10417.538032] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [10417.547187] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [10417.556051] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [10417.564707] [] ? wake_up_state+0x20/0x20 [10417.571918] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [10417.581290] [] kthread+0xcf/0xe0 [10417.587722] [] ? kthread_create_on_node+0x140/0x140 [10417.596007] [] ret_from_fork+0x58/0x90 [10417.603018] [] ? kthread_create_on_node+0x140/0x140 [10417.611294] Code: 88 11 00 00 01 4c 89 ef e8 f5 9f e9 df 44 8b 4d 88 48 85 db 74 23 f6 83 18 0a 00 00 08 74 1a 4d 85 e4 74 15 41 8b 87 6c 14 00 00 <41> 39 84 24 0c 0a 00 00 0f 82 d9 00 00 00 31 c0 41 f6 87 74 14 [10445.206349] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [10445.216014] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [10445.348133] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [10445.361426] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [10445.377682] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [10445.387069] RIP: 0010:[] [] osd_inode_iteration+0x264/0xcc0 [osd_ldiskfs] [10445.399318] RSP: 0018:ffff880703603d18 EFLAGS: 00000286 [10445.406263] RAX: ffff8800b2a37468 RBX: 000000009f8f2001 RCX: 0000000000000000 [10445.415246] RDX: ffff880703603d78 RSI: ffff880419dc3a00 RDI: ffff88042bd79800 [10445.424217] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [10445.433178] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [10445.442133] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [10445.451087] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [10445.461121] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10445.468532] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [10445.477508] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [10445.486467] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [10445.495415] Stack: [10445.498642] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [10445.507945] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [10445.517231] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [10445.526515] Call Trace: [10445.530244] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [10445.539418] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [10445.548299] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [10445.556974] [] ? wake_up_state+0x20/0x20 [10445.564208] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [10445.573579] [] kthread+0xcf/0xe0 [10445.580018] [] ? kthread_create_on_node+0x140/0x140 [10445.588318] [] ret_from_fork+0x58/0x90 [10445.595339] [] ? kthread_create_on_node+0x140/0x140 [10445.603624] Code: 75 98 48 8b 7d 88 e8 dc 2d f5 ff 48 85 c0 48 89 45 90 0f 84 b3 08 00 00 44 8b 7d a0 48 8b 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 77 5b f4 ff 42 8d 14 38 48 8b 45 88 48 8b 80 a8 03 00 00 48 [10473.207317] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [10473.216972] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [10473.349154] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [10473.362468] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [10473.378772] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [10473.388185] RIP: 0010:[] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [10473.400446] RSP: 0018:ffff880703603d18 EFLAGS: 00000292 [10473.407417] RAX: 0000000000000000 RBX: 000000009f8f2001 RCX: 0000000000000000 [10473.416401] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [10473.425387] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [10473.434355] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffff880703603d57 [10473.443312] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [10473.452270] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [10473.462308] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10473.469720] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [10473.478691] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [10473.487667] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [10473.496622] Stack: [10473.499853] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [10473.509154] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [10473.518451] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [10473.527749] Call Trace: [10473.531479] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [10473.540649] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [10473.549533] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [10473.558216] [] ? wake_up_state+0x20/0x20 [10473.565434] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [10473.574789] [] kthread+0xcf/0xe0 [10473.581223] [] ? kthread_create_on_node+0x140/0x140 [10473.589510] [] ret_from_fork+0x58/0x90 [10473.596526] [] ? kthread_create_on_node+0x140/0x140 [10473.604793] Code: 00 00 45 89 e9 4c 8d 85 67 ff ff ff 48 8b 4d a8 48 8d 55 88 48 8b b5 38 ff ff ff 48 8b bd 48 ff ff ff 48 8b 85 28 ff ff ff ff d0 <85> c0 41 89 c5 75 1c 8b 85 58 ff ff ff 41 3b 04 24 72 10 48 8b [10501.208292] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [10501.217946] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [10501.350183] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [10501.363509] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [10501.379846] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [10501.389266] RIP: 0010:[] [] osd_inode_iteration+0x282/0xcc0 [osd_ldiskfs] [10501.401519] RSP: 0018:ffff880703603d18 EFLAGS: 00000287 [10501.408487] RAX: ffff88042bd78800 RBX: 000000009f8f2001 RCX: ffff88042bd78800 [10501.417480] RDX: 00000000000072cf RSI: ffff880419dc3a00 RDI: ffff88042bd79800 [10501.426466] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [10501.435453] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000001 [10501.444419] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [10501.453372] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [10501.463412] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10501.470827] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [10501.479812] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [10501.488781] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [10501.497730] Stack: [10501.500953] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [10501.510232] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [10501.519537] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [10501.528836] Call Trace: [10501.532567] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [10501.541743] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [10501.550616] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [10501.559308] [] ? wake_up_state+0x20/0x20 [10501.566520] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [10501.575875] [] kthread+0xcf/0xe0 [10501.582303] [] ? kthread_create_on_node+0x140/0x140 [10501.590584] [] ret_from_fork+0x58/0x90 [10501.597601] [] ? kthread_create_on_node+0x140/0x140 [10501.605879] Code: 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 e8 77 5b f4 ff 42 8d 14 38 48 8b 45 88 48 8b 80 a8 03 00 00 48 3b 50 20 0f 83 57 01 00 00 <44> 0f b6 85 67 ff ff ff 48 8d 4d a8 48 8d 55 88 48 8b b5 38 ff [10529.209275] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [10529.218933] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [10529.351295] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [10529.364613] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [10529.380931] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [10529.390341] RIP: 0010:[] [] osd_scrub_exec+0xf7/0x5b0 [osd_ldiskfs] [10529.402024] RSP: 0018:ffff880703603c78 EFLAGS: 00000246 [10529.408991] RAX: 0000000000280001 RBX: ffff880703603c37 RCX: 0000000000000000 [10529.417992] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [10529.426979] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [10529.435950] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [10529.444909] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [10529.453871] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [10529.463905] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10529.471318] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [10529.480292] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [10529.489263] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [10529.498232] Stack: [10529.501476] 0000000000000004 0000000000000001 ffff880703603df0 ffff880703603d98 [10529.510775] ffff880703603d57 000000009f8f2001 0000000000000004 0000000000000001 [10529.520068] 0000000000000004 ffff880703603d98 ffff880703603d78 0000000000000000 [10529.529373] Call Trace: [10529.533130] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [10529.542301] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [10529.551473] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [10529.560350] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [10529.569031] [] ? wake_up_state+0x20/0x20 [10529.576246] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [10529.585622] [] kthread+0xcf/0xe0 [10529.592050] [] ? kthread_create_on_node+0x140/0x140 [10529.600349] [] ret_from_fork+0x58/0x90 [10529.607374] [] ? kthread_create_on_node+0x140/0x140 [10529.615657] Code: 8b 4d 88 48 85 db 74 23 f6 83 18 0a 00 00 08 74 1a 4d 85 e4 74 15 41 8b 87 6c 14 00 00 41 39 84 24 0c 0a 00 00 0f 82 d9 00 00 00 <31> c0 41 f6 87 74 14 00 00 04 0f 85 6e ff ff ff 41 83 f9 02 0f [10543.763775] INFO: rcu_sched self-detected stall on CPU { 24} (t=3120072 jiffies g=907112 c=907111 q=2916916) [10543.776187] Task dump for CPU 24: [10543.780929] OI_scrub R running task 0 5551 2 0x00000088 [10543.789862] ffff88083fde6dd0 00000000fb6db880 ffff88082da03db0 ffffffff810c47e8 [10543.799254] 0000000000000018 ffffffff81a1e7c0 ffff88082da03dc8 ffffffff810c8079 [10543.808560] 0000000000000009 ffff88082da03df8 ffffffff811378e0 ffff88082da101c0 [10543.817835] Call Trace: [10543.821516] [] sched_show_task+0xa8/0x110 [10543.829490] [] dump_cpu_task+0x39/0x70 [10543.836455] [] rcu_dump_cpu_stacks+0x90/0xd0 [10543.843986] [] rcu_check_callbacks+0x442/0x720 [10543.851675] [] ? tick_sched_handle.isra.13+0x60/0x60 [10543.859926] [] update_process_times+0x47/0x80 [10543.867474] [] tick_sched_handle.isra.13+0x25/0x60 [10543.875496] [] tick_sched_timer+0x41/0x70 [10543.882624] [] __hrtimer_run_queues+0xd2/0x260 [10543.890215] [] hrtimer_interrupt+0xb0/0x1e0 [10543.897494] [] ? call_softirq+0x1c/0x30 [10543.904379] [] local_apic_timer_interrupt+0x37/0x60 [10543.912405] [] smp_apic_timer_interrupt+0x3f/0x60 [10543.920235] [] apic_timer_interrupt+0x6d/0x80 [10543.927658] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [10543.937285] [] ? osd_scrub_exec+0x14/0x5b0 [osd_ldiskfs] [10543.945803] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [10543.954709] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [10543.963621] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [10543.972227] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [10543.980641] [] ? wake_up_state+0x20/0x20 [10543.987588] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [10543.996678] [] kthread+0xcf/0xe0 [10544.002856] [] ? kthread_create_on_node+0x140/0x140 [10544.010873] [] ret_from_fork+0x58/0x90 [10544.017619] [] ? kthread_create_on_node+0x140/0x140 [10569.210661] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [10569.220064] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [10569.350004] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [10569.363194] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [10569.379239] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [10569.388536] RIP: 0010:[] [] osd_scrub_exec+0x5/0x5b0 [osd_ldiskfs] [10569.399998] RSP: 0018:ffff880703603d10 EFLAGS: 00000293 [10569.406866] RAX: ffffffffa121b990 RBX: 000000009f8f2001 RCX: 0000000000000000 [10569.415762] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [10569.424657] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [10569.433552] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000001 [10569.442440] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [10569.451331] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [10569.461294] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10569.468638] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [10569.477556] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [10569.486461] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [10569.495366] Stack: [10569.498542] ffffffffa121d1e9 ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 [10569.507795] 00000000810d354f ffff8803162f6000 ffff8800b2a37468 0000000020000000 [10569.517048] 010000000000000c 0000000000000000 0000000000000000 ffff8800b2a36000 [10569.526293] Call Trace: [10569.529991] [] ? osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [10569.539325] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [10569.548465] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [10569.557314] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [10569.565967] [] ? wake_up_state+0x20/0x20 [10569.573148] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [10569.582473] [] kthread+0xcf/0xe0 [10569.588880] [] ? kthread_create_on_node+0x140/0x140 [10569.597130] [] ret_from_fork+0x58/0x90 [10569.604115] [] ? kthread_create_on_node+0x140/0x140 [10569.612383] Code: 05 85 a5 02 00 00 00 02 00 89 54 24 08 48 8b 40 40 ba 07 00 00 00 48 89 04 24 31 c0 e8 a5 03 98 ff e9 c7 fd ff ff 66 66 66 66 90 <55> 48 89 e5 41 57 49 89 f7 41 56 41 55 41 54 45 31 e4 53 48 83 [10597.211642] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [10597.221250] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [10597.353468] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [10597.366768] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [10597.383023] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [10597.392411] RIP: 0010:[] [] osd_scrub_exec+0x96/0x5b0 [osd_ldiskfs] [10597.404060] RSP: 0018:ffff880703603d10 EFLAGS: 00000292 [10597.411007] RAX: 0000000000000000 RBX: ffff880703603d57 RCX: 0000000000000000 [10597.419993] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [10597.428968] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [10597.437936] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [10597.446892] R13: ffffea00027e3c80 R14: 000000009f8f2001 R15: ffff880703603d08 [10597.455850] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [10597.465885] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10597.473301] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [10597.482292] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [10597.491259] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [10597.500220] Stack: [10597.503441] ffffffffa121d1e9 ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 [10597.512747] 00000000810d354f ffff8803162f6000 ffff8800b2a37468 0000000020000000 [10597.522051] 010000000000000c 0000000000000000 0000000000000000 ffff8800b2a36000 [10597.531358] Call Trace: [10597.535106] [] ? osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [10597.544472] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [10597.553648] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [10597.562522] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [10597.571207] [] ? wake_up_state+0x20/0x20 [10597.578426] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [10597.587794] [] kthread+0xcf/0xe0 [10597.594242] [] ? kthread_create_on_node+0x140/0x140 [10597.602554] [] ret_from_fork+0x58/0x90 [10597.609575] [] ? kthread_create_on_node+0x140/0x140 [10597.617857] Code: 00 00 41 80 a7 74 14 00 00 fe 48 8b 4d d0 65 48 33 0c 25 28 00 00 00 0f 85 b1 03 00 00 48 83 c4 68 5b 41 5c 41 5d 41 5e 41 5f 5d 66 0f 1f 84 00 00 00 00 00 4c 8d ae f0 10 00 00 44 89 4d 88 [10625.212618] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [10625.222284] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [10625.354758] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [10625.368065] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [10625.384337] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [10625.393743] RIP: 0010:[] [] osd_scrub_exec+0x23/0x5b0 [osd_ldiskfs] [10625.405403] RSP: 0018:ffff880703603c78 EFLAGS: 00000296 [10625.412355] RAX: ffffffffa121b990 RBX: ffff880703603c37 RCX: 0000000000000000 [10625.421344] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [10625.430321] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [10625.439285] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [10625.448237] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [10625.457198] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [10625.467238] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10625.474659] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [10625.483641] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [10625.492613] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [10625.501568] Stack: [10625.504780] ffff880703603d08 0000000000000001 ffffea00027e3c80 000000009f8f2001 [10625.514068] ffff880703603d57 ffff880703603d57 0000000000000000 0000000000000000 [10625.523373] ffff880703603d78 ffff8800b2a36000 ffff8803162f6000 ffffffffffffff10 [10625.532666] Call Trace: [10625.536400] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [10625.545564] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [10625.554722] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [10625.563590] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [10625.572258] [] ? wake_up_state+0x20/0x20 [10625.579468] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [10625.588830] [] kthread+0xcf/0xe0 [10625.595262] [] ? kthread_create_on_node+0x140/0x140 [10625.603547] [] ret_from_fork+0x58/0x90 [10625.610550] [] ? kthread_create_on_node+0x140/0x140 [10625.618827] Code: 03 98 ff e9 c7 fd ff ff 66 66 66 66 90 55 48 89 e5 41 57 49 89 f7 41 56 41 55 41 54 45 31 e4 53 48 83 ec 68 48 8b 9e 08 10 00 00 <4c> 89 45 90 65 48 8b 04 25 28 00 00 00 48 89 45 d0 31 c0 48 8d [10653.213594] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [10653.223244] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [10653.355075] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [10653.368364] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [10653.384630] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [10653.394008] RIP: 0010:[] [] osd_inode_iteration+0x490/0xcc0 [osd_ldiskfs] [10653.406239] RSP: 0018:ffff880703603d18 EFLAGS: 00000293 [10653.413185] RAX: 0000000000000004 RBX: 000000009f8f2001 RCX: 0000000000000000 [10653.422159] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [10653.431131] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [10653.440082] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [10653.449028] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [10653.457968] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [10653.467999] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10653.475410] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [10653.484380] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [10653.493338] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [10653.502289] Stack: [10653.505500] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [10653.514795] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [10653.524072] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [10653.533369] Call Trace: [10653.537080] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [10653.546244] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [10653.555114] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [10653.563780] [] ? wake_up_state+0x20/0x20 [10653.570986] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [10653.580341] [] kthread+0xcf/0xe0 [10653.586778] [] ? kthread_create_on_node+0x140/0x140 [10653.595070] [] ret_from_fork+0x58/0x90 [10653.602088] [] ? kthread_create_on_node+0x140/0x140 [10653.610374] Code: 0f fc ff ff 0f 1f 80 00 00 00 00 45 89 e9 4c 8d 85 67 ff ff ff 48 8b 4d a8 48 8d 55 88 48 8b b5 38 ff ff ff 48 8b bd 48 ff ff ff <48> 8b 85 28 ff ff ff ff d0 85 c0 41 89 c5 75 1c 8b 85 58 ff ff [10681.214565] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [10681.224245] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [10681.356243] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [10681.369552] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [10681.385849] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [10681.395254] RIP: 0010:[] [] osd_scrub_exec+0x1c/0x5b0 [osd_ldiskfs] [10681.406922] RSP: 0018:ffff880703603c78 EFLAGS: 00000296 [10681.413882] RAX: ffffffffa121b990 RBX: ffff880703603c37 RCX: 0000000000000000 [10681.422882] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [10681.431867] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [10681.440835] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [10681.449801] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [10681.458756] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [10681.468792] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10681.476203] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [10681.485182] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [10681.494150] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [10681.503108] Stack: [10681.506321] 0000000000000004 0000000000000001 ffff880703603df0 ffff880703603d98 [10681.515603] ffff880703603d57 000000009f8f2001 0000000000000004 ffff880703603d57 [10681.524890] ffff8800b2a37468 0000000000000000 ffff880703603d78 ffff8800b2a36000 [10681.534184] Call Trace: [10681.537921] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [10681.547087] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [10681.556242] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [10681.565115] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [10681.573795] [] ? wake_up_state+0x20/0x20 [10681.581003] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [10681.590354] [] kthread+0xcf/0xe0 [10681.596783] [] ? kthread_create_on_node+0x140/0x140 [10681.605084] [] ret_from_fork+0x58/0x90 [10681.612096] [] ? kthread_create_on_node+0x140/0x140 [10681.620361] Code: 89 04 24 31 c0 e8 a5 03 98 ff e9 c7 fd ff ff 66 66 66 66 90 55 48 89 e5 41 57 49 89 f7 41 56 41 55 41 54 45 31 e4 53 48 83 ec 68 <48> 8b 9e 08 10 00 00 4c 89 45 90 65 48 8b 04 25 28 00 00 00 48 [10709.215546] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [10709.225199] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [10709.357261] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [10709.370544] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [10709.386795] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [10709.396167] RIP: 0010:[] [] osd_scrub_exec+0xf7/0x5b0 [osd_ldiskfs] [10709.407812] RSP: 0018:ffff880703603c78 EFLAGS: 00000246 [10709.414747] RAX: 0000000000280001 RBX: ffff880703603c37 RCX: 0000000000000000 [10709.423727] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [10709.432695] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [10709.441654] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [10709.450613] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [10709.459577] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [10709.469616] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10709.477030] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [10709.486001] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [10709.494953] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [10709.503894] Stack: [10709.507098] 0000000000000004 0000000000000001 ffff880703603df0 ffff880703603d98 [10709.516388] ffff880703603d57 000000009f8f2001 0000000000000004 0000000000000001 [10709.525695] 0000000000000004 ffff880703603d98 ffff880703603d78 0000000000000000 [10709.534987] Call Trace: [10709.538705] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [10709.547878] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [10709.557040] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [10709.565911] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [10709.574597] [] ? wake_up_state+0x20/0x20 [10709.581815] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [10709.591169] [] kthread+0xcf/0xe0 [10709.597601] [] ? kthread_create_on_node+0x140/0x140 [10709.605882] [] ret_from_fork+0x58/0x90 [10709.612893] [] ? kthread_create_on_node+0x140/0x140 [10709.621171] Code: 8b 4d 88 48 85 db 74 23 f6 83 18 0a 00 00 08 74 1a 4d 85 e4 74 15 41 8b 87 6c 14 00 00 41 39 84 24 0c 0a 00 00 0f 82 d9 00 00 00 <31> c0 41 f6 87 74 14 00 00 04 0f 85 6e ff ff ff 41 83 f9 02 0f [10723.773051] INFO: rcu_sched self-detected stall on CPU { 24} (t=3300075 jiffies g=907112 c=907111 q=3476409) [10723.785411] Task dump for CPU 24: [10723.790137] OI_scrub R running task 0 5551 2 0x00000088 [10723.799055] ffff88083fde6dd0 00000000fb6db880 ffff88082da03db0 ffffffff810c47e8 [10723.808390] 0000000000000018 ffffffff81a1e7c0 ffff88082da03dc8 ffffffff810c8079 [10723.817691] 0000000000000009 ffff88082da03df8 ffffffff811378e0 ffff88082da101c0 [10723.826979] Call Trace: [10723.830668] [] sched_show_task+0xa8/0x110 [10723.838663] [] dump_cpu_task+0x39/0x70 [10723.845632] [] rcu_dump_cpu_stacks+0x90/0xd0 [10723.853157] [] rcu_check_callbacks+0x442/0x720 [10723.860860] [] ? tick_sched_handle.isra.13+0x60/0x60 [10723.869128] [] update_process_times+0x47/0x80 [10723.876685] [] tick_sched_handle.isra.13+0x25/0x60 [10723.884705] [] tick_sched_timer+0x41/0x70 [10723.891831] [] __hrtimer_run_queues+0xd2/0x260 [10723.899422] [] hrtimer_interrupt+0xb0/0x1e0 [10723.906695] [] ? call_softirq+0x1c/0x30 [10723.913575] [] local_apic_timer_interrupt+0x37/0x60 [10723.921600] [] smp_apic_timer_interrupt+0x3f/0x60 [10723.929411] [] apic_timer_interrupt+0x6d/0x80 [10723.936848] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [10723.946481] [] ? osd_scrub_exec+0x12/0x5b0 [osd_ldiskfs] [10723.954998] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [10723.963914] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [10723.972814] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [10723.981423] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [10723.989829] [] ? wake_up_state+0x20/0x20 [10723.996776] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [10724.005878] [] kthread+0xcf/0xe0 [10724.012054] [] ? kthread_create_on_node+0x140/0x140 [10724.020071] [] ret_from_fork+0x58/0x90 [10724.026822] [] ? kthread_create_on_node+0x140/0x140 [10749.216940] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [10749.226394] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [10749.356196] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [10749.369377] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [10749.385434] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [10749.394712] RIP: 0010:[] [] osd_scrub_exec+0x95/0x5b0 [osd_ldiskfs] [10749.406284] RSP: 0018:ffff880703603d08 EFLAGS: 00000292 [10749.413149] RAX: 0000000000000000 RBX: 000000009f8f2001 RCX: 0000000000000000 [10749.422045] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [10749.430956] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [10749.439859] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [10749.448748] R13: 0000000000000018 R14: ffff880703603c78 R15: 0000000000000296 [10749.457639] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [10749.467593] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10749.474927] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [10749.483825] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [10749.492727] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [10749.501621] Stack: [10749.504787] ffff880703603df0 ffffffffa121d1e9 ffffffffa121b990 ffffffffa1217a20 [10749.514020] ffff8800b2a36000 00000000810d354f ffff8803162f6000 ffff8800b2a37468 [10749.523264] 0000000020000000 010000000000000c 0000000000000000 0000000000000000 [10749.532513] Call Trace: [10749.536192] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [10749.545322] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [10749.554446] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [10749.563282] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [10749.571918] [] ? wake_up_state+0x20/0x20 [10749.579086] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [10749.588392] [] kthread+0xcf/0xe0 [10749.594795] [] ? kthread_create_on_node+0x140/0x140 [10749.603048] [] ret_from_fork+0x58/0x90 [10749.610027] [] ? kthread_create_on_node+0x140/0x140 [10749.618262] Code: 01 00 00 41 80 a7 74 14 00 00 fe 48 8b 4d d0 65 48 33 0c 25 28 00 00 00 0f 85 b1 03 00 00 48 83 c4 68 5b 41 5c 41 5d 41 5e 41 5f <5d> c3 66 0f 1f 84 00 00 00 00 00 4c 8d ae f0 10 00 00 44 89 4d [10777.217907] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [10777.227572] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [10777.359410] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [10777.372704] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [10777.388945] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [10777.398329] RIP: 0010:[] [] ldiskfs_itable_unused_count+0xd/0x30 [ldiskfs] [10777.410668] RSP: 0018:ffff880703603d08 EFLAGS: 00000286 [10777.417635] RAX: ffff8800b2a37468 RBX: ffff880703603d57 RCX: ffff88042bd78800 [10777.426607] RDX: ffff880703603d78 RSI: ffff880419dc3a00 RDI: ffff88042bd79800 [10777.435574] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [10777.444522] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [10777.453471] R13: ffffea00027e3c80 R14: ffff880703603c70 R15: ffff880703603d08 [10777.462417] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [10777.472428] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10777.479845] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [10777.488811] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [10777.497764] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [10777.506711] Stack: [10777.509920] ffff880703603df0 ffffffffa121cfb9 ffffffffa121b990 ffffffffa1217a20 [10777.519205] ffff8800b2a36000 00000000810d354f ffff8803162f6000 ffff8800b2a37468 [10777.528504] 0000000020000000 010000000000000c 0000000000000000 0000000000000000 [10777.537804] Call Trace: [10777.541540] [] ? osd_inode_iteration+0x269/0xcc0 [osd_ldiskfs] [10777.550906] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [10777.560072] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [10777.568935] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [10777.577598] [] ? wake_up_state+0x20/0x20 [10777.584811] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [10777.594175] [] kthread+0xcf/0xe0 [10777.600614] [] ? kthread_create_on_node+0x140/0x140 [10777.608893] [] ret_from_fork+0x58/0x90 [10777.615903] [] ? kthread_create_on_node+0x140/0x140 [10777.624174] Code: 10 48 89 e5 48 83 39 3f 76 07 0f b7 46 30 c1 e0 10 09 d0 5d c3 66 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 8b 8f a8 03 00 00 <31> c0 0f b7 56 1c 48 89 e5 48 83 39 3f 76 07 0f b7 46 32 c1 e0 [10805.218891] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [10805.228550] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [10805.360338] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [10805.373668] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [10805.389927] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [10805.399323] RIP: 0010:[] [] osd_inode_iteration+0x255/0xcc0 [osd_ldiskfs] [10805.411565] RSP: 0018:ffff880703603d18 EFLAGS: 00000286 [10805.418512] RAX: ffff8800b2a37468 RBX: 000000009f8f2001 RCX: 0000000000000000 [10805.427501] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [10805.436477] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [10805.445438] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [10805.454389] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [10805.463345] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [10805.473379] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10805.480799] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [10805.489771] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [10805.498722] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [10805.507669] Stack: [10805.510881] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [10805.520175] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [10805.529483] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [10805.538793] Call Trace: [10805.542523] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [10805.551691] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [10805.560567] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [10805.569240] [] ? wake_up_state+0x20/0x20 [10805.576449] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [10805.585795] [] kthread+0xcf/0xe0 [10805.592226] [] ? kthread_create_on_node+0x140/0x140 [10805.600512] [] ret_from_fork+0x58/0x90 [10805.607528] [] ? kthread_create_on_node+0x140/0x140 [10805.615802] Code: 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 00 8b 75 98 48 8b 7d 88 e8 dc 2d f5 ff 48 85 c0 48 89 45 90 0f 84 b3 08 00 00 44 8b 7d a0 <48> 8b 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 e8 77 5b f4 ff 42 [10825.856229] perf: interrupt took too long (4030 > 4028), lowering kernel.perf_event_max_sample_rate to 49000 [10853.220561] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [10853.230232] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [10853.362727] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [10853.376057] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [10853.392348] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [10853.401756] RIP: 0010:[] [] osd_scrub_exec+0xcf/0x5b0 [osd_ldiskfs] [10853.413442] RSP: 0018:ffff880703603c78 EFLAGS: 00000246 [10853.420415] RAX: ffff8803162f1008 RBX: ffff880703603c37 RCX: 0000000000000000 [10853.429419] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [10853.438410] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [10853.447385] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [10853.456352] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [10853.465325] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [10853.475369] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10853.482802] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [10853.491800] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [10853.500781] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [10853.509741] Stack: [10853.512982] ffffea00027e3c80 000000009f8f2001 ffff880703603df0 ffff880703603d57 [10853.522307] ffff880703603d57 000000009f8f2001 0000000000000004 ffff880703603d57 [10853.531631] ffff8800b2a37468 0000000000000000 ffff880703603d78 ffff8800b2a36000 [10853.540920] Call Trace: [10853.544665] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [10853.553856] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [10853.563041] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [10853.571925] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [10853.580631] [] ? wake_up_state+0x20/0x20 [10853.587856] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [10853.597234] [] kthread+0xcf/0xe0 [10853.603680] [] ? kthread_create_on_node+0x140/0x140 [10853.611992] [] ret_from_fork+0x58/0x90 [10853.619018] [] ? kthread_create_on_node+0x140/0x140 [10853.627313] Code: 10 00 00 44 89 4d 88 4c 89 ef e8 1d fb 46 e0 41 83 87 68 14 00 00 01 49 83 87 88 11 00 00 01 4c 89 ef e8 f5 9f e9 df 44 8b 4d 88 <48> 85 db 74 23 f6 83 18 0a 00 00 08 74 1a 4d 85 e4 74 15 41 8b [10881.221544] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [10881.231218] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [10881.363455] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [10881.376735] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [10881.392986] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [10881.402370] RIP: 0010:[] [] osd_inode_iteration+0x251/0xcc0 [osd_ldiskfs] [10881.414611] RSP: 0018:ffff880703603d18 EFLAGS: 00000286 [10881.421558] RAX: ffff8800b2a37468 RBX: ffff880703603d57 RCX: 0000000000000000 [10881.430540] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [10881.439504] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [10881.448455] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [10881.457403] R13: ffffea00027e3c80 R14: 000000009f8f2001 R15: 0000000000000018 [10881.466353] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [10881.476381] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10881.483785] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [10881.492755] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [10881.501710] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [10881.510660] Stack: [10881.513866] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [10881.523160] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [10881.532450] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [10881.541731] Call Trace: [10881.545460] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [10881.554624] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [10881.563492] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [10881.572172] [] ? wake_up_state+0x20/0x20 [10881.579383] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [10881.588735] [] kthread+0xcf/0xe0 [10881.595164] [] ? kthread_create_on_node+0x140/0x140 [10881.603455] [] ret_from_fork+0x58/0x90 [10881.610472] [] ? kthread_create_on_node+0x140/0x140 [10881.618744] Code: b0 00 00 00 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 00 8b 75 98 48 8b 7d 88 e8 dc 2d f5 ff 48 85 c0 48 89 45 90 0f 84 b3 08 00 00 <44> 8b 7d a0 48 8b 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 e8 77 [10903.782330] INFO: rcu_sched self-detected stall on CPU { 24} (t=3480078 jiffies g=907112 c=907111 q=4879055) [10903.794692] Task dump for CPU 24: [10903.799431] OI_scrub R running task 0 5551 2 0x00000088 [10903.808393] ffff88083fde6dd0 00000000fb6db880 ffff88082da03db0 ffffffff810c47e8 [10903.817711] 0000000000000018 ffffffff81a1e7c0 ffff88082da03dc8 ffffffff810c8079 [10903.827020] 0000000000000009 ffff88082da03df8 ffffffff811378e0 ffff88082da101c0 [10903.836305] Call Trace: [10903.840010] [] sched_show_task+0xa8/0x110 [10903.848010] [] dump_cpu_task+0x39/0x70 [10903.854999] [] rcu_dump_cpu_stacks+0x90/0xd0 [10903.862531] [] rcu_check_callbacks+0x442/0x720 [10903.870234] [] ? tick_sched_handle.isra.13+0x60/0x60 [10903.878500] [] update_process_times+0x47/0x80 [10903.886064] [] tick_sched_handle.isra.13+0x25/0x60 [10903.894086] [] tick_sched_timer+0x41/0x70 [10903.901218] [] __hrtimer_run_queues+0xd2/0x260 [10903.908818] [] hrtimer_interrupt+0xb0/0x1e0 [10903.916107] [] ? call_softirq+0x1c/0x30 [10903.922986] [] local_apic_timer_interrupt+0x37/0x60 [10903.931021] [] smp_apic_timer_interrupt+0x3f/0x60 [10903.938847] [] apic_timer_interrupt+0x6d/0x80 [10903.946269] [] ? osd_scrub_next+0x14d/0x3c0 [osd_ldiskfs] [10903.955599] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [10903.964498] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [10903.973395] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [10903.981991] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [10903.990398] [] ? wake_up_state+0x20/0x20 [10903.997344] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [10904.006425] [] kthread+0xcf/0xe0 [10904.012595] [] ? kthread_create_on_node+0x140/0x140 [10904.020605] [] ret_from_fork+0x58/0x90 [10904.027358] [] ? kthread_create_on_node+0x140/0x140 [10929.223216] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [10929.232609] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [10929.362863] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [10929.376050] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [10929.392125] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [10929.401422] RIP: 0010:[] [] osd_inode_iteration+0x255/0xcc0 [osd_ldiskfs] [10929.413570] RSP: 0018:ffff880703603d18 EFLAGS: 00000286 [10929.420434] RAX: ffff8800b2a37468 RBX: 000000009f8f2001 RCX: 0000000000000000 [10929.429342] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [10929.438237] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [10929.447136] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffff880703603d57 [10929.456040] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [10929.464941] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [10929.474920] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10929.482269] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [10929.491200] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [10929.500105] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [10929.509008] Stack: [10929.512173] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [10929.521409] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [10929.530643] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [10929.539892] Call Trace: [10929.543590] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [10929.552731] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [10929.561581] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [10929.570226] [] ? wake_up_state+0x20/0x20 [10929.577418] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [10929.586740] [] kthread+0xcf/0xe0 [10929.593140] [] ? kthread_create_on_node+0x140/0x140 [10929.601391] [] ret_from_fork+0x58/0x90 [10929.608377] [] ? kthread_create_on_node+0x140/0x140 [10929.616632] Code: 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 00 8b 75 98 48 8b 7d 88 e8 dc 2d f5 ff 48 85 c0 48 89 45 90 0f 84 b3 08 00 00 44 8b 7d a0 <48> 8b 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 e8 77 5b f4 ff 42 [10934.532800] Lustre: soaked-MDT0000: Client 91eacee7-b83c-19ec-9326-47c5ebda930d (at 192.168.1.116@o2ib100) reconnecting [10934.546060] Lustre: soaked-MDT0000: Connection restored to 91eacee7-b83c-19ec-9326-47c5ebda930d (at 192.168.1.116@o2ib100) [10957.224185] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [10957.233862] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [10957.365840] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [10957.379151] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [10957.395463] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [10957.404874] RIP: 0010:[] [] osd_scrub_next+0x2b/0x3c0 [osd_ldiskfs] [10957.416552] RSP: 0018:ffff880703603c70 EFLAGS: 00000282 [10957.423523] RAX: 00000000fb6db880 RBX: ffff880703603c37 RCX: ffff880703603d98 [10957.432530] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [10957.441524] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [10957.450512] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [10957.459502] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [10957.468494] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [10957.478569] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10957.486013] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [10957.495018] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [10957.504009] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [10957.512989] Stack: [10957.516248] 000000009f8f2001 ffff880703603d08 000000009f8f2001 ffffea00027e3c80 [10957.525586] 000000009f8f2001 ffff880703603d57 0000000000000001 ffffffffa1217a20 [10957.534903] ffff880703603d98 ffff880703603d78 ffff8800b2a36000 ffff8803162f6000 [10957.544224] Call Trace: [10957.547975] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [10957.556881] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [10957.566071] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [10957.575261] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [10957.584149] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [10957.592843] [] ? wake_up_state+0x20/0x20 [10957.600079] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [10957.609444] [] kthread+0xcf/0xe0 [10957.615903] [] ? kthread_create_on_node+0x140/0x140 [10957.624208] [] ret_from_fork+0x58/0x90 [10957.631249] [] ? kthread_create_on_node+0x140/0x140 [10957.639548] Code: 66 66 66 66 90 55 48 89 e5 41 57 49 89 f7 41 56 41 55 49 89 fd 41 54 49 89 d4 53 48 89 cb 48 83 ec 70 65 48 8b 04 25 28 00 00 00 <48> 89 45 d0 31 c0 48 8b 05 f8 59 9c ff 48 85 c0 75 7b 41 f6 87 [10985.225153] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [10985.234781] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [10985.366298] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [10985.379566] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [10985.395770] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [10985.405131] RIP: 0010:[] [] osd_scrub_exec+0xe9/0x5b0 [osd_ldiskfs] [10985.416754] RSP: 0018:ffff880703603c78 EFLAGS: 00000282 [10985.423678] RAX: 0000000000280001 RBX: ffff880703603c37 RCX: 0000000000000000 [10985.432634] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [10985.441581] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [10985.450522] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [10985.459464] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [10985.468410] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [10985.478442] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10985.485853] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [10985.494815] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [10985.503851] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [10985.512814] Stack: [10985.516083] 0000000000000004 0000000000000001 ffff880703603df0 ffff880703603d98 [10985.525432] ffff880703603d57 000000009f8f2001 0000000000000004 ffff880703603d57 [10985.534871] ffff8800b2a37468 0000000000000000 ffff880703603d78 ffff880419dc3a00 [10985.544158] Call Trace: [10985.548004] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [10985.557178] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [10985.566344] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [10985.575206] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [10985.583990] [] ? wake_up_state+0x20/0x20 [10985.591337] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [10985.600697] [] kthread+0xcf/0xe0 [10985.607437] [] ? kthread_create_on_node+0x140/0x140 [10985.615723] [] ret_from_fork+0x58/0x90 [10985.622741] [] ? kthread_create_on_node+0x140/0x140 [10985.631028] Code: 88 11 00 00 01 4c 89 ef e8 f5 9f e9 df 44 8b 4d 88 48 85 db 74 23 f6 83 18 0a 00 00 08 74 1a 4d 85 e4 74 15 41 8b 87 6c 14 00 00 <41> 39 84 24 0c 0a 00 00 0f 82 d9 00 00 00 31 c0 41 f6 87 74 14 [11013.226129] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [11013.235811] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [11013.367626] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [11013.380910] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [11013.397154] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [11013.406543] RIP: 0010:[] [] osd_scrub_exec+0xf7/0x5b0 [osd_ldiskfs] [11013.418203] RSP: 0018:ffff880703603c78 EFLAGS: 00000246 [11013.425139] RAX: 0000000000280001 RBX: ffff880703603c37 RCX: 0000000000000000 [11013.434104] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [11013.443065] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [11013.452021] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [11013.460968] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [11013.469916] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [11013.479941] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11013.487342] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [11013.496312] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [11013.505264] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [11013.514207] Stack: [11013.517411] ffff880703603d08 000000009f8f2001 ffff880703603df0 000000009f8f2001 [11013.526688] ffff880703603d57 000000009f8f2001 0000000000000004 ffff880703603d57 [11013.535988] ffff88042bd78800 ffff88042bd78800 00000000000072cf ffff880419dc3a00 [11013.545266] Call Trace: [11013.548974] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [11013.558133] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [11013.567286] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [11013.576139] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [11013.584801] [] ? wake_up_state+0x20/0x20 [11013.592008] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [11013.601349] [] kthread+0xcf/0xe0 [11013.607771] [] ? kthread_create_on_node+0x140/0x140 [11013.616041] [] ret_from_fork+0x58/0x90 [11013.623052] [] ? kthread_create_on_node+0x140/0x140 [11013.631315] Code: 8b 4d 88 48 85 db 74 23 f6 83 18 0a 00 00 08 74 1a 4d 85 e4 74 15 41 8b 87 6c 14 00 00 41 39 84 24 0c 0a 00 00 0f 82 d9 00 00 00 <31> c0 41 f6 87 74 14 00 00 04 0f 85 6e ff ff ff 41 83 f9 02 0f [11041.227110] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [11041.236799] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [11041.368518] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [11041.381807] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [11041.398071] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [11041.407788] RIP: 0010:[] [] osd_inode_iteration+0x49e/0xcc0 [osd_ldiskfs] [11041.420373] RSP: 0018:ffff880703603d18 EFLAGS: 00000246 [11041.427512] RAX: 0000000000000000 RBX: 000000009f8f2001 RCX: 0000000000000000 [11041.436592] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [11041.445756] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [11041.454914] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffff880703603d57 [11041.464162] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [11041.473191] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [11041.483471] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11041.490902] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [11041.499873] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [11041.508967] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [11041.518095] Stack: [11041.521368] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [11041.530688] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [11041.539976] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [11041.549268] Call Trace: [11041.552999] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [11041.562181] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [11041.571057] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [11041.579730] [] ? wake_up_state+0x20/0x20 [11041.586933] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [11041.596300] [] kthread+0xcf/0xe0 [11041.602744] [] ? kthread_create_on_node+0x140/0x140 [11041.611029] [] ret_from_fork+0x58/0x90 [11041.618047] [] ? kthread_create_on_node+0x140/0x140 [11041.626332] Code: 4c 8d 85 67 ff ff ff 48 8b 4d a8 48 8d 55 88 48 8b b5 38 ff ff ff 48 8b bd 48 ff ff ff 48 8b 85 28 ff ff ff ff d0 85 c0 41 89 c5 <75> 1c 8b 85 58 ff ff ff 41 3b 04 24 72 10 48 8b 85 50 ff ff ff [11069.228080] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [11069.237693] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [11069.369611] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [11069.382923] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [11069.399242] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [11069.408648] RIP: 0010:[] [] osd_inode_iteration+0x251/0xcc0 [osd_ldiskfs] [11069.420917] RSP: 0018:ffff880703603d18 EFLAGS: 00000286 [11069.427891] RAX: ffff8800b2a37468 RBX: 000000009f8f2001 RCX: 0000000000000000 [11069.436884] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [11069.445867] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [11069.454837] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [11069.463792] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [11069.472743] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [11069.482769] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11069.490187] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [11069.499157] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [11069.508128] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [11069.517086] Stack: [11069.520302] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [11069.529600] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [11069.538886] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [11069.548188] Call Trace: [11069.551908] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [11069.561078] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [11069.569957] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [11069.578642] [] ? wake_up_state+0x20/0x20 [11069.585854] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [11069.595208] [] kthread+0xcf/0xe0 [11069.601638] [] ? kthread_create_on_node+0x140/0x140 [11069.609915] [] ret_from_fork+0x58/0x90 [11069.616926] [] ? kthread_create_on_node+0x140/0x140 [11069.625192] Code: b0 00 00 00 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 00 8b 75 98 48 8b 7d 88 e8 dc 2d f5 ff 48 85 c0 48 89 45 90 0f 84 b3 08 00 00 <44> 8b 7d a0 48 8b 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 e8 77 [11083.791587] INFO: rcu_sched self-detected stall on CPU { 24} (t=3660081 jiffies g=907112 c=907111 q=5905472) [11083.803940] Task dump for CPU 24: [11083.808688] OI_scrub R running task 0 5551 2 0x00000088 [11083.817614] ffff88083fde6dd0 00000000fb6db880 ffff88082da03db0 ffffffff810c47e8 [11083.826933] 0000000000000018 ffffffff81a1e7c0 ffff88082da03dc8 ffffffff810c8079 [11083.836248] 0000000000000009 ffff88082da03df8 ffffffff811378e0 ffff88082da101c0 [11083.845559] Call Trace: [11083.849274] [] sched_show_task+0xa8/0x110 [11083.857279] [] dump_cpu_task+0x39/0x70 [11083.864271] [] rcu_dump_cpu_stacks+0x90/0xd0 [11083.871807] [] rcu_check_callbacks+0x442/0x720 [11083.879523] [] ? tick_sched_handle.isra.13+0x60/0x60 [11083.887800] [] update_process_times+0x47/0x80 [11083.895370] [] tick_sched_handle.isra.13+0x25/0x60 [11083.903400] [] tick_sched_timer+0x41/0x70 [11083.910541] [] __hrtimer_run_queues+0xd2/0x260 [11083.918142] [] hrtimer_interrupt+0xb0/0x1e0 [11083.925425] [] ? call_softirq+0x1c/0x30 [11083.932322] [] local_apic_timer_interrupt+0x37/0x60 [11083.940362] [] smp_apic_timer_interrupt+0x3f/0x60 [11083.948205] [] apic_timer_interrupt+0x6d/0x80 [11083.955629] [] ? osd_scrub_next+0x169/0x3c0 [osd_ldiskfs] [11083.964961] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [11083.973862] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [11083.982759] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [11083.991362] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [11083.999767] [] ? wake_up_state+0x20/0x20 [11084.006714] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [11084.015800] [] kthread+0xcf/0xe0 [11084.021960] [] ? kthread_create_on_node+0x140/0x140 [11084.029976] [] ret_from_fork+0x58/0x90 [11084.036734] [] ? kthread_create_on_node+0x140/0x140 [11109.229472] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [11109.238868] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [11109.368571] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [11109.381759] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [11109.397834] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [11109.407133] RIP: 0010:[] [] osd_inode_iteration+0x255/0xcc0 [osd_ldiskfs] [11109.419303] RSP: 0018:ffff880703603d18 EFLAGS: 00000286 [11109.426166] RAX: ffff8800b2a37468 RBX: ffff880703603d98 RCX: 0000000000000000 [11109.435079] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [11109.443987] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [11109.452895] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000001 [11109.461801] R13: 0000000000000004 R14: 000000009f8f2001 R15: 0000000000000018 [11109.470704] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [11109.480670] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11109.488008] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [11109.496935] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [11109.505838] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [11109.514725] Stack: [11109.517886] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [11109.527135] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [11109.536396] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [11109.545669] Call Trace: [11109.549351] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [11109.558476] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [11109.567332] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [11109.576096] [] ? wake_up_state+0x20/0x20 [11109.583306] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [11109.592618] [] kthread+0xcf/0xe0 [11109.599009] [] ? kthread_create_on_node+0x140/0x140 [11109.607248] [] ret_from_fork+0x58/0x90 [11109.614231] [] ? kthread_create_on_node+0x140/0x140 [11109.622468] Code: 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 00 8b 75 98 48 8b 7d 88 e8 dc 2d f5 ff 48 85 c0 48 89 45 90 0f 84 b3 08 00 00 44 8b 7d a0 <48> 8b 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 e8 77 5b f4 ff 42 [11129.530033] Lustre: soaked-MDT0000: Connection restored to 7ea3a484-c027-1559-8163-dc85944c882c (at 192.168.1.143@o2ib100) [11137.230449] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [11137.240115] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [11137.372046] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [11137.385354] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [11137.401670] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [11137.411073] RIP: 0010:[] [] osd_scrub_next+0x2f/0x3c0 [osd_ldiskfs] [11137.422755] RSP: 0018:ffff880703603c70 EFLAGS: 00000282 [11137.429712] RAX: 00000000fb6db880 RBX: ffff880703603c37 RCX: ffff880703603d98 [11137.438708] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [11137.447691] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [11137.456658] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [11137.465624] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [11137.474591] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [11137.484641] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11137.492075] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [11137.501073] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [11137.510061] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [11137.519039] Stack: [11137.522278] 000000009f8f2001 ffff880703603d08 ffff880703603d57 ffff880703603df0 [11137.531610] 000000009f8f2001 ffff880703603d57 000000009f8f2001 0000000000000004 [11137.540935] ffff880703603d57 0000000020000000 0000000000000000 ffff880703603d78 [11137.550268] Call Trace: [11137.554016] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [11137.563209] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [11137.572399] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [11137.581289] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [11137.589989] [] ? wake_up_state+0x20/0x20 [11137.597220] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [11137.606604] [] kthread+0xcf/0xe0 [11137.613058] [] ? kthread_create_on_node+0x140/0x140 [11137.621370] [] ret_from_fork+0x58/0x90 [11137.628408] [] ? kthread_create_on_node+0x140/0x140 [11137.636704] Code: 90 55 48 89 e5 41 57 49 89 f7 41 56 41 55 49 89 fd 41 54 49 89 d4 53 48 89 cb 48 83 ec 70 65 48 8b 04 25 28 00 00 00 48 89 45 d0 <31> c0 48 8b 05 f8 59 9c ff 48 85 c0 75 7b 41 f6 87 58 10 00 00 [11165.231430] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [11165.241114] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [11165.372816] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [11165.386202] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [11165.402458] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [11165.411842] RIP: 0010:[] [] osd_scrub_exec+0xf7/0x5b0 [osd_ldiskfs] [11165.423503] RSP: 0018:ffff880703603c78 EFLAGS: 00000246 [11165.430444] RAX: 0000000000280001 RBX: ffff880703603c37 RCX: 0000000000000000 [11165.439423] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [11165.448387] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [11165.457336] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [11165.466289] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [11165.475241] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [11165.485269] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11165.492678] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [11165.501641] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [11165.510594] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [11165.519535] Stack: [11165.522751] ffffea00027e3c80 000000009f8f2001 0000000000000004 0000000000000001 [11165.532045] ffff880703603d57 ffff880703603d98 ffff880703603d78 ffff8800b2a36000 [11165.541378] ffff8803162f6000 ffffffffffffff10 ffffffffa1217a32 0000000000000010 [11165.550678] Call Trace: [11165.554399] [] ? osd_scrub_next+0x12/0x3c0 [osd_ldiskfs] [11165.563182] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [11165.572349] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [11165.581508] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [11165.590369] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [11165.599050] [] ? wake_up_state+0x20/0x20 [11165.606259] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [11165.615617] [] kthread+0xcf/0xe0 [11165.622049] [] ? kthread_create_on_node+0x140/0x140 [11165.630335] [] ret_from_fork+0x58/0x90 [11165.637346] [] ? kthread_create_on_node+0x140/0x140 [11165.645620] Code: 8b 4d 88 48 85 db 74 23 f6 83 18 0a 00 00 08 74 1a 4d 85 e4 74 15 41 8b 87 6c 14 00 00 41 39 84 24 0c 0a 00 00 0f 82 d9 00 00 00 <31> c0 41 f6 87 74 14 00 00 04 0f 85 6e ff ff ff 41 83 f9 02 0f [11193.232400] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [11193.242056] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [11193.373579] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [11193.386865] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [11193.403092] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [11193.412466] RIP: 0010:[] [] osd_scrub_next+0x12/0x3c0 [osd_ldiskfs] [11193.424098] RSP: 0018:ffff880703603cf0 EFLAGS: 00000287 [11193.431040] RAX: ffffffffa1217a20 RBX: 000000009f8f2001 RCX: ffff880703603d98 [11193.440009] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [11193.448972] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [11193.457916] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffff880703603c70 [11193.466876] R13: ffff880703603d08 R14: 0000000000000010 R15: ffffffffa1217a4f [11193.475840] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [11193.485863] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11193.493274] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [11193.502236] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [11193.511190] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [11193.520134] Stack: [11193.523349] 0000000000000000 00000000ffffffff 0000000000000001 ffff880703603df0 [11193.532659] ffffffffa121cff9 ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 [11193.541970] 00000000810d354f ffff8803162f6000 ffff8800b2a37468 0000000020000000 [11193.551249] Call Trace: [11193.554971] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [11193.564144] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [11193.573320] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [11193.582190] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [11193.590865] [] ? wake_up_state+0x20/0x20 [11193.598076] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [11193.607434] [] kthread+0xcf/0xe0 [11193.613874] [] ? kthread_create_on_node+0x140/0x140 [11193.622159] [] ret_from_fork+0x58/0x90 [11193.629172] [] ? kthread_create_on_node+0x140/0x140 [11193.637455] Code: 25 fc ff ff 48 83 c4 08 5b 41 5c 41 5d 41 5e 5d c3 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 89 e5 41 57 49 89 f7 41 56 41 55 <49> 89 fd 41 54 49 89 d4 53 48 89 cb 48 83 ec 70 65 48 8b 04 25 [11221.233380] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [11221.243073] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [11221.375053] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [11221.388344] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [11221.404595] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [11221.413983] RIP: 0010:[] [] osd_inode_iteration+0x28a/0xcc0 [osd_ldiskfs] [11221.426245] RSP: 0018:ffff880703603d18 EFLAGS: 00000287 [11221.433199] RAX: ffff88042bd78800 RBX: 0000000000000000 RCX: ffff88042bd78800 [11221.442182] RDX: 00000000000072cf RSI: ffff880419dc3a00 RDI: ffff88042bd79800 [11221.451161] RBP: ffff880703603df0 R08: 0000000000000001 R09: 0000000000000004 [11221.460132] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffff880703603d57 [11221.469101] R13: 0000000000000004 R14: 000000009f8f2001 R15: ffffea00027e3c80 [11221.478061] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [11221.488103] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11221.495515] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [11221.504485] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [11221.513446] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [11221.522395] Stack: [11221.525609] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [11221.534909] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [11221.544213] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [11221.553503] Call Trace: [11221.557230] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [11221.566396] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [11221.575267] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [11221.583937] [] ? wake_up_state+0x20/0x20 [11221.591145] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [11221.600502] [] kthread+0xcf/0xe0 [11221.606934] [] ? kthread_create_on_node+0x140/0x140 [11221.615222] [] ret_from_fork+0x58/0x90 [11221.622237] [] ? kthread_create_on_node+0x140/0x140 [11221.630512] Code: a8 00 00 00 00 e8 77 5b f4 ff 42 8d 14 38 48 8b 45 88 48 8b 80 a8 03 00 00 48 3b 50 20 0f 83 57 01 00 00 44 0f b6 85 67 ff ff ff <48> 8d 4d a8 48 8d 55 88 48 8b b5 38 ff ff ff 48 8b bd 48 ff ff [11249.234349] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [11249.244012] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [11249.375975] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [11249.389300] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [11249.405600] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [11249.415012] RIP: 0010:[] [] osd_scrub_next+0x12/0x3c0 [osd_ldiskfs] [11249.426698] RSP: 0018:ffff880703603cf0 EFLAGS: 00000287 [11249.433659] RAX: ffffffffa1217a20 RBX: 000000009f8f2001 RCX: ffff880703603d98 [11249.442648] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [11249.451626] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [11249.460592] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffff880703603c78 [11249.469545] R13: 0000000000000246 R14: 0000000000000010 R15: ffffffffa121ba12 [11249.478502] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [11249.488534] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11249.495944] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [11249.504904] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [11249.513865] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [11249.522818] Stack: [11249.526027] 0000000000000000 00000000ffffffff 0000000000000001 ffff880703603df0 [11249.535324] ffffffffa121cff9 ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 [11249.544624] 00000000810d354f ffff8803162f6000 ffff8800b2a37468 0000000020000000 [11249.553934] Call Trace: [11249.557660] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [11249.566834] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [11249.575995] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [11249.584867] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [11249.593538] [] ? wake_up_state+0x20/0x20 [11249.600753] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [11249.610106] [] kthread+0xcf/0xe0 [11249.616536] [] ? kthread_create_on_node+0x140/0x140 [11249.624820] [] ret_from_fork+0x58/0x90 [11249.631837] [] ? kthread_create_on_node+0x140/0x140 [11249.640104] Code: 25 fc ff ff 48 83 c4 08 5b 41 5c 41 5d 41 5e 5d c3 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 89 e5 41 57 49 89 f7 41 56 41 55 <49> 89 fd 41 54 49 89 d4 53 48 89 cb 48 83 ec 70 65 48 8b 04 25 [11263.800871] INFO: rcu_sched self-detected stall on CPU { 24} (t=3840084 jiffies g=907112 c=907111 q=6877352) [11263.813258] Task dump for CPU 24: [11263.817996] OI_scrub R running task 0 5551 2 0x00000088 [11263.826928] ffff88083fde6dd0 00000000fb6db880 ffff88082da03db0 ffffffff810c47e8 [11263.836243] 0000000000000018 ffffffff81a1e7c0 ffff88082da03dc8 ffffffff810c8079 [11263.845556] 0000000000000009 ffff88082da03df8 ffffffff811378e0 ffff88082da101c0 [11263.854834] Call Trace: [11263.858516] [] sched_show_task+0xa8/0x110 [11263.866478] [] dump_cpu_task+0x39/0x70 [11263.873446] [] rcu_dump_cpu_stacks+0x90/0xd0 [11263.880972] [] rcu_check_callbacks+0x442/0x720 [11263.888671] [] ? tick_sched_handle.isra.13+0x60/0x60 [11263.896921] [] update_process_times+0x47/0x80 [11263.904467] [] tick_sched_handle.isra.13+0x25/0x60 [11263.912479] [] tick_sched_timer+0x41/0x70 [11263.919595] [] __hrtimer_run_queues+0xd2/0x260 [11263.927176] [] hrtimer_interrupt+0xb0/0x1e0 [11263.934464] [] ? call_softirq+0x1c/0x30 [11263.941346] [] local_apic_timer_interrupt+0x37/0x60 [11263.949379] [] smp_apic_timer_interrupt+0x3f/0x60 [11263.957205] [] apic_timer_interrupt+0x6d/0x80 [11263.964626] [] ? ldiskfs_itable_unused_count+0x23/0x30 [ldiskfs] [11263.974656] [] osd_inode_iteration+0x269/0xcc0 [osd_ldiskfs] [11263.983563] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [11263.992465] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [11264.001077] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [11264.009484] [] ? wake_up_state+0x20/0x20 [11264.016434] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [11264.025520] [] kthread+0xcf/0xe0 [11264.031697] [] ? kthread_create_on_node+0x140/0x140 [11264.039726] [] ret_from_fork+0x58/0x90 [11264.046483] [] ? kthread_create_on_node+0x140/0x140 [11289.235743] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [11289.245166] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [11289.374875] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [11289.388049] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [11289.404115] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [11289.413415] RIP: 0010:[] [] osd_scrub_next+0x9/0x3c0 [osd_ldiskfs] [11289.424885] RSP: 0018:ffff880703603d08 EFLAGS: 00000287 [11289.431739] RAX: ffffffffa1217a20 RBX: 000000009f8f2001 RCX: ffff880703603d98 [11289.440645] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [11289.449537] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [11289.458438] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [11289.467339] R13: 0000000000000018 R14: ffff880703603c70 R15: 0000000000000282 [11289.476224] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [11289.486179] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11289.493519] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [11289.502418] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [11289.511319] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [11289.520205] Stack: [11289.523361] ffff880703603df0 ffffffffa121cff9 ffffffffa121b990 ffffffffa1217a20 [11289.532571] ffff8800b2a36000 00000000810d354f ffff8803162f6000 ffff8800b2a37468 [11289.541814] 0000000020000000 010000000000000c 0000000000000000 0000000000000000 [11289.551072] Call Trace: [11289.554757] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [11289.563883] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [11289.573005] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [11289.581837] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [11289.590472] [] ? wake_up_state+0x20/0x20 [11289.597644] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [11289.606960] [] kthread+0xcf/0xe0 [11289.613348] [] ? kthread_create_on_node+0x140/0x140 [11289.621586] [] ret_from_fork+0x58/0x90 [11289.628564] [] ? kthread_create_on_node+0x140/0x140 [11289.636796] Code: 48 8d 4c 02 10 48 01 c2 e8 25 fc ff ff 48 83 c4 08 5b 41 5c 41 5d 41 5e 5d c3 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 89 e5 <41> 57 49 89 f7 41 56 41 55 49 89 fd 41 54 49 89 d4 53 48 89 cb [11317.236725] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [11317.246345] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [11317.378088] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [11317.391396] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [11317.407642] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [11317.417018] RIP: 0010:[] [] osd_inode_iteration+0x473/0xcc0 [osd_ldiskfs] [11317.429271] RSP: 0018:ffff880703603d18 EFLAGS: 00000293 [11317.436206] RAX: 0000000000000004 RBX: ffff880703603d98 RCX: ffff880703603d98 [11317.445187] RDX: ffff880703603d78 RSI: 0000000000000000 RDI: ffff8803162f6000 [11317.454153] RBP: ffff880703603df0 R08: 0000000000000001 R09: 0000000000000004 [11317.463108] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000001 [11317.472055] R13: 0000000000000004 R14: 000000009f8f2001 R15: 0000000000000018 [11317.480995] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [11317.491014] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11317.498406] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [11317.507368] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [11317.516321] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [11317.525259] Stack: [11317.528464] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [11317.537744] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [11317.547032] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [11317.556299] Call Trace: [11317.560014] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [11317.569176] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [11317.578044] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [11317.586712] [] ? wake_up_state+0x20/0x20 [11317.593908] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [11317.603253] [] kthread+0xcf/0xe0 [11317.609691] [] ? kthread_create_on_node+0x140/0x140 [11317.617973] [] ret_from_fork+0x58/0x90 [11317.624976] [] ? kthread_create_on_node+0x140/0x140 [11317.633240] Code: 00 00 48 c7 05 9b 86 02 00 00 00 00 00 c7 05 89 86 02 00 01 00 00 00 e8 7c eb 97 ff e9 0f fc ff ff 0f 1f 80 00 00 00 00 45 89 e9 <4c> 8d 85 67 ff ff ff 48 8b 4d a8 48 8d 55 88 48 8b b5 38 ff ff [11332.823838] perf: interrupt took too long (5039 > 5037), lowering kernel.perf_event_max_sample_rate to 39000 [11357.238111] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [11357.247813] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [11357.379703] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [11357.393006] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [11357.409288] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [11357.418687] RIP: 0010:[] [] osd_scrub_exec+0x30/0x5b0 [osd_ldiskfs] [11357.430343] RSP: 0018:ffff880703603c78 EFLAGS: 00000296 [11357.437288] RAX: 00000000fb6db880 RBX: ffff880703603c37 RCX: 0000000000000000 [11357.446279] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [11357.455256] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [11357.464219] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [11357.473186] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [11357.482146] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [11357.492188] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11357.499614] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [11357.508605] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [11357.517582] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [11357.526554] Stack: [11357.529785] 0000000000000004 0000000000000001 ffff880703603df0 ffff880703603d98 [11357.539108] ffff880703603d57 000000009f8f2001 0000000000000004 ffff880703603d57 [11357.548426] ffff88042bd78800 ffff88042bd78800 00000000000072cf ffff880419dc3a00 [11357.557744] Call Trace: [11357.561479] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [11357.570670] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [11357.579862] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [11357.588743] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [11357.597424] [] ? wake_up_state+0x20/0x20 [11357.604655] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [11357.614019] [] kthread+0xcf/0xe0 [11357.620465] [] ? kthread_create_on_node+0x140/0x140 [11357.628762] [] ret_from_fork+0x58/0x90 [11357.635792] [] ? kthread_create_on_node+0x140/0x140 [11357.644092] Code: 55 48 89 e5 41 57 49 89 f7 41 56 41 55 41 54 45 31 e4 53 48 83 ec 68 48 8b 9e 08 10 00 00 4c 89 45 90 65 48 8b 04 25 28 00 00 00 <48> 89 45 d0 31 c0 48 8d 43 08 48 85 db 4c 0f 45 e0 41 83 f9 04 [11385.239093] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [11385.248775] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [11385.380726] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [11385.394016] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [11385.410258] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [11385.419643] RIP: 0010:[] [] osd_inode_iteration+0x264/0xcc0 [osd_ldiskfs] [11385.431887] RSP: 0018:ffff880703603d18 EFLAGS: 00000286 [11385.438830] RAX: ffff8800b2a37468 RBX: 000000009f8f2001 RCX: 0000000000000000 [11385.447809] RDX: ffff880703603d78 RSI: ffff880419dc3a00 RDI: ffff88042bd79800 [11385.456779] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [11385.465737] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [11385.474682] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [11385.483628] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [11385.493658] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11385.501067] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [11385.510034] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [11385.518985] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [11385.527923] Stack: [11385.531137] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [11385.540432] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [11385.549724] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [11385.559020] Call Trace: [11385.562738] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [11385.571898] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [11385.580776] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [11385.589443] [] ? wake_up_state+0x20/0x20 [11385.596655] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [11385.606008] [] kthread+0xcf/0xe0 [11385.612438] [] ? kthread_create_on_node+0x140/0x140 [11385.620730] [] ret_from_fork+0x58/0x90 [11385.627749] [] ? kthread_create_on_node+0x140/0x140 [11385.636027] Code: 75 98 48 8b 7d 88 e8 dc 2d f5 ff 48 85 c0 48 89 45 90 0f 84 b3 08 00 00 44 8b 7d a0 48 8b 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 77 5b f4 ff 42 8d 14 38 48 8b 45 88 48 8b 80 a8 03 00 00 48 [11412.545528] Lustre: soaked-MDT0000: Client 91eacee7-b83c-19ec-9326-47c5ebda930d (at 192.168.1.116@o2ib100) reconnecting [11412.558814] Lustre: Skipped 1 previous similar message [11412.565640] Lustre: soaked-MDT0000: Connection restored to 91eacee7-b83c-19ec-9326-47c5ebda930d (at 192.168.1.116@o2ib100) [11413.240063] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [11413.249725] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [11413.381777] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [11413.395083] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [11413.411375] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [11413.420779] RIP: 0010:[] [] osd_inode_iteration+0x489/0xcc0 [osd_ldiskfs] [11413.433038] RSP: 0018:ffff880703603d18 EFLAGS: 00000293 [11413.440021] RAX: 0000000000000004 RBX: 000000009f8f2001 RCX: 0000000000000000 [11413.449040] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [11413.458052] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [11413.467053] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [11413.476056] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [11413.485061] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [11413.495150] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11413.502607] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [11413.511623] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [11413.520626] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [11413.529628] Stack: [11413.532900] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [11413.542247] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [11413.551589] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [11413.560931] Call Trace: [11413.564699] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [11413.573912] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [11413.582835] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [11413.591556] [] ? wake_up_state+0x20/0x20 [11413.598818] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [11413.608224] [] kthread+0xcf/0xe0 [11413.614712] [] ? kthread_create_on_node+0x140/0x140 [11413.623037] [] ret_from_fork+0x58/0x90 [11413.630102] [] ? kthread_create_on_node+0x140/0x140 [11413.638444] Code: 00 e8 7c eb 97 ff e9 0f fc ff ff 0f 1f 80 00 00 00 00 45 89 e9 4c 8d 85 67 ff ff ff 48 8b 4d a8 48 8d 55 88 48 8b b5 38 ff ff ff <48> 8b bd 48 ff ff ff 48 8b 85 28 ff ff ff ff d0 85 c0 41 89 c5 [11441.241045] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [11441.250738] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [11441.382767] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [11441.396077] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [11441.412387] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [11441.421789] RIP: 0010:[] [] osd_scrub_next+0x2b/0x3c0 [osd_ldiskfs] [11441.433457] RSP: 0018:ffff880703603c70 EFLAGS: 00000282 [11441.440425] RAX: 00000000fb6db880 RBX: ffff880703603c37 RCX: ffff880703603d98 [11441.449417] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [11441.458394] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [11441.467362] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [11441.476323] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [11441.485278] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [11441.495294] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11441.502700] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [11441.511666] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [11441.520631] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [11441.529586] Stack: [11441.532791] 000000009f8f2001 ffff880703603d08 0000000000000001 ffff880703603df0 [11441.542082] 000000009f8f2001 ffff880703603d57 000000009f8f2001 0000000000000004 [11441.551374] ffff880703603d57 ffff8800b2a37468 0000000000000000 ffff880703603d78 [11441.560663] Call Trace: [11441.564387] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [11441.573556] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [11441.582716] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [11441.591583] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [11441.600247] [] ? wake_up_state+0x20/0x20 [11441.607458] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [11441.616799] [] kthread+0xcf/0xe0 [11441.623232] [] ? kthread_create_on_node+0x140/0x140 [11441.631503] [] ret_from_fork+0x58/0x90 [11441.638507] [] ? kthread_create_on_node+0x140/0x140 [11441.646780] Code: 66 66 66 66 90 55 48 89 e5 41 57 49 89 f7 41 56 41 55 49 89 fd 41 54 49 89 d4 53 48 89 cb 48 83 ec 70 65 48 8b 04 25 28 00 00 00 <48> 89 45 d0 31 c0 48 8b 05 f8 59 9c ff 48 85 c0 75 7b 41 f6 87 [11443.810127] INFO: rcu_sched self-detected stall on CPU { 24} (t=4020087 jiffies g=907112 c=907111 q=7512209) [11443.822505] Task dump for CPU 24: [11443.827223] OI_scrub R running task 0 5551 2 0x00000088 [11443.836145] ffff88083fde6dd0 00000000fb6db880 ffff88082da03db0 ffffffff810c47e8 [11443.845475] 0000000000000018 ffffffff81a1e7c0 ffff88082da03dc8 ffffffff810c8079 [11443.854763] 0000000000000009 ffff88082da03df8 ffffffff811378e0 ffff88082da101c0 [11443.864038] Call Trace: [11443.867713] [] sched_show_task+0xa8/0x110 [11443.875685] [] dump_cpu_task+0x39/0x70 [11443.882643] [] rcu_dump_cpu_stacks+0x90/0xd0 [11443.890157] [] rcu_check_callbacks+0x442/0x720 [11443.897838] [] ? tick_sched_handle.isra.13+0x60/0x60 [11443.906084] [] update_process_times+0x47/0x80 [11443.913626] [] tick_sched_handle.isra.13+0x25/0x60 [11443.921641] [] tick_sched_timer+0x41/0x70 [11443.928766] [] __hrtimer_run_queues+0xd2/0x260 [11443.936345] [] hrtimer_interrupt+0xb0/0x1e0 [11443.943613] [] ? call_softirq+0x1c/0x30 [11443.950482] [] local_apic_timer_interrupt+0x37/0x60 [11443.958503] [] smp_apic_timer_interrupt+0x3f/0x60 [11443.966325] [] apic_timer_interrupt+0x6d/0x80 [11443.973744] [] ? osd_inode_iteration+0x271/0xcc0 [osd_ldiskfs] [11443.983561] [] ? osd_inode_iteration+0x269/0xcc0 [osd_ldiskfs] [11443.992658] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [11444.001558] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [11444.010166] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [11444.018568] [] ? wake_up_state+0x20/0x20 [11444.025510] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [11444.034601] [] kthread+0xcf/0xe0 [11444.040765] [] ? kthread_create_on_node+0x140/0x140 [11444.048797] [] ret_from_fork+0x58/0x90 [11444.055544] [] ? kthread_create_on_node+0x140/0x140 [11469.242020] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [11469.251457] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [11469.381350] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [11469.394537] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [11469.410632] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [11469.419936] RIP: 0010:[] [] osd_scrub_next+0xb/0x3c0 [osd_ldiskfs] [11469.431398] RSP: 0018:ffff880703603d00 EFLAGS: 00000287 [11469.438257] RAX: ffffffffa1217a20 RBX: 000000009f8f2001 RCX: ffff880703603d98 [11469.447162] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [11469.456076] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [11469.464972] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [11469.473872] R13: ffff880703603d08 R14: ffff880703603c70 R15: 0000000000000282 [11469.482761] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [11469.492727] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11469.500074] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [11469.508985] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [11469.517879] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [11469.526781] Stack: [11469.529960] 0000000000000001 ffff880703603df0 ffffffffa121cff9 ffffffffa121b990 [11469.539228] ffffffffa1217a20 ffff8800b2a36000 00000000810d354f ffff8803162f6000 [11469.548478] ffff8800b2a37468 0000000020000000 010000000000000c 0000000000000000 [11469.557742] Call Trace: [11469.561434] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [11469.570571] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [11469.579705] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [11469.588545] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [11469.597198] [] ? wake_up_state+0x20/0x20 [11469.604385] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [11469.613705] [] kthread+0xcf/0xe0 [11469.620099] [] ? kthread_create_on_node+0x140/0x140 [11469.628350] [] ret_from_fork+0x58/0x90 [11469.635325] [] ? kthread_create_on_node+0x140/0x140 [11469.643563] Code: 4c 02 10 48 01 c2 e8 25 fc ff ff 48 83 c4 08 5b 41 5c 41 5d 41 5e 5d c3 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 89 e5 41 57 <49> 89 f7 41 56 41 55 49 89 fd 41 54 49 89 d4 53 48 89 cb 48 83 [11497.242988] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [11497.252635] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [11497.384482] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [11497.397767] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [11497.414014] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [11497.423395] RIP: 0010:[] [] osd_scrub_exec+0xf7/0x5b0 [osd_ldiskfs] [11497.435049] RSP: 0018:ffff880703603c78 EFLAGS: 00000246 [11497.441984] RAX: 0000000000280001 RBX: ffff880703603c37 RCX: 0000000000000000 [11497.450958] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [11497.459919] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [11497.468867] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [11497.477814] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [11497.486762] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [11497.496788] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11497.504198] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [11497.513164] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [11497.522119] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [11497.531057] Stack: [11497.534266] 0000000000000004 0000000000000001 ffff880703603df0 ffff880703603d98 [11497.543569] ffff880703603d57 000000009f8f2001 0000000000000004 0000000000000001 [11497.552859] ffff88042bd78800 ffff88042bd78800 00000000000072cf ffff880419dc3a00 [11497.562149] Call Trace: [11497.565869] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [11497.575043] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [11497.584217] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [11497.593069] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [11497.601746] [] ? wake_up_state+0x20/0x20 [11497.608962] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [11497.618314] [] kthread+0xcf/0xe0 [11497.624757] [] ? kthread_create_on_node+0x140/0x140 [11497.633029] [] ret_from_fork+0x58/0x90 [11497.640045] [] ? kthread_create_on_node+0x140/0x140 [11497.648334] Code: 8b 4d 88 48 85 db 74 23 f6 83 18 0a 00 00 08 74 1a 4d 85 e4 74 15 41 8b 87 6c 14 00 00 41 39 84 24 0c 0a 00 00 0f 82 d9 00 00 00 <31> c0 41 f6 87 74 14 00 00 04 0f 85 6e ff ff ff 41 83 f9 02 0f [11525.243964] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [11525.253644] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [11525.385519] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [11525.398806] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [11525.415079] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [11525.424468] RIP: 0010:[] [] osd_inode_iteration+0x26d/0xcc0 [osd_ldiskfs] [11525.436708] RSP: 0018:ffff880703603d18 EFLAGS: 00000202 [11525.443654] RAX: 00000000000072ce RBX: 000000009f8f2001 RCX: ffff88042bd78800 [11525.452633] RDX: 00000000000072cf RSI: ffff880419dc3a00 RDI: ffff88042bd79800 [11525.461601] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [11525.470561] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000001 [11525.479509] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [11525.488455] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [11525.498485] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11525.505891] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [11525.514865] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [11525.523814] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [11525.532750] Stack: [11525.535957] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [11525.545248] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [11525.554543] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [11525.563842] Call Trace: [11525.567565] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [11525.576721] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [11525.585593] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [11525.594255] [] ? wake_up_state+0x20/0x20 [11525.601463] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [11525.610811] [] kthread+0xcf/0xe0 [11525.617246] [] ? kthread_create_on_node+0x140/0x140 [11525.625516] [] ret_from_fork+0x58/0x90 [11525.632522] [] ? kthread_create_on_node+0x140/0x140 [11525.640795] Code: f5 ff 48 85 c0 48 89 45 90 0f 84 b3 08 00 00 44 8b 7d a0 48 8b 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 e8 77 5b f4 ff 42 8d 14 38 <48> 8b 45 88 48 8b 80 a8 03 00 00 48 3b 50 20 0f 83 57 01 00 00 [11553.244946] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [11553.254612] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [11553.386621] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [11553.399926] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [11553.416220] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [11553.425624] RIP: 0010:[] [] osd_inode_iteration+0x264/0xcc0 [osd_ldiskfs] [11553.437863] RSP: 0018:ffff880703603d18 EFLAGS: 00000286 [11553.444826] RAX: ffff8800b2a37468 RBX: 000000009f8f2001 RCX: 0000000000000000 [11553.453811] RDX: ffff880703603d78 RSI: ffff880419dc3a00 RDI: ffff88042bd79800 [11553.462788] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [11553.471747] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffff880703603d57 [11553.480704] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [11553.489659] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [11553.499693] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11553.507106] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [11553.516075] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [11553.525051] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [11553.534010] Stack: [11553.537229] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [11553.546523] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [11553.555813] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [11553.565116] Call Trace: [11553.568860] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [11553.578038] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [11553.586914] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [11553.595608] [] ? wake_up_state+0x20/0x20 [11553.602834] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [11553.612185] [] kthread+0xcf/0xe0 [11553.618626] [] ? kthread_create_on_node+0x140/0x140 [11553.626917] [] ret_from_fork+0x58/0x90 [11553.633941] [] ? kthread_create_on_node+0x140/0x140 [11553.642216] Code: 75 98 48 8b 7d 88 e8 dc 2d f5 ff 48 85 c0 48 89 45 90 0f 84 b3 08 00 00 44 8b 7d a0 48 8b 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 77 5b f4 ff 42 8d 14 38 48 8b 45 88 48 8b 80 a8 03 00 00 48 [11581.245926] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [11581.255597] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [11581.388183] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [11581.401508] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [11581.417832] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [11581.427254] RIP: 0010:[] [] osd_inode_iteration+0x2be/0xcc0 [osd_ldiskfs] [11581.439521] RSP: 0018:ffff880703603d18 EFLAGS: 00000297 [11581.446496] RAX: 0000000000000004 RBX: ffff880703603d98 RCX: ffff880703603d98 [11581.455496] RDX: ffff880703603d78 RSI: 0000000000000000 RDI: ffff8803162f6000 [11581.464495] RBP: ffff880703603df0 R08: 0000000000000001 R09: 0000000000000004 [11581.473469] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000001 [11581.482432] R13: 0000000000000004 R14: 000000009f8f2001 R15: ffffea00027e3c80 [11581.491388] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [11581.501411] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11581.508822] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [11581.517798] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [11581.526762] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [11581.535719] Stack: [11581.538934] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [11581.548237] ffff8803162f6000 ffff8800b2a37468[11581.551605] Lustre: soaked-MDT0000: Connection restored to 91eacee7-b83c-19ec-9326-47c5ebda930d (at 192.168.1.116@o2ib100) 0000000020000000 010000000000000c [11581.571917] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [11581.581232] Call Trace: [11581.584988] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [11581.594185] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [11581.603084] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [11581.611784] [] ? wake_up_state+0x20/0x20 [11581.619025] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [11581.628403] [] kthread+0xcf/0xe0 [11581.634866] [] ? kthread_create_on_node+0x140/0x140 [11581.643171] [] ret_from_fork+0x58/0x90 [11581.650210] [] ? kthread_create_on_node+0x140/0x140 [11581.658511] Code: 8b b5 38 ff ff ff 48 8b bd 48 ff ff ff 48 8b 85 30 ff ff ff ff d0 83 f8 03 41 89 c5 0f 84 4b 02 00 00 0f 8e 15 01 00 00 83 f8 05 <66> 90 0f 84 8a 02 00 00 83 f8 06 0f 85 a1 01 00 00 48 8b 7d 90 [11609.246890] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [11609.256565] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [11609.388541] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [11609.401846] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [11609.418139] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [11609.427539] RIP: 0010:[] [] osd_scrub_next+0x14d/0x3c0 [osd_ldiskfs] [11609.439303] RSP: 0018:ffff880703603c70 EFLAGS: 00000202 [11609.446267] RAX: 0000000000000004 RBX: ffff880703603c37 RCX: ffff880703603d98 [11609.455263] RDX: ffff880703603d78 RSI: 00000000fb6db880 RDI: ffff8803162f6000 [11609.464250] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [11609.473243] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [11609.482223] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [11609.491212] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [11609.501275] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11609.508717] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [11609.517729] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [11609.526721] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [11609.535699] Stack: [11609.538949] 000000009f8f2001 ffff880703603d08 0000000000000001 ffff880703603df0 [11609.548297] 000000009f8f2001 ffff880703603d57 000000009f8f2001 0000000000000004 [11609.557629] ffff880703603d57 ffff88042bd78800 ffff88042bd78800 00000000000072cf [11609.566939] Call Trace: [11609.570685] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [11609.579853] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [11609.589016] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [11609.597879] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [11609.606544] [] ? wake_up_state+0x20/0x20 [11609.613745] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [11609.623086] [] kthread+0xcf/0xe0 [11609.629515] [] ? kthread_create_on_node+0x140/0x140 [11609.637791] [] ret_from_fork+0x58/0x90 [11609.644802] [] ? kthread_create_on_node+0x140/0x140 [11609.653063] Code: 49 8d 9f 10 11 00 00 48 89 df e8 df 6a 47 e0 41 c7 87 58 10 00 00 02 00 00 00 48 89 df e8 fc 69 47 e0 b8 05 00 00 00 48 8b 75 d0 <65> 48 33 34 25 28 00 00 00 0f 85 4c 02 00 00 48 83 c4 70 5b 41 [11623.819412] INFO: rcu_sched self-detected stall on CPU { 24} (t=4200090 jiffies g=907112 c=907111 q=8013065) [11623.831774] Task dump for CPU 24: [11623.836517] OI_scrub R running task 0 5551 2 0x00000088 [11623.845430] ffff88083fde6dd0 00000000fb6db880 ffff88082da03db0 ffffffff810c47e8 [11623.854745] 0000000000000018 ffffffff81a1e7c0 ffff88082da03dc8 ffffffff810c8079 [11623.864041] 0000000000000009 ffff88082da03df8 ffffffff811378e0 ffff88082da101c0 [11623.873308] Call Trace: [11623.876982] [] sched_show_task+0xa8/0x110 [11623.884951] [] dump_cpu_task+0x39/0x70 [11623.891914] [] rcu_dump_cpu_stacks+0x90/0xd0 [11623.899437] [] rcu_check_callbacks+0x442/0x720 [11623.907131] [] ? tick_sched_handle.isra.13+0x60/0x60 [11623.915390] [] update_process_times+0x47/0x80 [11623.922940] [] tick_sched_handle.isra.13+0x25/0x60 [11623.930953] [] tick_sched_timer+0x41/0x70 [11623.938091] [] __hrtimer_run_queues+0xd2/0x260 [11623.945675] [] hrtimer_interrupt+0xb0/0x1e0 [11623.952957] [] ? call_softirq+0x1c/0x30 [11623.959832] [] local_apic_timer_interrupt+0x37/0x60 [11623.967857] [] smp_apic_timer_interrupt+0x3f/0x60 [11623.975670] [] apic_timer_interrupt+0x6d/0x80 [11623.983087] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [11623.992705] [] ? osd_scrub_exec+0x14/0x5b0 [osd_ldiskfs] [11624.001223] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [11624.010119] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [11624.019023] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [11624.027627] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [11624.036028] [] ? wake_up_state+0x20/0x20 [11624.042968] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [11624.052074] [] kthread+0xcf/0xe0 [11624.058252] [] ? kthread_create_on_node+0x140/0x140 [11624.066278] [] ret_from_fork+0x58/0x90 [11624.073032] [] ? kthread_create_on_node+0x140/0x140 [11649.248288] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [11649.257684] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [11649.387709] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [11649.400884] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [11649.416927] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [11649.426214] RIP: 0010:[] [] osd_scrub_next+0x1b/0x3c0 [osd_ldiskfs] [11649.437776] RSP: 0018:ffff880703603ce0 EFLAGS: 00000287 [11649.444639] RAX: ffffffffa1217a20 RBX: ffff880703603c70 RCX: ffff880703603d98 [11649.453542] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [11649.462445] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [11649.471338] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000010 [11649.480237] R13: ffffffffa1217b6d R14: ffffffffffffff10 R15: ffff8803162f6000 [11649.489129] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [11649.499096] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11649.506445] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [11649.515371] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [11649.524283] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [11649.533175] Stack: [11649.536354] ffff880419dc3a00 ffff8800b2a3746c 0000000000000000 00000000ffffffff [11649.545619] 0000000000000001 ffff880703603df0 ffffffffa121cff9 ffffffffa121b990 [11649.554883] ffffffffa1217a20 ffff8800b2a36000 00000000810d354f ffff8803162f6000 [11649.564119] Call Trace: [11649.567819] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [11649.576965] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [11649.586100] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [11649.594948] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [11649.603594] [] ? wake_up_state+0x20/0x20 [11649.610769] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [11649.620093] [] kthread+0xcf/0xe0 [11649.626498] [] ? kthread_create_on_node+0x140/0x140 [11649.634757] [] ret_from_fork+0x58/0x90 [11649.641744] [] ? kthread_create_on_node+0x140/0x140 [11649.649994] Code: 41 5c 41 5d 41 5e 5d c3 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 89 e5 41 57 49 89 f7 41 56 41 55 49 89 fd 41 54 49 89 d4 53 <48> 89 cb 48 83 ec 70 65 48 8b 04 25 28 00 00 00 48 89 45 d0 31 [11677.249258] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [11677.258874] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [11677.390612] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [11677.403910] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [11677.420178] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [11677.429562] RIP: 0010:[] [] osd_inode_iteration+0x489/0xcc0 [osd_ldiskfs] [11677.441813] RSP: 0018:ffff880703603d18 EFLAGS: 00000293 [11677.448762] RAX: 0000000000000004 RBX: 000000009f8f2001 RCX: 0000000000000000 [11677.457747] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [11677.466722] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [11677.475685] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [11677.484643] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [11677.493603] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [11677.503647] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11677.511063] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [11677.520037] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [11677.529004] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [11677.537957] Stack: [11677.541167] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [11677.550460] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [11677.559762] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [11677.569039] Call Trace: [11677.572761] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [11677.581931] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [11677.590807] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [11677.599483] [] ? wake_up_state+0x20/0x20 [11677.606694] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [11677.616060] [] kthread+0xcf/0xe0 [11677.622497] [] ? kthread_create_on_node+0x140/0x140 [11677.630794] [] ret_from_fork+0x58/0x90 [11677.637812] [] ? kthread_create_on_node+0x140/0x140 [11677.646095] Code: 00 e8 7c eb 97 ff e9 0f fc ff ff 0f 1f 80 00 00 00 00 45 89 e9 4c 8d 85 67 ff ff ff 48 8b 4d a8 48 8d 55 88 48 8b b5 38 ff ff ff <48> 8b bd 48 ff ff ff 48 8b 85 28 ff ff ff ff d0 85 c0 41 89 c5 [11705.250235] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [11705.259925] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [11705.391818] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [11705.405125] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [11705.421421] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [11705.430816] RIP: 0010:[] [] osd_inode_iteration+0x2ac/0xcc0 [osd_ldiskfs] [11705.443069] RSP: 0018:ffff880703603d18 EFLAGS: 00000202 [11705.450031] RAX: 0000000000000004 RBX: 000000009f8f2001 RCX: ffff880703603d98 [11705.459011] RDX: ffff880703603d78 RSI: 0000000000000000 RDI: ffff8803162f6000 [11705.467991] RBP: ffff880703603df0 R08: 0000000000000001 R09: 0000000000000004 [11705.476960] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000001 [11705.485914] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [11705.494857] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [11705.504876] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11705.512279] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [11705.521237] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [11705.530194] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [11705.539143] Stack: [11705.542351] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [11705.551644] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [11705.560929] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [11705.570232] Call Trace: [11705.573949] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [11705.583110] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [11705.591976] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [11705.600647] [] ? wake_up_state+0x20/0x20 [11705.607855] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [11705.617221] [] kthread+0xcf/0xe0 [11705.623644] [] ? kthread_create_on_node+0x140/0x140 [11705.631923] [] ret_from_fork+0x58/0x90 [11705.638938] [] ? kthread_create_on_node+0x140/0x140 [11705.647212] Code: 00 44 0f b6 85 67 ff ff ff 48 8d 4d a8 48 8d 55 88 48 8b b5 38 ff ff ff 48 8b bd 48 ff ff ff 48 8b 85 30 ff ff ff ff d0 83 f8 03 <41> 89 c5 0f 84 4b 02 00 00 0f 8e 15 01 00 00 83 f8 05 66 90 0f [11733.251217] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [11733.260874] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [11733.392776] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [11733.406078] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [11733.422367] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [11733.431773] RIP: 0010:[] [] osd_inode_iteration+0x28a/0xcc0 [osd_ldiskfs] [11733.444028] RSP: 0018:ffff880703603d18 EFLAGS: 00000287 [11733.450986] RAX: ffff88042bd78800 RBX: 000000009f8f2001 RCX: ffff88042bd78800 [11733.459970] RDX: 00000000000072cf RSI: ffff880419dc3a00 RDI: ffff88042bd79800 [11733.468946] RBP: ffff880703603df0 R08: 0000000000000001 R09: 0000000000000004 [11733.477916] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [11733.486868] R13: ffff880703603df0 R14: 000000009f8f2001 R15: 0000000000000018 [11733.495827] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [11733.505855] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11733.513283] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [11733.522257] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [11733.531230] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [11733.540186] Stack: [11733.543407] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [11733.552703] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [11733.562008] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [11733.571322] Call Trace: [11733.575061] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [11733.584235] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [11733.593116] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [11733.601803] [] ? wake_up_state+0x20/0x20 [11733.609030] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [11733.618393] [] kthread+0xcf/0xe0 [11733.624830] [] ? kthread_create_on_node+0x140/0x140 [11733.633130] [] ret_from_fork+0x58/0x90 [11733.640140] [] ? kthread_create_on_node+0x140/0x140 [11733.648417] Code: a8 00 00 00 00 e8 77 5b f4 ff 42 8d 14 38 48 8b 45 88 48 8b 80 a8 03 00 00 48 3b 50 20 0f 83 57 01 00 00 44 0f b6 85 67 ff ff ff <48> 8d 4d a8 48 8d 55 88 48 8b b5 38 ff ff ff 48 8b bd 48 ff ff [11761.252185] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [11761.261851] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [11761.393804] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [11761.407118] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [11761.423439] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [11761.432847] RIP: 0010:[] [] osd_scrub_exec+0xd4/0x5b0 [osd_ldiskfs] [11761.444517] RSP: 0018:ffff880703603c78 EFLAGS: 00000286 [11761.451489] RAX: ffff8803162f1008 RBX: ffff880703603c37 RCX: 0000000000000000 [11761.460481] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [11761.469464] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [11761.478430] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [11761.487386] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [11761.496337] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [11761.506374] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11761.513779] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [11761.522754] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [11761.531717] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [11761.540663] Stack: [11761.543883] 0000000000000004 0000000000000001 0000000000000004 ffff880703603d98 [11761.553170] ffff880703603d57 0000000000000000 ffff8803162f6000 ffffffffffffff10 [11761.562455] ffffffffa1217b80 0000000000000010 0000000000000282 ffff880703603ce0 [11761.571750] Call Trace: [11761.575477] [] ? osd_scrub_next+0x160/0x3c0 [osd_ldiskfs] [11761.584344] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [11761.593508] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [11761.602667] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [11761.611529] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [11761.620192] [] ? wake_up_state+0x20/0x20 [11761.627395] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [11761.636738] [] kthread+0xcf/0xe0 [11761.643171] [] ? kthread_create_on_node+0x140/0x140 [11761.651450] [] ret_from_fork+0x58/0x90 [11761.658456] [] ? kthread_create_on_node+0x140/0x140 [11761.666721] Code: 4d 88 4c 89 ef e8 1d fb 46 e0 41 83 87 68 14 00 00 01 49 83 87 88 11 00 00 01 4c 89 ef e8 f5 9f e9 df 44 8b 4d 88 48 85 db 74 23 83 18 0a 00 00 08 74 1a 4d 85 e4 74 15 41 8b 87 6c 14 00 00 [11789.253161] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [11789.262830] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [11789.394327] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [11789.407600] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [11789.423807] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [11789.433180] RIP: 0010:[] [] ldiskfs_itable_unused_count+0x23/0x30 [ldiskfs] [11789.445602] RSP: 0018:ffff880703603d08 EFLAGS: 00000297 [11789.452535] RAX: 0000000000000000 RBX: 000000009f8f2001 RCX: ffff88042bd78800 [11789.461501] RDX: 00000000000072ce RSI: ffff880419dc3a00 RDI: ffff88042bd79800 [11789.470453] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [11789.479403] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [11789.488352] R13: ffff880703603d08 R14: ffff880703603c70 R15: 0000000000000282 [11789.497304] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [11789.507333] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11789.514745] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [11789.523700] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [11789.532649] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [11789.541586] Stack: [11789.544788] ffff880703603df0 ffffffffa121cfb9 ffffffffa121b990 ffffffffa1217a20 [11789.554083] ffff8800b2a36000 00000000810d354f ffff8803162f6000 ffff8800b2a37468 [11789.563376] 0000000020000000 010000000000000c 0000000000000000 0000000000000000 [11789.572667] Call Trace: [11789.576394] [] osd_inode_iteration+0x269/0xcc0 [osd_ldiskfs] [11789.585550] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [11789.594704] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [11789.603562] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [11789.612221] [] ? wake_up_state+0x20/0x20 [11789.619429] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [11789.628775] [] kthread+0xcf/0xe0 [11789.635219] [] ? kthread_create_on_node+0x140/0x140 [11789.643500] [] ret_from_fork+0x58/0x90 [11789.650509] [] ? kthread_create_on_node+0x140/0x140 [11789.658783] Code: 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 8b 8f a8 03 00 00 31 c0 0f b7 56 1c 48 89 e5 48 83 39 3f 76 07 0f b7 46 32 c1 e0 10 <09> d0 5d c3 66 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 89 16 [11803.828669] INFO: rcu_sched self-detected stall on CPU { 24} (t=4380093 jiffies g=907112 c=907111 q=8604937) [11803.841047] Task dump for CPU 24: [11803.845787] OI_scrub R running task 0 5551 2 0x00000088 [11803.854725] ffff88083fde6dd0 00000000fb6db880 ffff88082da03db0 ffffffff810c47e8 [11803.864043] 0000000000000018 ffffffff81a1e7c0 ffff88082da03dc8 ffffffff810c8079 [11803.873358] 0000000000000009 ffff88082da03df8 ffffffff811378e0 ffff88082da101c0 [11803.882648] Call Trace: [11803.886331] [] sched_show_task+0xa8/0x110 [11803.894318] [] dump_cpu_task+0x39/0x70 [11803.901280] [] rcu_dump_cpu_stacks+0x90/0xd0 [11803.908793] [] rcu_check_callbacks+0x442/0x720 [11803.916479] [] ? tick_sched_handle.isra.13+0x60/0x60 [11803.924731] [] update_process_times+0x47/0x80 [11803.932270] [] tick_sched_handle.isra.13+0x25/0x60 [11803.940290] [] tick_sched_timer+0x41/0x70 [11803.947407] [] __hrtimer_run_queues+0xd2/0x260 [11803.954993] [] hrtimer_interrupt+0xb0/0x1e0 [11803.962266] [] ? call_softirq+0x1c/0x30 [11803.969139] [] local_apic_timer_interrupt+0x37/0x60 [11803.977162] [] smp_apic_timer_interrupt+0x3f/0x60 [11803.984992] [] apic_timer_interrupt+0x6d/0x80 [11803.992415] [] ? osd_inode_iteration+0x2be/0xcc0 [osd_ldiskfs] [11804.002231] [] ? osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [11804.011338] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [11804.020243] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [11804.028857] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [11804.037264] [] ? wake_up_state+0x20/0x20 [11804.044214] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [11804.053300] [] kthread+0xcf/0xe0 [11804.059469] [] ? kthread_create_on_node+0x140/0x140 [11804.067490] [] ret_from_fork+0x58/0x90 [11804.074244] [] ? kthread_create_on_node+0x140/0x140 [11829.254554] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [11829.263999] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [11829.393719] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [11829.406897] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [11829.422975] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [11829.432271] RIP: 0010:[] [] osd_scrub_next+0x14d/0x3c0 [osd_ldiskfs] [11829.443935] RSP: 0018:ffff880703603c70 EFLAGS: 00000202 [11829.450794] RAX: 0000000000000004 RBX: ffff880703603c37 RCX: ffff880703603d98 [11829.459690] RDX: ffff880703603d78 RSI: 00000000fb6db880 RDI: ffff8803162f6000 [11829.468590] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [11829.477487] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [11829.486386] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [11829.495281] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [11829.505240] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11829.512586] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [11829.521493] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [11829.530402] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [11829.539301] Stack: [11829.542472] 000000009f8f2001 ffff880703603d08 0000000000000001 ffff880703603df0 [11829.551729] 000000009f8f2001 ffff880703603d57 000000009f8f2001 0000000000000004 [11829.560971] ffff880703603d57 0000000000000000 0000000000000000 ffff880703603d78 [11829.570219] Call Trace: [11829.573910] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [11829.583051] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [11829.592188] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [11829.601026] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [11829.609671] [] ? wake_up_state+0x20/0x20 [11829.616866] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [11829.626179] [] kthread+0xcf/0xe0 [11829.632575] [] ? kthread_create_on_node+0x140/0x140 [11829.640818] [] ret_from_fork+0x58/0x90 [11829.647794] [] ? kthread_create_on_node+0x140/0x140 [11829.656032] Code: 49 8d 9f 10 11 00 00 48 89 df e8 df 6a 47 e0 41 c7 87 58 10 00 00 02 00 00 00 48 89 df e8 fc 69 47 e0 b8 05 00 00 00 48 8b 75 d0 <65> 48 33 34 25 28 00 00 00 0f 85 4c 02 00 00 48 83 c4 70 5b 41 [11857.255529] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [11857.265210] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [11857.396889] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [11857.410172] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [11857.426417] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [11857.435796] RIP: 0010:[] [] osd_inode_iteration+0x28a/0xcc0 [osd_ldiskfs] [11857.448035] RSP: 0018:ffff880703603d18 EFLAGS: 00000287 [11857.454984] RAX: ffff88042bd78800 RBX: 0000000000000001 RCX: ffff88042bd78800 [11857.463968] RDX: 00000000000072cf RSI: ffff880419dc3a00 RDI: ffff88042bd79800 [11857.472931] RBP: ffff880703603df0 R08: 0000000000000001 R09: 0000000000000004 [11857.481886] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [11857.490837] R13: ffffea00027e3c80 R14: 000000009f8f2001 R15: ffff880703603d08 [11857.499795] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [11857.509827] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11857.517238] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [11857.526207] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [11857.535166] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [11857.544112] Stack: [11857.547315] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [11857.556612] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [11857.565914] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [11857.575203] Call Trace: [11857.578923] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [11857.588103] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [11857.596985] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [11857.605668] [] ? wake_up_state+0x20/0x20 [11857.612883] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [11857.622240] [] kthread+0xcf/0xe0 [11857.628678] [] ? kthread_create_on_node+0x140/0x140 [11857.636970] [] ret_from_fork+0x58/0x90 [11857.643990] [] ? kthread_create_on_node+0x140/0x140 [11857.652268] Code: a8 00 00 00 00 e8 77 5b f4 ff 42 8d 14 38 48 8b 45 88 48 8b 80 a8 03 00 00 48 3b 50 20 0f 83 57 01 00 00 44 0f b6 85 67 ff ff ff <48> 8d 4d a8 48 8d 55 88 48 8b b5 38 ff ff ff 48 8b bd 48 ff ff [11885.256505] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [11885.266132] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [11885.398122] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [11885.411444] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [11885.427748] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [11885.437159] RIP: 0010:[] [] osd_scrub_next+0x149/0x3c0 [osd_ldiskfs] [11885.448931] RSP: 0018:ffff880703603c70 EFLAGS: 00000202 [11885.455898] RAX: 0000000000000004 RBX: ffff880703603c37 RCX: ffff880703603d98 [11885.464897] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [11885.473887] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [11885.482855] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [11885.491824] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [11885.500780] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [11885.510815] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11885.518227] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [11885.527196] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [11885.536161] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [11885.545110] Stack: [11885.548325] 000000009f8f2001 0000000000000004 0000000000000001 0000000000000004 [11885.557607] ffff880703603d98 ffff880703603d57 0000000000000000 ffff8803162f6000 [11885.566902] ffffffffffffff10 ffffffffa1217b80 0000000000000010 0000000000000282 [11885.576198] Call Trace: [11885.579930] [] ? osd_scrub_next+0x160/0x3c0 [osd_ldiskfs] [11885.588811] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [11885.597978] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [11885.607142] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [11885.616016] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [11885.624681] [] ? wake_up_state+0x20/0x20 [11885.631889] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [11885.641245] [] kthread+0xcf/0xe0 [11885.647674] [] ? kthread_create_on_node+0x140/0x140 [11885.655956] [] ret_from_fork+0x58/0x90 [11885.662963] [] ? kthread_create_on_node+0x140/0x140 [11885.671223] Code: 45 90 74 49 49 8d 9f 10 11 00 00 48 89 df e8 df 6a 47 e0 41 c7 87 58 10 00 00 02 00 00 00 48 89 df e8 fc 69 47 e0 b8 05 00 00 00 <48> 8b 75 d0 65 48 33 34 25 28 00 00 00 0f 85 4c 02 00 00 48 83 [11911.485625] Lustre: soaked-MDT0000: Connection restored to 91eacee7-b83c-19ec-9326-47c5ebda930d (at 192.168.1.116@o2ib100) [11913.257481] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [11913.267187] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [11913.398846] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [11913.412131] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [11913.428365] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [11913.437745] RIP: 0010:[] [] ldiskfs_itable_unused_count+0x6/0x30 [ldiskfs] [11913.450082] RSP: 0018:ffff880703603d08 EFLAGS: 00000286 [11913.457032] RAX: ffff8800b2a37468 RBX: 000000009f8f2001 RCX: 0000000000000000 [11913.466018] RDX: ffff880703603d78 RSI: ffff880419dc3a00 RDI: ffff88042bd79800 [11913.475000] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [11913.483970] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [11913.492951] R13: 0000000000000018 R14: ffff880703603c78 R15: 0000000000000246 [11913.501931] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [11913.511971] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11913.519381] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [11913.528353] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [11913.537321] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [11913.546285] Stack: [11913.549537] ffff880703603df0 ffffffffa121cfb9 ffffffffa121b990 ffffffffa1217a20 [11913.558831] ffff8800b2a36000 00000000810d354f ffff8803162f6000 ffff8800b2a37468 [11913.568130] 0000000020000000 010000000000000c 0000000000000000 0000000000000000 [11913.577427] Call Trace: [11913.581179] [] ? osd_inode_iteration+0x269/0xcc0 [osd_ldiskfs] [11913.590554] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [11913.599731] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [11913.608621] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [11913.617315] [] ? wake_up_state+0x20/0x20 [11913.624559] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [11913.633932] [] kthread+0xcf/0xe0 [11913.640386] [] ? kthread_create_on_node+0x140/0x140 [11913.648696] [] ret_from_fork+0x58/0x90 [11913.655727] [] ? kthread_create_on_node+0x140/0x140 [11913.664022] Code: 00 00 31 c0 0f b7 56 10 48 89 e5 48 83 39 3f 76 07 0f b7 46 30 c1 e0 10 09 d0 5d c3 66 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 <48> 8b 8f a8 03 00 00 31 c0 0f b7 56 1c 48 89 e5 48 83 39 3f 76 [11941.258456] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [11941.268116] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [11941.399876] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [11941.413170] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [11941.429433] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [11941.438816] RIP: 0010:[] [] osd_inode_iteration+0x473/0xcc0 [osd_ldiskfs] [11941.451057] RSP: 0018:ffff880703603d18 EFLAGS: 00000293 [11941.458005] RAX: 0000000000000004 RBX: 000000009f8f2001 RCX: ffff880703603d98 [11941.466986] RDX: ffff880703603d78 RSI: 0000000000000000 RDI: ffff8803162f6000 [11941.475948] RBP: ffff880703603df0 R08: 0000000000000001 R09: 0000000000000004 [11941.484895] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000001 [11941.493843] R13: ffff880703603df0 R14: 000000009f8f2001 R15: ffffea00027e3c80 [11941.502793] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [11941.512824] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11941.520219] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [11941.529192] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [11941.538151] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [11941.547114] Stack: [11941.550317] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [11941.559612] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [11941.568917] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [11941.578217] Call Trace: [11941.581928] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [11941.591095] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [11941.599968] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [11941.608646] [] ? wake_up_state+0x20/0x20 [11941.615860] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [11941.625212] [] kthread+0xcf/0xe0 [11941.631642] [] ? kthread_create_on_node+0x140/0x140 [11941.639920] [] ret_from_fork+0x58/0x90 [11941.646931] [] ? kthread_create_on_node+0x140/0x140 [11941.655208] Code: 00 00 48 c7 05 9b 86 02 00 00 00 00 00 c7 05 89 86 02 00 01 00 00 00 e8 7c eb 97 ff e9 0f fc ff ff 0f 1f 80 00 00 00 00 45 89 e9 <4c> 8d 85 67 ff ff ff 48 8b 4d a8 48 8d 55 88 48 8b b5 38 ff ff [11969.259438] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [11969.269078] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [11969.401075] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [11969.414380] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [11969.430682] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [11969.440091] RIP: 0010:[] [] osd_inode_iteration+0x473/0xcc0 [osd_ldiskfs] [11969.452346] RSP: 0018:ffff880703603d18 EFLAGS: 00000293 [11969.459314] RAX: 0000000000000004 RBX: 000000009f8f2001 RCX: ffff880703603d98 [11969.468307] RDX: ffff880703603d78 RSI: 0000000000000000 RDI: ffff8803162f6000 [11969.477291] RBP: ffff880703603df0 R08: 0000000000000001 R09: 0000000000000004 [11969.486249] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffff880703603d57 [11969.495200] R13: ffff880703603df0 R14: 000000009f8f2001 R15: 0000000000000018 [11969.504149] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [11969.514174] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11969.521590] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [11969.530567] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [11969.539534] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [11969.548483] Stack: [11969.551701] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [11969.560994] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [11969.570278] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [11969.579595] Call Trace: [11969.583326] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [11969.592500] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [11969.601373] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [11969.610054] [] ? wake_up_state+0x20/0x20 [11969.617281] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [11969.626639] [] kthread+0xcf/0xe0 [11969.633070] [] ? kthread_create_on_node+0x140/0x140 [11969.641359] [] ret_from_fork+0x58/0x90 [11969.648379] [] ? kthread_create_on_node+0x140/0x140 [11969.656654] Code: 00 00 48 c7 05 9b 86 02 00 00 00 00 00 c7 05 89 86 02 00 01 00 00 00 e8 7c eb 97 ff e9 0f fc ff ff 0f 1f 80 00 00 00 00 45 89 e9 <4c> 8d 85 67 ff ff ff 48 8b 4d a8 48 8d 55 88 48 8b b5 38 ff ff [11983.837939] INFO: rcu_sched self-detected stall on CPU { 24} (t=4560096 jiffies g=907112 c=907111 q=9683906) [11983.850317] Task dump for CPU 24: [11983.855078] OI_scrub R running task 0 5551 2 0x00000088 [11983.864020] ffff88083fde6dd0 00000000fb6db880 ffff88082da03db0 ffffffff810c47e8 [11983.873340] 0000000000000018 ffffffff81a1e7c0 ffff88082da03dc8 ffffffff810c8079 [11983.882678] 0000000000000009 ffff88082da03df8 ffffffff811378e0 ffff88082da101c0 [11983.891984] Call Trace: [11983.895688] [] sched_show_task+0xa8/0x110 [11983.903681] [] dump_cpu_task+0x39/0x70 [11983.910657] [] rcu_dump_cpu_stacks+0x90/0xd0 [11983.918202] [] rcu_check_callbacks+0x442/0x720 [11983.925908] [] ? tick_sched_handle.isra.13+0x60/0x60 [11983.934185] [] update_process_times+0x47/0x80 [11983.941761] [] tick_sched_handle.isra.13+0x25/0x60 [11983.949792] [] tick_sched_timer+0x41/0x70 [11983.956927] [] __hrtimer_run_queues+0xd2/0x260 [11983.964536] [] hrtimer_interrupt+0xb0/0x1e0 [11983.971830] [] ? call_softirq+0x1c/0x30 [11983.978726] [] local_apic_timer_interrupt+0x37/0x60 [11983.986780] [] smp_apic_timer_interrupt+0x3f/0x60 [11983.994629] [] apic_timer_interrupt+0x6d/0x80 [11984.002065] [] ? osd_inode_iteration+0x26d/0xcc0 [osd_ldiskfs] [11984.011886] [] ? osd_inode_iteration+0x269/0xcc0 [osd_ldiskfs] [11984.020984] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [11984.029893] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [11984.038509] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [11984.046923] [] ? wake_up_state+0x20/0x20 [11984.053883] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [11984.062973] [] kthread+0xcf/0xe0 [11984.069149] [] ? kthread_create_on_node+0x140/0x140 [11984.077176] [] ret_from_fork+0x58/0x90 [11984.083943] [] ? kthread_create_on_node+0x140/0x140 [12009.260833] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [12009.270264] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [12009.400034] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [12009.413214] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [12009.429289] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [12009.438585] RIP: 0010:[] [] osd_inode_iteration+0x255/0xcc0 [osd_ldiskfs] [12009.450733] RSP: 0018:ffff880703603d18 EFLAGS: 00000286 [12009.457590] RAX: ffff8800b2a37468 RBX: 000000009f8f2001 RCX: 0000000000000000 [12009.466498] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [12009.475408] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [12009.484306] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000001 [12009.493206] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [12009.502103] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [12009.512067] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12009.519409] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [12009.528311] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12009.537215] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [12009.546110] Stack: [12009.549274] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [12009.558498] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [12009.567749] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [12009.577023] Call Trace: [12009.580718] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [12009.589863] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [12009.598709] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [12009.607365] [] ? wake_up_state+0x20/0x20 [12009.614557] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [12009.623894] [] kthread+0xcf/0xe0 [12009.630290] [] ? kthread_create_on_node+0x140/0x140 [12009.638534] [] ret_from_fork+0x58/0x90 [12009.645517] [] ? kthread_create_on_node+0x140/0x140 [12009.653764] Code: 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 00 8b 75 98 48 8b 7d 88 e8 dc 2d f5 ff 48 85 c0 48 89 45 90 0f 84 b3 08 00 00 44 8b 7d a0 <48> 8b 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 e8 77 5b f4 ff 42 [12037.261800] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [12037.271477] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [12037.403502] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [12037.416808] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [12037.433108] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [12037.442509] RIP: 0010:[] [] osd_scrub_next+0x3d/0x3c0 [osd_ldiskfs] [12037.454189] RSP: 0018:ffff880703603c70 EFLAGS: 00000246 [12037.461141] RAX: 0000000000000000 RBX: ffff880703603c37 RCX: ffff880703603d98 [12037.470132] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [12037.479113] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [12037.488064] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [12037.497015] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [12037.505961] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [12037.515985] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12037.523390] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [12037.532356] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12037.541321] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [12037.550271] Stack: [12037.553486] 000000009f8f2001 ffff880703603d08 ffff880703603d57 ffff880703603df0 [12037.562769] 000000009f8f2001 ffff880703603d57 000000009f8f2001 0000000000000004 [12037.572069] ffff880703603d57 ffff88042bd78800 ffff88042bd78800 00000000000072cf [12037.581379] Call Trace: [12037.585103] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [12037.594271] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [12037.603437] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [12037.612306] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [12037.620970] [] ? wake_up_state+0x20/0x20 [12037.628174] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [12037.637518] [] kthread+0xcf/0xe0 [12037.643946] [] ? kthread_create_on_node+0x140/0x140 [12037.652227] [] ret_from_fork+0x58/0x90 [12037.659232] [] ? kthread_create_on_node+0x140/0x140 [12037.667496] Code: 49 89 fd 41 54 49 89 d4 53 48 89 cb 48 83 ec 70 65 48 8b 04 25 28 00 00 00 48 89 45 d0 31 c0 48 8b 05 f8 59 9c ff 48 85 c0 75 7b <41> f6 87 58 10 00 00 08 0f 84 62 03 00 00 4d 8d b7 e0 10 00 00 [12065.262776] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [12065.272470] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [12065.404143] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [12065.417427] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [12065.433670] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [12065.443051] RIP: 0010:[] [] osd_inode_iteration+0x4ac/0xcc0 [osd_ldiskfs] [12065.455282] RSP: 0018:ffff880703603d18 EFLAGS: 00000216 [12065.462227] RAX: 0000000020000000 RBX: 0000000000000001 RCX: 0000000000000000 [12065.471201] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [12065.480165] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [12065.489122] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [12065.498071] R13: ffffea00027e3c80 R14: 000000009f8f2001 R15: 0000000000000018 [12065.507020] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [12065.517046] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12065.524451] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [12065.533414] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12065.542358] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [12065.551296] Stack: [12065.554501] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [12065.563784] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [12065.573083] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [12065.582355] Call Trace: [12065.586069] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [12065.595233] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [12065.604104] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [12065.612774] [] ? wake_up_state+0x20/0x20 [12065.619981] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [12065.629346] [] kthread+0xcf/0xe0 [12065.635776] [] ? kthread_create_on_node+0x140/0x140 [12065.644055] [] ret_from_fork+0x58/0x90 [12065.651070] [] ? kthread_create_on_node+0x140/0x140 [12065.659351] Code: 88 48 8b b5 38 ff ff ff 48 8b bd 48 ff ff ff 48 8b 85 28 ff ff ff ff d0 85 c0 41 89 c5 75 1c 8b 85 58 ff ff ff 41 3b 04 24 72 10 <48> 8b 85 50 ff ff ff 44 3b 30 0f 87 95 fd ff ff 45 31 ff e9 a2 [12078.568324] Lustre: soaked-MDT0000: Client af5c0dbc-bc84-dcde-3672-645b97702b86 (at 192.168.1.122@o2ib100) reconnecting [12078.583209] Lustre: Skipped 2 previous similar messages [12093.263752] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [12093.273463] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [12093.405225] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [12093.418520] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [12093.434800] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [12093.444195] RIP: 0010:[] [] osd_inode_iteration+0x251/0xcc0 [osd_ldiskfs] [12093.456444] RSP: 0018:ffff880703603d18 EFLAGS: 00000286 [12093.463407] RAX: ffff8800b2a37468 RBX: ffff880703603d98 RCX: 0000000000000000 [12093.472402] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [12093.481391] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [12093.490374] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000001 [12093.499355] R13: 0000000000000004 R14: 000000009f8f2001 R15: 0000000000000018 [12093.508335] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [12093.518379] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12093.525807] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [12093.534783] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12093.543764] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [12093.552727] Stack: [12093.555960] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [12093.565270] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [12093.574579] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [12093.583887] Call Trace: [12093.587626] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [12093.596822] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [12093.605706] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [12093.614387] [] ? wake_up_state+0x20/0x20 [12093.621614] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [12093.630969] [] kthread+0xcf/0xe0 [12093.637420] [] ? kthread_create_on_node+0x140/0x140 [12093.645713] [] ret_from_fork+0x58/0x90 [12093.652738] [] ? kthread_create_on_node+0x140/0x140 [12093.661019] Code: b0 00 00 00 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 00 8b 75 98 48 8b 7d 88 e8 dc 2d f5 ff 48 85 c0 48 89 45 90 0f 84 b3 08 00 00 <44> 8b 7d a0 48 8b 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 e8 77 [12121.264726] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [12121.274436] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [12121.406348] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [12121.419646] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [12121.435923] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [12121.445317] RIP: 0010:[] [] osd_inode_iteration+0x473/0xcc0 [osd_ldiskfs] [12121.457556] RSP: 0018:ffff880703603d18 EFLAGS: 00000293 [12121.464516] RAX: 0000000000000004 RBX: 000000009f8f2001 RCX: ffff880703603d98 [12121.473496] RDX: ffff880703603d78 RSI: 0000000000000000 RDI: ffff8803162f6000 [12121.482465] RBP: ffff880703603df0 R08: 0000000000000001 R09: 0000000000000004 [12121.491416] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffff880703603d57 [12121.500355] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [12121.509294] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [12121.519313] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12121.526714] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [12121.535673] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12121.544634] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [12121.553583] Stack: [12121.556798] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [12121.566094] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [12121.575384] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [12121.584697] Call Trace: [12121.588422] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [12121.597590] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [12121.606470] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [12121.615159] [] ? wake_up_state+0x20/0x20 [12121.622374] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [12121.631722] [] kthread+0xcf/0xe0 [12121.638156] [] ? kthread_create_on_node+0x140/0x140 [12121.646434] [] ret_from_fork+0x58/0x90 [12121.653446] [] ? kthread_create_on_node+0x140/0x140 [12121.661726] Code: 00 00 48 c7 05 9b 86 02 00 00 00 00 00 c7 05 89 86 02 00 01 00 00 00 e8 7c eb 97 ff e9 0f fc ff ff 0f 1f 80 00 00 00 00 45 89 e9 <4c> 8d 85 67 ff ff ff 48 8b 4d a8 48 8d 55 88 48 8b b5 38 ff ff [12149.265708] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [12149.275384] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [12149.407428] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [12149.420739] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [12149.437033] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [12149.446437] RIP: 0010:[] [] osd_inode_iteration+0x264/0xcc0 [osd_ldiskfs] [12149.458695] RSP: 0018:ffff880703603d18 EFLAGS: 00000286 [12149.465666] RAX: ffff8800b2a37468 RBX: 000000009f8f2001 RCX: 0000000000000000 [12149.474656] RDX: ffff880703603d78 RSI: ffff880419dc3a00 RDI: ffff88042bd79800 [12149.483632] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [12149.492605] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [12149.501569] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [12149.510513] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [12149.520544] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12149.527948] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [12149.536910] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12149.545875] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [12149.554830] Stack: [12149.558051] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [12149.567344] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [12149.576644] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [12149.585973] Call Trace: [12149.589701] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [12149.598872] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [12149.607747] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [12149.616427] [] ? wake_up_state+0x20/0x20 [12149.623643] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [12149.632992] [] kthread+0xcf/0xe0 [12149.639431] [] ? kthread_create_on_node+0x140/0x140 [12149.647709] [] ret_from_fork+0x58/0x90 [12149.654730] [] ? kthread_create_on_node+0x140/0x140 [12149.663006] Code: 75 98 48 8b 7d 88 e8 dc 2d f5 ff 48 85 c0 48 89 45 90 0f 84 b3 08 00 00 44 8b 7d a0 48 8b 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 77 5b f4 ff 42 8d 14 38 48 8b 45 88 48 8b 80 a8 03 00 00 48 [12163.847209] INFO: rcu_sched self-detected stall on CPU { 24} (t=4740099 jiffies g=907112 c=907111 q=10773653) [12163.859658] Task dump for CPU 24: [12163.864408] OI_scrub R running task 0 5551 2 0x00000088 [12163.873335] ffff88083fde6dd0 00000000fb6db880 ffff88082da03db0 ffffffff810c47e8 [12163.882668] 0000000000000018 ffffffff81a1e7c0 ffff88082da03dc8 ffffffff810c8079 [12163.891978] 0000000000000009 ffff88082da03df8 ffffffff811378e0 ffff88082da101c0 [12163.901285] Call Trace: [12163.904985] [] sched_show_task+0xa8/0x110 [12163.912979] [] dump_cpu_task+0x39/0x70 [12163.919954] [] rcu_dump_cpu_stacks+0x90/0xd0 [12163.927496] [] rcu_check_callbacks+0x442/0x720 [12163.935200] [] ? tick_sched_handle.isra.13+0x60/0x60 [12163.943468] [] update_process_times+0x47/0x80 [12163.951037] [] tick_sched_handle.isra.13+0x25/0x60 [12163.959069] [] tick_sched_timer+0x41/0x70 [12163.966195] [] __hrtimer_run_queues+0xd2/0x260 [12163.973801] [] hrtimer_interrupt+0xb0/0x1e0 [12163.981098] [] ? call_softirq+0x1c/0x30 [12163.987982] [] local_apic_timer_interrupt+0x37/0x60 [12163.996024] [] smp_apic_timer_interrupt+0x3f/0x60 [12164.003852] [] apic_timer_interrupt+0x6d/0x80 [12164.011284] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [12164.020901] [] ? osd_scrub_exec+0x9/0x5b0 [osd_ldiskfs] [12164.029318] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [12164.038222] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [12164.047123] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [12164.055742] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [12164.064144] [] ? wake_up_state+0x20/0x20 [12164.071087] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [12164.080167] [] kthread+0xcf/0xe0 [12164.086334] [] ? kthread_create_on_node+0x140/0x140 [12164.094355] [] ret_from_fork+0x58/0x90 [12164.101106] [] ? kthread_create_on_node+0x140/0x140 [12189.267096] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [12189.276529] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [12189.406129] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [12189.419305] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [12189.435345] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [12189.444628] RIP: 0010:[] [] ldiskfs_itable_unused_count+0x16/0x30 [ldiskfs] [12189.456954] RSP: 0018:ffff880703603d08 EFLAGS: 00000246 [12189.463814] RAX: 0000000000000000 RBX: 0000000000000001 RCX: ffff88042bd78800 [12189.472711] RDX: 00000000000072ce RSI: ffff880419dc3a00 RDI: ffff88042bd79800 [12189.481683] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [12189.490582] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [12189.499473] R13: 0000000000000018 R14: ffff880703603c78 R15: 0000000000000246 [12189.508358] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [12189.518305] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12189.525643] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [12189.534552] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12189.543449] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [12189.552337] Stack: [12189.555497] ffff880703603df0 ffffffffa121cfb9 ffffffffa121b990 ffffffffa1217a20 [12189.564733] ffff8800b2a36000 00000000810d354f ffff8803162f6000 ffff8800b2a37468 [12189.573981] 0000000020000000 010000000000000c 0000000000000000 0000000000000000 [12189.583238] Call Trace: [12189.586933] [] osd_inode_iteration+0x269/0xcc0 [osd_ldiskfs] [12189.596069] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [12189.605196] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [12189.614033] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [12189.622665] [] ? wake_up_state+0x20/0x20 [12189.629831] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [12189.639135] [] kthread+0xcf/0xe0 [12189.645525] [] ? kthread_create_on_node+0x140/0x140 [12189.653757] [] ret_from_fork+0x58/0x90 [12189.660731] [] ? kthread_create_on_node+0x140/0x140 [12189.668965] Code: 07 0f b7 46 30 c1 e0 10 09 d0 5d c3 66 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 8b 8f a8 03 00 00 31 c0 0f b7 56 1c 48 89 e5 <48> 83 39 3f 76 07 0f b7 46 32 c1 e0 10 09 d0 5d c3 66 0f 1f 84 [12217.268072] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [12217.277695] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [12217.409406] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [12217.422696] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [12217.438951] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [12217.448329] RIP: 0010:[] [] osd_inode_iteration+0x2be/0xcc0 [osd_ldiskfs] [12217.460563] RSP: 0018:ffff880703603d18 EFLAGS: 00000297 [12217.467500] RAX: 0000000000000004 RBX: 000000009f8f2001 RCX: ffff880703603d98 [12217.476481] RDX: ffff880703603d78 RSI: 0000000000000000 RDI: ffff8803162f6000 [12217.485454] RBP: ffff880703603df0 R08: 0000000000000001 R09: 0000000000000004 [12217.494409] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffff880703603d57 [12217.503341] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [12217.512287] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [12217.522307] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12217.529715] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [12217.538676] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12217.547633] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [12217.556575] Stack: [12217.559783] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [12217.569064] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [12217.578378] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [12217.587670] Call Trace: [12217.591393] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [12217.600555] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [12217.609427] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [12217.618107] [] ? wake_up_state+0x20/0x20 [12217.625319] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [12217.634667] [] kthread+0xcf/0xe0 [12217.641098] [] ? kthread_create_on_node+0x140/0x140 [12217.649382] [] ret_from_fork+0x58/0x90 [12217.656385] [] ? kthread_create_on_node+0x140/0x140 [12217.664657] Code: 8b b5 38 ff ff ff 48 8b bd 48 ff ff ff 48 8b 85 30 ff ff ff ff d0 83 f8 03 41 89 c5 0f 84 4b 02 00 00 0f 8e 15 01 00 00 83 f8 05 <66> 90 0f 84 8a 02 00 00 83 f8 06 0f 85 a1 01 00 00 48 8b 7d 90 [12245.269046] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [12245.278709] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [12245.410609] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [12245.423919] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [12245.440210] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [12245.449618] RIP: 0010:[] [] osd_inode_iteration+0x282/0xcc0 [osd_ldiskfs] [12245.461873] RSP: 0018:ffff880703603d18 EFLAGS: 00000287 [12245.468836] RAX: ffff88042bd78800 RBX: ffff880703603d98 RCX: ffff88042bd78800 [12245.477826] RDX: 00000000000072cf RSI: ffff880419dc3a00 RDI: ffff88042bd79800 [12245.486806] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [12245.495788] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000001 [12245.504740] R13: 0000000000000004 R14: 000000009f8f2001 R15: 0000000000000018 [12245.513687] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [12245.523707] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12245.531110] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [12245.540069] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12245.549024] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [12245.557964] Stack: [12245.561168] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [12245.570441] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [12245.579727] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [12245.589019] Call Trace: [12245.592742] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [12245.601901] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [12245.610767] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [12245.619436] [] ? wake_up_state+0x20/0x20 [12245.626643] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [12245.635982] [] kthread+0xcf/0xe0 [12245.642404] [] ? kthread_create_on_node+0x140/0x140 [12245.650675] [] ret_from_fork+0x58/0x90 [12245.657680] [] ? kthread_create_on_node+0x140/0x140 [12245.665945] Code: 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 e8 77 5b f4 ff 42 8d 14 38 48 8b 45 88 48 8b 80 a8 03 00 00 48 3b 50 20 0f 83 57 01 00 00 <44> 0f b6 85 67 ff ff ff 48 8d 4d a8 48 8d 55 88 48 8b b5 38 ff [12273.270021] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [12273.279687] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [12273.411523] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [12273.424827] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [12273.441103] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [12273.450509] RIP: 0010:[] [] osd_inode_iteration+0x282/0xcc0 [osd_ldiskfs] [12273.462751] RSP: 0018:ffff880703603d18 EFLAGS: 00000287 [12273.469708] RAX: ffff88042bd78800 RBX: 000000009f8f2001 RCX: ffff88042bd78800 [12273.478690] RDX: 00000000000072cf RSI: ffff880419dc3a00 RDI: ffff88042bd79800 [12273.487671] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [12273.496631] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [12273.505579] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [12273.514526] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [12273.524540] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12273.531943] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [12273.540898] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12273.549861] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [12273.558813] Stack: [12273.562035] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [12273.571327] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [12273.580616] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [12273.589918] Call Trace: [12273.593645] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [12273.602813] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [12273.611690] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [12273.620378] [] ? wake_up_state+0x20/0x20 [12273.627596] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [12273.636938] [] kthread+0xcf/0xe0 [12273.643374] [] ? kthread_create_on_node+0x140/0x140 [12273.651657] [] ret_from_fork+0x58/0x90 [12273.658672] [] ? kthread_create_on_node+0x140/0x140 [12273.666947] Code: 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 e8 77 5b f4 ff 42 8d 14 38 48 8b 45 88 48 8b 80 a8 03 00 00 48 3b 50 20 0f 83 57 01 00 00 <44> 0f b6 85 67 ff ff ff 48 8d 4d a8 48 8d 55 88 48 8b b5 38 ff [12301.270998] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [12301.280618] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [12301.412444] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [12301.425753] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [12301.442050] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [12301.451458] RIP: 0010:[] [] osd_inode_iteration+0x251/0xcc0 [osd_ldiskfs] [12301.463718] RSP: 0018:ffff880703603d18 EFLAGS: 00000286 [12301.470685] RAX: ffff8800b2a37468 RBX: 0000000000000001 RCX: 0000000000000000 [12301.479676] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [12301.488658] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [12301.497625] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [12301.506578] R13: ffffea00027e3c80 R14: 000000009f8f2001 R15: ffff880703603d08 [12301.515532] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [12301.525562] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12301.532972] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [12301.541938] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12301.550902] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [12301.559851] Stack: [12301.563072] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [12301.572358] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [12301.581633] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [12301.590935] Call Trace: [12301.594661] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [12301.603830] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [12301.612714] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [12301.621387] [] ? wake_up_state+0x20/0x20 [12301.628596] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [12301.637954] [] kthread+0xcf/0xe0 [12301.644387] [] ? kthread_create_on_node+0x140/0x140 [12301.652668] [] ret_from_fork+0x58/0x90 [12301.659682] [] ? kthread_create_on_node+0x140/0x140 [12301.667955] Code: b0 00 00 00 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 00 8b 75 98 48 8b 7d 88 e8 dc 2d f5 ff 48 85 c0 48 89 45 90 0f 84 b3 08 00 00 <44> 8b 7d a0 48 8b 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 e8 77 [12329.271972] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [12329.281647] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [12329.413521] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [12329.426829] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [12329.443128] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [12329.452534] RIP: 0010:[] [] osd_inode_iteration+0x251/0xcc0 [osd_ldiskfs] [12329.464794] RSP: 0018:ffff880703603d18 EFLAGS: 00000286 [12329.471758] RAX: ffff8800b2a37468 RBX: 000000009f8f2001 RCX: 0000000000000000 [12329.480750] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [12329.489741] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [12329.498706] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [12329.507666] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [12329.516618] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [12329.526646] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12329.534054] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [12329.543032] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12329.551996] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [12329.560945] Stack: [12329.564158] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [12329.573439] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [12329.582734] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [12329.592026] Call Trace: [12329.595758] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [12329.604922] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [12329.613796] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [12329.622472] [] ? wake_up_state+0x20/0x20 [12329.629688] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [12329.639044] [] kthread+0xcf/0xe0 [12329.645480] [] ? kthread_create_on_node+0x140/0x140 [12329.653759] [] ret_from_fork+0x58/0x90 [12329.660771] [] ? kthread_create_on_node+0x140/0x140 [12329.669042] Code: b0 00 00 00 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 00 8b 75 98 48 8b 7d 88 e8 dc 2d f5 ff 48 85 c0 48 89 45 90 0f 84 b3 08 00 00 <44> 8b 7d a0 48 8b 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 e8 77 [12343.856480] INFO: rcu_sched self-detected stall on CPU { 24} (t=4920102 jiffies g=907112 c=907111 q=11879033) [12343.869035] Task dump for CPU 24: [12343.873790] OI_scrub R running task 0 5551 2 0x00000088 [12343.882719] ffff88083fde6dd0 00000000fb6db880 ffff88082da03db0 ffffffff810c47e8 [12343.892056] 0000000000000018 ffffffff81a1e7c0 ffff88082da03dc8 ffffffff810c8079 [12343.901382] 0000000000000009 ffff88082da03df8 ffffffff811378e0 ffff88082da101c0 [12343.910682] Call Trace: [12343.914382] [] sched_show_task+0xa8/0x110 [12343.922377] [] dump_cpu_task+0x39/0x70 [12343.929357] [] rcu_dump_cpu_stacks+0x90/0xd0 [12343.936892] [] rcu_check_callbacks+0x442/0x720 [12343.944606] [] ? tick_sched_handle.isra.13+0x60/0x60 [12343.952883] [] update_process_times+0x47/0x80 [12343.960453] [] tick_sched_handle.isra.13+0x25/0x60 [12343.968484] [] tick_sched_timer+0x41/0x70 [12343.975611] [] __hrtimer_run_queues+0xd2/0x260 [12343.983210] [] hrtimer_interrupt+0xb0/0x1e0 [12343.990513] [] ? call_softirq+0x1c/0x30 [12343.997392] [] local_apic_timer_interrupt+0x37/0x60 [12344.005425] [] smp_apic_timer_interrupt+0x3f/0x60 [12344.013261] [] apic_timer_interrupt+0x6d/0x80 [12344.020685] [] ? osd_inode_iteration+0x2a0/0xcc0 [osd_ldiskfs] [12344.030498] [] ? osd_inode_iteration+0x269/0xcc0 [osd_ldiskfs] [12344.039595] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [12344.048499] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [12344.057110] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [12344.065524] [] ? wake_up_state+0x20/0x20 [12344.072474] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [12344.081562] [] kthread+0xcf/0xe0 [12344.087730] [] ? kthread_create_on_node+0x140/0x140 [12344.095747] [] ret_from_fork+0x58/0x90 [12344.102497] [] ? kthread_create_on_node+0x140/0x140 [12369.273366] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [12369.282758] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [12369.412406] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [12369.425591] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [12369.441674] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [12369.450969] RIP: 0010:[] [] osd_scrub_exec+0xdb/0x5b0 [osd_ldiskfs] [12369.462530] RSP: 0018:ffff880703603c78 EFLAGS: 00000202 [12369.469387] RAX: ffff8803162f1008 RBX: ffff880703603c37 RCX: 0000000000000000 [12369.478291] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [12369.487181] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [12369.496077] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [12369.504974] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [12369.513868] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [12369.523836] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12369.531179] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [12369.540086] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12369.548994] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [12369.557900] Stack: [12369.561067] ffff880703603d08 0000000000000001 ffff880703603df0 000000009f8f2001 [12369.570305] ffff880703603d57 000000009f8f2001 0000000000000004 ffff880703603d57 [12369.579569] 0000000020000000 0000000000000000 ffff880703603d78 ffff8800b2a36000 [12369.588833] Call Trace: [12369.592510] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [12369.601648] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [12369.610781] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [12369.619631] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [12369.628279] [] ? wake_up_state+0x20/0x20 [12369.635460] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [12369.644772] [] kthread+0xcf/0xe0 [12369.651169] [] ? kthread_create_on_node+0x140/0x140 [12369.659420] [] ret_from_fork+0x58/0x90 [12369.666398] [] ? kthread_create_on_node+0x140/0x140 [12369.674638] Code: fb 46 e0 41 83 87 68 14 00 00 01 49 83 87 88 11 00 00 01 4c 89 ef e8 f5 9f e9 df 44 8b 4d 88 48 85 db 74 23 f6 83 18 0a 00 00 08 <74> 1a 4d 85 e4 74 15 41 8b 87 6c 14 00 00 41 39 84 24 0c 0a 00 [12397.274341] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [12397.283932] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [12397.415627] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [12397.428917] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [12397.445160] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [12397.454541] RIP: 0010:[] [] osd_scrub_next+0x2b/0x3c0 [osd_ldiskfs] [12397.466192] RSP: 0018:ffff880703603c70 EFLAGS: 00000282 [12397.473131] RAX: 00000000fb6db880 RBX: ffff880703603c37 RCX: ffff880703603d98 [12397.482111] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [12397.491073] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [12397.500022] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [12397.508971] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [12397.517920] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [12397.527950] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12397.535357] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [12397.544327] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12397.553279] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [12397.562226] Stack: [12397.565437] 000000009f8f2001 ffff880703603d08 000000009f8f2001 ffff880703603df0 [12397.574726] 000000009f8f2001 ffff880703603d57 000000009f8f2001 0000000000000004 [12397.584016] 0000000000000001 ffff88042bd78800 ffff880703603d98 ffff880703603d78 [12397.593307] Call Trace: [12397.597023] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [12397.606180] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [12397.615334] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [12397.624190] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [12397.632843] [] ? wake_up_state+0x20/0x20 [12397.640049] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [12397.649397] [] kthread+0xcf/0xe0 [12397.655820] [] ? kthread_create_on_node+0x140/0x140 [12397.664088] [] ret_from_fork+0x58/0x90 [12397.671095] [] ? kthread_create_on_node+0x140/0x140 [12397.679353] Code: 66 66 66 66 90 55 48 89 e5 41 57 49 89 f7 41 56 41 55 49 89 fd 41 54 49 89 d4 53 48 89 cb 48 83 ec 70 65 48 8b 04 25 28 00 00 00 <48> 89 45 d0 31 c0 48 8b 05 f8 59 9c ff 48 85 c0 75 7b 41 f6 87 [12425.275316] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [12425.284956] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [12425.416590] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [12425.429868] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [12425.446102] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [12425.455478] RIP: 0010:[] [] osd_inode_iteration+0x473/0xcc0 [osd_ldiskfs] [12425.467709] RSP: 0018:ffff880703603d18 EFLAGS: 00000293 [12425.474646] RAX: 0000000000000004 RBX: ffff880703603d98 RCX: ffff880703603d98 [12425.483619] RDX: ffff880703603d78 RSI: 0000000000000000 RDI: ffff8803162f6000 [12425.492571] RBP: ffff880703603df0 R08: 0000000000000001 R09: 0000000000000004 [12425.501516] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000001 [12425.510462] R13: 0000000000000004 R14: 000000009f8f2001 R15: ffffea00027e3c80 [12425.519394] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [12425.529419] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12425.536811] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [12425.545769] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12425.554719] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [12425.563654] Stack: [12425.566850] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [12425.576120] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [12425.585419] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [12425.594709] Call Trace: [12425.598420] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [12425.607577] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [12425.616458] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [12425.625123] [] ? wake_up_state+0x20/0x20 [12425.632323] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [12425.641670] [] kthread+0xcf/0xe0 [12425.648095] [] ? kthread_create_on_node+0x140/0x140 [12425.656368] [] ret_from_fork+0x58/0x90 [12425.663374] [] ? kthread_create_on_node+0x140/0x140 [12425.671648] Code: 00 00 48 c7 05 9b 86 02 00 00 00 00 00 c7 05 89 86 02 00 01 00 00 00 e8 7c eb 97 ff e9 0f fc ff ff 0f 1f 80 00 00 00 00 45 89 e9 <4c> 8d 85 67 ff ff ff 48 8b 4d a8 48 8d 55 88 48 8b b5 38 ff ff [12453.276293] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [12453.285944] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [12453.417832] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [12453.431135] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [12453.447424] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [12453.456833] RIP: 0010:[] [] osd_scrub_exec+0xf9/0x5b0 [osd_ldiskfs] [12453.468503] RSP: 0018:ffff880703603c78 EFLAGS: 00000246 [12453.475457] RAX: 0000000000000000 RBX: ffff880703603c37 RCX: 0000000000000000 [12453.484446] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [12453.493427] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [12453.502376] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [12453.511326] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [12453.520271] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [12453.530290] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12453.537698] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [12453.546662] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12453.555622] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [12453.564569] Stack: [12453.567783] ffff880703603d08 ffff880703603d57 ffff880703603df0 000000009f8f2001 [12453.577072] ffff880703603d57 000000009f8f2001 0000000000000004 ffff880703603d57 [12453.586364] 0000000000000000 0000000000000000 ffff880703603d78 ffff8800b2a36000 [12453.595661] Call Trace: [12453.599388] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [12453.608556] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [12453.617720] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [12453.626594] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [12453.635283] [] ? wake_up_state+0x20/0x20 [12453.642492] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [12453.651839] [] kthread+0xcf/0xe0 [12453.658267] [] ? kthread_create_on_node+0x140/0x140 [12453.666548] [] ret_from_fork+0x58/0x90 [12453.673559] [] ? kthread_create_on_node+0x140/0x140 [12453.681828] Code: 88 48 85 db 74 23 f6 83 18 0a 00 00 08 74 1a 4d 85 e4 74 15 41 8b 87 6c 14 00 00 41 39 84 24 0c 0a 00 00 0f 82 d9 00 00 00 31 c0 <41> f6 87 74 14 00 00 04 0f 85 6e ff ff ff 41 83 f9 02 0f 84 64 [12481.277268] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [12481.286943] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [12481.418594] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [12481.431881] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [12481.448133] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [12481.457518] RIP: 0010:[] [] osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [12481.469260] RSP: 0018:ffff880703603d10 EFLAGS: 00000287 [12481.476201] RAX: ffffffffa1217a20 RBX: 000000009f8f2001 RCX: ffff880703603d98 [12481.485179] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [12481.494154] RBP: ffff880703603df0 R08: 0000000000000001 R09: 0000000000000004 [12481.503111] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000001 [12481.512062] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [12481.521016] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [12481.531043] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12481.538453] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [12481.547418] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12481.556378] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [12481.565317] Stack: [12481.568530] ffffffffa121cff9 ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 [12481.577816] 00000000810d354f ffff8803162f6000 ffff8800b2a37468 0000000020000000 [12481.587104] 010000000000000c 0000000000000000 0000000000000000 ffff8800b2a36000 [12481.596393] Call Trace: [12481.600111] [] ? osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [12481.609471] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [12481.618626] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [12481.627492] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [12481.636155] [] ? wake_up_state+0x20/0x20 [12481.643372] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [12481.652735] [] kthread+0xcf/0xe0 [12481.659166] [] ? kthread_create_on_node+0x140/0x140 [12481.667445] [] ret_from_fork+0x58/0x90 [12481.674457] [] ? kthread_create_on_node+0x140/0x140 [12481.682727] Code: f7 48 8d 04 80 48 c1 e0 03 48 8d 4c 02 10 48 01 c2 e8 25 fc ff ff 48 83 c4 08 5b 41 5c 41 5d 41 5e 5d c3 0f 1f 84 00 00 00 00 00 <66> 66 66 66 90 55 48 89 e5 41 57 49 89 f7 41 56 41 55 49 89 fd [12509.278242] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [12509.287999] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [12509.419781] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [12509.433072] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [12509.449331] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [12509.458717] RIP: 0010:[] [] osd_inode_iteration+0x269/0xcc0 [osd_ldiskfs] [12509.470976] RSP: 0018:ffff880703603d18 EFLAGS: 00000202 [12509.477927] RAX: 00000000000072ce RBX: 000000009f8f2001 RCX: ffff88042bd78800 [12509.486910] RDX: 00000000000072ce RSI: ffff880419dc3a00 RDI: ffff88042bd79800 [12509.495870] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [12509.504835] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffff880703603d57 [12509.513787] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [12509.522749] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [12509.532781] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12509.540192] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [12509.549164] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12509.558126] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [12509.567076] Stack: [12509.570292] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [12509.579599] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [12509.588895] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [12509.598204] Call Trace: [12509.601929] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [12509.611098] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [12509.619976] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [12509.628649] [] ? wake_up_state+0x20/0x20 [12509.635859] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [12509.645198] [] kthread+0xcf/0xe0 [12509.651630] [] ? kthread_create_on_node+0x140/0x140 [12509.659915] [] ret_from_fork+0x58/0x90 [12509.666931] [] ? kthread_create_on_node+0x140/0x140 [12509.675212] Code: 88 e8 dc 2d f5 ff 48 85 c0 48 89 45 90 0f 84 b3 08 00 00 44 8b 7d a0 48 8b 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 e8 77 5b f4 ff <42> 8d 14 38 48 8b 45 88 48 8b 80 a8 03 00 00 48 3b 50 20 0f 83 [12523.865751] INFO: rcu_sched self-detected stall on CPU { 24} (t=5100105 jiffies g=907112 c=907111 q=13351514) [12523.878243] Task dump for CPU 24: [12523.882999] OI_scrub R running task 0 5551 2 0x00000088 [12523.891935] ffff88083fde6dd0 00000000fb6db880 ffff88082da03db0 ffffffff810c47e8 [12523.901272] 0000000000000018 ffffffff81a1e7c0 ffff88082da03dc8 ffffffff810c8079 [12523.910592] 0000000000000009 ffff88082da03df8 ffffffff811378e0 ffff88082da101c0 [12523.919898] Call Trace: [12523.923604] [] sched_show_task+0xa8/0x110 [12523.931603] [] dump_cpu_task+0x39/0x70 [12523.938580] [] rcu_dump_cpu_stacks+0x90/0xd0 [12523.946114] [] rcu_check_callbacks+0x442/0x720 [12523.953822] [] ? tick_sched_handle.isra.13+0x60/0x60 [12523.962101] [] update_process_times+0x47/0x80 [12523.969667] [] tick_sched_handle.isra.13+0x25/0x60 [12523.977699] [] tick_sched_timer+0x41/0x70 [12523.984846] [] __hrtimer_run_queues+0xd2/0x260 [12523.992449] [] hrtimer_interrupt+0xb0/0x1e0 [12523.999747] [] ? call_softirq+0x1c/0x30 [12524.006630] [] local_apic_timer_interrupt+0x37/0x60 [12524.014667] [] smp_apic_timer_interrupt+0x3f/0x60 [12524.022496] [] apic_timer_interrupt+0x6d/0x80 [12524.029923] [] ? osd_inode_iteration+0x4a0/0xcc0 [osd_ldiskfs] [12524.039746] [] ? osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [12524.048850] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [12524.057758] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [12524.066383] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [12524.074797] [] ? wake_up_state+0x20/0x20 [12524.081760] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [12524.090848] [] kthread+0xcf/0xe0 [12524.097022] [] ? kthread_create_on_node+0x140/0x140 [12524.105032] [] ret_from_fork+0x58/0x90 [12524.111793] [] ? kthread_create_on_node+0x140/0x140 [12549.279635] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [12549.289106] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [12549.418758] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [12549.431934] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [12549.448014] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [12549.457313] RIP: 0010:[] [] osd_scrub_exec+0x34/0x5b0 [osd_ldiskfs] [12549.468877] RSP: 0018:ffff880703603c78 EFLAGS: 00000296 [12549.475738] RAX: 00000000fb6db880 RBX: ffff880703603c37 RCX: 0000000000000000 [12549.484649] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [12549.493549] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [12549.502448] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [12549.511348] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [12549.520242] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [12549.530209] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12549.537560] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [12549.546466] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12549.555373] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [12549.564280] Stack: [12549.567450] 0000000000000004 ffff880703603d57 ffff880703603df0 0000000000000000 [12549.576720] ffff880703603d57 000000009f8f2001 0000000000000004 ffff880703603d57 [12549.585977] ffff8800b2a37468 0000000000000000 ffff880703603d78 ffff880419dc3a00 [12549.595242] Call Trace: [12549.598946] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [12549.608087] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [12549.617232] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [12549.626075] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [12549.634746] [] ? wake_up_state+0x20/0x20 [12549.641933] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [12549.651250] [] kthread+0xcf/0xe0 [12549.657645] [] ? kthread_create_on_node+0x140/0x140 [12549.665903] [] ret_from_fork+0x58/0x90 [12549.672891] [] ? kthread_create_on_node+0x140/0x140 [12549.681135] Code: 41 57 49 89 f7 41 56 41 55 41 54 45 31 e4 53 48 83 ec 68 48 8b 9e 08 10 00 00 4c 89 45 90 65 48 8b 04 25 28 00 00 00 48 89 45 d0 <31> c0 48 8d 43 08 48 85 db 4c 0f 45 e0 41 83 f9 04 0f 84 84 00 [12577.280611] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [12577.290217] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [12577.421899] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [12577.435188] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [12577.451440] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [12577.460825] RIP: 0010:[] [] osd_inode_iteration+0x26d/0xcc0 [osd_ldiskfs] [12577.473065] RSP: 0018:ffff880703603d18 EFLAGS: 00000202 [12577.480009] RAX: 00000000000072ce RBX: 000000009f8f2001 RCX: ffff88042bd78800 [12577.488997] RDX: 00000000000072cf RSI: ffff880419dc3a00 RDI: ffff88042bd79800 [12577.497963] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [12577.506921] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [12577.515876] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [12577.524824] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [12577.534868] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12577.542295] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [12577.551267] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12577.560229] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [12577.569172] Stack: [12577.572389] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [12577.581679] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [12577.590974] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [12577.600267] Call Trace: [12577.603985] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [12577.613147] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [12577.622023] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [12577.630693] [] ? wake_up_state+0x20/0x20 [12577.637897] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [12577.647249] [] kthread+0xcf/0xe0 [12577.653668] [] ? kthread_create_on_node+0x140/0x140 [12577.661952] [] ret_from_fork+0x58/0x90 [12577.668963] [] ? kthread_create_on_node+0x140/0x140 [12577.677233] Code: f5 ff 48 85 c0 48 89 45 90 0f 84 b3 08 00 00 44 8b 7d a0 48 8b 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 e8 77 5b f4 ff 42 8d 14 38 <48> 8b 45 88 48 8b 80 a8 03 00 00 48 3b 50 20 0f 83 57 01 00 00 [12605.281586] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [12605.291246] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [12605.423158] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [12605.436467] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [12605.452748] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [12605.462155] RIP: 0010:[] [] osd_scrub_exec+0xf7/0x5b0 [osd_ldiskfs] [12605.473822] RSP: 0018:ffff880703603c78 EFLAGS: 00000246 [12605.480785] RAX: 0000000000280001 RBX: ffff880703603c37 RCX: 0000000000000000 [12605.489762] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [12605.498735] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [12605.507692] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [12605.516649] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [12605.525611] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [12605.535643] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12605.543057] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [12605.552030] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12605.560998] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [12605.569947] Stack: [12605.573159] ffffea00027e3c80 000000009f8f2001 ffff880703603df0 ffff880703603d57 [12605.582446] ffff880703603d57 000000009f8f2001 0000000000000004 ffff880703603d57 [12605.591716] ffff88042bd78800 ffff88042bd78800 00000000000072cf ffff880419dc3a00 [12605.600992] Call Trace: [12605.604718] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [12605.613876] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [12605.623028] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [12605.631892] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [12605.640563] [] ? wake_up_state+0x20/0x20 [12605.647762] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [12605.657099] [] kthread+0xcf/0xe0 [12605.663523] [] ? kthread_create_on_node+0x140/0x140 [12605.671792] [] ret_from_fork+0x58/0x90 [12605.678792] [] ? kthread_create_on_node+0x140/0x140 [12605.687051] Code: 8b 4d 88 48 85 db 74 23 f6 83 18 0a 00 00 08 74 1a 4d 85 e4 74 15 41 8b 87 6c 14 00 00 41 39 84 24 0c 0a 00 00 0f 82 d9 00 00 00 <31> c0 41 f6 87 74 14 00 00 04 0f 85 6e ff ff ff 41 83 f9 02 0f [12633.282562] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [12633.292237] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [12633.423764] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [12633.437043] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [12633.453282] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [12633.462658] RIP: 0010:[] [] osd_inode_iteration+0x4a6/0xcc0 [osd_ldiskfs] [12633.474903] RSP: 0018:ffff880703603d18 EFLAGS: 00000246 [12633.481828] RAX: 0000000020000000 RBX: ffff880703603d98 RCX: 0000000000000000 [12633.490801] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [12633.499753] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [12633.508700] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000001 [12633.517632] R13: 0000000000000004 R14: 000000009f8f2001 R15: ffffea00027e3c80 [12633.526578] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [12633.536594] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12633.543989] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [12633.552935] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12633.561877] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [12633.570807] Stack: [12633.574011] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [12633.583290] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [12633.592580] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [12633.601851] Call Trace: [12633.605552] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [12633.614701] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [12633.623566] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [12633.632236] [] ? wake_up_state+0x20/0x20 [12633.639443] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [12633.648784] [] kthread+0xcf/0xe0 [12633.655212] [] ? kthread_create_on_node+0x140/0x140 [12633.663498] [] ret_from_fork+0x58/0x90 [12633.670505] [] ? kthread_create_on_node+0x140/0x140 [12633.678779] Code: 8b 4d a8 48 8d 55 88 48 8b b5 38 ff ff ff 48 8b bd 48 ff ff ff 48 8b 85 28 ff ff ff ff d0 85 c0 41 89 c5 75 1c 8b 85 58 ff ff ff <41> 3b 04 24 72 10 48 8b 85 50 ff ff ff 44 3b 30 0f 87 95 fd ff [12661.283537] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [12661.293223] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [12661.425131] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [12661.438431] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [12661.454723] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [12661.464126] RIP: 0010:[] [] osd_inode_iteration+0x251/0xcc0 [osd_ldiskfs] [12661.476353] RSP: 0018:ffff880703603d18 EFLAGS: 00000286 [12661.483309] RAX: ffff8800b2a37468 RBX: ffff880703603d57 RCX: 0000000000000000 [12661.492303] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [12661.501285] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [12661.510233] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [12661.519179] R13: ffffea00027e3c80 R14: 000000009f8f2001 R15: ffff880703603d08 [12661.528120] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [12661.538139] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12661.545537] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [12661.554511] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12661.563553] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [12661.572499] Stack: [12661.575708] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [12661.585004] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [12661.594286] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [12661.603575] Call Trace: [12661.607308] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [12661.616472] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [12661.625344] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [12661.634020] [] ? wake_up_state+0x20/0x20 [12661.641240] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [12661.650600] [] kthread+0xcf/0xe0 [12661.657035] [] ? kthread_create_on_node+0x140/0x140 [12661.665326] [] ret_from_fork+0x58/0x90 [12661.672336] [] ? kthread_create_on_node+0x140/0x140 [12661.680605] Code: b0 00 00 00 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 00 8b 75 98 48 8b 7d 88 e8 dc 2d f5 ff 48 85 c0 48 89 45 90 0f 84 b3 08 00 00 <44> 8b 7d a0 48 8b 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 e8 77 [12689.284512] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [12689.294162] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [12689.426076] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [12689.439391] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [12689.455681] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [12689.465097] RIP: 0010:[] [] osd_inode_iteration+0x255/0xcc0 [osd_ldiskfs] [12689.477362] RSP: 0018:ffff880703603d18 EFLAGS: 00000286 [12689.484327] RAX: ffff8800b2a37468 RBX: ffff880703603d98 RCX: 0000000000000000 [12689.493318] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [12689.502297] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [12689.511269] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000001 [12689.520212] R13: 0000000000000004 R14: 000000009f8f2001 R15: 0000000000000018 [12689.529156] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [12689.539174] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12689.546580] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [12689.555546] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12689.564499] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [12689.573444] Stack: [12689.576647] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [12689.585918] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [12689.595201] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [12689.604497] Call Trace: [12689.608212] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [12689.617367] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [12689.626230] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [12689.634905] [] ? wake_up_state+0x20/0x20 [12689.642111] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [12689.651455] [] kthread+0xcf/0xe0 [12689.657877] [] ? kthread_create_on_node+0x140/0x140 [12689.666146] [] ret_from_fork+0x58/0x90 [12689.673148] [] ? kthread_create_on_node+0x140/0x140 [12689.681410] Code: 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 00 8b 75 98 48 8b 7d 88 e8 dc 2d f5 ff 48 85 c0 48 89 45 90 0f 84 b3 08 00 00 44 8b 7d a0 <48> 8b 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 e8 77 5b f4 ff 42 [12703.875030] INFO: rcu_sched self-detected stall on CPU { 24} (t=5280108 jiffies g=907112 c=907111 q=14494531) [12703.887510] Task dump for CPU 24: [12703.892253] OI_scrub R running task 0 5551 2 0x00000088 [12703.901175] ffff88083fde6dd0 00000000fb6db880 ffff88082da03db0 ffffffff810c47e8 [12703.910512] 0000000000000018 ffffffff81a1e7c0 ffff88082da03dc8 ffffffff810c8079 [12703.919832] 0000000000000009 ffff88082da03df8 ffffffff811378e0 ffff88082da101c0 [12703.929123] Call Trace: [12703.932815] [] sched_show_task+0xa8/0x110 [12703.940816] [] dump_cpu_task+0x39/0x70 [12703.947781] [] rcu_dump_cpu_stacks+0x90/0xd0 [12703.955307] [] rcu_check_callbacks+0x442/0x720 [12703.963013] [] ? tick_sched_handle.isra.13+0x60/0x60 [12703.971274] [] update_process_times+0x47/0x80 [12703.978833] [] tick_sched_handle.isra.13+0x25/0x60 [12703.986856] [] tick_sched_timer+0x41/0x70 [12703.993982] [] __hrtimer_run_queues+0xd2/0x260 [12704.001575] [] hrtimer_interrupt+0xb0/0x1e0 [12704.008861] [] ? call_softirq+0x1c/0x30 [12704.015736] [] local_apic_timer_interrupt+0x37/0x60 [12704.023779] [] smp_apic_timer_interrupt+0x3f/0x60 [12704.031600] [] apic_timer_interrupt+0x6d/0x80 [12704.039015] [] ? osd_scrub_next+0x350/0x3c0 [osd_ldiskfs] [12704.048335] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [12704.057228] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [12704.066119] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [12704.074720] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [12704.083135] [] ? wake_up_state+0x20/0x20 [12704.090088] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [12704.099171] [] kthread+0xcf/0xe0 [12704.105343] [] ? kthread_create_on_node+0x140/0x140 [12704.113363] [] ret_from_fork+0x58/0x90 [12704.120119] [] ? kthread_create_on_node+0x140/0x140 [12719.514071] Lustre: soaked-MDT0000: Client e211cd09-6b0c-e43b-059c-f657101042a0 (at 192.168.1.117@o2ib100) reconnecting [12719.528038] Lustre: Skipped 1 previous similar message [12719.535516] Lustre: soaked-MDT0000: Connection restored to e211cd09-6b0c-e43b-059c-f657101042a0 (at 192.168.1.117@o2ib100) [12719.549633] Lustre: Skipped 2 previous similar messages [12729.285906] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [12729.295372] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [12729.425421] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [12729.438612] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [12729.454695] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [12729.464008] RIP: 0010:[] [] osd_scrub_exec+0xf7/0x5b0 [osd_ldiskfs] [12729.475608] RSP: 0018:ffff880703603c78 EFLAGS: 00000246 [12729.482513] RAX: 0000000000280001 RBX: ffff880703603c37 RCX: 0000000000000000 [12729.491460] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [12729.500403] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [12729.509342] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [12729.518288] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [12729.527243] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [12729.537268] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12729.544670] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [12729.553640] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12729.562613] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [12729.571570] Stack: [12729.574799] 0000000000000004 0000000000000001 ffff880703603df0 ffff880703603d98 [12729.584113] ffff880703603d57 000000009f8f2001 0000000000000004 ffff880703603d57 [12729.593437] 0000000020000000 0000000000000000 ffff880703603d78 ffff8800b2a36000 [12729.602730] Call Trace: [12729.606484] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [12729.615682] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [12729.624855] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [12729.633748] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [12729.642437] [] ? wake_up_state+0x20/0x20 [12729.649675] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [12729.659057] [] kthread+0xcf/0xe0 [12729.665516] [] ? kthread_create_on_node+0x140/0x140 [12729.673814] [] ret_from_fork+0x58/0x90 [12729.680845] [] ? kthread_create_on_node+0x140/0x140 [12729.689137] Code: 8b 4d 88 48 85 db 74 23 f6 83 18 0a 00 00 08 74 1a 4d 85 e4 74 15 41 8b 87 6c 14 00 00 41 39 84 24 0c 0a 00 00 0f 82 d9 00 00 00 <31> c0 41 f6 87 74 14 00 00 04 0f 85 6e ff ff ff 41 83 f9 02 0f [12757.286890] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [12757.296557] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [12757.428393] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [12757.441679] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [12757.457938] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [12757.467325] RIP: 0010:[] [] osd_inode_iteration+0x28a/0xcc0 [osd_ldiskfs] [12757.479563] RSP: 0018:ffff880703603d18 EFLAGS: 00000287 [12757.486502] RAX: ffff88042bd78800 RBX: 000000009f8f2001 RCX: ffff88042bd78800 [12757.495480] RDX: 00000000000072cf RSI: ffff880419dc3a00 RDI: ffff88042bd79800 [12757.504445] RBP: ffff880703603df0 R08: 0000000000000001 R09: 0000000000000004 [12757.513417] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [12757.522378] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [12757.531338] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [12757.541375] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12757.548792] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [12757.557774] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12757.566739] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [12757.575691] Stack: [12757.578901] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [12757.588212] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [12757.597512] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [12757.606825] Call Trace: [12757.610558] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [12757.619733] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [12757.628605] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [12757.637285] [] ? wake_up_state+0x20/0x20 [12757.644500] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [12757.653860] [] kthread+0xcf/0xe0 [12757.660300] [] ? kthread_create_on_node+0x140/0x140 [12757.668588] [] ret_from_fork+0x58/0x90 [12757.675610] [] ? kthread_create_on_node+0x140/0x140 [12757.683891] Code: a8 00 00 00 00 e8 77 5b f4 ff 42 8d 14 38 48 8b 45 88 48 8b 80 a8 03 00 00 48 3b 50 20 0f 83 57 01 00 00 44 0f b6 85 67 ff ff ff <48> 8d 4d a8 48 8d 55 88 48 8b b5 38 ff ff ff 48 8b bd 48 ff ff [12785.287856] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [12785.297507] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [12785.429694] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [12785.443006] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [12785.459313] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [12785.468721] RIP: 0010:[] [] osd_scrub_next+0x149/0x3c0 [osd_ldiskfs] [12785.480494] RSP: 0018:ffff880703603c70 EFLAGS: 00000202 [12785.487458] RAX: 0000000000000004 RBX: ffff880703603c37 RCX: ffff880703603d98 [12785.496448] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [12785.505444] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [12785.514424] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [12785.523380] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [12785.532334] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [12785.542373] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12785.549771] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [12785.558737] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12785.567705] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [12785.576655] Stack: [12785.579869] 000000009f8f2001 ffff880703603d08 000000009f8f2001 ffff880703603df0 [12785.589157] 000000009f8f2001 ffff880703603d57 000000009f8f2001 0000000000000004 [12785.598432] ffff880703603d57 ffff88042bd78800 ffff88042bd78800 00000000000072cf [12785.607731] Call Trace: [12785.611459] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [12785.620619] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [12785.629785] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [12785.638657] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [12785.647320] [] ? wake_up_state+0x20/0x20 [12785.654520] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [12785.663863] [] kthread+0xcf/0xe0 [12785.670285] [] ? kthread_create_on_node+0x140/0x140 [12785.678560] [] ret_from_fork+0x58/0x90 [12785.685563] [] ? kthread_create_on_node+0x140/0x140 [12785.693825] Code: 45 90 74 49 49 8d 9f 10 11 00 00 48 89 df e8 df 6a 47 e0 41 c7 87 58 10 00 00 02 00 00 00 48 89 df e8 fc 69 47 e0 b8 05 00 00 00 <48> 8b 75 d0 65 48 33 34 25 28 00 00 00 0f 85 4c 02 00 00 48 83 [12813.288831] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [12813.298478] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [12813.430275] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [12813.443548] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [12813.459775] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [12813.469162] RIP: 0010:[] [] osd_inode_iteration+0x4ac/0xcc0 [osd_ldiskfs] [12813.481399] RSP: 0018:ffff880703603d18 EFLAGS: 00000216 [12813.488338] RAX: 0000000020000000 RBX: 000000009f8f2001 RCX: 0000000000000000 [12813.497322] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [12813.506290] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [12813.515239] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [12813.524182] R13: ffff880703603d08 R14: 000000009f8f2001 R15: ffff880703603d08 [12813.533134] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [12813.543159] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12813.550560] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [12813.559527] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12813.568488] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [12813.577440] Stack: [12813.580658] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [12813.589956] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [12813.599245] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [12813.608541] Call Trace: [12813.612259] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [12813.621426] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [12813.630305] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [12813.638977] [] ? wake_up_state+0x20/0x20 [12813.646185] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [12813.655544] [] kthread+0xcf/0xe0 [12813.661980] [] ? kthread_create_on_node+0x140/0x140 [12813.670270] [] ret_from_fork+0x58/0x90 [12813.677287] [] ? kthread_create_on_node+0x140/0x140 [12813.685571] Code: 88 48 8b b5 38 ff ff ff 48 8b bd 48 ff ff ff 48 8b 85 28 ff ff ff ff d0 85 c0 41 89 c5 75 1c 8b 85 58 ff ff ff 41 3b 04 24 72 10 <48> 8b 85 50 ff ff ff 44 3b 30 0f 87 95 fd ff ff 45 31 ff e9 a2 [12841.289821] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [12841.299509] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [12841.431746] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [12841.445057] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [12841.461354] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [12841.470758] RIP: 0010:[] [] osd_scrub_exec+0x3d/0x5b0 [osd_ldiskfs] [12841.482429] RSP: 0018:ffff880703603c78 EFLAGS: 00000286 [12841.489397] RAX: ffff8803162f1008 RBX: ffff880703603c37 RCX: 0000000000000000 [12841.498393] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [12841.507381] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [12841.516348] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [12841.525307] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [12841.534257] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [12841.544282] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12841.551687] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [12841.560652] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12841.569617] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [12841.578567] Stack: [12841.581796] 0000000000000004 0000000000000001 ffffffffa1217a20 ffff880703603d98 [12841.591080] ffff880703603d57 ffff8800b2a36000 ffff8803162f6000 ffffffffffffff10 [12841.600387] ffffffffa1217a3b 0000000000000010 0000000000000287 ffff880703603ce0 [12841.609676] Call Trace: [12841.613407] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [12841.622280] [] ? osd_scrub_next+0x1b/0x3c0 [osd_ldiskfs] [12841.631058] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [12841.640222] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [12841.649385] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [12841.658248] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [12841.666917] [] ? wake_up_state+0x20/0x20 [12841.674120] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [12841.683472] [] kthread+0xcf/0xe0 [12841.689903] [] ? kthread_create_on_node+0x140/0x140 [12841.698173] [] ret_from_fork+0x58/0x90 [12841.705185] [] ? kthread_create_on_node+0x140/0x140 [12841.713459] Code: 41 54 45 31 e4 53 48 83 ec 68 48 8b 9e 08 10 00 00 4c 89 45 90 65 48 8b 04 25 28 00 00 00 48 89 45 d0 31 c0 48 8d 43 08 48 85 db <4c> 0f 45 e0 41 83 f9 04 0f 84 84 00 00 00 41 83 f9 07 74 4f 41 [12869.290789] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [12869.300417] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [12869.432078] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [12869.445342] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [12869.461545] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [12869.470902] RIP: 0010:[] [] osd_scrub_exec+0x95/0x5b0 [osd_ldiskfs] [12869.482531] RSP: 0018:ffff880703603d08 EFLAGS: 00000292 [12869.489450] RAX: 0000000000000000 RBX: ffff880703603d57 RCX: 0000000000000000 [12869.498414] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [12869.507365] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [12869.516324] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [12869.525282] R13: ffffea00027e3c80 R14: ffff880703603c70 R15: ffff880703603d08 [12869.534242] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [12869.544278] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12869.551685] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [12869.560647] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12869.569596] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [12869.578546] Stack: [12869.581779] ffff880703603df0 ffffffffa121d1e9 ffffffffa121b990 ffffffffa1217a20 [12869.591092] ffff8800b2a36000 00000000810d354f ffff8803162f6000 ffff8800b2a37468 [12869.600379] 0000000020000000 010000000000000c 0000000000000000 0000000000000000 [12869.609656] Call Trace: [12869.613376] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [12869.622540] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [12869.631688] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [12869.640543] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [12869.649222] [] ? wake_up_state+0x20/0x20 [12869.656418] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [12869.665769] [] kthread+0xcf/0xe0 [12869.672207] [] ? kthread_create_on_node+0x140/0x140 [12869.680491] [] ret_from_fork+0x58/0x90 [12869.687506] [] ? kthread_create_on_node+0x140/0x140 [12869.695792] Code: 01 00 00 41 80 a7 74 14 00 00 fe 48 8b 4d d0 65 48 33 0c 25 28 00 00 00 0f 85 b1 03 00 00 48 83 c4 68 5b 41 5c 41 5d 41 5e 41 5f <5d> c3 66 0f 1f 84 00 00 00 00 00 4c 8d ae f0 10 00 00 44 89 4d [12883.884298] INFO: rcu_sched self-detected stall on CPU { 24} (t=5460111 jiffies g=907112 c=907111 q=15277119) [12883.896715] Task dump for CPU 24: [12883.901434] OI_scrub R running task 0 5551 2 0x00000088 [12883.910341] ffff88083fde6dd0 00000000fb6db880 ffff88082da03db0 ffffffff810c47e8 [12883.919661] 0000000000000018 ffffffff81a1e7c0 ffff88082da03dc8 ffffffff810c8079 [12883.928958] 0000000000000009 ffff88082da03df8 ffffffff811378e0 ffff88082da101c0 [12883.938250] Call Trace: [12883.941932] [] sched_show_task+0xa8/0x110 [12883.949919] [] dump_cpu_task+0x39/0x70 [12883.956871] [] rcu_dump_cpu_stacks+0x90/0xd0 [12883.964384] [] rcu_check_callbacks+0x442/0x720 [12883.972074] [] ? tick_sched_handle.isra.13+0x60/0x60 [12883.980327] [] update_process_times+0x47/0x80 [12883.987880] [] tick_sched_handle.isra.13+0x25/0x60 [12883.995890] [] tick_sched_timer+0x41/0x70 [12884.003012] [] __hrtimer_run_queues+0xd2/0x260 [12884.010600] [] hrtimer_interrupt+0xb0/0x1e0 [12884.017871] [] ? call_softirq+0x1c/0x30 [12884.024745] [] local_apic_timer_interrupt+0x37/0x60 [12884.032775] [] smp_apic_timer_interrupt+0x3f/0x60 [12884.040591] [] apic_timer_interrupt+0x6d/0x80 [12884.048021] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [12884.057658] [] ? osd_scrub_exec+0x12/0x5b0 [osd_ldiskfs] [12884.066172] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [12884.075083] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [12884.083985] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [12884.092602] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [12884.101007] [] ? wake_up_state+0x20/0x20 [12884.107956] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [12884.117038] [] kthread+0xcf/0xe0 [12884.123218] [] ? kthread_create_on_node+0x140/0x140 [12884.131232] [] ret_from_fork+0x58/0x90 [12884.137993] [] ? kthread_create_on_node+0x140/0x140 [12909.292184] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [12909.301577] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [12909.431348] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [12909.444530] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [12909.460567] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [12909.469846] RIP: 0010:[] [] osd_inode_iteration+0x2be/0xcc0 [osd_ldiskfs] [12909.481987] RSP: 0018:ffff880703603d18 EFLAGS: 00000297 [12909.488853] RAX: 0000000000000004 RBX: 000000009f8f2001 RCX: ffff880703603d98 [12909.497749] RDX: ffff880703603d78 RSI: 0000000000000000 RDI: ffff8803162f6000 [12909.506648] RBP: ffff880703603df0 R08: 0000000000000001 R09: 0000000000000004 [12909.515550] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000001 [12909.524439] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [12909.533323] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [12909.543287] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12909.550631] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [12909.559537] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12909.568436] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [12909.577333] Stack: [12909.580509] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [12909.589757] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [12909.598993] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [12909.608258] Call Trace: [12909.611943] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [12909.621071] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [12909.629908] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [12909.638550] [] ? wake_up_state+0x20/0x20 [12909.645735] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [12909.655059] [] kthread+0xcf/0xe0 [12909.661451] [] ? kthread_create_on_node+0x140/0x140 [12909.669714] [] ret_from_fork+0x58/0x90 [12909.676697] [] ? kthread_create_on_node+0x140/0x140 [12909.684943] Code: 8b b5 38 ff ff ff 48 8b bd 48 ff ff ff 48 8b 85 30 ff ff ff ff d0 83 f8 03 41 89 c5 0f 84 4b 02 00 00 0f 8e 15 01 00 00 83 f8 05 <66> 90 0f 84 8a 02 00 00 83 f8 06 0f 85 a1 01 00 00 48 8b 7d 90 [12937.293158] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [12937.302797] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [12937.434998] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [12937.448317] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [12937.464610] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [12937.474024] RIP: 0010:[] [] osd_inode_iteration+0x4a6/0xcc0 [osd_ldiskfs] [12937.486273] RSP: 0018:ffff880703603d18 EFLAGS: 00000246 [12937.493230] RAX: 0000000020000000 RBX: 0000000000000000 RCX: 0000000000000000 [12937.502232] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [12937.511218] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [12937.520177] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffff880703603d57 [12937.529132] R13: 0000000000000004 R14: 000000009f8f2001 R15: ffffea00027e3c80 [12937.538085] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [12937.548112] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12937.555515] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [12937.564487] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12937.573455] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [12937.582402] Stack: [12937.585621] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [12937.594901] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [12937.604180] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [12937.613473] Call Trace: [12937.617217] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [12937.626381] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [12937.635261] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [12937.643953] [] ? wake_up_state+0x20/0x20 [12937.651166] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [12937.660534] [] kthread+0xcf/0xe0 [12937.666971] [] ? kthread_create_on_node+0x140/0x140 [12937.675259] [] ret_from_fork+0x58/0x90 [12937.682278] [] ? kthread_create_on_node+0x140/0x140 [12937.690549] Code: 8b 4d a8 48 8d 55 88 48 8b b5 38 ff ff ff 48 8b bd 48 ff ff ff 48 8b 85 28 ff ff ff ff d0 85 c0 41 89 c5 75 1c 8b 85 58 ff ff ff <41> 3b 04 24 72 10 48 8b 85 50 ff ff ff 44 3b 30 0f 87 95 fd ff [12965.294134] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [12965.303818] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [12965.435850] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [12965.449156] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [12965.465443] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [12965.474835] RIP: 0010:[] [] osd_scrub_exec+0xf7/0x5b0 [osd_ldiskfs] [12965.486497] RSP: 0018:ffff880703603c78 EFLAGS: 00000246 [12965.493454] RAX: 0000000000280001 RBX: ffff880703603c37 RCX: 0000000000000000 [12965.502434] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [12965.511408] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [12965.520365] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [12965.529312] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [12965.538267] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [12965.548293] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12965.555695] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [12965.564671] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12965.573636] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [12965.582589] Stack: [12965.585807] ffff880703603df0 0000000000000001 ffff880703603df0 000000009f8f2001 [12965.595108] ffff880703603d57 000000009f8f2001 0000000000000004 ffff880703603d57 [12965.604400] ffff88042bd78800 ffff88042bd78800 00000000000072cf ffff880419dc3a00 [12965.613699] Call Trace: [12965.617428] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [12965.626583] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [12965.635739] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [12965.644615] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [12965.653290] [] ? wake_up_state+0x20/0x20 [12965.660498] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [12965.669856] [] kthread+0xcf/0xe0 [12965.676286] [] ? kthread_create_on_node+0x140/0x140 [12965.684567] [] ret_from_fork+0x58/0x90 [12965.691578] [] ? kthread_create_on_node+0x140/0x140 [12965.699834] Code: 8b 4d 88 48 85 db 74 23 f6 83 18 0a 00 00 08 74 1a 4d 85 e4 74 15 41 8b 87 6c 14 00 00 41 39 84 24 0c 0a 00 00 0f 82 d9 00 00 00 <31> c0 41 f6 87 74 14 00 00 04 0f 85 6e ff ff ff 41 83 f9 02 0f [12993.295109] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [12993.304759] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [12993.436954] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [12993.450243] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [12993.466504] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [12993.475891] RIP: 0010:[] [] osd_scrub_exec+0x95/0x5b0 [osd_ldiskfs] [12993.487546] RSP: 0018:ffff880703603d08 EFLAGS: 00000292 [12993.494488] RAX: 0000000000000000 RBX: 0000000000000001 RCX: 0000000000000000 [12993.503465] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [12993.512438] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [12993.521399] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [12993.530350] R13: ffffea00027e3c80 R14: ffff880703603c78 R15: ffff880703603d08 [12993.539299] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [12993.549327] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12993.556735] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [12993.565709] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [12993.574664] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [12993.583607] Stack: [12993.586823] ffff880703603df0 ffffffffa121d1e9 ffffffffa121b990 ffffffffa1217a20 [12993.596094] ffff8800b2a36000 00000000810d354f ffff8803162f6000 ffff8800b2a37468 [12993.605381] 0000000020000000 010000000000000c 0000000000000000 0000000000000000 [12993.614662] Call Trace: [12993.618389] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [12993.627553] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [12993.636718] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [12993.645588] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [12993.654264] [] ? wake_up_state+0x20/0x20 [12993.661475] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [12993.670831] [] kthread+0xcf/0xe0 [12993.677270] [] ? kthread_create_on_node+0x140/0x140 [12993.685552] [] ret_from_fork+0x58/0x90 [12993.692566] [] ? kthread_create_on_node+0x140/0x140 [12993.700840] Code: 01 00 00 41 80 a7 74 14 00 00 fe 48 8b 4d d0 65 48 33 0c 25 28 00 00 00 0f 85 b1 03 00 00 48 83 c4 68 5b 41 5c 41 5d 41 5e 41 5f <5d> c3 66 0f 1f 84 00 00 00 00 00 4c 8d ae f0 10 00 00 44 89 4d [13021.296077] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [13021.305734] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [13021.437866] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [13021.451159] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [13021.467436] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [13021.476833] RIP: 0010:[] [] osd_scrub_next+0x161/0x3c0 [osd_ldiskfs] [13021.488603] RSP: 0018:ffff880703603ce8 EFLAGS: 00000282 [13021.495547] RAX: 0000000000000004 RBX: ffff880703603c78 RCX: ffff880703603d98 [13021.504535] RDX: ffff880703603d78 RSI: 0000000000000000 RDI: ffff8803162f6000 [13021.513498] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [13021.522455] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000010 [13021.531406] R13: ffffffffa121b9c4 R14: ffffffffffffff10 R15: ffff8803162f6000 [13021.540358] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [13021.550390] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13021.557795] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [13021.566757] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13021.575717] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [13021.584662] Stack: [13021.587875] ffff8800b2a3746c 0000000000000000 00000000ffffffff 0000000000000001 [13021.597176] ffff880703603df0 ffffffffa121cff9 ffffffffa121b990 ffffffffa1217a20 [13021.606474] ffff8800b2a36000 00000000810d354f ffff8803162f6000 ffff8800b2a37468 [13021.615770] Call Trace: [13021.619499] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [13021.628668] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [13021.637832] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [13021.646700] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [13021.655367] [] ? wake_up_state+0x20/0x20 [13021.662580] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [13021.671937] [] kthread+0xcf/0xe0 [13021.678381] [] ? kthread_create_on_node+0x140/0x140 [13021.686671] [] ret_from_fork+0x58/0x90 [13021.693687] [] ? kthread_create_on_node+0x140/0x140 [13021.701959] Code: 00 00 02 00 00 00 48 89 df e8 fc 69 47 e0 b8 05 00 00 00 48 8b 75 d0 65 48 33 34 25 28 00 00 00 0f 85 4c 02 00 00 48 83 c4 70 5b <41> 5c 41 5d 41 5e 41 5f 5d c3 48 8b 05 be 58 9c ff 48 85 c0 0f [13049.297053] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [13049.306735] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [13049.438732] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [13049.452020] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [13049.468293] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [13049.477697] RIP: 0010:[] [] osd_scrub_exec+0x23/0x5b0 [osd_ldiskfs] [13049.489356] RSP: 0018:ffff880703603c78 EFLAGS: 00000296 [13049.496302] RAX: ffffffffa121b990 RBX: ffff880703603c37 RCX: 0000000000000000 [13049.505292] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [13049.514252] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [13049.523208] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [13049.532158] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [13049.541111] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [13049.551150] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13049.558567] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [13049.567542] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13049.576513] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [13049.585465] Stack: [13049.588682] ffff880703603d08 000000009f8f2001 ffff880703603df0 000000009f8f2001 [13049.597977] ffff880703603d57 000000009f8f2001 0000000000000004 ffff880703603d57 [13049.607267] ffff8800b2a37468 0000000000000000 ffff880703603d78 ffff8800b2a36000 [13049.616565] Call Trace: [13049.620287] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [13049.629454] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [13049.638615] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [13049.647478] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [13049.656140] [] ? wake_up_state+0x20/0x20 [13049.663349] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [13049.672707] [] kthread+0xcf/0xe0 [13049.679137] [] ? kthread_create_on_node+0x140/0x140 [13049.687414] [] ret_from_fork+0x58/0x90 [13049.694426] [] ? kthread_create_on_node+0x140/0x140 [13049.702694] Code: 03 98 ff e9 c7 fd ff ff 66 66 66 66 90 55 48 89 e5 41 57 49 89 f7 41 56 41 55 41 54 45 31 e4 53 48 83 ec 68 48 8b 9e 08 10 00 00 <4c> 89 45 90 65 48 8b 04 25 28 00 00 00 48 89 45 d0 31 c0 48 8d [13063.893562] INFO: rcu_sched self-detected stall on CPU { 24} (t=5640114 jiffies g=907112 c=907111 q=15703893) [13063.906080] Task dump for CPU 24: [13063.910821] OI_scrub R running task 0 5551 2 0x00000088 [13063.919745] ffff88083fde6dd0 00000000fb6db880 ffff88082da03db0 ffffffff810c47e8 [13063.929067] 0000000000000018 ffffffff81a1e7c0 ffff88082da03dc8 ffffffff810c8079 [13063.938370] 0000000000000009 ffff88082da03df8 ffffffff811378e0 ffff88082da101c0 [13063.947636] Call Trace: [13063.951316] [] sched_show_task+0xa8/0x110 [13063.959297] [] dump_cpu_task+0x39/0x70 [13063.966254] [] rcu_dump_cpu_stacks+0x90/0xd0 [13063.973779] [] rcu_check_callbacks+0x442/0x720 [13063.981468] [] ? tick_sched_handle.isra.13+0x60/0x60 [13063.989729] [] update_process_times+0x47/0x80 [13063.997282] [] tick_sched_handle.isra.13+0x25/0x60 [13064.005307] [] tick_sched_timer+0x41/0x70 [13064.012431] [] __hrtimer_run_queues+0xd2/0x260 [13064.020014] [] hrtimer_interrupt+0xb0/0x1e0 [13064.027281] [] ? call_softirq+0x1c/0x30 [13064.034140] [] local_apic_timer_interrupt+0x37/0x60 [13064.042160] [] smp_apic_timer_interrupt+0x3f/0x60 [13064.049989] [] apic_timer_interrupt+0x6d/0x80 [13064.057418] [] ? osd_scrub_exec+0xf7/0x5b0 [osd_ldiskfs] [13064.066652] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [13064.075549] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [13064.084440] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [13064.093049] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [13064.101450] [] ? wake_up_state+0x20/0x20 [13064.108401] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [13064.117487] [] kthread+0xcf/0xe0 [13064.123647] [] ? kthread_create_on_node+0x140/0x140 [13064.131658] [] ret_from_fork+0x58/0x90 [13064.138398] [] ? kthread_create_on_node+0x140/0x140 [13089.298448] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [13089.307873] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [13089.437773] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [13089.450950] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [13089.467018] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [13089.476307] RIP: 0010:[] [] osd_scrub_next+0x167/0x3c0 [osd_ldiskfs] [13089.487974] RSP: 0018:ffff880703603d00 EFLAGS: 00000282 [13089.494827] RAX: 0000000000000004 RBX: 0000000000000001 RCX: ffff880703603d98 [13089.503738] RDX: ffff880703603d78 RSI: 0000000000000000 RDI: ffff8803162f6000 [13089.512627] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [13089.521520] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [13089.530410] R13: 0000000000000018 R14: ffff880703603c78 R15: 0000000000000296 [13089.539296] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [13089.549249] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13089.556574] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [13089.565472] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13089.574369] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [13089.583248] Stack: [13089.586409] 0000000000000001 ffff880703603df0 ffffffffa121cff9 ffffffffa121b990 [13089.595641] ffffffffa1217a20 ffff8800b2a36000 00000000810d354f ffff8803162f6000 [13089.604867] ffff8800b2a37468 0000000020000000 010000000000000c 0000000000000000 [13089.614094] Call Trace: [13089.617753] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [13089.626872] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [13089.635987] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [13089.644811] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [13089.653455] [] ? wake_up_state+0x20/0x20 [13089.660627] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [13089.669931] [] kthread+0xcf/0xe0 [13089.676326] [] ? kthread_create_on_node+0x140/0x140 [13089.684576] [] ret_from_fork+0x58/0x90 [13089.691550] [] ? kthread_create_on_node+0x140/0x140 [13089.699791] Code: 48 89 df e8 fc 69 47 e0 b8 05 00 00 00 48 8b 75 d0 65 48 33 34 25 28 00 00 00 0f 85 4c 02 00 00 48 83 c4 70 5b 41 5c 41 5d 41 5e <41> 5f 5d c3 48 8b 05 be 58 9c ff 48 85 c0 0f 84 c2 fe ff ff 0f [13117.299422] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [13117.309070] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [13117.440718] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [13117.453993] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [13117.470224] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [13117.479602] RIP: 0010:[] [] osd_scrub_next+0x160/0x3c0 [osd_ldiskfs] [13117.491347] RSP: 0018:ffff880703603ce0 EFLAGS: 00000282 [13117.498275] RAX: 0000000000000004 RBX: ffff880703603c78 RCX: ffff880703603d98 [13117.507240] RDX: ffff880703603d78 RSI: 0000000000000000 RDI: ffff8803162f6000 [13117.516201] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [13117.525138] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000010 [13117.534073] R13: ffffffffa121ba87 R14: ffffffffffffff10 R15: ffff8803162f6000 [13117.543009] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [13117.553027] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13117.560434] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [13117.569393] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13117.578351] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [13117.587305] Stack: [13117.590518] ffff880419dc3a00 ffff8800b2a3746c 0000000000000000 00000000ffffffff [13117.599816] 0000000000000001 ffff880703603df0 ffffffffa121cff9 ffffffffa121b990 [13117.609117] ffffffffa1217a20 ffff8800b2a36000 00000000810d354f ffff8803162f6000 [13117.618412] Call Trace: [13117.622132] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [13117.631297] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [13117.640458] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [13117.649321] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [13117.657983] [] ? wake_up_state+0x20/0x20 [13117.665198] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [13117.674550] [] kthread+0xcf/0xe0 [13117.680977] [] ? kthread_create_on_node+0x140/0x140 [13117.689283] [] ret_from_fork+0x58/0x90 [13117.696299] [] ? kthread_create_on_node+0x140/0x140 [13117.704574] Code: 10 00 00 02 00 00 00 48 89 df e8 fc 69 47 e0 b8 05 00 00 00 48 8b 75 d0 65 48 33 34 25 28 00 00 00 0f 85 4c 02 00 00 48 83 c4 70 <5b> 41 5c 41 5d 41 5e 41 5f 5d c3 48 8b 05 be 58 9c ff 48 85 c0 [13145.300397] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [13145.310074] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [13145.441818] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [13145.455109] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [13145.471374] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [13145.480764] RIP: 0010:[] [] ldiskfs_itable_unused_count+0x6/0x30 [ldiskfs] [13145.493112] RSP: 0018:ffff880703603d08 EFLAGS: 00000286 [13145.500064] RAX: ffff8800b2a37468 RBX: 0000000000000001 RCX: 0000000000000000 [13145.509044] RDX: ffff880703603d78 RSI: ffff880419dc3a00 RDI: ffff88042bd79800 [13145.518030] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [13145.526985] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [13145.535927] R13: 0000000000000018 R14: ffff880703603c78 R15: 0000000000000282 [13145.544873] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [13145.554899] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13145.562310] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [13145.571278] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13145.580239] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [13145.589188] Stack: [13145.592407] ffff880703603df0 ffffffffa121cfb9 ffffffffa121b990 ffffffffa1217a20 [13145.601701] ffff8800b2a36000 00000000810d354f ffff8803162f6000 ffff8800b2a37468 [13145.610999] 0000000020000000 010000000000000c 0000000000000000 0000000000000000 [13145.620295] Call Trace: [13145.624031] [] ? osd_inode_iteration+0x269/0xcc0 [osd_ldiskfs] [13145.633389] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [13145.642548] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [13145.651418] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [13145.660079] [] ? wake_up_state+0x20/0x20 [13145.667287] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [13145.676640] [] kthread+0xcf/0xe0 [13145.683076] [] ? kthread_create_on_node+0x140/0x140 [13145.691359] [] ret_from_fork+0x58/0x90 [13145.698368] [] ? kthread_create_on_node+0x140/0x140 [13145.706642] Code: 00 00 31 c0 0f b7 56 10 48 89 e5 48 83 39 3f 76 07 0f b7 46 30 c1 e0 10 09 d0 5d c3 66 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 <48> 8b 8f a8 03 00 00 31 c0 0f b7 56 1c 48 89 e5 48 83 39 3f 76 [13173.301372] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [13173.311009] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [13173.442884] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [13173.456182] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [13173.472446] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [13173.481838] RIP: 0010:[] [] osd_scrub_exec+0xd4/0x5b0 [osd_ldiskfs] [13173.493500] RSP: 0018:ffff880703603c78 EFLAGS: 00000286 [13173.500439] RAX: ffff8803162f1008 RBX: ffff880703603c37 RCX: 0000000000000000 [13173.509423] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [13173.518386] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [13173.527335] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [13173.536281] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [13173.545232] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [13173.555265] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13173.562670] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [13173.571638] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13173.580597] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [13173.589546] Stack: [13173.592760] ffff880703603df0 ffff880703603d57 ffff880703603df0 000000009f8f2001 [13173.602054] ffff880703603d57 000000009f8f2001 0000000000000004 0000000000000001 [13173.611362] 0000000000000004 ffff880703603d98 ffff880703603d78 0000000000000000 [13173.620650] Call Trace: [13173.624373] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [13173.633555] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [13173.642717] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [13173.651582] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [13173.660244] [] ? wake_up_state+0x20/0x20 [13173.667452] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [13173.676804] [] kthread+0xcf/0xe0 [13173.683232] [] ? kthread_create_on_node+0x140/0x140 [13173.691523] [] ret_from_fork+0x58/0x90 [13173.698532] [] ? kthread_create_on_node+0x140/0x140 [13173.706798] Code: 4d 88 4c 89 ef e8 1d fb 46 e0 41 83 87 68 14 00 00 01 49 83 87 88 11 00 00 01 4c 89 ef e8 f5 9f e9 df 44 8b 4d 88 48 85 db 74 23 83 18 0a 00 00 08 74 1a 4d 85 e4 74 15 41 8b 87 6c 14 00 00 [13201.302348] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [13201.312026] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [13201.443757] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [13201.457044] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [13201.473293] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [13201.482676] RIP: 0010:[] [] osd_scrub_exec+0x3d/0x5b0 [osd_ldiskfs] [13201.494328] RSP: 0018:ffff880703603c78 EFLAGS: 00000286 [13201.501266] RAX: ffff8803162f1008 RBX: ffff880703603c37 RCX: 0000000000000000 [13201.510251] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [13201.519220] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [13201.528168] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [13201.537121] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [13201.546076] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [13201.556103] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13201.563508] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [13201.572475] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13201.581441] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [13201.590382] Stack: [13201.593589] ffff880703603df0 000000009f8f2001 ffff880703603df0 000000009f8f2001 [13201.602887] ffff880703603d57 000000009f8f2001 0000000000000004 ffff880703603d57 [13201.612170] ffff88042bd78800 ffff88042bd78800 00000000000072cf ffff880419dc3a00 [13201.621464] Call Trace: [13201.625198] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [13201.634358] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [13201.643514] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [13201.652379] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [13201.661040] [] ? wake_up_state+0x20/0x20 [13201.668247] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [13201.677587] [] kthread+0xcf/0xe0 [13201.684017] [] ? kthread_create_on_node+0x140/0x140 [13201.692293] [] ret_from_fork+0x58/0x90 [13201.699304] [] ? kthread_create_on_node+0x140/0x140 [13201.707564] Code: 41 54 45 31 e4 53 48 83 ec 68 48 8b 9e 08 10 00 00 4c 89 45 90 65 48 8b 04 25 28 00 00 00 48 89 45 d0 31 c0 48 8d 43 08 48 85 db <4c> 0f 45 e0 41 83 f9 04 0f 84 84 00 00 00 41 83 f9 07 74 4f 41 [13229.303323] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [13229.312997] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [13229.444770] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [13229.458062] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [13229.474300] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [13229.483691] RIP: 0010:[] [] osd_scrub_exec+0x1c/0x5b0 [osd_ldiskfs] [13229.495340] RSP: 0018:ffff880703603c78 EFLAGS: 00000296 [13229.502273] RAX: ffffffffa121b990 RBX: ffff880703603c37 RCX: 0000000000000000 [13229.511244] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [13229.520201] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [13229.529152] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [13229.538094] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [13229.547037] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [13229.557060] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13229.564460] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [13229.573431] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13229.582378] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [13229.591321] Stack: [13229.594525] ffff880703603d08 0000000000000001 ffffea00027e3c80 000000009f8f2001 [13229.603808] ffff880703603d57 ffff880703603d57 00000000000072ce ffff88042bd78800 [13229.613103] 00000000000072ce ffff880419dc3a00 ffff88042bd79800 ffffffffffffff10 [13229.622405] Call Trace: [13229.626125] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [13229.635300] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [13229.644459] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [13229.653319] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [13229.661987] [] ? wake_up_state+0x20/0x20 [13229.669194] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [13229.678541] [] kthread+0xcf/0xe0 [13229.684977] [] ? kthread_create_on_node+0x140/0x140 [13229.693274] [] ret_from_fork+0x58/0x90 [13229.700282] [] ? kthread_create_on_node+0x140/0x140 [13229.708554] Code: 89 04 24 31 c0 e8 a5 03 98 ff e9 c7 fd ff ff 66 66 66 66 90 55 48 89 e5 41 57 49 89 f7 41 56 41 55 41 54 45 31 e4 53 48 83 ec 68 <48> 8b 9e 08 10 00 00 4c 89 45 90 65 48 8b 04 25 28 00 00 00 48 [13243.902832] INFO: rcu_sched self-detected stall on CPU { 24} (t=5820117 jiffies g=907112 c=907111 q=16299401) [13243.915297] Task dump for CPU 24: [13243.920023] OI_scrub R running task 0 5551 2 0x00000088 [13243.928960] ffff88083fde6dd0 00000000fb6db880 ffff88082da03db0 ffffffff810c47e8 [13243.938277] 0000000000000018 ffffffff81a1e7c0 ffff88082da03dc8 ffffffff810c8079 [13243.947556] 0000000000000009 ffff88082da03df8 ffffffff811378e0 ffff88082da101c0 [13243.956839] Call Trace: [13243.960524] [] sched_show_task+0xa8/0x110 [13243.968518] [] dump_cpu_task+0x39/0x70 [13243.975483] [] rcu_dump_cpu_stacks+0x90/0xd0 [13243.983007] [] rcu_check_callbacks+0x442/0x720 [13243.990704] [] ? tick_sched_handle.isra.13+0x60/0x60 [13243.998972] [] update_process_times+0x47/0x80 [13244.006527] [] tick_sched_handle.isra.13+0x25/0x60 [13244.014540] [] tick_sched_timer+0x41/0x70 [13244.021669] [] __hrtimer_run_queues+0xd2/0x260 [13244.029257] [] hrtimer_interrupt+0xb0/0x1e0 [13244.036545] [] ? call_softirq+0x1c/0x30 [13244.043435] [] local_apic_timer_interrupt+0x37/0x60 [13244.051469] [] smp_apic_timer_interrupt+0x3f/0x60 [13244.059287] [] apic_timer_interrupt+0x6d/0x80 [13244.066710] [] ? osd_scrub_next+0x2b/0x3c0 [osd_ldiskfs] [13244.075956] [] ? osd_scrub_exec+0x8d/0x5b0 [osd_ldiskfs] [13244.084471] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [13244.093375] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [13244.102278] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [13244.110881] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [13244.119293] [] ? wake_up_state+0x20/0x20 [13244.126241] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [13244.135341] [] kthread+0xcf/0xe0 [13244.141521] [] ? kthread_create_on_node+0x140/0x140 [13244.149554] [] ret_from_fork+0x58/0x90 [13244.156305] [] ? kthread_create_on_node+0x140/0x140 [13269.304723] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [13269.314164] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [13269.443867] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [13269.457038] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [13269.473064] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [13269.482341] RIP: 0010:[] [] osd_scrub_exec+0x9/0x5b0 [osd_ldiskfs] [13269.493802] RSP: 0018:ffff880703603d08 EFLAGS: 00000293 [13269.500654] RAX: ffffffffa121b990 RBX: 000000009f8f2001 RCX: 0000000000000000 [13269.509547] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [13269.518448] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [13269.527349] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [13269.536242] R13: 0000000000000018 R14: ffff880703603c78 R15: 0000000000000282 [13269.545122] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [13269.555073] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13269.562404] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [13269.571300] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13269.580193] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [13269.589076] Stack: [13269.592229] ffff880703603df0 ffffffffa121d1e9 ffffffffa121b990 ffffffffa1217a20 [13269.601479] ffff8800b2a36000 00000000810d354f ffff8803162f6000 ffff8800b2a37468 [13269.610720] 0000000020000000 010000000000000c 0000000000000000 0000000000000000 [13269.619973] Call Trace: [13269.623651] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [13269.632780] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [13269.641907] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [13269.650739] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [13269.659379] [] ? wake_up_state+0x20/0x20 [13269.666544] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [13269.675845] [] kthread+0xcf/0xe0 [13269.682240] [] ? kthread_create_on_node+0x140/0x140 [13269.690484] [] ret_from_fork+0x58/0x90 [13269.697462] [] ? kthread_create_on_node+0x140/0x140 [13269.705687] Code: 00 00 00 02 00 89 54 24 08 48 8b 40 40 ba 07 00 00 00 48 89 04 24 31 c0 e8 a5 03 98 ff e9 c7 fd ff ff 66 66 66 66 90 55 48 89 e5 <41> 57 49 89 f7 41 56 41 55 41 54 45 31 e4 53 48 83 ec 68 48 8b [13297.305699] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [13297.315338] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [13297.447098] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [13297.460372] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [13297.476608] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [13297.485976] RIP: 0010:[] [] osd_scrub_next+0x350/0x3c0 [osd_ldiskfs] [13297.497714] RSP: 0018:ffff880703603c70 EFLAGS: 00000246 [13297.504641] RAX: 0000000000000000 RBX: ffff880703603c37 RCX: ffff880703603d98 [13297.513610] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [13297.522569] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [13297.531516] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [13297.540461] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [13297.549403] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [13297.559433] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13297.566837] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [13297.575801] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13297.584761] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [13297.593705] Stack: [13297.596924] 000000009f8f2001 ffff880703603d08 000000009f8f2001 ffff880703603df0 [13297.606222] 000000009f8f2001 ffff880703603d57 000000009f8f2001 0000000000000004 [13297.615527] ffff880703603d57 0000000000000000 0000000000000000 ffff880703603d78 [13297.624827] Call Trace: [13297.628542] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [13297.637702] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [13297.646857] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [13297.655714] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [13297.664369] [] ? wake_up_state+0x20/0x20 [13297.671572] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [13297.680913] [] kthread+0xcf/0xe0 [13297.687336] [] ? kthread_create_on_node+0x140/0x140 [13297.695614] [] ret_from_fork+0x58/0x90 [13297.702618] [] ? kthread_create_on_node+0x140/0x140 [13297.710882] Code: 97 ff 48 8b 55 90 48 8b 4d 88 44 8b 85 74 ff ff ff e9 1d ff ff ff 48 89 d7 44 89 45 90 e8 f9 67 47 e0 44 8b 45 90 0f 1f 44 00 00 <45> 84 c0 b8 04 00 00 00 0f 85 eb fd ff ff 49 8d b7 6c 14 00 00 [13325.306667] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [13325.316359] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [13325.448145] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [13325.461437] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [13325.477678] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [13325.487057] RIP: 0010:[] [] osd_scrub_exec+0x82/0x5b0 [osd_ldiskfs] [13325.498697] RSP: 0018:ffff880703603c78 EFLAGS: 00000246 [13325.505633] RAX: 0000000000000000 RBX: ffff880703603c37 RCX: 0000000000000000 [13325.514613] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [13325.523580] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [13325.532525] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [13325.541467] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [13325.550410] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [13325.560431] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13325.567831] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [13325.576789] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13325.585735] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [13325.594670] Stack: [13325.597870] ffff880703603d08 000000009f8f2001 ffff880703603df0 000000009f8f2001 [13325.607138] ffff880703603d57 000000009f8f2001 0000000000000004 0000000000000001 [13325.616433] 0000000000000004 ffff880703603d98 ffff880703603d78 0000000000000000 [13325.625721] Call Trace: [13325.629427] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [13325.638598] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [13325.647751] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [13325.656605] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [13325.665259] [] ? wake_up_state+0x20/0x20 [13325.672460] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [13325.681808] [] kthread+0xcf/0xe0 [13325.688231] [] ? kthread_create_on_node+0x140/0x140 [13325.696501] [] ret_from_fork+0x58/0x90 [13325.703504] [] ? kthread_create_on_node+0x140/0x140 [13325.711769] Code: 48 89 ca 44 89 c9 e8 0e c6 ff ff 85 c0 41 89 c1 0f 84 a3 01 00 00 41 80 a7 74 14 00 00 fe 48 8b 4d d0 65 48 33 0c 25 28 00 00 00 <0f> 85 b1 03 00 00 48 83 c4 68 5b 41 5c 41 5d 41 5e 41 5f 5d c3 [13353.307643] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [13353.317359] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [13353.449125] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [13353.462409] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [13353.478654] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [13353.488023] RIP: 0010:[] [] osd_inode_iteration+0x269/0xcc0 [osd_ldiskfs] [13353.500248] RSP: 0018:ffff880703603d18 EFLAGS: 00000202 [13353.507185] RAX: 00000000000072ce RBX: 000000009f8f2001 RCX: ffff88042bd78800 [13353.516165] RDX: 00000000000072ce RSI: ffff880419dc3a00 RDI: ffff88042bd79800 [13353.525121] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [13353.534068] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [13353.543012] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [13353.551961] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [13353.561982] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13353.569387] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [13353.578345] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13353.587290] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [13353.596229] Stack: [13353.599444] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [13353.608725] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [13353.618025] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [13353.627317] Call Trace: [13353.631039] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [13353.640209] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [13353.649077] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [13353.657751] [] ? wake_up_state+0x20/0x20 [13353.664955] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [13353.674309] [] kthread+0xcf/0xe0 [13353.680740] [] ? kthread_create_on_node+0x140/0x140 [13353.689024] [] ret_from_fork+0x58/0x90 [13353.696036] [] ? kthread_create_on_node+0x140/0x140 [13353.704314] Code: 88 e8 dc 2d f5 ff 48 85 c0 48 89 45 90 0f 84 b3 08 00 00 44 8b 7d a0 48 8b 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 e8 77 5b f4 ff <42> 8d 14 38 48 8b 45 88 48 8b 80 a8 03 00 00 48 3b 50 20 0f 83 [13373.553202] Lustre: soaked-MDT0000: Client e211cd09-6b0c-e43b-059c-f657101042a0 (at 192.168.1.117@o2ib100) reconnecting [13373.566529] Lustre: Skipped 1 previous similar message [13373.573360] Lustre: soaked-MDT0000: Connection restored to e211cd09-6b0c-e43b-059c-f657101042a0 (at 192.168.1.117@o2ib100) [13373.586773] Lustre: Skipped 1 previous similar message [13381.308618] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [13381.318309] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [13381.450048] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [13381.463337] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [13381.479591] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [13381.488987] RIP: 0010:[] [] osd_scrub_next+0x1b/0x3c0 [osd_ldiskfs] [13381.500653] RSP: 0018:ffff880703603ce0 EFLAGS: 00000287 [13381.507616] RAX: ffffffffa1217a20 RBX: ffff880703603c78 RCX: ffff880703603d98 [13381.516619] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [13381.525627] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [13381.534637] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000010 [13381.543644] R13: ffffffffa121ba64 R14: ffffffffffffff10 R15: ffff8803162f6000 [13381.552650] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [13381.562723] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13381.570173] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [13381.579182] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13381.588185] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [13381.597192] Stack: [13381.600451] ffff880419dc3a00 ffff8800b2a3746c 0000000000000000 00000000ffffffff [13381.609794] 0000000000000001 ffff880703603df0 ffffffffa121cff9 ffffffffa121b990 [13381.619138] ffffffffa1217a20 ffff8800b2a36000 00000000810d354f ffff8803162f6000 [13381.628497] Call Trace: [13381.632283] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [13381.641496] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [13381.650715] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [13381.659633] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [13381.668363] [] ? wake_up_state+0x20/0x20 [13381.675625] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [13381.685014] [] kthread+0xcf/0xe0 [13381.691495] [] ? kthread_create_on_node+0x140/0x140 [13381.699827] [] ret_from_fork+0x58/0x90 [13381.706892] [] ? kthread_create_on_node+0x140/0x140 [13381.715210] Code: 41 5c 41 5d 41 5e 5d c3 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 89 e5 41 57 49 89 f7 41 56 41 55 49 89 fd 41 54 49 89 d4 53 <48> 89 cb 48 83 ec 70 65 48 8b 04 25 28 00 00 00 48 89 45 d0 31 [13409.309601] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [13409.319277] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [13409.451128] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [13409.464420] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [13409.480677] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [13409.490066] RIP: 0010:[] [] osd_scrub_next+0x1e/0x3c0 [osd_ldiskfs] [13409.501719] RSP: 0018:ffff880703603ce0 EFLAGS: 00000287 [13409.508667] RAX: ffffffffa1217a20 RBX: ffff880703603c70 RCX: ffff880703603d98 [13409.517649] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [13409.526611] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [13409.535568] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000010 [13409.544512] R13: ffffffffa1217a4b R14: ffffffffffffff10 R15: ffff8803162f6000 [13409.553468] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [13409.563502] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13409.570913] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [13409.579877] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13409.588823] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [13409.597757] Stack: [13409.600968] ffff880419dc3a00 ffff8800b2a3746c 0000000000000000 00000000ffffffff [13409.610251] 0000000000000001 ffff880703603df0 ffffffffa121cff9 ffffffffa121b990 [13409.619561] ffffffffa1217a20 ffff8800b2a36000 00000000810d354f ffff8803162f6000 [13409.628878] Call Trace: [13409.632610] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [13409.641781] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [13409.650947] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [13409.659800] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [13409.668472] [] ? wake_up_state+0x20/0x20 [13409.675679] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [13409.685033] [] kthread+0xcf/0xe0 [13409.691470] [] ? kthread_create_on_node+0x140/0x140 [13409.699748] [] ret_from_fork+0x58/0x90 [13409.706765] [] ? kthread_create_on_node+0x140/0x140 [13409.715034] Code: 5d 41 5e 5d c3 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 89 e5 41 57 49 89 f7 41 56 41 55 49 89 fd 41 54 49 89 d4 53 48 89 cb <48> 83 ec 70 65 48 8b 04 25 28 00 00 00 48 89 45 d0 31 c0 48 8b [13423.912103] INFO: rcu_sched self-detected stall on CPU { 24} (t=6000120 jiffies g=907112 c=907111 q=17487651) [13423.924595] Task dump for CPU 24: [13423.929321] OI_scrub R running task 0 5551 2 0x00000088 [13423.938250] ffff88083fde6dd0 00000000fb6db880 ffff88082da03db0 ffffffff810c47e8 [13423.947574] 0000000000000018 ffffffff81a1e7c0 ffff88082da03dc8 ffffffff810c8079 [13423.956874] 0000000000000009 ffff88082da03df8 ffffffff811378e0 ffff88082da101c0 [13423.966156] Call Trace: [13423.969836] [] sched_show_task+0xa8/0x110 [13423.977809] [] dump_cpu_task+0x39/0x70 [13423.984763] [] rcu_dump_cpu_stacks+0x90/0xd0 [13423.992281] [] rcu_check_callbacks+0x442/0x720 [13423.999970] [] ? tick_sched_handle.isra.13+0x60/0x60 [13424.008224] [] update_process_times+0x47/0x80 [13424.015781] [] tick_sched_handle.isra.13+0x25/0x60 [13424.023795] [] tick_sched_timer+0x41/0x70 [13424.030918] [] __hrtimer_run_queues+0xd2/0x260 [13424.038508] [] hrtimer_interrupt+0xb0/0x1e0 [13424.045790] [] ? call_softirq+0x1c/0x30 [13424.052672] [] local_apic_timer_interrupt+0x37/0x60 [13424.060697] [] smp_apic_timer_interrupt+0x3f/0x60 [13424.068510] [] apic_timer_interrupt+0x6d/0x80 [13424.075940] [] ? osd_scrub_exec+0xd4/0x5b0 [osd_ldiskfs] [13424.085186] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [13424.094091] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [13424.102998] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [13424.111613] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [13424.120015] [] ? wake_up_state+0x20/0x20 [13424.126959] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [13424.136044] [] kthread+0xcf/0xe0 [13424.142211] [] ? kthread_create_on_node+0x140/0x140 [13424.150232] [] ret_from_fork+0x58/0x90 [13424.156982] [] ? kthread_create_on_node+0x140/0x140 [13449.310988] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [13449.320402] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [13449.450001] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [13449.463183] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [13449.479246] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [13449.488543] RIP: 0010:[] [] osd_inode_iteration+0x473/0xcc0 [osd_ldiskfs] [13449.500690] RSP: 0018:ffff880703603d18 EFLAGS: 00000293 [13449.507546] RAX: 0000000000000004 RBX: 000000009f8f2001 RCX: ffff880703603d98 [13449.516456] RDX: ffff880703603d78 RSI: 0000000000000000 RDI: ffff8803162f6000 [13449.525347] RBP: ffff880703603df0 R08: 0000000000000001 R09: 0000000000000004 [13449.534235] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000001 [13449.543133] R13: ffff880703603d08 R14: 000000009f8f2001 R15: ffffea00027e3c80 [13449.552018] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [13449.561984] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13449.569325] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [13449.578222] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13449.587118] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [13449.596012] Stack: [13449.599171] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [13449.608409] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [13449.617663] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [13449.626919] Call Trace: [13449.630607] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [13449.639726] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [13449.648567] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [13449.657220] [] ? wake_up_state+0x20/0x20 [13449.664406] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [13449.673727] [] kthread+0xcf/0xe0 [13449.680122] [] ? kthread_create_on_node+0x140/0x140 [13449.688375] [] ret_from_fork+0x58/0x90 [13449.695355] [] ? kthread_create_on_node+0x140/0x140 [13449.703596] Code: 00 00 48 c7 05 9b 86 02 00 00 00 00 00 c7 05 89 86 02 00 01 00 00 00 e8 7c eb 97 ff e9 0f fc ff ff 0f 1f 80 00 00 00 00 45 89 e9 <4c> 8d 85 67 ff ff ff 48 8b 4d a8 48 8d 55 88 48 8b b5 38 ff ff [13459.556421] Lustre: 3694:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1494366804/real 1494366804] req@ffff88028d932d00 x1566943502434336/t0(0) o104->soaked-MDT0000@192.168.1.118@o2ib100:15/16 lens 296/224 e 0 to 1 dl 1494366811 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [13477.311965] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [13477.321663] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [13477.453686] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [13477.466998] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [13477.483303] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [13477.492717] RIP: 0010:[] [] osd_inode_iteration+0x47a/0xcc0 [osd_ldiskfs] [13477.504990] RSP: 0018:ffff880703603d18 EFLAGS: 00000293 [13477.511972] RAX: 0000000000000004 RBX: 000000009f8f2001 RCX: ffff880703603d98 [13477.520990] RDX: ffff880703603d78 RSI: 0000000000000000 RDI: ffff8803162f6000 [13477.530001] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [13477.539010] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [13477.548030] R13: ffff880703603d08 R14: 000000009f8f2001 R15: ffff880703603d08 [13477.557045] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [13477.567136] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13477.574603] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [13477.583621] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13477.592631] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [13477.601636] Stack: [13477.604924] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [13477.614278] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [13477.623625] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [13477.632970] Call Trace: [13477.636758] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [13477.645984] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [13477.654905] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [13477.663634] [] ? wake_up_state+0x20/0x20 [13477.670901] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [13477.680312] [] kthread+0xcf/0xe0 [13477.686802] [] ? kthread_create_on_node+0x140/0x140 [13477.695136] [] ret_from_fork+0x58/0x90 [13477.702213] [] ? kthread_create_on_node+0x140/0x140 [13477.710536] Code: 02 00 00 00 00 00 c7 05 89 86 02 00 01 00 00 00 e8 7c eb 97 ff e9 0f fc ff ff 0f 1f 80 00 00 00 00 45 89 e9 4c 8d 85 67 ff ff ff <48> 8b 4d a8 48 8d 55 88 48 8b b5 38 ff ff ff 48 8b bd 48 ff ff [13505.312939] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [13505.322653] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [13505.454675] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [13505.467986] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [13505.484289] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [13505.493698] RIP: 0010:[] [] osd_scrub_next+0x3d/0x3c0 [osd_ldiskfs] [13505.505375] RSP: 0018:ffff880703603c70 EFLAGS: 00000246 [13505.512337] RAX: 0000000000000000 RBX: ffff880703603c37 RCX: ffff880703603d98 [13505.521326] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [13505.530305] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [13505.539278] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [13505.548234] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [13505.557194] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [13505.567223] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13505.574632] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [13505.583602] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13505.592567] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [13505.601516] Stack: [13505.604735] 000000009f8f2001 ffff880703603d08 0000000000000001 ffff880703603df0 [13505.614007] 000000009f8f2001 ffff880703603d57 000000009f8f2001 0000000000000004 [13505.623299] ffff880703603d57 ffff88042bd78800 ffff88042bd78800 00000000000072cf [13505.632599] Call Trace: [13505.636320] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [13505.645482] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [13505.654646] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [13505.663511] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [13505.672174] [] ? wake_up_state+0x20/0x20 [13505.679375] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [13505.688714] [] kthread+0xcf/0xe0 [13505.695138] [] ? kthread_create_on_node+0x140/0x140 [13505.703418] [] ret_from_fork+0x58/0x90 [13505.710421] [] ? kthread_create_on_node+0x140/0x140 [13505.718688] Code: 49 89 fd 41 54 49 89 d4 53 48 89 cb 48 83 ec 70 65 48 8b 04 25 28 00 00 00 48 89 45 d0 31 c0 48 8b 05 f8 59 9c ff 48 85 c0 75 7b <41> f6 87 58 10 00 00 08 0f 84 62 03 00 00 4d 8d b7 e0 10 00 00 [13533.313914] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [13533.323577] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [13533.455299] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [13533.468581] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [13533.484813] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [13533.494192] RIP: 0010:[] [] osd_inode_iteration+0x47a/0xcc0 [osd_ldiskfs] [13533.506417] RSP: 0018:ffff880703603d18 EFLAGS: 00000293 [13533.513354] RAX: 0000000000000004 RBX: 000000009f8f2001 RCX: ffff880703603d98 [13533.522329] RDX: ffff880703603d78 RSI: 0000000000000000 RDI: ffff8803162f6000 [13533.531291] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [13533.540240] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [13533.549185] R13: ffff880703603d08 R14: 000000009f8f2001 R15: ffff880703603d08 [13533.558132] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [13533.568159] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13533.575563] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [13533.584529] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13533.593473] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [13533.602413] Stack: [13533.605617] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [13533.614903] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [13533.624223] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [13533.633503] Call Trace: [13533.637217] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [13533.646390] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [13533.655264] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [13533.663936] [] ? wake_up_state+0x20/0x20 [13533.671137] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [13533.680483] [] kthread+0xcf/0xe0 [13533.686917] [] ? kthread_create_on_node+0x140/0x140 [13533.695198] [] ret_from_fork+0x58/0x90 [13533.702223] [] ? kthread_create_on_node+0x140/0x140 [13533.710489] Code: 02 00 00 00 00 00 c7 05 89 86 02 00 01 00 00 00 e8 7c eb 97 ff e9 0f fc ff ff 0f 1f 80 00 00 00 00 45 89 e9 4c 8d 85 67 ff ff ff <48> 8b 4d a8 48 8d 55 88 48 8b b5 38 ff ff ff 48 8b bd 48 ff ff [13561.314889] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [13561.324557] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [13561.456559] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [13561.469867] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [13561.486154] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [13561.495562] RIP: 0010:[] [] ldiskfs_used_dirs_count+0x30/0x30 [ldiskfs] [13561.507627] RSP: 0018:ffff880703603d10 EFLAGS: 00000286 [13561.514592] RAX: ffff8800b2a37468 RBX: ffff880703603d57 RCX: 0000000000000000 [13561.523583] RDX: ffff880703603d78 RSI: ffff880419dc3a00 RDI: ffff88042bd79800 [13561.532565] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [13561.541533] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [13561.550490] R13: ffffea00027e3c80 R14: 000000009f8f2001 R15: 0000000000000018 [13561.559440] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [13561.569476] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13561.576882] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [13561.585849] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13561.594809] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [13561.603755] Stack: [13561.606963] ffffffffa121cfb9 ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 [13561.616261] 00000000810d354f ffff8803162f6000 ffff8800b2a37468 0000000020000000 [13561.625556] 010000000000000c 0000000000000000 0000000000000000 ffff8800b2a36000 [13561.634858] Call Trace: [13561.638603] [] ? osd_inode_iteration+0x269/0xcc0 [osd_ldiskfs] [13561.647967] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [13561.657133] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [13561.666009] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [13561.674685] [] ? wake_up_state+0x20/0x20 [13561.681894] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [13561.691236] [] kthread+0xcf/0xe0 [13561.697677] [] ? kthread_create_on_node+0x140/0x140 [13561.705958] [] ret_from_fork+0x58/0x90 [13561.712970] [] ? kthread_create_on_node+0x140/0x140 [13561.721246] Code: 55 48 8b 8f a8 03 00 00 31 c0 0f b7 56 10 48 89 e5 48 83 39 3f 76 07 0f b7 46 30 c1 e0 10 09 d0 5d c3 66 0f 1f 84 00 00 00 00 00 <66> 66 66 66 90 55 48 8b 8f a8 03 00 00 31 c0 0f b7 56 1c 48 89 [13589.315865] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [13589.325550] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [13589.457328] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [13589.470619] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [13589.486883] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [13589.496272] RIP: 0010:[] [] osd_scrub_next+0x3d/0x3c0 [osd_ldiskfs] [13589.507922] RSP: 0018:ffff880703603c70 EFLAGS: 00000246 [13589.514865] RAX: 0000000000000000 RBX: ffff880703603c37 RCX: ffff880703603d98 [13589.523847] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [13589.532819] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [13589.541774] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [13589.550720] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [13589.559665] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [13589.569700] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13589.577122] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [13589.586098] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13589.595060] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [13589.604010] Stack: [13589.607227] 000000009f8f2001 ffff880703603d08 ffff880703603d57 ffffea00027e3c80 [13589.616522] 000000009f8f2001 ffff880703603d57 ffff880703603d57 ffffffffa121b990 [13589.625834] 0000000000000000 ffff880703603d78 ffff8800b2a36000 ffff8803162f6000 [13589.635126] Call Trace: [13589.638846] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [13589.648005] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [13589.657164] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [13589.666308] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [13589.675152] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [13589.683802] [] ? wake_up_state+0x20/0x20 [13589.690996] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [13589.700327] [] kthread+0xcf/0xe0 [13589.706741] [] ? kthread_create_on_node+0x140/0x140 [13589.715014] [] ret_from_fork+0x58/0x90 [13589.722005] [] ? kthread_create_on_node+0x140/0x140 [13589.730259] Code: 49 89 fd 41 54 49 89 d4 53 48 89 cb 48 83 ec 70 65 48 8b 04 25 28 00 00 00 48 89 45 d0 31 c0 48 8b 05 f8 59 9c ff 48 85 c0 75 7b <41> f6 87 58 10 00 00 08 0f 84 62 03 00 00 4d 8d b7 e0 10 00 00 [13603.921386] INFO: rcu_sched self-detected stall on CPU { 24} (t=6180123 jiffies g=907112 c=907111 q=19283214) [13603.933835] Task dump for CPU 24: [13603.938544] OI_scrub R running task 0 5551 2 0x00000088 [13603.947461] ffff88083fde6dd0 00000000fb6db880 ffff88082da03db0 ffffffff810c47e8 [13603.956762] 0000000000000018 ffffffff81a1e7c0 ffff88082da03dc8 ffffffff810c8079 [13603.966040] 0000000000000009 ffff88082da03df8 ffffffff811378e0 ffff88082da101c0 [13603.975285] Call Trace: [13603.978940] [] sched_show_task+0xa8/0x110 [13603.986898] [] dump_cpu_task+0x39/0x70 [13603.993828] [] rcu_dump_cpu_stacks+0x90/0xd0 [13604.001319] [] rcu_check_callbacks+0x442/0x720 [13604.008982] [] ? tick_sched_handle.isra.13+0x60/0x60 [13604.017206] [] update_process_times+0x47/0x80 [13604.024726] [] tick_sched_handle.isra.13+0x25/0x60 [13604.032713] [] tick_sched_timer+0x41/0x70 [13604.039806] [] __hrtimer_run_queues+0xd2/0x260 [13604.047363] [] hrtimer_interrupt+0xb0/0x1e0 [13604.054626] [] ? call_softirq+0x1c/0x30 [13604.061486] [] local_apic_timer_interrupt+0x37/0x60 [13604.069508] [] smp_apic_timer_interrupt+0x3f/0x60 [13604.077329] [] apic_timer_interrupt+0x6d/0x80 [13604.084745] [] ? osd_inode_iteration+0x489/0xcc0 [osd_ldiskfs] [13604.094561] [] ? osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [13604.103652] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [13604.112549] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [13604.121144] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [13604.129537] [] ? wake_up_state+0x20/0x20 [13604.136479] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [13604.145568] [] kthread+0xcf/0xe0 [13604.151736] [] ? kthread_create_on_node+0x140/0x140 [13604.159754] [] ret_from_fork+0x58/0x90 [13604.166495] [] ? kthread_create_on_node+0x140/0x140 [13629.317264] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [13629.326675] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [13629.456991] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [13629.470172] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [13629.486251] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [13629.495559] RIP: 0010:[] [] osd_inode_iteration+0x4ac/0xcc0 [osd_ldiskfs] [13629.507710] RSP: 0018:ffff880703603d18 EFLAGS: 00000216 [13629.514579] RAX: 0000000020000000 RBX: 000000009f8f2001 RCX: 0000000000000000 [13629.523495] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [13629.532404] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [13629.541317] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffff880703603d57 [13629.550228] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [13629.559136] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [13629.569123] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13629.576481] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [13629.585403] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13629.594332] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [13629.603249] Stack: [13629.606431] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [13629.615694] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [13629.624949] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [13629.634214] Call Trace: [13629.637905] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [13629.647054] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [13629.655903] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [13629.664572] [] ? wake_up_state+0x20/0x20 [13629.671779] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [13629.681120] [] kthread+0xcf/0xe0 [13629.687533] [] ? kthread_create_on_node+0x140/0x140 [13629.695799] [] ret_from_fork+0x58/0x90 [13629.702792] [] ? kthread_create_on_node+0x140/0x140 [13629.711048] Code: 88 48 8b b5 38 ff ff ff 48 8b bd 48 ff ff ff 48 8b 85 28 ff ff ff ff d0 85 c0 41 89 c5 75 1c 8b 85 58 ff ff ff 41 3b 04 24 72 10 <48> 8b 85 50 ff ff ff 44 3b 30 0f 87 95 fd ff ff 45 31 ff e9 a2 [13657.318239] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [13657.327886] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [13657.459789] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [13657.473106] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [13657.489431] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [13657.498837] RIP: 0010:[] [] osd_inode_iteration+0x4a0/0xcc0 [osd_ldiskfs] [13657.511108] RSP: 0018:ffff880703603d18 EFLAGS: 00000246 [13657.518081] RAX: 0000000000000000 RBX: 000000009f8f2001 RCX: 0000000000000000 [13657.527080] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [13657.536063] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [13657.545038] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [13657.554001] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [13657.562961] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [13657.572996] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13657.580411] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [13657.589391] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13657.598363] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [13657.607322] Stack: [13657.610542] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [13657.619836] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [13657.629124] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [13657.638432] Call Trace: [13657.642158] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [13657.651336] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [13657.660214] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [13657.668892] [] ? wake_up_state+0x20/0x20 [13657.676112] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [13657.685462] [] kthread+0xcf/0xe0 [13657.691896] [] ? kthread_create_on_node+0x140/0x140 [13657.700180] [] ret_from_fork+0x58/0x90 [13657.707203] [] ? kthread_create_on_node+0x140/0x140 [13657.715486] Code: 85 67 ff ff ff 48 8b 4d a8 48 8d 55 88 48 8b b5 38 ff ff ff 48 8b bd 48 ff ff ff 48 8b 85 28 ff ff ff ff d0 85 c0 41 89 c5 75 1c <8b> 85 58 ff ff ff 41 3b 04 24 72 10 48 8b 85 50 ff ff ff 44 3b [13685.319213] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [13685.328834] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [13685.460808] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [13685.474115] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [13685.490398] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [13685.499813] RIP: 0010:[] [] osd_scrub_exec+0x95/0x5b0 [osd_ldiskfs] [13685.511494] RSP: 0018:ffff880703603d08 EFLAGS: 00000292 [13685.518460] RAX: 0000000000000000 RBX: 0000000000000001 RCX: 0000000000000000 [13685.527448] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [13685.536430] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [13685.545389] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [13685.554348] R13: 0000000000000018 R14: ffff880703603c78 R15: 0000000000000246 [13685.563294] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [13685.573306] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13685.580707] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [13685.589667] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13685.598625] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [13685.607567] Stack: [13685.610774] ffff880703603df0 ffffffffa121d1e9 ffffffffa121b990 ffffffffa1217a20 [13685.620056] ffff8800b2a36000 00000000810d354f ffff8803162f6000 ffff8800b2a37468 [13685.629345] 0000000020000000 010000000000000c 0000000000000000 0000000000000000 [13685.638639] Call Trace: [13685.642364] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [13685.651520] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [13685.660678] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [13685.669546] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [13685.678209] [] ? wake_up_state+0x20/0x20 [13685.685410] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [13685.694758] [] kthread+0xcf/0xe0 [13685.701182] [] ? kthread_create_on_node+0x140/0x140 [13685.709467] [] ret_from_fork+0x58/0x90 [13685.716474] [] ? kthread_create_on_node+0x140/0x140 [13685.724736] Code: 01 00 00 41 80 a7 74 14 00 00 fe 48 8b 4d d0 65 48 33 0c 25 28 00 00 00 0f 85 b1 03 00 00 48 83 c4 68 5b 41 5c 41 5d 41 5e 41 5f <5d> c3 66 0f 1f 84 00 00 00 00 00 4c 8d ae f0 10 00 00 44 89 4d [13713.320184] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [13713.329789] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [13713.461453] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [13713.474730] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [13713.490960] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [13713.500334] RIP: 0010:[] [] osd_scrub_exec+0x96/0x5b0 [osd_ldiskfs] [13713.511973] RSP: 0018:ffff880703603d10 EFLAGS: 00000292 [13713.518905] RAX: 0000000000000000 RBX: 000000009f8f2001 RCX: 0000000000000000 [13713.527872] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [13713.536827] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [13713.545771] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [13713.554718] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [13713.563659] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [13713.573683] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13713.581079] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [13713.590040] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13713.599002] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [13713.607943] Stack: [13713.611150] ffffffffa121d1e9 ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 [13713.620441] 00000000810d354f ffff8803162f6000 ffff8800b2a37468 0000000020000000 [13713.629747] 010000000000000c 0000000000000000 0000000000000000 ffff8800b2a36000 [13713.639037] Call Trace: [13713.642754] [] ? osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [13713.652113] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [13713.661278] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [13713.670142] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [13713.678813] [] ? wake_up_state+0x20/0x20 [13713.686023] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [13713.695365] [] kthread+0xcf/0xe0 [13713.701798] [] ? kthread_create_on_node+0x140/0x140 [13713.710077] [] ret_from_fork+0x58/0x90 [13713.717090] [] ? kthread_create_on_node+0x140/0x140 [13713.725371] Code: 00 00 41 80 a7 74 14 00 00 fe 48 8b 4d d0 65 48 33 0c 25 28 00 00 00 0f 85 b1 03 00 00 48 83 c4 68 5b 41 5c 41 5d 41 5e 41 5f 5d 66 0f 1f 84 00 00 00 00 00 4c 8d ae f0 10 00 00 44 89 4d 88 [13741.321167] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [13741.330815] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [13741.462640] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [13741.475932] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [13741.492197] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [13741.501584] RIP: 0010:[] [] osd_scrub_exec+0x1c/0x5b0 [osd_ldiskfs] [13741.513250] RSP: 0018:ffff880703603c78 EFLAGS: 00000296 [13741.520197] RAX: ffffffffa121b990 RBX: ffff880703603c37 RCX: 0000000000000000 [13741.529179] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [13741.538153] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [13741.547115] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [13741.556068] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [13741.565019] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [13741.575057] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13741.582466] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [13741.591444] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13741.600412] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [13741.609361] Stack: [13741.612582] ffffea00027e3c80 000000009f8f2001 0000000000000004 0000000000000001 [13741.621881] ffff880703603d57 ffff880703603d98 ffff880703603d78 ffff8800b2a36000 [13741.631189] ffff8803162f6000 ffffffffffffff10 ffffffffa1217a32 0000000000000010 [13741.640496] Call Trace: [13741.644220] [] ? osd_scrub_next+0x12/0x3c0 [osd_ldiskfs] [13741.653013] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [13741.662185] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [13741.671345] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [13741.680208] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [13741.688888] [] ? wake_up_state+0x20/0x20 [13741.696098] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [13741.705461] [] kthread+0xcf/0xe0 [13741.711900] [] ? kthread_create_on_node+0x140/0x140 [13741.720178] [] ret_from_fork+0x58/0x90 [13741.727195] [] ? kthread_create_on_node+0x140/0x140 [13741.735474] Code: 89 04 24 31 c0 e8 a5 03 98 ff e9 c7 fd ff ff 66 66 66 66 90 55 48 89 e5 41 57 49 89 f7 41 56 41 55 41 54 45 31 e4 53 48 83 ec 68 <48> 8b 9e 08 10 00 00 4c 89 45 90 65 48 8b 04 25 28 00 00 00 48 [13769.322143] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [13769.331792] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [13769.463570] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [13769.476842] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [13769.493081] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [13769.502452] RIP: 0010:[] [] osd_scrub_exec+0x8d/0x5b0 [osd_ldiskfs] [13769.514095] RSP: 0018:ffff880703603ce8 EFLAGS: 00000292 [13769.521031] RAX: 0000000000000000 RBX: ffff880703603c78 RCX: 0000000000000000 [13769.530004] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [13769.538968] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [13769.547921] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000010 [13769.556870] R13: ffffffffa121ba87 R14: ffffffffffffff10 R15: ffff8803162f6000 [13769.565829] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [13769.575856] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13769.583270] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [13769.592233] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13769.601192] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [13769.610131] Stack: [13769.613358] ffff8800b2a3746c 0000000000000004 00000000ffffffff 0000000000000001 [13769.622659] ffff880703603df0 ffffffffa121d1e9 ffffffffa121b990 ffffffffa1217a20 [13769.631961] ffff8800b2a36000 00000000810d354f ffff8803162f6000 ffff8800b2a37468 [13769.641248] Call Trace: [13769.644974] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [13769.654148] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [13769.663303] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [13769.672161] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [13769.680830] [] ? wake_up_state+0x20/0x20 [13769.688046] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [13769.697392] [] kthread+0xcf/0xe0 [13769.703822] [] ? kthread_create_on_node+0x140/0x140 [13769.712111] [] ret_from_fork+0x58/0x90 [13769.719128] [] ? kthread_create_on_node+0x140/0x140 [13769.727397] Code: 85 c0 41 89 c1 0f 84 a3 01 00 00 41 80 a7 74 14 00 00 fe 48 8b 4d d0 65 48 33 0c 25 28 00 00 00 0f 85 b1 03 00 00 48 83 c4 68 5b <41> 5c 41 5d 41 5e 41 5f 5d c3 66 0f 1f 84 00 00 00 00 00 4c 8d [13783.930657] INFO: rcu_sched self-detected stall on CPU { 24} (t=6360126 jiffies g=907112 c=907111 q=19635298) [13783.943127] Task dump for CPU 24: [13783.947863] OI_scrub R running task 0 5551 2 0x00000088 [13783.956793] ffff88083fde6dd0 00000000fb6db880 ffff88082da03db0 ffffffff810c47e8 [13783.966125] 0000000000000018 ffffffff81a1e7c0 ffff88082da03dc8 ffffffff810c8079 [13783.975421] 0000000000000009 ffff88082da03df8 ffffffff811378e0 ffff88082da101c0 [13783.984707] Call Trace: [13783.988403] [] sched_show_task+0xa8/0x110 [13783.996378] [] dump_cpu_task+0x39/0x70 [13784.003348] [] rcu_dump_cpu_stacks+0x90/0xd0 [13784.010870] [] rcu_check_callbacks+0x442/0x720 [13784.018568] [] ? tick_sched_handle.isra.13+0x60/0x60 [13784.026818] [] update_process_times+0x47/0x80 [13784.034371] [] tick_sched_handle.isra.13+0x25/0x60 [13784.042383] [] tick_sched_timer+0x41/0x70 [13784.049518] [] __hrtimer_run_queues+0xd2/0x260 [13784.057114] [] hrtimer_interrupt+0xb0/0x1e0 [13784.064394] [] ? call_softirq+0x1c/0x30 [13784.071276] [] local_apic_timer_interrupt+0x37/0x60 [13784.079306] [] smp_apic_timer_interrupt+0x3f/0x60 [13784.087124] [] apic_timer_interrupt+0x6d/0x80 [13784.094550] [] ? osd_inode_iteration+0x2be/0xcc0 [osd_ldiskfs] [13784.104370] [] ? osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [13784.113476] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [13784.122383] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [13784.130998] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [13784.139415] [] ? wake_up_state+0x20/0x20 [13784.146367] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [13784.155463] [] kthread+0xcf/0xe0 [13784.161639] [] ? kthread_create_on_node+0x140/0x140 [13784.169664] [] ret_from_fork+0x58/0x90 [13784.176425] [] ? kthread_create_on_node+0x140/0x140 [13809.323529] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [13809.332951] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [13809.462700] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [13809.475886] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [13809.491957] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [13809.501261] RIP: 0010:[] [] osd_inode_iteration+0x2be/0xcc0 [osd_ldiskfs] [13809.513404] RSP: 0018:ffff880703603d18 EFLAGS: 00000297 [13809.520265] RAX: 0000000000000004 RBX: 000000009f8f2001 RCX: ffff880703603d98 [13809.529172] RDX: ffff880703603d78 RSI: 0000000000000000 RDI: ffff8803162f6000 [13809.538087] RBP: ffff880703603df0 R08: 0000000000000001 R09: 0000000000000004 [13809.546990] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [13809.555881] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [13809.564775] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [13809.574740] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13809.582084] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [13809.590988] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13809.599893] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [13809.608788] Stack: [13809.611961] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [13809.621200] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [13809.630455] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [13809.639708] Call Trace: [13809.643382] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [13809.652515] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [13809.661365] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [13809.670020] [] ? wake_up_state+0x20/0x20 [13809.677211] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [13809.686540] [] kthread+0xcf/0xe0 [13809.692946] [] ? kthread_create_on_node+0x140/0x140 [13809.701196] [] ret_from_fork+0x58/0x90 [13809.708185] [] ? kthread_create_on_node+0x140/0x140 [13809.716432] Code: 8b b5 38 ff ff ff 48 8b bd 48 ff ff ff 48 8b 85 30 ff ff ff ff d0 83 f8 03 41 89 c5 0f 84 4b 02 00 00 0f 8e 15 01 00 00 83 f8 05 <66> 90 0f 84 8a 02 00 00 83 f8 06 0f 85 a1 01 00 00 48 8b 7d 90 [13837.324505] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [13837.334146] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [13837.466086] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [13837.479383] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [13837.495670] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [13837.505073] RIP: 0010:[] [] osd_scrub_next+0x3d/0x3c0 [osd_ldiskfs] [13837.516743] RSP: 0018:ffff880703603c70 EFLAGS: 00000246 [13837.523701] RAX: 0000000000000000 RBX: ffff880703603c37 RCX: ffff880703603d98 [13837.532685] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [13837.541660] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [13837.550626] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [13837.559578] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [13837.568534] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [13837.578561] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13837.585975] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [13837.594943] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13837.603913] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [13837.612870] Stack: [13837.616084] 000000009f8f2001 ffff880703603d08 0000000000000001 ffff880703603df0 [13837.625376] 000000009f8f2001 ffff880703603d57 000000009f8f2001 0000000000000004 [13837.634659] ffff880703603d57 0000000000000004 0000000000000000 ffff880703603d78 [13837.643972] Call Trace: [13837.647702] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [13837.656865] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [13837.666024] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [13837.674891] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [13837.683565] [] ? wake_up_state+0x20/0x20 [13837.690768] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [13837.700112] [] kthread+0xcf/0xe0 [13837.706544] [] ? kthread_create_on_node+0x140/0x140 [13837.714817] [] ret_from_fork+0x58/0x90 [13837.721822] [] ? kthread_create_on_node+0x140/0x140 [13837.730092] Code: 49 89 fd 41 54 49 89 d4 53 48 89 cb 48 83 ec 70 65 48 8b 04 25 28 00 00 00 48 89 45 d0 31 c0 48 8b 05 f8 59 9c ff 48 85 c0 75 7b <41> f6 87 58 10 00 00 08 0f 84 62 03 00 00 4d 8d b7 e0 10 00 00 [13865.325479] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [13865.335106] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [13865.466818] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [13865.480106] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [13865.496347] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [13865.505727] RIP: 0010:[] [] osd_scrub_exec+0xd4/0x5b0 [osd_ldiskfs] [13865.517374] RSP: 0018:ffff880703603c78 EFLAGS: 00000286 [13865.524313] RAX: ffff8803162f1008 RBX: ffff880703603c37 RCX: 0000000000000000 [13865.533286] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [13865.542254] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [13865.551200] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [13865.560138] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [13865.569075] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [13865.579100] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13865.586501] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [13865.595464] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13865.604415] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [13865.613354] Stack: [13865.616568] ffff880703603d08 0000000000000001 ffff880703603df0 000000009f8f2001 [13865.625855] ffff880703603d57 000000009f8f2001 0000000000000004 ffff880703603d57 [13865.635140] ffff8800b2a37468 0000000000000000 ffff880703603d78 ffff8800b2a36000 [13865.644420] Call Trace: [13865.648130] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [13865.657297] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [13865.666452] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [13865.675310] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [13865.683963] [] ? wake_up_state+0x20/0x20 [13865.691168] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [13865.700517] [] kthread+0xcf/0xe0 [13865.706943] [] ? kthread_create_on_node+0x140/0x140 [13865.715215] [] ret_from_fork+0x58/0x90 [13865.722219] [] ? kthread_create_on_node+0x140/0x140 [13865.730481] Code: 4d 88 4c 89 ef e8 1d fb 46 e0 41 83 87 68 14 00 00 01 49 83 87 88 11 00 00 01 4c 89 ef e8 f5 9f e9 df 44 8b 4d 88 48 85 db 74 23 83 18 0a 00 00 08 74 1a 4d 85 e4 74 15 41 8b 87 6c 14 00 00 [13893.326455] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [13893.336147] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [13893.467888] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [13893.481171] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [13893.497417] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [13893.506807] RIP: 0010:[] [] osd_inode_iteration+0x255/0xcc0 [osd_ldiskfs] [13893.519035] RSP: 0018:ffff880703603d18 EFLAGS: 00000286 [13893.525983] RAX: ffff8800b2a37468 RBX: 000000009f8f2001 RCX: 0000000000000000 [13893.534961] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [13893.543931] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [13893.552888] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [13893.561834] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [13893.570789] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [13893.580822] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13893.588226] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [13893.597198] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13893.606151] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [13893.615091] Stack: [13893.618297] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [13893.627598] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [13893.636877] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [13893.646157] Call Trace: [13893.649870] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [13893.659039] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [13893.667917] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [13893.676585] [] ? wake_up_state+0x20/0x20 [13893.683790] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [13893.693139] [] kthread+0xcf/0xe0 [13893.699579] [] ? kthread_create_on_node+0x140/0x140 [13893.707858] [] ret_from_fork+0x58/0x90 [13893.714873] [] ? kthread_create_on_node+0x140/0x140 [13893.723158] Code: 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 00 8b 75 98 48 8b 7d 88 e8 dc 2d f5 ff 48 85 c0 48 89 45 90 0f 84 b3 08 00 00 44 8b 7d a0 <48> 8b 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 e8 77 5b f4 ff 42 [13921.327432] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [13921.337135] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [13921.468993] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [13921.482310] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [13921.498591] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [13921.508007] RIP: 0010:[] [] osd_scrub_next+0x160/0x3c0 [osd_ldiskfs] [13921.519769] RSP: 0018:ffff880703603ce0 EFLAGS: 00000282 [13921.526729] RAX: 0000000000000004 RBX: ffff880703603c78 RCX: ffff880703603d98 [13921.535715] RDX: ffff880703603d78 RSI: 0000000000000000 RDI: ffff8803162f6000 [13921.544696] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [13921.553669] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000010 [13921.562619] R13: ffffffffa121ba89 R14: ffffffffffffff10 R15: ffff8803162f6000 [13921.571571] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [13921.581597] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13921.589007] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [13921.597967] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13921.606929] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [13921.615873] Stack: [13921.619081] ffff880419dc3a00 ffff8800b2a3746c 0000000000000000 00000000ffffffff [13921.628354] 0000000000000001 ffff880703603df0 ffffffffa121cff9 ffffffffa121b990 [13921.637643] ffffffffa1217a20 ffff8800b2a36000 00000000810d354f ffff8803162f6000 [13921.646948] Call Trace: [13921.650678] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [13921.659852] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [13921.669006] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [13921.677880] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [13921.686557] [] ? wake_up_state+0x20/0x20 [13921.693764] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [13921.703108] [] kthread+0xcf/0xe0 [13921.709543] [] ? kthread_create_on_node+0x140/0x140 [13921.717821] [] ret_from_fork+0x58/0x90 [13921.724833] [] ? kthread_create_on_node+0x140/0x140 [13921.733101] Code: 10 00 00 02 00 00 00 48 89 df e8 fc 69 47 e0 b8 05 00 00 00 48 8b 75 d0 65 48 33 34 25 28 00 00 00 0f 85 4c 02 00 00 48 83 c4 70 <5b> 41 5c 41 5d 41 5e 41 5f 5d c3 48 8b 05 be 58 9c ff 48 85 c0 [13949.328405] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [13949.338087] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [13949.469845] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [13949.483133] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [13949.499379] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [13949.508763] RIP: 0010:[] [] osd_inode_iteration+0x255/0xcc0 [osd_ldiskfs] [13949.521002] RSP: 0018:ffff880703603d18 EFLAGS: 00000286 [13949.527945] RAX: ffff8800b2a37468 RBX: 000000009f8f2001 RCX: 0000000000000000 [13949.536924] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [13949.545900] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [13949.554854] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000001 [13949.563804] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [13949.572752] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [13949.582786] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13949.590197] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [13949.599162] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13949.608122] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [13949.617064] Stack: [13949.620275] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [13949.629576] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [13949.638878] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [13949.648190] Call Trace: [13949.651907] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [13949.661071] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [13949.669940] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [13949.678605] [] ? wake_up_state+0x20/0x20 [13949.685807] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [13949.695154] [] kthread+0xcf/0xe0 [13949.701589] [] ? kthread_create_on_node+0x140/0x140 [13949.709875] [] ret_from_fork+0x58/0x90 [13949.716884] [] ? kthread_create_on_node+0x140/0x140 [13949.725162] Code: 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 00 8b 75 98 48 8b 7d 88 e8 dc 2d f5 ff 48 85 c0 48 89 45 90 0f 84 b3 08 00 00 44 8b 7d a0 <48> 8b 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 e8 77 5b f4 ff 42 [13963.939914] INFO: rcu_sched self-detected stall on CPU { 24} (t=6540129 jiffies g=907112 c=907111 q=21002172) [13963.952393] Task dump for CPU 24: [13963.957150] OI_scrub R running task 0 5551 2 0x00000088 [13963.966092] ffff88083fde6dd0 00000000fb6db880 ffff88082da03db0 ffffffff810c47e8 [13963.975419] 0000000000000018 ffffffff81a1e7c0 ffff88082da03dc8 ffffffff810c8079 [13963.984733] 0000000000000009 ffff88082da03df8 ffffffff811378e0 ffff88082da101c0 [13963.994016] Call Trace: [13963.997724] [] sched_show_task+0xa8/0x110 [13964.005727] [] dump_cpu_task+0x39/0x70 [13964.012711] [] rcu_dump_cpu_stacks+0x90/0xd0 [13964.020247] [] rcu_check_callbacks+0x442/0x720 [13964.027947] [] ? tick_sched_handle.isra.13+0x60/0x60 [13964.036218] [] update_process_times+0x47/0x80 [13964.043781] [] tick_sched_handle.isra.13+0x25/0x60 [13964.051815] [] tick_sched_timer+0x41/0x70 [13964.058950] [] __hrtimer_run_queues+0xd2/0x260 [13964.066548] [] hrtimer_interrupt+0xb0/0x1e0 [13964.073837] [] ? call_softirq+0x1c/0x30 [13964.080717] [] local_apic_timer_interrupt+0x37/0x60 [13964.088752] [] smp_apic_timer_interrupt+0x3f/0x60 [13964.096580] [] apic_timer_interrupt+0x6d/0x80 [13964.104002] [] ? ldiskfs_itable_unused_count+0x23/0x30 [ldiskfs] [13964.114031] [] osd_inode_iteration+0x269/0xcc0 [osd_ldiskfs] [13964.122940] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [13964.131839] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [13964.140449] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [13964.148858] [] ? wake_up_state+0x20/0x20 [13964.155808] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [13964.164886] [] kthread+0xcf/0xe0 [13964.171051] [] ? kthread_create_on_node+0x140/0x140 [13964.179067] [] ret_from_fork+0x58/0x90 [13964.185823] [] ? kthread_create_on_node+0x140/0x140 [13989.329799] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [13989.339217] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [13989.468859] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [13989.482030] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [13989.498093] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [13989.507378] RIP: 0010:[] [] osd_scrub_exec+0xf7/0x5b0 [osd_ldiskfs] [13989.518936] RSP: 0018:ffff880703603c78 EFLAGS: 00000246 [13989.525790] RAX: 0000000000280001 RBX: ffff880703603c37 RCX: 0000000000000000 [13989.534692] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [13989.543588] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [13989.552490] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [13989.561387] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [13989.570284] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [13989.580257] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13989.587612] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [13989.596522] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13989.605426] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [13989.614314] Stack: [13989.617480] ffff880703603df0 000000009f8f2001 ffffea00027e3c80 000000009f8f2001 [13989.626731] ffff880703603d57 ffff880703603d57 ffff8800b2a37468 0000000000000000 [13989.636002] ffff880703603d78 ffff880419dc3a00 ffff88042bd79800 ffffffffffffff10 [13989.645255] Call Trace: [13989.648937] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [13989.658066] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [13989.667192] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [13989.676027] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [13989.684675] [] ? wake_up_state+0x20/0x20 [13989.691859] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [13989.701168] [] kthread+0xcf/0xe0 [13989.707566] [] ? kthread_create_on_node+0x140/0x140 [13989.715814] [] ret_from_fork+0x58/0x90 [13989.722788] [] ? kthread_create_on_node+0x140/0x140 [13989.731026] Code: 8b 4d 88 48 85 db 74 23 f6 83 18 0a 00 00 08 74 1a 4d 85 e4 74 15 41 8b 87 6c 14 00 00 41 39 84 24 0c 0a 00 00 0f 82 d9 00 00 00 <31> c0 41 f6 87 74 14 00 00 04 0f 85 6e ff ff ff 41 83 f9 02 0f [14017.330775] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [14017.340428] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [14017.472161] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [14017.485453] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [14017.501705] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [14017.511085] RIP: 0010:[] [] osd_scrub_exec+0x96/0x5b0 [osd_ldiskfs] [14017.522726] RSP: 0018:ffff880703603d10 EFLAGS: 00000292 [14017.529652] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000 [14017.538622] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [14017.547580] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [14017.556526] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffff880703603d57 [14017.565469] R13: 0000000000000004 R14: 000000009f8f2001 R15: 0000000000000018 [14017.574421] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [14017.584439] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14017.591841] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [14017.600796] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14017.609740] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [14017.618671] Stack: [14017.621869] ffffffffa121d1e9 ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 [14017.631129] 00000000810d354f ffff8803162f6000 ffff8800b2a37468 0000000020000000 [14017.640417] 010000000000000c 0000000000000000 0000000000000000 ffff8800b2a36000 [14017.649696] Call Trace: [14017.653413] [] ? osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [14017.662771] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [14017.671933] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [14017.680794] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [14017.689457] [] ? wake_up_state+0x20/0x20 [14017.696673] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [14017.706021] [] kthread+0xcf/0xe0 [14017.712457] [] ? kthread_create_on_node+0x140/0x140 [14017.720742] [] ret_from_fork+0x58/0x90 [14017.727751] [] ? kthread_create_on_node+0x140/0x140 [14017.736019] Code: 00 00 41 80 a7 74 14 00 00 fe 48 8b 4d d0 65 48 33 0c 25 28 00 00 00 0f 85 b1 03 00 00 48 83 c4 68 5b 41 5c 41 5d 41 5e 41 5f 5d 66 0f 1f 84 00 00 00 00 00 4c 8d ae f0 10 00 00 44 89 4d 88 [14045.331751] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [14045.341410] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [14045.473342] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [14045.486652] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [14045.502904] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [14045.512294] RIP: 0010:[] [] osd_scrub_next+0x3d/0x3c0 [osd_ldiskfs] [14045.523949] RSP: 0018:ffff880703603c70 EFLAGS: 00000246 [14045.530893] RAX: 0000000000000000 RBX: ffff880703603c37 RCX: ffff880703603d98 [14045.539861] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [14045.548825] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [14045.557778] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [14045.566730] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [14045.575679] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [14045.585717] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14045.593132] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [14045.602113] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14045.611072] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [14045.620028] Stack: [14045.623264] 000000009f8f2001 0000000000000004 0000000000000001 ffffffffa1217a20 [14045.632574] ffff880703603d98 ffff880703603d57 ffff8800b2a36000 ffff8803162f6000 [14045.641873] ffffffffffffff10 ffffffffa1217a3b 0000000000000010 0000000000000287 [14045.651185] Call Trace: [14045.654926] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [14045.663807] [] ? osd_scrub_next+0x1b/0x3c0 [osd_ldiskfs] [14045.672606] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [14045.681779] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [14045.690948] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [14045.699814] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [14045.708497] [] ? wake_up_state+0x20/0x20 [14045.715708] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [14045.725070] [] kthread+0xcf/0xe0 [14045.731515] [] ? kthread_create_on_node+0x140/0x140 [14045.739814] [] ret_from_fork+0x58/0x90 [14045.746834] [] ? kthread_create_on_node+0x140/0x140 [14045.755123] Code: 49 89 fd 41 54 49 89 d4 53 48 89 cb 48 83 ec 70 65 48 8b 04 25 28 00 00 00 48 89 45 d0 31 c0 48 8b 05 f8 59 9c ff 48 85 c0 75 7b <41> f6 87 58 10 00 00 08 0f 84 62 03 00 00 4d 8d b7 e0 10 00 00 [14073.332724] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [14073.342395] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [14073.473932] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [14073.487185] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [14073.503388] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [14073.512743] RIP: 0010:[] [] osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [14073.524765] RSP: 0018:ffff880703603d10 EFLAGS: 00000293 [14073.531691] RAX: ffffffffa121b990 RBX: 000000009f8f2001 RCX: 0000000000000000 [14073.540656] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [14073.549615] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [14073.558577] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffff880703603d57 [14073.567543] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [14073.576500] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [14073.586541] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14073.593947] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [14073.602908] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14073.611862] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [14073.620813] Stack: [14073.624040] ffffffffa121d1e9 ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 [14073.633334] 00000000810d354f ffff8803162f6000 ffff8800b2a37468 0000000020000000 [14073.642635] 010000000000000c 0000000000000000 0000000000000000 ffff8800b2a36000 [14073.651932] Call Trace: [14073.655654] [] ? osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [14073.665012] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [14073.674169] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [14073.683035] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [14073.691712] [] ? wake_up_state+0x20/0x20 [14073.698927] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [14073.708284] [] kthread+0xcf/0xe0 [14073.714723] [] ? kthread_create_on_node+0x140/0x140 [14073.723016] [] ret_from_fork+0x58/0x90 [14073.730039] [] ? kthread_create_on_node+0x140/0x140 [14073.738317] Code: b5 07 00 00 c7 05 85 a5 02 00 00 00 02 00 89 54 24 08 48 8b 40 40 ba 07 00 00 00 48 89 04 24 31 c0 e8 a5 03 98 ff e9 c7 fd ff ff <66> 66 66 66 90 55 48 89 e5 41 57 49 89 f7 41 56 41 55 41 54 45 [14101.333701] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [14101.343380] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [14101.475521] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [14101.488836] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [14101.505119] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [14101.514518] RIP: 0010:[] [] osd_scrub_exec+0xd4/0x5b0 [osd_ldiskfs] [14101.526185] RSP: 0018:ffff880703603c78 EFLAGS: 00000286 [14101.533140] RAX: ffff8803162f1008 RBX: ffff880703603c37 RCX: 0000000000000000 [14101.542133] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [14101.551106] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [14101.560061] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [14101.569024] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [14101.577988] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [14101.588013] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14101.595424] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [14101.604409] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14101.613372] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [14101.622327] Stack: [14101.625557] ffffea00027e3c80 000000009f8f2001 ffff880703603df0 0000000000000001 [14101.634860] ffff880703603d57 000000009f8f2001 0000000000000004 ffff880703603d57 [14101.644154] ffffffffa121b990 0000000000000000 ffff880703603d78 ffff8800b2a36000 [14101.653456] Call Trace: [14101.657176] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [14101.666343] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [14101.675504] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [14101.684656] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [14101.693515] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [14101.702181] [] ? wake_up_state+0x20/0x20 [14101.709388] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [14101.718727] [] kthread+0xcf/0xe0 [14101.725149] [] ? kthread_create_on_node+0x140/0x140 [14101.733439] [] ret_from_fork+0x58/0x90 [14101.740446] [] ? kthread_create_on_node+0x140/0x140 [14101.748707] Code: 4d 88 4c 89 ef e8 1d fb 46 e0 41 83 87 68 14 00 00 01 49 83 87 88 11 00 00 01 4c 89 ef e8 f5 9f e9 df 44 8b 4d 88 48 85 db 74 23 83 18 0a 00 00 08 74 1a 4d 85 e4 74 15 41 8b 87 6c 14 00 00 [14129.334675] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [14129.344350] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [14129.475910] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [14129.489170] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [14129.505369] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [14129.514732] RIP: 0010:[] [] osd_scrub_exec+0xe9/0x5b0 [osd_ldiskfs] [14129.526365] RSP: 0018:ffff880703603c78 EFLAGS: 00000282 [14129.533294] RAX: 0000000000280001 RBX: ffff880703603c37 RCX: 0000000000000000 [14129.542259] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [14129.551216] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [14129.560163] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [14129.569096] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [14129.578052] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [14129.588080] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14129.595481] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [14129.604434] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14129.613386] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [14129.622320] Stack: [14129.625527] 0000000000000004 0000000000000001 ffff880703603df0 ffff880703603d98 [14129.634820] ffff880703603d57 000000009f8f2001 0000000000000004 ffff880703603d57 [14129.644107] ffff8800b2a37468 0000000000000000 ffff880703603d78 ffff8800b2a36000 [14129.653381] Call Trace: [14129.657096] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [14129.666253] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [14129.675400] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [14129.684257] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [14129.692923] [] ? wake_up_state+0x20/0x20 [14129.700126] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [14129.709469] [] kthread+0xcf/0xe0 [14129.715887] [] ? kthread_create_on_node+0x140/0x140 [14129.724183] [] ret_from_fork+0x58/0x90 [14129.731194] [] ? kthread_create_on_node+0x140/0x140 [14129.739459] Code: 88 11 00 00 01 4c 89 ef e8 f5 9f e9 df 44 8b 4d 88 48 85 db 74 23 f6 83 18 0a 00 00 08 74 1a 4d 85 e4 74 15 41 8b 87 6c 14 00 00 <41> 39 84 24 0c 0a 00 00 0f 82 d9 00 00 00 31 c0 41 f6 87 74 14 [14143.949185] INFO: rcu_sched self-detected stall on CPU { 24} (t=6720132 jiffies g=907112 c=907111 q=21876797) [14143.961685] Task dump for CPU 24: [14143.966412] OI_scrub R running task 0 5551 2 0x00000088 [14143.975322] ffff88083fde6dd0 00000000fb6db880 ffff88082da03db0 ffffffff810c47e8 [14143.984639] 0000000000000018 ffffffff81a1e7c0 ffff88082da03dc8 ffffffff810c8079 [14143.993933] 0000000000000009 ffff88082da03df8 ffffffff811378e0 ffff88082da101c0 [14144.003202] Call Trace: [14144.006869] [] sched_show_task+0xa8/0x110 [14144.014844] [] dump_cpu_task+0x39/0x70 [14144.021797] [] rcu_dump_cpu_stacks+0x90/0xd0 [14144.029310] [] rcu_check_callbacks+0x442/0x720 [14144.036996] [] ? tick_sched_handle.isra.13+0x60/0x60 [14144.045242] [] update_process_times+0x47/0x80 [14144.052792] [] tick_sched_handle.isra.13+0x25/0x60 [14144.060808] [] tick_sched_timer+0x41/0x70 [14144.067933] [] __hrtimer_run_queues+0xd2/0x260 [14144.075529] [] hrtimer_interrupt+0xb0/0x1e0 [14144.082813] [] ? call_softirq+0x1c/0x30 [14144.089689] [] local_apic_timer_interrupt+0x37/0x60 [14144.097717] [] smp_apic_timer_interrupt+0x3f/0x60 [14144.105541] [] apic_timer_interrupt+0x6d/0x80 [14144.112964] [] ? osd_scrub_next+0x149/0x3c0 [osd_ldiskfs] [14144.122291] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [14144.130900] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [14144.139801] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [14144.148705] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [14144.157308] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [14144.165702] [] ? wake_up_state+0x20/0x20 [14144.172637] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [14144.181722] [] kthread+0xcf/0xe0 [14144.187892] [] ? kthread_create_on_node+0x140/0x140 [14144.195904] [] ret_from_fork+0x58/0x90 [14144.202644] [] ? kthread_create_on_node+0x140/0x140 [14169.336075] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [14169.345473] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [14169.475607] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [14169.488783] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [14169.504823] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [14169.514103] RIP: 0010:[] [] osd_inode_iteration+0x2c0/0xcc0 [osd_ldiskfs] [14169.526251] RSP: 0018:ffff880703603d18 EFLAGS: 00000297 [14169.533114] RAX: 0000000000000004 RBX: 000000009f8f2001 RCX: ffff880703603d98 [14169.542016] RDX: ffff880703603d78 RSI: 0000000000000000 RDI: ffff8803162f6000 [14169.550919] RBP: ffff880703603df0 R08: 0000000000000001 R09: 0000000000000004 [14169.559819] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000001 [14169.568710] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [14169.577596] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [14169.587558] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14169.594899] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [14169.603813] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14169.612717] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [14169.621612] Stack: [14169.624785] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [14169.634025] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [14169.643279] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [14169.652517] Call Trace: [14169.656205] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [14169.665351] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [14169.674189] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [14169.682842] [] ? wake_up_state+0x20/0x20 [14169.690022] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [14169.699351] [] kthread+0xcf/0xe0 [14169.705746] [] ? kthread_create_on_node+0x140/0x140 [14169.714007] [] ret_from_fork+0x58/0x90 [14169.720989] [] ? kthread_create_on_node+0x140/0x140 [14169.729238] Code: 38 ff ff ff 48 8b bd 48 ff ff ff 48 8b 85 30 ff ff ff ff d0 83 f8 03 41 89 c5 0f 84 4b 02 00 00 0f 8e 15 01 00 00 83 f8 05 66 90 <0f> 84 8a 02 00 00 83 f8 06 0f 85 a1 01 00 00 48 8b 7d 90 48 85 [14197.337049] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [14197.346643] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [14197.478574] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [14197.491882] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [14197.508175] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [14197.517579] RIP: 0010:[] [] osd_scrub_exec+0xf7/0x5b0 [osd_ldiskfs] [14197.529244] RSP: 0018:ffff880703603c78 EFLAGS: 00000246 [14197.536205] RAX: 0000000000280001 RBX: ffff880703603c37 RCX: 0000000000000000 [14197.545192] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [14197.554178] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [14197.563147] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [14197.572099] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [14197.581048] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [14197.591076] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14197.598487] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [14197.607460] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14197.616431] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [14197.625393] Stack: [14197.628615] ffff880703603d08 ffff880703603d57 ffff880703603df0 000000009f8f2001 [14197.637902] ffff880703603d57 000000009f8f2001 0000000000000004 ffff880703603d57 [14197.647198] 0000000000000000 0000000000000000 ffff880703603d78 ffff8800b2a36000 [14197.656493] Call Trace: [14197.660216] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [14197.669390] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [14197.678561] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [14197.687433] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [14197.696120] [] ? wake_up_state+0x20/0x20 [14197.703345] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [14197.712720] [] kthread+0xcf/0xe0 [14197.719156] [] ? kthread_create_on_node+0x140/0x140 [14197.727453] [] ret_from_fork+0x58/0x90 [14197.734478] [] ? kthread_create_on_node+0x140/0x140 [14197.742760] Code: 8b 4d 88 48 85 db 74 23 f6 83 18 0a 00 00 08 74 1a 4d 85 e4 74 15 41 8b 87 6c 14 00 00 41 39 84 24 0c 0a 00 00 0f 82 d9 00 00 00 <31> c0 41 f6 87 74 14 00 00 04 0f 85 6e ff ff ff 41 83 f9 02 0f [14225.338019] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [14225.347633] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [14225.479386] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [14225.492674] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [14225.508936] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [14225.518320] RIP: 0010:[] [] osd_inode_iteration+0x282/0xcc0 [osd_ldiskfs] [14225.530553] RSP: 0018:ffff880703603d18 EFLAGS: 00000287 [14225.537494] RAX: ffff88042bd78800 RBX: ffff880703603d57 RCX: ffff88042bd78800 [14225.546482] RDX: 00000000000072cf RSI: ffff880419dc3a00 RDI: ffff88042bd79800 [14225.555448] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [14225.564408] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [14225.573359] R13: ffffea00027e3c80 R14: 000000009f8f2001 R15: 0000000000000018 [14225.582310] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [14225.592340] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14225.599748] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [14225.608719] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14225.617680] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [14225.626624] Stack: [14225.629840] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [14225.639132] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [14225.648434] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [14225.657739] Call Trace: [14225.661461] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [14225.670623] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [14225.679493] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [14225.688167] [] ? wake_up_state+0x20/0x20 [14225.695373] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [14225.704734] [] kthread+0xcf/0xe0 [14225.711172] [] ? kthread_create_on_node+0x140/0x140 [14225.719455] [] ret_from_fork+0x58/0x90 [14225.726467] [] ? kthread_create_on_node+0x140/0x140 [14225.734745] Code: 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 e8 77 5b f4 ff 42 8d 14 38 48 8b 45 88 48 8b 80 a8 03 00 00 48 3b 50 20 0f 83 57 01 00 00 <44> 0f b6 85 67 ff ff ff 48 8d 4d a8 48 8d 55 88 48 8b b5 38 ff [14253.339001] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [14253.348632] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [14253.481082] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [14253.494396] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [14253.510694] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [14253.520104] RIP: 0010:[] [] osd_inode_iteration+0x251/0xcc0 [osd_ldiskfs] [14253.532371] RSP: 0018:ffff880703603d18 EFLAGS: 00000286 [14253.539334] RAX: ffff8800b2a37468 RBX: 000000009f8f2001 RCX: 0000000000000000 [14253.548330] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [14253.557320] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [14253.566291] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000001 [14253.575249] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [14253.584201] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [14253.594233] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14253.601641] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [14253.610618] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14253.619585] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [14253.628537] Stack: [14253.631754] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [14253.641058] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [14253.650337] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [14253.659634] Call Trace: [14253.663351] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [14253.672521] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [14253.681398] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [14253.690078] [] ? wake_up_state+0x20/0x20 [14253.697296] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [14253.706657] [] kthread+0xcf/0xe0 [14253.713091] [] ? kthread_create_on_node+0x140/0x140 [14253.721392] [] ret_from_fork+0x58/0x90 [14253.728410] [] ? kthread_create_on_node+0x140/0x140 [14253.736684] Code: b0 00 00 00 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 00 8b 75 98 48 8b 7d 88 e8 dc 2d f5 ff 48 85 c0 48 89 45 90 0f 84 b3 08 00 00 <44> 8b 7d a0 48 8b 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 e8 77 [14281.339977] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [14281.349598] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [14281.481457] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [14281.494765] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [14281.511067] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [14281.520472] RIP: 0010:[] [] osd_inode_iteration+0x299/0xcc0 [osd_ldiskfs] [14281.532732] RSP: 0018:ffff880703603d18 EFLAGS: 00000287 [14281.539697] RAX: ffff88042bd78800 RBX: 000000009f8f2001 RCX: ffff880703603d98 [14281.548690] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff88042bd79800 [14281.557673] RBP: ffff880703603df0 R08: 0000000000000001 R09: 0000000000000004 [14281.566640] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [14281.575596] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [14281.584549] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [14281.594576] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14281.601994] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [14281.610967] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14281.619937] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [14281.628892] Stack: [14281.632112] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [14281.641409] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [14281.650698] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [14281.659996] Call Trace: [14281.663724] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [14281.672897] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [14281.681779] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [14281.690461] [] ? wake_up_state+0x20/0x20 [14281.697683] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [14281.707041] [] kthread+0xcf/0xe0 [14281.713471] [] ? kthread_create_on_node+0x140/0x140 [14281.721756] [] ret_from_fork+0x58/0x90 [14281.728783] [] ? kthread_create_on_node+0x140/0x140 [14281.737070] Code: 8b 45 88 48 8b 80 a8 03 00 00 48 3b 50 20 0f 83 57 01 00 00 44 0f b6 85 67 ff ff ff 48 8d 4d a8 48 8d 55 88 48 8b b5 38 ff ff ff <48> 8b bd 48 ff ff ff 48 8b 85 30 ff ff ff ff d0 83 f8 03 41 89 [14309.340950] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [14309.350596] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [14309.482796] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [14309.496111] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [14309.512426] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [14309.521836] RIP: 0010:[] [] osd_scrub_exec+0xf7/0x5b0 [osd_ldiskfs] [14309.533519] RSP: 0018:ffff880703603c78 EFLAGS: 00000246 [14309.540479] RAX: 0000000000280001 RBX: ffff880703603c37 RCX: 0000000000000000 [14309.549474] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [14309.558454] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [14309.567424] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [14309.576384] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [14309.585340] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [14309.595372] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14309.602779] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [14309.611749] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14309.620712] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [14309.629659] Stack: [14309.632878] ffff880703603d08 000000009f8f2001 ffffea00027e3c80 000000009f8f2001 [14309.642166] ffff880703603d57 0000000000000001 ffffffffa1217a20 ffff880703603d98 [14309.651456] ffff880703603d78 ffff8800b2a36000 ffff8803162f6000 ffffffffffffff10 [14309.660754] Call Trace: [14309.664485] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [14309.673361] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [14309.682531] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [14309.691696] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [14309.700562] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [14309.709234] [] ? wake_up_state+0x20/0x20 [14309.716441] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [14309.725800] [] kthread+0xcf/0xe0 [14309.732230] [] ? kthread_create_on_node+0x140/0x140 [14309.740511] [] ret_from_fork+0x58/0x90 [14309.747519] [] ? kthread_create_on_node+0x140/0x140 [14309.755788] Code: 8b 4d 88 48 85 db 74 23 f6 83 18 0a 00 00 08 74 1a 4d 85 e4 74 15 41 8b 87 6c 14 00 00 41 39 84 24 0c 0a 00 00 0f 82 d9 00 00 00 <31> c0 41 f6 87 74 14 00 00 04 0f 85 6e ff ff ff 41 83 f9 02 0f [14323.958461] INFO: rcu_sched self-detected stall on CPU { 24} (t=6900135 jiffies g=907112 c=907111 q=22047866) [14323.970884] Task dump for CPU 24: [14323.975601] OI_scrub R running task 0 5551 2 0x00000088 [14323.984513] ffff88083fde6dd0 00000000fb6db880 ffff88082da03db0 ffffffff810c47e8 [14323.993812] 0000000000000018 ffffffff81a1e7c0 ffff88082da03dc8 ffffffff810c8079 [14324.003082] 0000000000000009 ffff88082da03df8 ffffffff811378e0 ffff88082da101c0 [14324.012325] Call Trace: [14324.015997] [] sched_show_task+0xa8/0x110 [14324.023953] [] dump_cpu_task+0x39/0x70 [14324.030891] [] rcu_dump_cpu_stacks+0x90/0xd0 [14324.038389] [] rcu_check_callbacks+0x442/0x720 [14324.046059] [] ? tick_sched_handle.isra.13+0x60/0x60 [14324.054289] [] update_process_times+0x47/0x80 [14324.061810] [] tick_sched_handle.isra.13+0x25/0x60 [14324.069805] [] tick_sched_timer+0x41/0x70 [14324.076905] [] __hrtimer_run_queues+0xd2/0x260 [14324.084478] [] hrtimer_interrupt+0xb0/0x1e0 [14324.091747] [] ? call_softirq+0x1c/0x30 [14324.098609] [] local_apic_timer_interrupt+0x37/0x60 [14324.106614] [] smp_apic_timer_interrupt+0x3f/0x60 [14324.114437] [] apic_timer_interrupt+0x6d/0x80 [14324.121863] [] ? osd_scrub_exec+0x95/0x5b0 [osd_ldiskfs] [14324.131101] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [14324.140008] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [14324.148913] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [14324.157513] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [14324.165910] [] ? wake_up_state+0x20/0x20 [14324.172850] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [14324.181939] [] kthread+0xcf/0xe0 [14324.188111] [] ? kthread_create_on_node+0x140/0x140 [14324.196124] [] ret_from_fork+0x58/0x90 [14324.202867] [] ? kthread_create_on_node+0x140/0x140 [14349.342345] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [14349.351714] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [14349.481628] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [14349.494800] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [14349.510864] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [14349.520161] RIP: 0010:[] [] osd_scrub_next+0x350/0x3c0 [osd_ldiskfs] [14349.531818] RSP: 0018:ffff880703603c70 EFLAGS: 00000246 [14349.538676] RAX: 0000000000000000 RBX: ffff880703603c37 RCX: ffff880703603d98 [14349.547570] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [14349.556467] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [14349.565369] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [14349.574268] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [14349.583163] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [14349.593124] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14349.600453] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [14349.609361] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14349.618267] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [14349.627168] Stack: [14349.630338] 000000009f8f2001 ffff880703603df0 000000009f8f2001 ffff880703603df0 [14349.639572] 000000009f8f2001 ffff880703603d57 000000009f8f2001 0000000000000004 [14349.648808] ffff880703603d57 ffff8800b2a37468 0000000000000000 ffff880703603d78 [14349.658057] Call Trace: [14349.661738] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [14349.670864] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [14349.679991] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [14349.688819] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [14349.697452] [] ? wake_up_state+0x20/0x20 [14349.704624] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [14349.713934] [] kthread+0xcf/0xe0 [14349.720331] [] ? kthread_create_on_node+0x140/0x140 [14349.728566] [] ret_from_fork+0x58/0x90 [14349.735539] [] ? kthread_create_on_node+0x140/0x140 [14349.743780] Code: 97 ff 48 8b 55 90 48 8b 4d 88 44 8b 85 74 ff ff ff e9 1d ff ff ff 48 89 d7 44 89 45 90 e8 f9 67 47 e0 44 8b 45 90 0f 1f 44 00 00 <45> 84 c0 b8 04 00 00 00 0f 85 eb fd ff ff 49 8d b7 6c 14 00 00 [14377.343322] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [14377.352942] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [14377.484694] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [14377.497974] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [14377.514216] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [14377.523601] RIP: 0010:[] [] osd_scrub_next+0x12/0x3c0 [osd_ldiskfs] [14377.535246] RSP: 0018:ffff880703603cf0 EFLAGS: 00000287 [14377.542178] RAX: ffffffffa1217a20 RBX: 000000009f8f2001 RCX: ffff880703603d98 [14377.551148] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [14377.560105] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [14377.569055] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffff880703603c78 [14377.578006] R13: 0000000000000296 R14: 0000000000000010 R15: ffffffffa121b9ac [14377.586948] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [14377.596967] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14377.604364] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [14377.613317] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14377.622260] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [14377.631191] Stack: [14377.634389] 0000000000000000 00000000ffffffff 0000000000000001 ffff880703603df0 [14377.643674] ffffffffa121cff9 ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 [14377.652968] 00000000810d354f ffff8803162f6000 ffff8800b2a37468 0000000020000000 [14377.662266] Call Trace: [14377.665977] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [14377.675144] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [14377.684304] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [14377.693168] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [14377.701836] [] ? wake_up_state+0x20/0x20 [14377.709044] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [14377.718390] [] kthread+0xcf/0xe0 [14377.724826] [] ? kthread_create_on_node+0x140/0x140 [14377.733104] [] ret_from_fork+0x58/0x90 [14377.740114] [] ? kthread_create_on_node+0x140/0x140 [14377.748381] Code: 25 fc ff ff 48 83 c4 08 5b 41 5c 41 5d 41 5e 5d c3 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 89 e5 41 57 49 89 f7 41 56 41 55 <49> 89 fd 41 54 49 89 d4 53 48 89 cb 48 83 ec 70 65 48 8b 04 25 [14405.344290] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [14405.353949] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [14405.485908] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [14405.499194] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [14405.515458] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [14405.524852] RIP: 0010:[] [] osd_inode_iteration+0x264/0xcc0 [osd_ldiskfs] [14405.537094] RSP: 0018:ffff880703603d18 EFLAGS: 00000286 [14405.544050] RAX: ffff8800b2a37468 RBX: 000000009f8f2001 RCX: 0000000000000000 [14405.553032] RDX: ffff880703603d78 RSI: ffff880419dc3a00 RDI: ffff88042bd79800 [14405.561997] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [14405.570952] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000001 [14405.579903] R13: ffff880703603d08 R14: 000000009f8f2001 R15: 0000000000000018 [14405.588859] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [14405.598893] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14405.606294] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [14405.615265] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14405.624221] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [14405.633165] Stack: [14405.636384] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [14405.645679] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [14405.654980] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [14405.664265] Call Trace: [14405.667991] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [14405.677162] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [14405.686039] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [14405.694710] [] ? wake_up_state+0x20/0x20 [14405.701913] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [14405.711262] [] kthread+0xcf/0xe0 [14405.717693] [] ? kthread_create_on_node+0x140/0x140 [14405.725977] [] ret_from_fork+0x58/0x90 [14405.732988] [] ? kthread_create_on_node+0x140/0x140 [14405.741265] Code: 75 98 48 8b 7d 88 e8 dc 2d f5 ff 48 85 c0 48 89 45 90 0f 84 b3 08 00 00 44 8b 7d a0 48 8b 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 77 5b f4 ff 42 8d 14 38 48 8b 45 88 48 8b 80 a8 03 00 00 48 [14433.345272] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [14433.354969] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [14433.486867] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [14433.500179] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [14433.516466] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [14433.525873] RIP: 0010:[] [] osd_inode_iteration+0x255/0xcc0 [osd_ldiskfs] [14433.538132] RSP: 0018:ffff880703603d18 EFLAGS: 00000286 [14433.545101] RAX: ffff8800b2a37468 RBX: 000000009f8f2001 RCX: 0000000000000000 [14433.554096] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [14433.563076] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [14433.572040] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 000000009f8f2001 [14433.580990] R13: ffff880703603d08 R14: 000000009f8f2001 R15: ffff880703603d08 [14433.589943] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [14433.599967] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14433.607373] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [14433.616346] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14433.625313] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [14433.634270] Stack: [14433.637501] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [14433.646783] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [14433.656078] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [14433.665376] Call Trace: [14433.669103] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [14433.678262] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [14433.687139] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [14433.695829] [] ? wake_up_state+0x20/0x20 [14433.703046] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [14433.712398] [] kthread+0xcf/0xe0 [14433.718835] [] ? kthread_create_on_node+0x140/0x140 [14433.727117] [] ret_from_fork+0x58/0x90 [14433.734132] [] ? kthread_create_on_node+0x140/0x140 [14433.742407] Code: 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 00 8b 75 98 48 8b 7d 88 e8 dc 2d f5 ff 48 85 c0 48 89 45 90 0f 84 b3 08 00 00 44 8b 7d a0 <48> 8b 7d 88 48 89 de 48 c7 45 a8 00 00 00 00 e8 77 5b f4 ff 42 [14461.346255] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [14461.355930] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [14461.487955] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [14461.501282] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [14461.517585] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [14461.527000] RIP: 0010:[] [] osd_scrub_next+0x12/0x3c0 [osd_ldiskfs] [14461.538669] RSP: 0018:ffff880703603cf0 EFLAGS: 00000287 [14461.545641] RAX: ffffffffa1217a20 RBX: 000000009f8f2001 RCX: ffff880703603d98 [14461.554637] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [14461.563619] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [14461.572588] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffff880703603c78 [14461.581543] R13: ffff880703603d08 R14: 0000000000000010 R15: ffffffffa121ba87 [14461.590487] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [14461.600516] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14461.607936] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [14461.616915] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14461.625888] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [14461.634852] Stack: [14461.638067] 0000000000000000 00000000ffffffff 0000000000000001 ffff880703603df0 [14461.647341] ffffffffa121cff9 ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 [14461.656636] 00000000810d354f ffff8803162f6000 ffff8800b2a37468 0000000020000000 [14461.665943] Call Trace: [14461.669682] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [14461.678855] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [14461.688030] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [14461.696917] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [14461.705592] [] ? wake_up_state+0x20/0x20 [14461.712811] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [14461.722177] [] kthread+0xcf/0xe0 [14461.728620] [] ? kthread_create_on_node+0x140/0x140 [14461.736902] [] ret_from_fork+0x58/0x90 [14461.743919] [] ? kthread_create_on_node+0x140/0x140 [14461.752203] Code: 25 fc ff ff 48 83 c4 08 5b 41 5c 41 5d 41 5e 5d c3 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 89 e5 41 57 49 89 f7 41 56 41 55 <49> 89 fd 41 54 49 89 d4 53 48 89 cb 48 83 ec 70 65 48 8b 04 25 [14489.347216] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [OI_scrub:5551] [14489.356851] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [14489.488705] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [14489.501998] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [14489.518268] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [14489.527665] RIP: 0010:[] [] osd_scrub_exec+0x1c/0x5b0 [osd_ldiskfs] [14489.539335] RSP: 0018:ffff880703603c78 EFLAGS: 00000296 [14489.546276] RAX: ffffffffa121b990 RBX: ffff880703603c37 RCX: 0000000000000000 [14489.555263] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [14489.564232] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [14489.573197] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [14489.582157] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [14489.591119] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [14489.601152] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14489.608570] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [14489.617539] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14489.626498] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [14489.635451] Stack: [14489.638666] ffffea00027e3c80 000000009f8f2001 0000000000000004 0000000000000001 [14489.647960] ffff880703603d57 ffff880703603d98 ffff880703603d78 ffff8800b2a36000 [14489.657279] ffff8803162f6000 ffffffffffffff10 ffffffffa1217a32 0000000000000010 [14489.666589] Call Trace: [14489.670324] [] ? osd_scrub_next+0x12/0x3c0 [osd_ldiskfs] [14489.679110] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [14489.688277] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [14489.697437] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [14489.706296] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [14489.714969] [] ? wake_up_state+0x20/0x20 [14489.722177] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [14489.731534] [] kthread+0xcf/0xe0 [14489.737971] [] ? kthread_create_on_node+0x140/0x140 [14489.746256] [] ret_from_fork+0x58/0x90 [14489.753263] [] ? kthread_create_on_node+0x140/0x140 [14489.761533] Code: 89 04 24 31 c0 e8 a5 03 98 ff e9 c7 fd ff ff 66 66 66 66 90 55 48 89 e5 41 57 49 89 f7 41 56 41 55 41 54 45 31 e4 53 48 83 ec 68 <48> 8b 9e 08 10 00 00 4c 89 45 90 65 48 8b 04 25 28 00 00 00 48 [14503.967727] INFO: rcu_sched self-detected stall on CPU { 24} (t=7080138 jiffies g=907112 c=907111 q=22723963) [14503.980179] Task dump for CPU 24: [14503.984897] OI_scrub R running task 0 5551 2 0x00000088 [14503.993812] ffff88083fde6dd0 00000000fb6db880 ffff88082da03db0 ffffffff810c47e8 [14504.003106] 0000000000000018 ffffffff81a1e7c0 ffff88082da03dc8 ffffffff810c8079 [14504.012381] 0000000000000009 ffff88082da03df8 ffffffff811378e0 ffff88082da101c0 [14504.021660] Call Trace: [14504.025340] [] sched_show_task+0xa8/0x110 [14504.033304] [] dump_cpu_task+0x39/0x70 [14504.040244] [] rcu_dump_cpu_stacks+0x90/0xd0 [14504.047742] [] rcu_check_callbacks+0x442/0x720 [14504.055413] [] ? tick_sched_handle.isra.13+0x60/0x60 [14504.063658] [] update_process_times+0x47/0x80 [14504.071188] [] tick_sched_handle.isra.13+0x25/0x60 [14504.079193] [] tick_sched_timer+0x41/0x70 [14504.086301] [] __hrtimer_run_queues+0xd2/0x260 [14504.093867] [] hrtimer_interrupt+0xb0/0x1e0 [14504.101134] [] ? call_softirq+0x1c/0x30 [14504.108012] [] local_apic_timer_interrupt+0x37/0x60 [14504.116028] [] smp_apic_timer_interrupt+0x3f/0x60 [14504.123860] [] apic_timer_interrupt+0x6d/0x80 [14504.131287] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [14504.140914] [] ? osd_scrub_exec+0x23/0x5b0 [osd_ldiskfs] [14504.149433] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [14504.158343] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [14504.167239] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [14504.175845] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [14504.184255] [] ? wake_up_state+0x20/0x20 [14504.191211] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [14504.200312] [] kthread+0xcf/0xe0 [14504.206491] [] ? kthread_create_on_node+0x140/0x140 [14504.214504] [] ret_from_fork+0x58/0x90 [14504.221257] [] ? kthread_create_on_node+0x140/0x140 [14529.348616] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [14529.358036] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [14529.488094] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [14529.501273] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [14529.517317] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [14529.526612] RIP: 0010:[] [] osd_scrub_next+0x149/0x3c0 [osd_ldiskfs] [14529.538276] RSP: 0018:ffff880703603c70 EFLAGS: 00000202 [14529.545146] RAX: 0000000000000004 RBX: ffff880703603c37 RCX: ffff880703603d98 [14529.554047] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [14529.562949] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [14529.571849] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [14529.580746] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [14529.589644] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [14529.599611] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14529.606960] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [14529.615874] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14529.624785] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [14529.633699] Stack: [14529.636879] 000000009f8f2001 ffff880703603d08 ffff880703603d57 ffff880703603df0 [14529.646129] 000000009f8f2001 ffff880703603d57 000000009f8f2001 0000000000000004 [14529.655382] ffff880703603d57 ffff8800b2a37468 0000000000000000 ffff880703603d78 [14529.664641] Call Trace: [14529.668345] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [14529.677485] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [14529.686621] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [14529.695468] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [14529.704112] [] ? wake_up_state+0x20/0x20 [14529.711291] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [14529.720611] [] kthread+0xcf/0xe0 [14529.727014] [] ? kthread_create_on_node+0x140/0x140 [14529.735277] [] ret_from_fork+0x58/0x90 [14529.742260] [] ? kthread_create_on_node+0x140/0x140 [14529.750507] Code: 45 90 74 49 49 8d 9f 10 11 00 00 48 89 df e8 df 6a 47 e0 41 c7 87 58 10 00 00 02 00 00 00 48 89 df e8 fc 69 47 e0 b8 05 00 00 00 <48> 8b 75 d0 65 48 33 34 25 28 00 00 00 0f 85 4c 02 00 00 48 83 [14557.349584] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [14557.359196] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [14557.491054] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [14557.504344] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [14557.520614] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [14557.530008] RIP: 0010:[] [] osd_scrub_exec+0x30/0x5b0 [osd_ldiskfs] [14557.541666] RSP: 0018:ffff880703603c78 EFLAGS: 00000296 [14557.548615] RAX: 00000000fb6db880 RBX: ffff880703603c37 RCX: 0000000000000000 [14557.557596] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [14557.566566] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [14557.575521] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [14557.584476] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [14557.593424] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [14557.603458] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14557.610868] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [14557.619839] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14557.628790] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [14557.637737] Stack: [14557.640940] ffff880703603d08 000000009f8f2001 ffffea00027e3c80 000000009f8f2001 [14557.650243] ffff880703603d57 ffff880703603d57 0000000000000000 0000000000000000 [14557.659546] ffff880703603d78 ffff8800b2a36000 ffff8803162f6000 ffffffffffffff10 [14557.668856] Call Trace: [14557.672572] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [14557.681743] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [14557.690907] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [14557.699773] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [14557.708434] [] ? wake_up_state+0x20/0x20 [14557.715645] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [14557.725004] [] kthread+0xcf/0xe0 [14557.731436] [] ? kthread_create_on_node+0x140/0x140 [14557.739729] [] ret_from_fork+0x58/0x90 [14557.746741] [] ? kthread_create_on_node+0x140/0x140 [14557.755012] Code: 55 48 89 e5 41 57 49 89 f7 41 56 41 55 41 54 45 31 e4 53 48 83 ec 68 48 8b 9e 08 10 00 00 4c 89 45 90 65 48 8b 04 25 28 00 00 00 <48> 89 45 d0 31 c0 48 8d 43 08 48 85 db 4c 0f 45 e0 41 83 f9 04 [14585.350568] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [14585.360283] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [14585.492107] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [14585.505395] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [14585.521652] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [14585.531039] RIP: 0010:[] [] osd_scrub_exec+0xf7/0x5b0 [osd_ldiskfs] [14585.542694] RSP: 0018:ffff880703603c78 EFLAGS: 00000246 [14585.549635] RAX: 0000000000280001 RBX: ffff880703603c37 RCX: 0000000000000000 [14585.558612] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [14585.567581] RBP: ffff880703603d08 R08: ffff880703603d57 R09: 0000000000000004 [14585.576534] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [14585.585487] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [14585.594438] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [14585.604462] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14585.611864] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [14585.620832] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14585.629788] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [14585.638725] Stack: [14585.641937] 0000000000000004 0000000000000001 ffff880703603df0 ffff880703603d98 [14585.651241] ffff880703603d57 000000009f8f2001 0000000000000004 0000000000000001 [14585.660525] 0000000000000004 ffff880703603d98 ffff880703603d78 0000000000000000 [14585.669828] Call Trace: [14585.673543] [] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] [14585.682713] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [14585.691881] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [14585.700744] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [14585.709415] [] ? wake_up_state+0x20/0x20 [14585.716624] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [14585.725996] [] kthread+0xcf/0xe0 [14585.732434] [] ? kthread_create_on_node+0x140/0x140 [14585.740725] [] ret_from_fork+0x58/0x90 [14585.747732] [] ? kthread_create_on_node+0x140/0x140 [14585.755999] Code: 8b 4d 88 48 85 db 74 23 f6 83 18 0a 00 00 08 74 1a 4d 85 e4 74 15 41 8b 87 6c 14 00 00 41 39 84 24 0c 0a 00 00 0f 82 d9 00 00 00 <31> c0 41 f6 87 74 14 00 00 04 0f 85 6e ff ff ff 41 83 f9 02 0f [14613.351543] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [14613.361201] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [14613.493418] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [14613.506717] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [14613.522993] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [14613.532384] RIP: 0010:[] [] osd_inode_iteration+0x4a6/0xcc0 [osd_ldiskfs] [14613.544634] RSP: 0018:ffff880703603d18 EFLAGS: 00000246 [14613.551590] RAX: 0000000020000000 RBX: ffff880703603d98 RCX: 0000000000000000 [14613.560571] RDX: ffff880703603d78 RSI: ffff8800b2a36000 RDI: ffff8803162f6000 [14613.569537] RBP: ffff880703603df0 R08: ffff880703603d57 R09: 0000000000000004 [14613.578501] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: 0000000000000001 [14613.587449] R13: 0000000000000004 R14: 000000009f8f2001 R15: ffffea00027e3c80 [14613.596405] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [14613.606437] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14613.613845] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [14613.622813] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14613.631770] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [14613.640716] Stack: [14613.643928] ffffffffa121b990 ffffffffa1217a20 ffff8800b2a36000 00000000810d354f [14613.653213] ffff8803162f6000 ffff8800b2a37468 0000000020000000 010000000000000c [14613.662488] 0000000000000000 0000000000000000 ffff8800b2a36000 0000000000000000 [14613.671801] Call Trace: [14613.675532] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [14613.684689] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [14613.693557] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [14613.702232] [] ? wake_up_state+0x20/0x20 [14613.709442] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [14613.718779] [] kthread+0xcf/0xe0 [14613.725213] [] ? kthread_create_on_node+0x140/0x140 [14613.733506] [] ret_from_fork+0x58/0x90 [14613.740520] [] ? kthread_create_on_node+0x140/0x140 [14613.748802] Code: 8b 4d a8 48 8d 55 88 48 8b b5 38 ff ff ff 48 8b bd 48 ff ff ff 48 8b 85 28 ff ff ff ff d0 85 c0 41 89 c5 75 1c 8b 85 58 ff ff ff <41> 3b 04 24 72 10 48 8b 85 50 ff ff ff 44 3b 30 0f 87 95 fd ff [14641.352518] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [14641.362160] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate 8021q garp mrp stp llc rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_round_robin ipmi_ssif sb_edac ipmi_devintf ntb sg iTCO_wdt ioatdma shpchp edac_core mei_me iTCO_vendor_support mei lpc_ich ipmi_si pcspkr i2c_i801 ipmi_msghandler wmi nfsd dm_multipath dm_mod nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mlx4_en mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm ahci crct10dif_pclmul crct10dif_common ptp libsas crc32c_intel libahci pps_core drm mlx4_core mpt2sas libata dca raid_class i2c_algo_bit scsi_transport_sas devlink i2c_core fjes [14641.494132] CPU: 24 PID: 5551 Comm: OI_scrub Tainted: P OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 [14641.507438] Hardware name: Intel Corporation SandyBridge Platform/To be filled by O.E.M., BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [14641.523721] task: ffff88083fde6dd0 ti: ffff880703600000 task.ti: ffff880703600000 [14641.533125] RIP: 0010:[] [] osd_scrub_next+0x14d/0x3c0 [osd_ldiskfs] [14641.544883] RSP: 0018:ffff880703603c70 EFLAGS: 00000202 [14641.551843] RAX: 0000000000000004 RBX: ffff880703603c37 RCX: ffff880703603d98 [14641.560821] RDX: ffff880703603d78 RSI: 00000000fb6db880 RDI: ffff8803162f6000 [14641.569800] RBP: ffff880703603d08 R08: 0000000000000001 R09: 0000000000000004 [14641.578768] R10: 000000009f8f2001 R11: ffffea00027e3c80 R12: ffffffffa12184d1 [14641.587722] R13: ffff880703603c68 R14: ffff8800b2a37110 R15: ffff88009f8f2dc0 [14641.596664] FS: 0000000000000000(0000) GS:ffff88082da00000(0000) knlGS:0000000000000000 [14641.606691] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14641.614100] CR2: 00007f465d19147e CR3: 00000000019be000 CR4: 00000000000407e0 [14641.623070] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [14641.632036] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [14641.640990] Stack: [14641.644208] 000000009f8f2001 ffff880703603d08 000000009f8f2001 ffff880703603df0 [14641.653493] 000000009f8f2001 ffff880703603d57 000000009f8f2001 0000000000000004 [14641.662783] 0000000000000001 0000000000000004 ffff880703603d98 ffff880703603d78 [14641.672077] Call Trace: [14641.675801] [] osd_inode_iteration+0x2a9/0xcc0 [osd_ldiskfs] [14641.684968] [] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] [14641.694120] [] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] [14641.702989] [] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] [14641.711665] [] ? wake_up_state+0x20/0x20 [14641.718870] [] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] [14641.728213] [] kthread+0xcf/0xe0 [14641.734633] [] ? kthread_create_on_node+0x140/0x140 [14641.742902] [] ret_from_fork+0x58/0x90 [14641.749917] [] ? kthread_create_on_node+0x140/0x140 [14641.758183] Code: 49 8d 9f 10 11 00 00 48 89 df e8 df 6a 47 e0 41 c7 87 58 10 00 00 02 00 00 00 48 89 df e8 fc 69 47 e0 b8 05 00 00 00 48 8b 75 d0 <65> 48 33 34 25 28 00 00 00 0f 85 4c 02 00 00 48 83 c4 70 5b 41 [14669.353493] NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [OI_scrub:5551] [14669.363149] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) p