Lustre: DEBUG MARKER: /usr/sbin/lctl mark -----============= acceptance-small: lfsck ============----- Sat Dec 15 13:41:34 PST 2012 Lustre: DEBUG MARKER: -----============= acceptance-small: lfsck ============----- Sat Dec 15 13:41:34 PST 2012 Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre' ' /proc/mounts); mpts=$(mount | grep -c /mnt/lustre' '); if [ $running -ne $mpts ]; then echo $(hostname) env are INSANE!; exit 1; fi Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre2' ' /proc/mounts); mpts=$(mount | grep -c /mnt/lustre2' '); if [ $running -ne $mpts ]; then echo $(hostname) env are INSANE!; exit 1; fi Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 Lustre: DEBUG MARKER: Using TIMEOUT=20 Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l Lustre: 21737:0:(debug.c:326:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. Lustre: 21737:0:(debug.c:326:libcfs_debug_str2mask()) Skipped 1 previous similar message Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre' ' /proc/mounts); if [ $running -ne 0 ] ; then echo Stopping client $(hostname) /mnt/lustre opts:-f; lsof -t /mnt/lustre || need_kill=no; if [ x-f != x -a x$need_kill != xno ]; then pids=$(lsof -t /mnt/lustre | sort -u); Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request Lustre: 4267:0:(llite_lib.c:2290:ll_dirty_page_discard_warn()) dirty page discard: 10.10.4.154@tcp:/lustre/fid: [0x200002b13:0x1224:0x0]/ may get corrupted (rc -108) Lustre: 4267:0:(llite_lib.c:2290:ll_dirty_page_discard_warn()) dirty page discard: 10.10.4.154@tcp:/lustre/fid: [0x200002b13:0x1237:0x0]/ may get corrupted (rc -108) Lustre: 22194:0:(cl_lock.c:2026:cl_lock_page_out()) Writing 1 pages error: -108 BUG: soft lockup - CPU#0 stuck for 67s! [umount:22194] Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] CPU 0 Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] Pid: 22194, comm: umount Not tainted 2.6.32-279.14.1.el6.x86_64 #1 Red Hat KVM RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff88007b7ef508 EFLAGS: 00000206 RAX: 0000000000000001 RBX: ffff88007b7ef508 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88007ae2c400 RDI: ffff880076b3a734 RBP: ffffffff8100bc0e R08: 0000000000000001 R09: 0000000000000001 R10: 0000000000000000 R11: 0000000000000001 R12: ffff8800760154e0 R13: ffff8800760154e0 R14: ffff880024017cf0 R15: ffff88007b7ef488 FS: 00007f7464baa740(0000) GS:ffff880002200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f74642c81b0 CR3: 0000000078444000 CR4: 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process umount (pid: 22194, threadinfo ffff88007b7ee000, task ffff88007d059500) Stack: ffff88007b7ef5a8 ffffffffa08ee29c ffff8800754f3d98 ffff880073c08400 ffff88007ae2c400 ffff880078d76c38 ffff88007ae2c510 ffff88007ae2c4f0 ffff880076b3a648 0000000100000001 ffff88007ae2c4f8 ffff88007d0449e8 Call Trace: [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b Code: 00 00 00 01 74 05 e8 62 e0 d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 3e 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> 1f 44 00 00 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Call Trace: [] ? osc_cl_page_osc+0x15/0x60 [osc] [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#0 stuck for 67s! [umount:22194] Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] CPU 0 Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] Pid: 22194, comm: umount Not tainted 2.6.32-279.14.1.el6.x86_64 #1 Red Hat KVM RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff88007b7ef508 EFLAGS: 00000206 RAX: 0000000000000001 RBX: ffff88007b7ef508 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88007ae2c400 RDI: ffff880076b3a734 RBP: ffffffff8100bc0e R08: 0000000000000001 R09: 0000000000000001 R10: 0000000000000000 R11: 0000000000000001 R12: ffff8800760154e0 R13: ffff8800760154e0 R14: ffff880024017cf0 R15: ffff88007b7ef488 FS: 00007f7464baa740(0000) GS:ffff880002200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f74642c81b0 CR3: 0000000078444000 CR4: 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process umount (pid: 22194, threadinfo ffff88007b7ee000, task ffff88007d059500) Stack: ffff88007b7ef5a8 ffffffffa08ee29c ffff8800754f3d98 ffff880073c08400 ffff88007ae2c400 ffff880078d76c38 ffff88007ae2c510 ffff88007ae2c4f0 ffff880076b3a648 0000000100000001 ffff88007ae2c4f8 ffff88007d0449e8 Call Trace: [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b Code: 00 00 00 01 74 05 e8 62 e0 d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 3e 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> 1f 44 00 00 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Call Trace: [] ? osc_cl_page_osc+0x15/0x60 [osc] [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#0 stuck for 67s! [umount:22194] Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] CPU 0 Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] Pid: 22194, comm: umount Not tainted 2.6.32-279.14.1.el6.x86_64 #1 Red Hat KVM RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff88007b7ef508 EFLAGS: 00000206 RAX: 0000000000000001 RBX: ffff88007b7ef508 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88007ae2c400 RDI: ffff880076b3a734 RBP: ffffffff8100bc0e R08: 0000000000000001 R09: 0000000000000001 R10: 0000000000000000 R11: 0000000000000001 R12: ffff8800760154e0 R13: ffff8800760154e0 R14: ffff880024017cf0 R15: ffff88007b7ef488 FS: 00007f7464baa740(0000) GS:ffff880002200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f74642c81b0 CR3: 0000000078444000 CR4: 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process umount (pid: 22194, threadinfo ffff88007b7ee000, task ffff88007d059500) Stack: ffff88007b7ef5a8 ffffffffa08ee29c ffff8800754f3d98 ffff880073c08400 ffff88007ae2c400 ffff880078d76c38 ffff88007ae2c510 ffff88007ae2c4f0 ffff880076b3a648 0000000100000001 ffff88007ae2c4f8 ffff88007d0449e8 Call Trace: [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b Code: 00 00 00 01 74 05 e8 62 e0 d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 3e 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> 1f 44 00 00 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Call Trace: [] ? osc_cl_page_osc+0x15/0x60 [osc] [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#0 stuck for 67s! [umount:22194] Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] CPU 0 Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] Pid: 22194, comm: umount Not tainted 2.6.32-279.14.1.el6.x86_64 #1 Red Hat KVM RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff88007b7ef508 EFLAGS: 00000206 RAX: 0000000000000001 RBX: ffff88007b7ef508 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88007ae2c400 RDI: ffff880076b3a734 RBP: ffffffff8100bc0e R08: 0000000000000001 R09: 0000000000000001 R10: 0000000000000000 R11: 0000000000000001 R12: ffff8800760154e0 R13: ffff8800760154e0 R14: ffff880024017cf0 R15: ffff88007b7ef488 FS: 00007f7464baa740(0000) GS:ffff880002200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f74642c81b0 CR3: 0000000078444000 CR4: 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process umount (pid: 22194, threadinfo ffff88007b7ee000, task ffff88007d059500) Stack: ffff88007b7ef5a8 ffffffffa08ee29c ffff8800754f3d98 ffff880073c08400 ffff88007ae2c400 ffff880078d76c38 ffff88007ae2c510 ffff88007ae2c4f0 ffff880076b3a648 0000000100000001 ffff88007ae2c4f8 ffff88007d0449e8 Call Trace: [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b Code: 00 00 00 01 74 05 e8 62 e0 d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 3e 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> 1f 44 00 00 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Call Trace: [] ? osc_cl_page_osc+0x15/0x60 [osc] [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#0 stuck for 67s! [umount:22194] Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] CPU 0 Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] Pid: 22194, comm: umount Not tainted 2.6.32-279.14.1.el6.x86_64 #1 Red Hat KVM RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff88007b7ef508 EFLAGS: 00000206 RAX: 0000000000000001 RBX: ffff88007b7ef508 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88007ae2c400 RDI: ffff880076b3a734 RBP: ffffffff8100bc0e R08: 0000000000000001 R09: 0000000000000001 R10: 0000000000000000 R11: 0000000000000001 R12: ffff8800760154e0 R13: ffff8800760154e0 R14: ffff880024017cf0 R15: ffff88007b7ef488 FS: 00007f7464baa740(0000) GS:ffff880002200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f74642c81b0 CR3: 0000000078444000 CR4: 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process umount (pid: 22194, threadinfo ffff88007b7ee000, task ffff88007d059500) Stack: ffff88007b7ef5a8 ffffffffa08ee29c ffff8800754f3d98 ffff880073c08400 ffff88007ae2c400 ffff880078d76c38 ffff88007ae2c510 ffff88007ae2c4f0 ffff880076b3a648 0000000100000001 ffff88007ae2c4f8 ffff88007d0449e8 Call Trace: [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b Code: 00 00 00 01 74 05 e8 62 e0 d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 3e 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> 1f 44 00 00 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Call Trace: [] ? osc_cl_page_osc+0x15/0x60 [osc] [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#0 stuck for 67s! [umount:22194] Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] CPU 0 Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] Pid: 22194, comm: umount Not tainted 2.6.32-279.14.1.el6.x86_64 #1 Red Hat KVM RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff88007b7ef508 EFLAGS: 00000206 RAX: 0000000000000001 RBX: ffff88007b7ef508 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88007ae2c400 RDI: ffff880076b3a734 RBP: ffffffff8100bc0e R08: 0000000000000001 R09: 0000000000000001 R10: 0000000000000000 R11: 0000000000000001 R12: ffff8800760154e0 R13: ffff8800760154e0 R14: ffff880024017cf0 R15: ffff88007b7ef488 FS: 00007f7464baa740(0000) GS:ffff880002200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f74642c81b0 CR3: 0000000078444000 CR4: 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process umount (pid: 22194, threadinfo ffff88007b7ee000, task ffff88007d059500) Stack: ffff88007b7ef5a8 ffffffffa08ee29c ffff8800754f3d98 ffff880073c08400 ffff88007ae2c400 ffff880078d76c38 ffff88007ae2c510 ffff88007ae2c4f0 ffff880076b3a648 0000000100000001 ffff88007ae2c4f8 ffff88007d0449e8 Call Trace: [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b Code: 00 00 00 01 74 05 e8 62 e0 d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 3e 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> 1f 44 00 00 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Call Trace: [] ? osc_cl_page_osc+0x15/0x60 [osc] [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#0 stuck for 67s! [umount:22194] Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] CPU 0 Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] Pid: 22194, comm: umount Not tainted 2.6.32-279.14.1.el6.x86_64 #1 Red Hat KVM RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff88007b7ef508 EFLAGS: 00000206 RAX: 0000000000000001 RBX: ffff88007b7ef508 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88007ae2c400 RDI: ffff880076b3a734 RBP: ffffffff8100bc0e R08: 0000000000000001 R09: 0000000000000001 R10: 0000000000000000 R11: 0000000000000001 R12: ffff8800760154e0 R13: ffff8800760154e0 R14: ffff880024017cf0 R15: ffff88007b7ef488 FS: 00007f7464baa740(0000) GS:ffff880002200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f74642c81b0 CR3: 0000000078444000 CR4: 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process umount (pid: 22194, threadinfo ffff88007b7ee000, task ffff88007d059500) Stack: ffff88007b7ef5a8 ffffffffa08ee29c ffff8800754f3d98 ffff880073c08400 ffff88007ae2c400 ffff880078d76c38 ffff88007ae2c510 ffff88007ae2c4f0 ffff880076b3a648 0000000100000001 ffff88007ae2c4f8 ffff88007d0449e8 Call Trace: [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b Code: 00 00 00 01 74 05 e8 62 e0 d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 3e 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> 1f 44 00 00 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Call Trace: [] ? osc_cl_page_osc+0x15/0x60 [osc] [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#0 stuck for 67s! [umount:22194] Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] CPU 0 Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] Pid: 22194, comm: umount Not tainted 2.6.32-279.14.1.el6.x86_64 #1 Red Hat KVM RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff88007b7ef508 EFLAGS: 00000206 RAX: 0000000000000001 RBX: ffff88007b7ef508 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88007ae2c400 RDI: ffff880076b3a734 RBP: ffffffff8100bc0e R08: 0000000000000001 R09: 0000000000000001 R10: 0000000000000000 R11: 0000000000000001 R12: ffff8800760154e0 R13: ffff8800760154e0 R14: ffff880024017cf0 R15: ffff88007b7ef488 FS: 00007f7464baa740(0000) GS:ffff880002200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f74642c81b0 CR3: 0000000078444000 CR4: 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process umount (pid: 22194, threadinfo ffff88007b7ee000, task ffff88007d059500) Stack: ffff88007b7ef5a8 ffffffffa08ee29c ffff8800754f3d98 ffff880073c08400 ffff88007ae2c400 ffff880078d76c38 ffff88007ae2c510 ffff88007ae2c4f0 ffff880076b3a648 0000000100000001 ffff88007ae2c4f8 ffff88007d0449e8 Call Trace: [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b Code: 00 00 00 01 74 05 e8 62 e0 d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 3e 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> 1f 44 00 00 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Call Trace: [] ? osc_cl_page_osc+0x15/0x60 [osc] [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#0 stuck for 67s! [umount:22194] Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] CPU 0 Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] Pid: 22194, comm: umount Not tainted 2.6.32-279.14.1.el6.x86_64 #1 Red Hat KVM RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff88007b7ef508 EFLAGS: 00000206 RAX: 0000000000000001 RBX: ffff88007b7ef508 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88007ae2c400 RDI: ffff880076b3a734 RBP: ffffffff8100bc0e R08: 0000000000000001 R09: 0000000000000001 R10: 0000000000000000 R11: 0000000000000001 R12: ffff8800760154e0 R13: ffff8800760154e0 R14: ffff880024017cf0 R15: ffff88007b7ef488 FS: 00007f7464baa740(0000) GS:ffff880002200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f74642c81b0 CR3: 0000000078444000 CR4: 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process umount (pid: 22194, threadinfo ffff88007b7ee000, task ffff88007d059500) Stack: ffff88007b7ef5a8 ffffffffa08ee29c ffff8800754f3d98 ffff880073c08400 ffff88007ae2c400 ffff880078d76c38 ffff88007ae2c510 ffff88007ae2c4f0 ffff880076b3a648 0000000100000001 ffff88007ae2c4f8 ffff88007d0449e8 Call Trace: [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b Code: 00 00 00 01 74 05 e8 62 e0 d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 3e 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> 1f 44 00 00 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Call Trace: [] ? osc_cl_page_osc+0x15/0x60 [osc] [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#0 stuck for 68s! [umount:22194] Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] CPU 0 Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] Pid: 22194, comm: umount Not tainted 2.6.32-279.14.1.el6.x86_64 #1 Red Hat KVM RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff88007b7ef508 EFLAGS: 00000206 RAX: 0000000000000001 RBX: ffff88007b7ef508 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88007ae2c400 RDI: ffff880076b3a734 RBP: ffffffff8100bc0e R08: 0000000000000001 R09: 0000000000000001 R10: 0000000000000000 R11: 0000000000000001 R12: ffff8800760154e0 R13: ffff8800760154e0 R14: ffff880024017cf0 R15: ffff88007b7ef488 FS: 00007f7464baa740(0000) GS:ffff880002200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f74642c81b0 CR3: 0000000078444000 CR4: 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process umount (pid: 22194, threadinfo ffff88007b7ee000, task ffff88007d059500) Stack: ffff88007b7ef5a8 ffffffffa08ee29c ffff8800754f3d98 ffff880073c08400 ffff88007ae2c400 ffff880078d76c38 ffff88007ae2c510 ffff88007ae2c4f0 ffff880076b3a648 0000000100000001 ffff88007ae2c4f8 ffff88007d0449e8 Call Trace: [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b Code: 00 00 00 01 74 05 e8 62 e0 d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 3e 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> 1f 44 00 00 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Call Trace: [] ? osc_cl_page_osc+0x15/0x60 [osc] [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#0 stuck for 67s! [umount:22194] Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] CPU 0 Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] Pid: 22194, comm: umount Not tainted 2.6.32-279.14.1.el6.x86_64 #1 Red Hat KVM RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff88007b7ef508 EFLAGS: 00000206 RAX: 0000000000000001 RBX: ffff88007b7ef508 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88007ae2c400 RDI: ffff880076b3a734 RBP: ffffffff8100bc0e R08: 0000000000000001 R09: 0000000000000001 R10: 0000000000000000 R11: 0000000000000001 R12: ffff8800760154e0 R13: ffff8800760154e0 R14: ffff880024017cf0 R15: ffff88007b7ef488 FS: 00007f7464baa740(0000) GS:ffff880002200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f74642c81b0 CR3: 0000000078444000 CR4: 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process umount (pid: 22194, threadinfo ffff88007b7ee000, task ffff88007d059500) Stack: ffff88007b7ef5a8 ffffffffa08ee29c ffff8800754f3d98 ffff880073c08400 ffff88007ae2c400 ffff880078d76c38 ffff88007ae2c510 ffff88007ae2c4f0 ffff880076b3a648 0000000100000001 ffff88007ae2c4f8 ffff88007d0449e8 Call Trace: [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b Code: 00 00 00 01 74 05 e8 62 e0 d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 3e 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> 1f 44 00 00 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Call Trace: [] ? osc_cl_page_osc+0x15/0x60 [osc] [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#0 stuck for 67s! [umount:22194] Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] CPU 0 Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] Pid: 22194, comm: umount Not tainted 2.6.32-279.14.1.el6.x86_64 #1 Red Hat KVM RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff88007b7ef508 EFLAGS: 00000206 RAX: 0000000000000001 RBX: ffff88007b7ef508 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88007ae2c400 RDI: ffff880076b3a734 RBP: ffffffff8100bc0e R08: 0000000000000001 R09: 0000000000000001 R10: 0000000000000000 R11: 0000000000000001 R12: ffff8800760154e0 R13: ffff8800760154e0 R14: ffff880024017cf0 R15: ffff88007b7ef488 FS: 00007f7464baa740(0000) GS:ffff880002200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f74642c81b0 CR3: 0000000078444000 CR4: 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process umount (pid: 22194, threadinfo ffff88007b7ee000, task ffff88007d059500) Stack: ffff88007b7ef5a8 ffffffffa08ee29c ffff8800754f3d98 ffff880073c08400 ffff88007ae2c400 ffff880078d76c38 ffff88007ae2c510 ffff88007ae2c4f0 ffff880076b3a648 0000000100000001 ffff88007ae2c4f8 ffff88007d0449e8 Call Trace: [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b Code: 00 00 00 01 74 05 e8 62 e0 d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 3e 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> 1f 44 00 00 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Call Trace: [] ? osc_cl_page_osc+0x15/0x60 [osc] [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#0 stuck for 67s! [umount:22194] Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] CPU 0 Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] Pid: 22194, comm: umount Not tainted 2.6.32-279.14.1.el6.x86_64 #1 Red Hat KVM RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff88007b7ef508 EFLAGS: 00000206 RAX: 0000000000000001 RBX: ffff88007b7ef508 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88007ae2c400 RDI: ffff880076b3a734 RBP: ffffffff8100bc0e R08: 0000000000000001 R09: 0000000000000001 R10: 0000000000000000 R11: 0000000000000001 R12: ffff8800760154e0 R13: ffff8800760154e0 R14: ffff880024017cf0 R15: ffff88007b7ef488 FS: 00007f7464baa740(0000) GS:ffff880002200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f74642c81b0 CR3: 0000000078444000 CR4: 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process umount (pid: 22194, threadinfo ffff88007b7ee000, task ffff88007d059500) Stack: ffff88007b7ef5a8 ffffffffa08ee29c ffff8800754f3d98 ffff880073c08400 ffff88007ae2c400 ffff880078d76c38 ffff88007ae2c510 ffff88007ae2c4f0 ffff880076b3a648 0000000100000001 ffff88007ae2c4f8 ffff88007d0449e8 Call Trace: [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b Code: 00 00 00 01 74 05 e8 62 e0 d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 3e 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> 1f 44 00 00 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Call Trace: [] ? osc_cl_page_osc+0x15/0x60 [osc] [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#0 stuck for 67s! [umount:22194] Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] CPU 0 Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] Pid: 22194, comm: umount Not tainted 2.6.32-279.14.1.el6.x86_64 #1 Red Hat KVM RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff88007b7ef508 EFLAGS: 00000206 RAX: 0000000000000001 RBX: ffff88007b7ef508 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88007ae2c400 RDI: ffff880076b3a734 RBP: ffffffff8100bc0e R08: 0000000000000001 R09: 0000000000000001 R10: 0000000000000000 R11: 0000000000000001 R12: ffff8800760154e0 R13: ffff8800760154e0 R14: ffff880024017cf0 R15: ffff88007b7ef488 FS: 00007f7464baa740(0000) GS:ffff880002200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f74642c81b0 CR3: 0000000078444000 CR4: 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process umount (pid: 22194, threadinfo ffff88007b7ee000, task ffff88007d059500) Stack: ffff88007b7ef5a8 ffffffffa08ee29c ffff8800754f3d98 ffff880073c08400 ffff88007ae2c400 ffff880078d76c38 ffff88007ae2c510 ffff88007ae2c4f0 ffff880076b3a648 0000000100000001 ffff88007ae2c4f8 ffff88007d0449e8 Call Trace: [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b Code: 00 00 00 01 74 05 e8 62 e0 d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 3e 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> 1f 44 00 00 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Call Trace: [] ? osc_cl_page_osc+0x15/0x60 [osc] [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#0 stuck for 67s! [umount:22194] Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] CPU 0 Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] Pid: 22194, comm: umount Not tainted 2.6.32-279.14.1.el6.x86_64 #1 Red Hat KVM RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff88007b7ef508 EFLAGS: 00000206 RAX: 0000000000000001 RBX: ffff88007b7ef508 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88007ae2c400 RDI: ffff880076b3a734 RBP: ffffffff8100bc0e R08: 0000000000000001 R09: 0000000000000001 R10: 0000000000000000 R11: 0000000000000001 R12: ffff8800760154e0 R13: ffff8800760154e0 R14: ffff880024017cf0 R15: ffff88007b7ef488 FS: 00007f7464baa740(0000) GS:ffff880002200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f74642c81b0 CR3: 0000000078444000 CR4: 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process umount (pid: 22194, threadinfo ffff88007b7ee000, task ffff88007d059500) Stack: ffff88007b7ef5a8 ffffffffa08ee29c ffff8800754f3d98 ffff880073c08400 ffff88007ae2c400 ffff880078d76c38 ffff88007ae2c510 ffff88007ae2c4f0 ffff880076b3a648 0000000100000001 ffff88007ae2c4f8 ffff88007d0449e8 Call Trace: [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b Code: 00 00 00 01 74 05 e8 62 e0 d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 3e 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> 1f 44 00 00 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Call Trace: [] ? osc_cl_page_osc+0x15/0x60 [osc] [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#0 stuck for 67s! [umount:22194] Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] CPU 0 Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] Pid: 22194, comm: umount Not tainted 2.6.32-279.14.1.el6.x86_64 #1 Red Hat KVM RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff88007b7ef508 EFLAGS: 00000206 RAX: 0000000000000001 RBX: ffff88007b7ef508 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88007ae2c400 RDI: ffff880076b3a734 RBP: ffffffff8100bc0e R08: 0000000000000001 R09: 0000000000000001 R10: 0000000000000000 R11: 0000000000000001 R12: ffff8800760154e0 R13: ffff8800760154e0 R14: ffff880024017cf0 R15: ffff88007b7ef488 FS: 00007f7464baa740(0000) GS:ffff880002200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f74642c81b0 CR3: 0000000078444000 CR4: 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process umount (pid: 22194, threadinfo ffff88007b7ee000, task ffff88007d059500) Stack: ffff88007b7ef5a8 ffffffffa08ee29c ffff8800754f3d98 ffff880073c08400 ffff88007ae2c400 ffff880078d76c38 ffff88007ae2c510 ffff88007ae2c4f0 ffff880076b3a648 0000000100000001 ffff88007ae2c4f8 ffff88007d0449e8 Call Trace: [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b Code: 00 00 00 01 74 05 e8 62 e0 d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 3e 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> 1f 44 00 00 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Call Trace: [] ? osc_cl_page_osc+0x15/0x60 [osc] [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#0 stuck for 67s! [umount:22194] Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] CPU 0 Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] Pid: 22194, comm: umount Not tainted 2.6.32-279.14.1.el6.x86_64 #1 Red Hat KVM RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff88007b7ef508 EFLAGS: 00000206 RAX: 0000000000000001 RBX: ffff88007b7ef508 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88007ae2c400 RDI: ffff880076b3a734 RBP: ffffffff8100bc0e R08: 0000000000000001 R09: 0000000000000001 R10: 0000000000000000 R11: 0000000000000001 R12: ffff8800760154e0 R13: ffff8800760154e0 R14: ffff880024017cf0 R15: ffff88007b7ef488 FS: 00007f7464baa740(0000) GS:ffff880002200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f74642c81b0 CR3: 0000000078444000 CR4: 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process umount (pid: 22194, threadinfo ffff88007b7ee000, task ffff88007d059500) Stack: ffff88007b7ef5a8 ffffffffa08ee29c ffff8800754f3d98 ffff880073c08400 ffff88007ae2c400 ffff880078d76c38 ffff88007ae2c510 ffff88007ae2c4f0 ffff880076b3a648 0000000100000001 ffff88007ae2c4f8 ffff88007d0449e8 Call Trace: [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b Code: 00 00 00 01 74 05 e8 62 e0 d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 3e 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> 1f 44 00 00 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Call Trace: [] ? osc_cl_page_osc+0x15/0x60 [osc] [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#0 stuck for 67s! [umount:22194] Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] CPU 0 Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] Pid: 22194, comm: umount Not tainted 2.6.32-279.14.1.el6.x86_64 #1 Red Hat KVM RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff88007b7ef508 EFLAGS: 00000206 RAX: 0000000000000001 RBX: ffff88007b7ef508 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88007ae2c400 RDI: ffff880076b3a734 RBP: ffffffff8100bc0e R08: 0000000000000001 R09: 0000000000000001 R10: 0000000000000000 R11: 0000000000000001 R12: ffff8800760154e0 R13: ffff8800760154e0 R14: ffff880024017cf0 R15: ffff88007b7ef488 FS: 00007f7464baa740(0000) GS:ffff880002200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f74642c81b0 CR3: 0000000078444000 CR4: 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process umount (pid: 22194, threadinfo ffff88007b7ee000, task ffff88007d059500) Stack: ffff88007b7ef5a8 ffffffffa08ee29c ffff8800754f3d98 ffff880073c08400 ffff88007ae2c400 ffff880078d76c38 ffff88007ae2c510 ffff88007ae2c4f0 ffff880076b3a648 0000000100000001 ffff88007ae2c4f8 ffff88007d0449e8 Call Trace: [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b Code: 00 00 00 01 74 05 e8 62 e0 d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 3e 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> 1f 44 00 00 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Call Trace: [] ? osc_cl_page_osc+0x15/0x60 [osc] [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#0 stuck for 67s! [umount:22194] Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] CPU 0 Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] Pid: 22194, comm: umount Not tainted 2.6.32-279.14.1.el6.x86_64 #1 Red Hat KVM RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff88007b7ef508 EFLAGS: 00000206 RAX: 0000000000000001 RBX: ffff88007b7ef508 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88007ae2c400 RDI: ffff880076b3a734 RBP: ffffffff8100bc0e R08: 0000000000000001 R09: 0000000000000001 R10: 0000000000000000 R11: 0000000000000001 R12: ffff8800760154e0 R13: ffff8800760154e0 R14: ffff880024017cf0 R15: ffff88007b7ef488 FS: 00007f7464baa740(0000) GS:ffff880002200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f74642c81b0 CR3: 0000000078444000 CR4: 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process umount (pid: 22194, threadinfo ffff88007b7ee000, task ffff88007d059500) Stack: ffff88007b7ef5a8 ffffffffa08ee29c ffff8800754f3d98 ffff880073c08400 ffff88007ae2c400 ffff880078d76c38 ffff88007ae2c510 ffff88007ae2c4f0 ffff880076b3a648 0000000100000001 ffff88007ae2c4f8 ffff88007d0449e8 Call Trace: [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b Code: 00 00 00 01 74 05 e8 62 e0 d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 3e 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> 1f 44 00 00 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Call Trace: [] ? osc_cl_page_osc+0x15/0x60 [osc] [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#0 stuck for 67s! [umount:22194] Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] CPU 0 Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] Pid: 22194, comm: umount Not tainted 2.6.32-279.14.1.el6.x86_64 #1 Red Hat KVM RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff88007b7ef508 EFLAGS: 00000206 RAX: 0000000000000001 RBX: ffff88007b7ef508 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88007ae2c400 RDI: ffff880076b3a734 RBP: ffffffff8100bc0e R08: 0000000000000001 R09: 0000000000000001 R10: 0000000000000000 R11: 0000000000000001 R12: ffff8800760154e0 R13: ffff8800760154e0 R14: ffff880024017cf0 R15: ffff88007b7ef488 FS: 00007f7464baa740(0000) GS:ffff880002200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f74642c81b0 CR3: 0000000078444000 CR4: 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process umount (pid: 22194, threadinfo ffff88007b7ee000, task ffff88007d059500) Stack: ffff88007b7ef5a8 ffffffffa08ee29c ffff8800754f3d98 ffff880073c08400 ffff88007ae2c400 ffff880078d76c38 ffff88007ae2c510 ffff88007ae2c4f0 ffff880076b3a648 0000000100000001 ffff88007ae2c4f8 ffff88007d0449e8 Call Trace: [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b Code: 00 00 00 01 74 05 e8 62 e0 d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 3e 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> 1f 44 00 00 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Call Trace: [] ? osc_cl_page_osc+0x15/0x60 [osc] [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#0 stuck for 67s! [umount:22194] Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] CPU 0 Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] Pid: 22194, comm: umount Not tainted 2.6.32-279.14.1.el6.x86_64 #1 Red Hat KVM RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff88007b7ef508 EFLAGS: 00000206 RAX: 0000000000000001 RBX: ffff88007b7ef508 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88007ae2c400 RDI: ffff880076b3a734 RBP: ffffffff8100bc0e R08: 0000000000000001 R09: 0000000000000001 R10: 0000000000000000 R11: 0000000000000001 R12: ffff8800760154e0 R13: ffff8800760154e0 R14: ffff880024017cf0 R15: ffff88007b7ef488 FS: 00007f7464baa740(0000) GS:ffff880002200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f74642c81b0 CR3: 0000000078444000 CR4: 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process umount (pid: 22194, threadinfo ffff88007b7ee000, task ffff88007d059500) Stack: ffff88007b7ef5a8 ffffffffa08ee29c ffff8800754f3d98 ffff880073c08400 ffff88007ae2c400 ffff880078d76c38 ffff88007ae2c510 ffff88007ae2c4f0 ffff880076b3a648 0000000100000001 ffff88007ae2c4f8 ffff88007d0449e8 Call Trace: [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b Code: 00 00 00 01 74 05 e8 62 e0 d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 3e 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> 1f 44 00 00 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Call Trace: [] ? osc_cl_page_osc+0x15/0x60 [osc] [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#0 stuck for 67s! [umount:22194] Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] CPU 0 Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] Pid: 22194, comm: umount Not tainted 2.6.32-279.14.1.el6.x86_64 #1 Red Hat KVM RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff88007b7ef508 EFLAGS: 00000206 RAX: 0000000000000001 RBX: ffff88007b7ef508 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88007ae2c400 RDI: ffff880076b3a734 RBP: ffffffff8100bc0e R08: 0000000000000001 R09: 0000000000000001 R10: 0000000000000000 R11: 0000000000000001 R12: ffff8800760154e0 R13: ffff8800760154e0 R14: ffff880024017cf0 R15: ffff88007b7ef488 FS: 00007f7464baa740(0000) GS:ffff880002200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f74642c81b0 CR3: 0000000078444000 CR4: 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process umount (pid: 22194, threadinfo ffff88007b7ee000, task ffff88007d059500) Stack: ffff88007b7ef5a8 ffffffffa08ee29c ffff8800754f3d98 ffff880073c08400 ffff88007ae2c400 ffff880078d76c38 ffff88007ae2c510 ffff88007ae2c4f0 ffff880076b3a648 0000000100000001 ffff88007ae2c4f8 ffff88007d0449e8 Call Trace: [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b Code: 00 00 00 01 74 05 e8 62 e0 d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 3e 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> 1f 44 00 00 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Call Trace: [] ? osc_cl_page_osc+0x15/0x60 [osc] [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#0 stuck for 68s! [umount:22194] Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] CPU 0 Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] Pid: 22194, comm: umount Not tainted 2.6.32-279.14.1.el6.x86_64 #1 Red Hat KVM RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff88007b7ef508 EFLAGS: 00000206 RAX: 0000000000000001 RBX: ffff88007b7ef508 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88007ae2c400 RDI: ffff880076b3a734 RBP: ffffffff8100bc0e R08: 0000000000000001 R09: 0000000000000001 R10: 0000000000000000 R11: 0000000000000001 R12: ffff8800760154e0 R13: ffff8800760154e0 R14: ffff880024017cf0 R15: ffff88007b7ef488 FS: 00007f7464baa740(0000) GS:ffff880002200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f74642c81b0 CR3: 0000000078444000 CR4: 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process umount (pid: 22194, threadinfo ffff88007b7ee000, task ffff88007d059500) Stack: ffff88007b7ef5a8 ffffffffa08ee29c ffff8800754f3d98 ffff880073c08400 ffff88007ae2c400 ffff880078d76c38 ffff88007ae2c510 ffff88007ae2c4f0 ffff880076b3a648 0000000100000001 ffff88007ae2c4f8 ffff88007d0449e8 Call Trace: [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b Code: 00 00 00 01 74 05 e8 62 e0 d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 3e 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> 1f 44 00 00 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Call Trace: [] ? osc_cl_page_osc+0x15/0x60 [osc] [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#0 stuck for 67s! [umount:22194] Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] CPU 0 Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] Pid: 22194, comm: umount Not tainted 2.6.32-279.14.1.el6.x86_64 #1 Red Hat KVM RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff88007b7ef508 EFLAGS: 00000206 RAX: 0000000000000001 RBX: ffff88007b7ef508 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88007ae2c400 RDI: ffff880076b3a734 RBP: ffffffff8100bc0e R08: 0000000000000001 R09: 0000000000000001 R10: 0000000000000000 R11: 0000000000000001 R12: ffff8800760154e0 R13: ffff8800760154e0 R14: ffff880024017cf0 R15: ffff88007b7ef488 FS: 00007f7464baa740(0000) GS:ffff880002200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f74642c81b0 CR3: 0000000078444000 CR4: 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process umount (pid: 22194, threadinfo ffff88007b7ee000, task ffff88007d059500) Stack: ffff88007b7ef5a8 ffffffffa08ee29c ffff8800754f3d98 ffff880073c08400 ffff88007ae2c400 ffff880078d76c38 ffff88007ae2c510 ffff88007ae2c4f0 ffff880076b3a648 0000000100000001 ffff88007ae2c4f8 ffff88007d0449e8 Call Trace: [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b Code: 00 00 00 01 74 05 e8 62 e0 d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 3e 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> 1f 44 00 00 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Call Trace: [] ? osc_cl_page_osc+0x15/0x60 [osc] [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#0 stuck for 67s! [umount:22194] Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] CPU 0 Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] Pid: 22194, comm: umount Not tainted 2.6.32-279.14.1.el6.x86_64 #1 Red Hat KVM RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff88007b7ef508 EFLAGS: 00000206 RAX: 0000000000000001 RBX: ffff88007b7ef508 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88007ae2c400 RDI: ffff880076b3a734 RBP: ffffffff8100bc0e R08: 0000000000000001 R09: 0000000000000001 R10: 0000000000000000 R11: 0000000000000001 R12: ffff8800760154e0 R13: ffff8800760154e0 R14: ffff880024017cf0 R15: ffff88007b7ef488 FS: 00007f7464baa740(0000) GS:ffff880002200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f74642c81b0 CR3: 0000000078444000 CR4: 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process umount (pid: 22194, threadinfo ffff88007b7ee000, task ffff88007d059500) Stack: ffff88007b7ef5a8 ffffffffa08ee29c ffff8800754f3d98 ffff880073c08400 ffff88007ae2c400 ffff880078d76c38 ffff88007ae2c510 ffff88007ae2c4f0 ffff880076b3a648 0000000100000001 ffff88007ae2c4f8 ffff88007d0449e8 Call Trace: [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b Code: 00 00 00 01 74 05 e8 62 e0 d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 3e 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> 1f 44 00 00 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Call Trace: [] ? osc_cl_page_osc+0x15/0x60 [osc] [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#0 stuck for 67s! [umount:22194] Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] CPU 0 Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] Pid: 22194, comm: umount Not tainted 2.6.32-279.14.1.el6.x86_64 #1 Red Hat KVM RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff88007b7ef508 EFLAGS: 00000206 RAX: 0000000000000001 RBX: ffff88007b7ef508 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88007ae2c400 RDI: ffff880076b3a734 RBP: ffffffff8100bc0e R08: 0000000000000001 R09: 0000000000000001 R10: 0000000000000000 R11: 0000000000000001 R12: ffff8800760154e0 R13: ffff8800760154e0 R14: ffff880024017cf0 R15: ffff88007b7ef488 FS: 00007f7464baa740(0000) GS:ffff880002200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f74642c81b0 CR3: 0000000078444000 CR4: 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process umount (pid: 22194, threadinfo ffff88007b7ee000, task ffff88007d059500) Stack: ffff88007b7ef5a8 ffffffffa08ee29c ffff8800754f3d98 ffff880073c08400 ffff88007ae2c400 ffff880078d76c38 ffff88007ae2c510 ffff88007ae2c4f0 ffff880076b3a648 0000000100000001 ffff88007ae2c4f8 ffff88007d0449e8 Call Trace: [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b Code: 00 00 00 01 74 05 e8 62 e0 d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 3e 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> 1f 44 00 00 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Call Trace: [] ? osc_cl_page_osc+0x15/0x60 [osc] [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#0 stuck for 67s! [umount:22194] Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] CPU 0 Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] Pid: 22194, comm: umount Not tainted 2.6.32-279.14.1.el6.x86_64 #1 Red Hat KVM RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff88007b7ef508 EFLAGS: 00000206 RAX: 0000000000000001 RBX: ffff88007b7ef508 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88007ae2c400 RDI: ffff880076b3a734 RBP: ffffffff8100bc0e R08: 0000000000000001 R09: 0000000000000001 R10: 0000000000000000 R11: 0000000000000001 R12: ffff8800760154e0 R13: ffff8800760154e0 R14: ffff880024017cf0 R15: ffff88007b7ef488 FS: 00007f7464baa740(0000) GS:ffff880002200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f74642c81b0 CR3: 0000000078444000 CR4: 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process umount (pid: 22194, threadinfo ffff88007b7ee000, task ffff88007d059500) Stack: ffff88007b7ef5a8 ffffffffa08ee29c ffff8800754f3d98 ffff880073c08400 ffff88007ae2c400 ffff880078d76c38 ffff88007ae2c510 ffff88007ae2c4f0 ffff880076b3a648 0000000100000001 ffff88007ae2c4f8 ffff88007d0449e8 Call Trace: [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b Code: 00 00 00 01 74 05 e8 62 e0 d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 3e 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> 1f 44 00 00 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Call Trace: [] ? osc_cl_page_osc+0x15/0x60 [osc] [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#0 stuck for 67s! [umount:22194] Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] CPU 0 Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] Pid: 22194, comm: umount Not tainted 2.6.32-279.14.1.el6.x86_64 #1 Red Hat KVM RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff88007b7ef508 EFLAGS: 00000206 RAX: 0000000000000001 RBX: ffff88007b7ef508 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88007ae2c400 RDI: ffff880076b3a734 RBP: ffffffff8100bc0e R08: 0000000000000001 R09: 0000000000000001 R10: 0000000000000000 R11: 0000000000000001 R12: ffff8800760154e0 R13: ffff8800760154e0 R14: ffff880024017cf0 R15: ffff88007b7ef488 FS: 00007f7464baa740(0000) GS:ffff880002200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f74642c81b0 CR3: 0000000078444000 CR4: 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process umount (pid: 22194, threadinfo ffff88007b7ee000, task ffff88007d059500) Stack: ffff88007b7ef5a8 ffffffffa08ee29c ffff8800754f3d98 ffff880073c08400 ffff88007ae2c400 ffff880078d76c38 ffff88007ae2c510 ffff88007ae2c4f0 ffff880076b3a648 0000000100000001 ffff88007ae2c4f8 ffff88007d0449e8 Call Trace: [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b Code: 00 00 00 01 74 05 e8 62 e0 d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 3e 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> 1f 44 00 00 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Call Trace: [] ? osc_cl_page_osc+0x15/0x60 [osc] [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#0 stuck for 67s! [umount:22194] Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] CPU 0 Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] Pid: 22194, comm: umount Not tainted 2.6.32-279.14.1.el6.x86_64 #1 Red Hat KVM RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff88007b7ef508 EFLAGS: 00000206 RAX: 0000000000000001 RBX: ffff88007b7ef508 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88007ae2c400 RDI: ffff880076b3a734 RBP: ffffffff8100bc0e R08: 0000000000000001 R09: 0000000000000001 R10: 0000000000000000 R11: 0000000000000001 R12: ffff8800760154e0 R13: ffff8800760154e0 R14: ffff880024017cf0 R15: ffff88007b7ef488 FS: 00007f7464baa740(0000) GS:ffff880002200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f74642c81b0 CR3: 0000000078444000 CR4: 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process umount (pid: 22194, threadinfo ffff88007b7ee000, task ffff88007d059500) Stack: ffff88007b7ef5a8 ffffffffa08ee29c ffff8800754f3d98 ffff880073c08400 ffff88007ae2c400 ffff880078d76c38 ffff88007ae2c510 ffff88007ae2c4f0 ffff880076b3a648 0000000100000001 ffff88007ae2c4f8 ffff88007d0449e8 Call Trace: [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b Code: 00 00 00 01 74 05 e8 62 e0 d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 3e 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> 1f 44 00 00 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Call Trace: [] ? osc_cl_page_osc+0x15/0x60 [osc] [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#0 stuck for 67s! [umount:22194] Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] CPU 0 Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] Pid: 22194, comm: umount Not tainted 2.6.32-279.14.1.el6.x86_64 #1 Red Hat KVM RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff88007b7ef508 EFLAGS: 00000206 RAX: 0000000000000001 RBX: ffff88007b7ef508 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88007ae2c400 RDI: ffff880076b3a734 RBP: ffffffff8100bc0e R08: 0000000000000001 R09: 0000000000000001 R10: 0000000000000000 R11: 0000000000000001 R12: ffff8800760154e0 R13: ffff8800760154e0 R14: ffff880024017cf0 R15: ffff88007b7ef488 FS: 00007f7464baa740(0000) GS:ffff880002200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f74642c81b0 CR3: 0000000078444000 CR4: 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process umount (pid: 22194, threadinfo ffff88007b7ee000, task ffff88007d059500) Stack: ffff88007b7ef5a8 ffffffffa08ee29c ffff8800754f3d98 ffff880073c08400 ffff88007ae2c400 ffff880078d76c38 ffff88007ae2c510 ffff88007ae2c4f0 ffff880076b3a648 0000000100000001 ffff88007ae2c4f8 ffff88007d0449e8 Call Trace: [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b Code: 00 00 00 01 74 05 e8 62 e0 d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 3e 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> 1f 44 00 00 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Call Trace: [] ? osc_cl_page_osc+0x15/0x60 [osc] [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#0 stuck for 67s! [umount:22194] Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] CPU 0 Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] Pid: 22194, comm: umount Not tainted 2.6.32-279.14.1.el6.x86_64 #1 Red Hat KVM RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff88007b7ef508 EFLAGS: 00000206 RAX: 0000000000000001 RBX: ffff88007b7ef508 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88007ae2c400 RDI: ffff880076b3a734 RBP: ffffffff8100bc0e R08: 0000000000000001 R09: 0000000000000001 R10: 0000000000000000 R11: 0000000000000001 R12: ffff8800760154e0 R13: ffff8800760154e0 R14: ffff880024017cf0 R15: ffff88007b7ef488 FS: 00007f7464baa740(0000) GS:ffff880002200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f74642c81b0 CR3: 0000000078444000 CR4: 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process umount (pid: 22194, threadinfo ffff88007b7ee000, task ffff88007d059500) Stack: ffff88007b7ef5a8 ffffffffa08ee29c ffff8800754f3d98 ffff880073c08400 ffff88007ae2c400 ffff880078d76c38 ffff88007ae2c510 ffff88007ae2c4f0 ffff880076b3a648 0000000100000001 ffff88007ae2c4f8 ffff88007d0449e8 Call Trace: [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b Code: 00 00 00 01 74 05 e8 62 e0 d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 3e 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> 1f 44 00 00 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Call Trace: [] ? osc_cl_page_osc+0x15/0x60 [osc] [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#0 stuck for 67s! [umount:22194] Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] CPU 0 Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] Pid: 22194, comm: umount Not tainted 2.6.32-279.14.1.el6.x86_64 #1 Red Hat KVM RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff88007b7ef508 EFLAGS: 00000206 RAX: 0000000000000001 RBX: ffff88007b7ef508 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88007ae2c400 RDI: ffff880076b3a734 RBP: ffffffff8100bc0e R08: 0000000000000001 R09: 0000000000000001 R10: 0000000000000000 R11: 0000000000000001 R12: ffff8800760154e0 R13: ffff8800760154e0 R14: ffff880024017cf0 R15: ffff88007b7ef488 FS: 00007f7464baa740(0000) GS:ffff880002200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f74642c81b0 CR3: 0000000078444000 CR4: 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process umount (pid: 22194, threadinfo ffff88007b7ee000, task ffff88007d059500) Stack: ffff88007b7ef5a8 ffffffffa08ee29c ffff8800754f3d98 ffff880073c08400 ffff88007ae2c400 ffff880078d76c38 ffff88007ae2c510 ffff88007ae2c4f0 ffff880076b3a648 0000000100000001 ffff88007ae2c4f8 ffff88007d0449e8 Call Trace: [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b Code: 00 00 00 01 74 05 e8 62 e0 d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 3e 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> 1f 44 00 00 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Call Trace: [] ? osc_cl_page_osc+0x15/0x60 [osc] [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#0 stuck for 67s! [umount:22194] Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] CPU 0 Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] Pid: 22194, comm: umount Not tainted 2.6.32-279.14.1.el6.x86_64 #1 Red Hat KVM RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff88007b7ef508 EFLAGS: 00000206 RAX: 0000000000000001 RBX: ffff88007b7ef508 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88007ae2c400 RDI: ffff880076b3a734 RBP: ffffffff8100bc0e R08: 0000000000000001 R09: 0000000000000001 R10: 0000000000000000 R11: 0000000000000001 R12: ffff8800760154e0 R13: ffff8800760154e0 R14: ffff880024017cf0 R15: ffff88007b7ef488 FS: 00007f7464baa740(0000) GS:ffff880002200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f74642c81b0 CR3: 0000000078444000 CR4: 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process umount (pid: 22194, threadinfo ffff88007b7ee000, task ffff88007d059500) Stack: ffff88007b7ef5a8 ffffffffa08ee29c ffff8800754f3d98 ffff880073c08400 ffff88007ae2c400 ffff880078d76c38 ffff88007ae2c510 ffff88007ae2c4f0 ffff880076b3a648 0000000100000001 ffff88007ae2c4f8 ffff88007d0449e8 Call Trace: [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b Code: 00 00 00 01 74 05 e8 62 e0 d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 3e 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> 1f 44 00 00 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Call Trace: [] ? osc_cl_page_osc+0x15/0x60 [osc] [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#0 stuck for 67s! [umount:22194] Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] CPU 0 Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] Pid: 22194, comm: umount Not tainted 2.6.32-279.14.1.el6.x86_64 #1 Red Hat KVM RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff88007b7ef508 EFLAGS: 00000206 RAX: 0000000000000001 RBX: ffff88007b7ef508 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88007ae2c400 RDI: ffff880076b3a734 RBP: ffffffff8100bc0e R08: 0000000000000001 R09: 0000000000000001 R10: 0000000000000000 R11: 0000000000000001 R12: ffff8800760154e0 R13: ffff8800760154e0 R14: ffff880024017cf0 R15: ffff88007b7ef488 FS: 00007f7464baa740(0000) GS:ffff880002200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f74642c81b0 CR3: 0000000078444000 CR4: 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process umount (pid: 22194, threadinfo ffff88007b7ee000, task ffff88007d059500) Stack: ffff88007b7ef5a8 ffffffffa08ee29c ffff8800754f3d98 ffff880073c08400 ffff88007ae2c400 ffff880078d76c38 ffff88007ae2c510 ffff88007ae2c4f0 ffff880076b3a648 0000000100000001 ffff88007ae2c4f8 ffff88007d0449e8 Call Trace: [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b Code: 00 00 00 01 74 05 e8 62 e0 d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 3e 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> 1f 44 00 00 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Call Trace: [] ? osc_cl_page_osc+0x15/0x60 [osc] [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#0 stuck for 67s! [umount:22194] Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] CPU 0 Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] Pid: 22194, comm: umount Not tainted 2.6.32-279.14.1.el6.x86_64 #1 Red Hat KVM RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff88007b7ef508 EFLAGS: 00000206 RAX: 0000000000000001 RBX: ffff88007b7ef508 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88007ae2c400 RDI: ffff880076b3a734 RBP: ffffffff8100bc0e R08: 0000000000000001 R09: 0000000000000001 R10: 0000000000000000 R11: 0000000000000001 R12: ffff8800760154e0 R13: ffff8800760154e0 R14: ffff880024017cf0 R15: ffff88007b7ef488 FS: 00007f7464baa740(0000) GS:ffff880002200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f74642c81b0 CR3: 0000000078444000 CR4: 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process umount (pid: 22194, threadinfo ffff88007b7ee000, task ffff88007d059500) Stack: ffff88007b7ef5a8 ffffffffa08ee29c ffff8800754f3d98 ffff880073c08400 ffff88007ae2c400 ffff880078d76c38 ffff88007ae2c510 ffff88007ae2c4f0 ffff880076b3a648 0000000100000001 ffff88007ae2c4f8 ffff88007d0449e8 Call Trace: [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b Code: 00 00 00 01 74 05 e8 62 e0 d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 3e 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> 1f 44 00 00 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Call Trace: [] ? osc_cl_page_osc+0x15/0x60 [osc] [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#0 stuck for 68s! [umount:22194] Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] CPU 0 Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] Pid: 22194, comm: umount Not tainted 2.6.32-279.14.1.el6.x86_64 #1 Red Hat KVM RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff88007b7ef508 EFLAGS: 00000206 RAX: 0000000000000001 RBX: ffff88007b7ef508 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88007ae2c400 RDI: ffff880076b3a734 RBP: ffffffff8100bc0e R08: 0000000000000001 R09: 0000000000000001 R10: 0000000000000000 R11: 0000000000000001 R12: ffff8800760154e0 R13: ffff8800760154e0 R14: ffff880024017cf0 R15: ffff88007b7ef488 FS: 00007f7464baa740(0000) GS:ffff880002200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f74642c81b0 CR3: 0000000078444000 CR4: 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process umount (pid: 22194, threadinfo ffff88007b7ee000, task ffff88007d059500) Stack: ffff88007b7ef5a8 ffffffffa08ee29c ffff8800754f3d98 ffff880073c08400 ffff88007ae2c400 ffff880078d76c38 ffff88007ae2c510 ffff88007ae2c4f0 ffff880076b3a648 0000000100000001 ffff88007ae2c4f8 ffff88007d0449e8 Call Trace: [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b Code: 00 00 00 01 74 05 e8 62 e0 d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 3e 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> 1f 44 00 00 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Call Trace: [] ? osc_cl_page_osc+0x15/0x60 [osc] [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#0 stuck for 67s! [umount:22194] Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] CPU 0 Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] Pid: 22194, comm: umount Not tainted 2.6.32-279.14.1.el6.x86_64 #1 Red Hat KVM RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff88007b7ef508 EFLAGS: 00000206 RAX: 0000000000000001 RBX: ffff88007b7ef508 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88007ae2c400 RDI: ffff880076b3a734 RBP: ffffffff8100bc0e R08: 0000000000000001 R09: 0000000000000001 R10: 0000000000000000 R11: 0000000000000001 R12: ffff8800760154e0 R13: ffff8800760154e0 R14: ffff880024017cf0 R15: ffff88007b7ef488 FS: 00007f7464baa740(0000) GS:ffff880002200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f74642c81b0 CR3: 0000000078444000 CR4: 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process umount (pid: 22194, threadinfo ffff88007b7ee000, task ffff88007d059500) Stack: ffff88007b7ef5a8 ffffffffa08ee29c ffff8800754f3d98 ffff880073c08400 ffff88007ae2c400 ffff880078d76c38 ffff88007ae2c510 ffff88007ae2c4f0 ffff880076b3a648 0000000100000001 ffff88007ae2c4f8 ffff88007d0449e8 Call Trace: [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b Code: 00 00 00 01 74 05 e8 62 e0 d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 3e 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> 1f 44 00 00 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Call Trace: [] ? osc_cl_page_osc+0x15/0x60 [osc] [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#0 stuck for 67s! [umount:22194] Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] CPU 0 Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] Pid: 22194, comm: umount Not tainted 2.6.32-279.14.1.el6.x86_64 #1 Red Hat KVM RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff88007b7ef508 EFLAGS: 00000206 RAX: 0000000000000001 RBX: ffff88007b7ef508 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88007ae2c400 RDI: ffff880076b3a734 RBP: ffffffff8100bc0e R08: 0000000000000001 R09: 0000000000000001 R10: 0000000000000000 R11: 0000000000000001 R12: ffff8800760154e0 R13: ffff8800760154e0 R14: ffff880024017cf0 R15: ffff88007b7ef488 FS: 00007f7464baa740(0000) GS:ffff880002200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f74642c81b0 CR3: 0000000078444000 CR4: 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process umount (pid: 22194, threadinfo ffff88007b7ee000, task ffff88007d059500) Stack: ffff88007b7ef5a8 ffffffffa08ee29c ffff8800754f3d98 ffff880073c08400 ffff88007ae2c400 ffff880078d76c38 ffff88007ae2c510 ffff88007ae2c4f0 ffff880076b3a648 0000000100000001 ffff88007ae2c4f8 ffff88007d0449e8 Call Trace: [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b Code: 00 00 00 01 74 05 e8 62 e0 d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 3e 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> 1f 44 00 00 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Call Trace: [] ? osc_cl_page_osc+0x15/0x60 [osc] [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#0 stuck for 67s! [umount:22194] Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] CPU 0 Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] Pid: 22194, comm: umount Not tainted 2.6.32-279.14.1.el6.x86_64 #1 Red Hat KVM RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff88007b7ef508 EFLAGS: 00000206 RAX: 0000000000000001 RBX: ffff88007b7ef508 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88007ae2c400 RDI: ffff880076b3a734 RBP: ffffffff8100bc0e R08: 0000000000000001 R09: 0000000000000001 R10: 0000000000000000 R11: 0000000000000001 R12: ffff8800760154e0 R13: ffff8800760154e0 R14: ffff880024017cf0 R15: ffff88007b7ef488 FS: 00007f7464baa740(0000) GS:ffff880002200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f74642c81b0 CR3: 0000000078444000 CR4: 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process umount (pid: 22194, threadinfo ffff88007b7ee000, task ffff88007d059500) Stack: ffff88007b7ef5a8 ffffffffa08ee29c ffff8800754f3d98 ffff880073c08400 ffff88007ae2c400 ffff880078d76c38 ffff88007ae2c510 ffff88007ae2c4f0 ffff880076b3a648 0000000100000001 ffff88007ae2c4f8 ffff88007d0449e8 Call Trace: [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b Code: 00 00 00 01 74 05 e8 62 e0 d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 3e 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> 1f 44 00 00 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Call Trace: [] ? osc_cl_page_osc+0x15/0x60 [osc] [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#0 stuck for 67s! [umount:22194] Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] CPU 0 Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] Pid: 22194, comm: umount Not tainted 2.6.32-279.14.1.el6.x86_64 #1 Red Hat KVM RIP: 0010:[] [] _spin_lock+0x26/0x30 RSP: 0018:ffff88007b7ef508 EFLAGS: 00000206 RAX: 0000000000000001 RBX: ffff88007b7ef508 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88007ae2c400 RDI: ffff880076b3a734 RBP: ffffffff8100bc0e R08: 0000000000000001 R09: 0000000000000001 R10: 0000000000000000 R11: 0000000000000001 R12: ffff8800760154e0 R13: ffff8800760154e0 R14: ffff880024017cf0 R15: ffff88007b7ef488 FS: 00007f7464baa740(0000) GS:ffff880002200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f74642c81b0 CR3: 0000000078444000 CR4: 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process umount (pid: 22194, threadinfo ffff88007b7ee000, task ffff88007d059500) Stack: ffff88007b7ef5a8 ffffffffa08ee29c ffff8800754f3d98 ffff880073c08400 ffff88007ae2c400 ffff880078d76c38 ffff88007ae2c510 ffff88007ae2c4f0 ffff880076b3a648 0000000100000001 ffff88007ae2c4f8 ffff88007d0449e8 Call Trace: [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b Code: e0 d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 3e 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f 1f 44 00 00 83 3f 00 <75> f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 44 00 00 3e 81 Call Trace: [] ? osc_cl_page_osc+0x15/0x60 [osc] [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#0 stuck for 67s! [umount:22194] Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] CPU 0 Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] Pid: 22194, comm: umount Not tainted 2.6.32-279.14.1.el6.x86_64 #1 Red Hat KVM RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff88007b7ef508 EFLAGS: 00000206 RAX: 0000000000000001 RBX: ffff88007b7ef508 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88007ae2c400 RDI: ffff880076b3a734 RBP: ffffffff8100bc0e R08: 0000000000000001 R09: 0000000000000001 R10: 0000000000000000 R11: 0000000000000001 R12: ffff8800760154e0 R13: ffff8800760154e0 R14: ffff880024017cf0 R15: ffff88007b7ef488 FS: 00007f7464baa740(0000) GS:ffff880002200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f74642c81b0 CR3: 0000000078444000 CR4: 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process umount (pid: 22194, threadinfo ffff88007b7ee000, task ffff88007d059500) Stack: ffff88007b7ef5a8 ffffffffa08ee29c ffff8800754f3d98 ffff880073c08400 ffff88007ae2c400 ffff880078d76c38 ffff88007ae2c510 ffff88007ae2c4f0 ffff880076b3a648 0000000100000001 ffff88007ae2c4f8 ffff88007d0449e8 Call Trace: [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b Code: 00 00 00 01 74 05 e8 62 e0 d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 3e 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> 1f 44 00 00 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Call Trace: [] ? osc_cl_page_osc+0x15/0x60 [osc] [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#0 stuck for 67s! [umount:22194] Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] CPU 0 Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] Pid: 22194, comm: umount Not tainted 2.6.32-279.14.1.el6.x86_64 #1 Red Hat KVM RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff88007b7ef508 EFLAGS: 00000206 RAX: 0000000000000001 RBX: ffff88007b7ef508 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88007ae2c400 RDI: ffff880076b3a734 RBP: ffffffff8100bc0e R08: 0000000000000001 R09: 0000000000000001 R10: 0000000000000000 R11: 0000000000000001 R12: ffff8800760154e0 R13: ffff8800760154e0 R14: ffff880024017cf0 R15: ffff88007b7ef488 FS: 00007f7464baa740(0000) GS:ffff880002200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f74642c81b0 CR3: 0000000078444000 CR4: 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process umount (pid: 22194, threadinfo ffff88007b7ee000, task ffff88007d059500) Stack: ffff88007b7ef5a8 ffffffffa08ee29c ffff8800754f3d98 ffff880073c08400 ffff88007ae2c400 ffff880078d76c38 ffff88007ae2c510 ffff88007ae2c4f0 ffff880076b3a648 0000000100000001 ffff88007ae2c4f8 ffff88007d0449e8 Call Trace: [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b Code: 00 00 00 01 74 05 e8 62 e0 d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 3e 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> 1f 44 00 00 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Call Trace: [] ? osc_cl_page_osc+0x15/0x60 [osc] [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#0 stuck for 67s! [umount:22194] Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] CPU 0 Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] Pid: 22194, comm: umount Not tainted 2.6.32-279.14.1.el6.x86_64 #1 Red Hat KVM RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff88007b7ef508 EFLAGS: 00000206 RAX: 0000000000000001 RBX: ffff88007b7ef508 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88007ae2c400 RDI: ffff880076b3a734 RBP: ffffffff8100bc0e R08: 0000000000000001 R09: 0000000000000001 R10: 0000000000000000 R11: 0000000000000001 R12: ffff8800760154e0 R13: ffff8800760154e0 R14: ffff880024017cf0 R15: ffff88007b7ef488 FS: 00007f7464baa740(0000) GS:ffff880002200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f74642c81b0 CR3: 0000000078444000 CR4: 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process umount (pid: 22194, threadinfo ffff88007b7ee000, task ffff88007d059500) Stack: ffff88007b7ef5a8 ffffffffa08ee29c ffff8800754f3d98 ffff880073c08400 ffff88007ae2c400 ffff880078d76c38 ffff88007ae2c510 ffff88007ae2c4f0 ffff880076b3a648 0000000100000001 ffff88007ae2c4f8 ffff88007d0449e8 Call Trace: [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b Code: 00 00 00 01 74 05 e8 62 e0 d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 3e 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> 1f 44 00 00 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Call Trace: [] ? osc_cl_page_osc+0x15/0x60 [osc] [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#0 stuck for 67s! [umount:22194] Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] CPU 0 Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] Pid: 22194, comm: umount Not tainted 2.6.32-279.14.1.el6.x86_64 #1 Red Hat KVM RIP: 0010:[] [] _spin_lock+0x26/0x30 RSP: 0018:ffff88007b7ef508 EFLAGS: 00000206 RAX: 0000000000000001 RBX: ffff88007b7ef508 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88007ae2c400 RDI: ffff880076b3a734 RBP: ffffffff8100bc0e R08: 0000000000000001 R09: 0000000000000001 R10: 0000000000000000 R11: 0000000000000001 R12: ffff8800760154e0 R13: ffff8800760154e0 R14: ffff880024017cf0 R15: ffff88007b7ef488 FS: 00007f7464baa740(0000) GS:ffff880002200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f74642c81b0 CR3: 0000000078444000 CR4: 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process umount (pid: 22194, threadinfo ffff88007b7ee000, task ffff88007d059500) Stack: ffff88007b7ef5a8 ffffffffa08ee29c ffff8800754f3d98 ffff880073c08400 ffff88007ae2c400 ffff880078d76c38 ffff88007ae2c510 ffff88007ae2c4f0 ffff880076b3a648 0000000100000001 ffff88007ae2c4f8 ffff88007d0449e8 Call Trace: [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b Code: e0 d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 3e 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f 1f 44 00 00 83 3f 00 <75> f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 44 00 00 3e 81 Call Trace: [] ? osc_cl_page_osc+0x15/0x60 [osc] [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#0 stuck for 67s! [umount:22194] Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] CPU 0 Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] Pid: 22194, comm: umount Not tainted 2.6.32-279.14.1.el6.x86_64 #1 Red Hat KVM RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff88007b7ef508 EFLAGS: 00000206 RAX: 0000000000000001 RBX: ffff88007b7ef508 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88007ae2c400 RDI: ffff880076b3a734 RBP: ffffffff8100bc0e R08: 0000000000000001 R09: 0000000000000001 R10: 0000000000000000 R11: 0000000000000001 R12: ffff8800760154e0 R13: ffff8800760154e0 R14: ffff880024017cf0 R15: ffff88007b7ef488 FS: 00007f7464baa740(0000) GS:ffff880002200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f74642c81b0 CR3: 0000000078444000 CR4: 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process umount (pid: 22194, threadinfo ffff88007b7ee000, task ffff88007d059500) Stack: ffff88007b7ef5a8 ffffffffa08ee29c ffff8800754f3d98 ffff880073c08400 ffff88007ae2c400 ffff880078d76c38 ffff88007ae2c510 ffff88007ae2c4f0 ffff880076b3a648 0000000100000001 ffff88007ae2c4f8 ffff88007d0449e8 Call Trace: [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b Code: 00 00 00 01 74 05 e8 62 e0 d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 3e 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> 1f 44 00 00 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Call Trace: [] ? osc_cl_page_osc+0x15/0x60 [osc] [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#0 stuck for 67s! [umount:22194] Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] CPU 0 Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] Pid: 22194, comm: umount Not tainted 2.6.32-279.14.1.el6.x86_64 #1 Red Hat KVM RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff88007b7ef508 EFLAGS: 00000206 RAX: 0000000000000001 RBX: ffff88007b7ef508 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88007ae2c400 RDI: ffff880076b3a734 RBP: ffffffff8100bc0e R08: 0000000000000001 R09: 0000000000000001 R10: 0000000000000000 R11: 0000000000000001 R12: ffff8800760154e0 R13: ffff8800760154e0 R14: ffff880024017cf0 R15: ffff88007b7ef488 FS: 00007f7464baa740(0000) GS:ffff880002200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f74642c81b0 CR3: 0000000078444000 CR4: 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process umount (pid: 22194, threadinfo ffff88007b7ee000, task ffff88007d059500) Stack: ffff88007b7ef5a8 ffffffffa08ee29c ffff8800754f3d98 ffff880073c08400 ffff88007ae2c400 ffff880078d76c38 ffff88007ae2c510 ffff88007ae2c4f0 ffff880076b3a648 0000000100000001 ffff88007ae2c4f8 ffff88007d0449e8 Call Trace: [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b Code: 00 00 00 01 74 05 e8 62 e0 d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 3e 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> 1f 44 00 00 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Call Trace: [] ? osc_cl_page_osc+0x15/0x60 [osc] [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#0 stuck for 67s! [umount:22194] Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] CPU 0 Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] Pid: 22194, comm: umount Not tainted 2.6.32-279.14.1.el6.x86_64 #1 Red Hat KVM RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff88007b7ef508 EFLAGS: 00000206 RAX: 0000000000000001 RBX: ffff88007b7ef508 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88007ae2c400 RDI: ffff880076b3a734 RBP: ffffffff8100bc0e R08: 0000000000000001 R09: 0000000000000001 R10: 0000000000000000 R11: 0000000000000001 R12: ffff8800760154e0 R13: ffff8800760154e0 R14: ffff880024017cf0 R15: ffff88007b7ef488 FS: 00007f7464baa740(0000) GS:ffff880002200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f74642c81b0 CR3: 0000000078444000 CR4: 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process umount (pid: 22194, threadinfo ffff88007b7ee000, task ffff88007d059500) Stack: ffff88007b7ef5a8 ffffffffa08ee29c ffff8800754f3d98 ffff880073c08400 ffff88007ae2c400 ffff880078d76c38 ffff88007ae2c510 ffff88007ae2c4f0 ffff880076b3a648 0000000100000001 ffff88007ae2c4f8 ffff88007d0449e8 Call Trace: [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b Code: 00 00 00 01 74 05 e8 62 e0 d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 3e 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> 1f 44 00 00 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Call Trace: [] ? osc_cl_page_osc+0x15/0x60 [osc] [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#0 stuck for 67s! [umount:22194] Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] CPU 0 Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] Pid: 22194, comm: umount Not tainted 2.6.32-279.14.1.el6.x86_64 #1 Red Hat KVM RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff88007b7ef508 EFLAGS: 00000206 RAX: 0000000000000001 RBX: ffff88007b7ef508 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88007ae2c400 RDI: ffff880076b3a734 RBP: ffffffff8100bc0e R08: 0000000000000001 R09: 0000000000000001 R10: 0000000000000000 R11: 0000000000000001 R12: ffff8800760154e0 R13: ffff8800760154e0 R14: ffff880024017cf0 R15: ffff88007b7ef488 FS: 00007f7464baa740(0000) GS:ffff880002200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f74642c81b0 CR3: 0000000078444000 CR4: 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process umount (pid: 22194, threadinfo ffff88007b7ee000, task ffff88007d059500) Stack: ffff88007b7ef5a8 ffffffffa08ee29c ffff8800754f3d98 ffff880073c08400 ffff88007ae2c400 ffff880078d76c38 ffff88007ae2c510 ffff88007ae2c4f0 ffff880076b3a648 0000000100000001 ffff88007ae2c4f8 ffff88007d0449e8 Call Trace: [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b Code: 00 00 00 01 74 05 e8 62 e0 d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 3e 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> 1f 44 00 00 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Call Trace: [] ? osc_cl_page_osc+0x15/0x60 [osc] [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#0 stuck for 68s! [umount:22194] Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] CPU 0 Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] Pid: 22194, comm: umount Not tainted 2.6.32-279.14.1.el6.x86_64 #1 Red Hat KVM RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff88007b7ef508 EFLAGS: 00000206 RAX: 0000000000000001 RBX: ffff88007b7ef508 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88007ae2c400 RDI: ffff880076b3a734 RBP: ffffffff8100bc0e R08: 0000000000000001 R09: 0000000000000001 R10: 0000000000000000 R11: 0000000000000001 R12: ffff8800760154e0 R13: ffff8800760154e0 R14: ffff880024017cf0 R15: ffff88007b7ef488 FS: 00007f7464baa740(0000) GS:ffff880002200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f74642c81b0 CR3: 0000000078444000 CR4: 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process umount (pid: 22194, threadinfo ffff88007b7ee000, task ffff88007d059500) Stack: ffff88007b7ef5a8 ffffffffa08ee29c ffff8800754f3d98 ffff880073c08400 ffff88007ae2c400 ffff880078d76c38 ffff88007ae2c510 ffff88007ae2c4f0 ffff880076b3a648 0000000100000001 ffff88007ae2c4f8 ffff88007d0449e8 Call Trace: [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b Code: 00 00 00 01 74 05 e8 62 e0 d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 3e 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> 1f 44 00 00 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Call Trace: [] ? osc_cl_page_osc+0x15/0x60 [osc] [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#0 stuck for 67s! [umount:22194] Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] CPU 0 Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] Pid: 22194, comm: umount Not tainted 2.6.32-279.14.1.el6.x86_64 #1 Red Hat KVM RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff88007b7ef508 EFLAGS: 00000206 RAX: 0000000000000001 RBX: ffff88007b7ef508 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88007ae2c400 RDI: ffff880076b3a734 RBP: ffffffff8100bc0e R08: 0000000000000001 R09: 0000000000000001 R10: 0000000000000000 R11: 0000000000000001 R12: ffff8800760154e0 R13: ffff8800760154e0 R14: ffff880024017cf0 R15: ffff88007b7ef488 FS: 00007f7464baa740(0000) GS:ffff880002200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f74642c81b0 CR3: 0000000078444000 CR4: 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process umount (pid: 22194, threadinfo ffff88007b7ee000, task ffff88007d059500) Stack: ffff88007b7ef5a8 ffffffffa08ee29c ffff8800754f3d98 ffff880073c08400 ffff88007ae2c400 ffff880078d76c38 ffff88007ae2c510 ffff88007ae2c4f0 ffff880076b3a648 0000000100000001 ffff88007ae2c4f8 ffff88007d0449e8 Call Trace: [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b Code: 00 00 00 01 74 05 e8 62 e0 d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 3e 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> 1f 44 00 00 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Call Trace: [] ? osc_cl_page_osc+0x15/0x60 [osc] [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#0 stuck for 67s! [umount:22194] Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] CPU 0 Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] Pid: 22194, comm: umount Not tainted 2.6.32-279.14.1.el6.x86_64 #1 Red Hat KVM RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff88007b7ef508 EFLAGS: 00000206 RAX: 0000000000000001 RBX: ffff88007b7ef508 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88007ae2c400 RDI: ffff880076b3a734 RBP: ffffffff8100bc0e R08: 0000000000000001 R09: 0000000000000001 R10: 0000000000000000 R11: 0000000000000001 R12: ffff8800760154e0 R13: ffff8800760154e0 R14: ffff880024017cf0 R15: ffff88007b7ef488 FS: 00007f7464baa740(0000) GS:ffff880002200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f74642c81b0 CR3: 0000000078444000 CR4: 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process umount (pid: 22194, threadinfo ffff88007b7ee000, task ffff88007d059500) Stack: ffff88007b7ef5a8 ffffffffa08ee29c ffff8800754f3d98 ffff880073c08400 ffff88007ae2c400 ffff880078d76c38 ffff88007ae2c510 ffff88007ae2c4f0 ffff880076b3a648 0000000100000001 ffff88007ae2c4f8 ffff88007d0449e8 Call Trace: [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b Code: 00 00 00 01 74 05 e8 62 e0 d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 3e 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> 1f 44 00 00 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Call Trace: [] ? osc_cl_page_osc+0x15/0x60 [osc] [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#0 stuck for 67s! [umount:22194] Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] CPU 0 Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] Pid: 22194, comm: umount Not tainted 2.6.32-279.14.1.el6.x86_64 #1 Red Hat KVM RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff88007b7ef508 EFLAGS: 00000206 RAX: 0000000000000001 RBX: ffff88007b7ef508 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88007ae2c400 RDI: ffff880076b3a734 RBP: ffffffff8100bc0e R08: 0000000000000001 R09: 0000000000000001 R10: 0000000000000000 R11: 0000000000000001 R12: ffff8800760154e0 R13: ffff8800760154e0 R14: ffff880024017cf0 R15: ffff88007b7ef488 FS: 00007f7464baa740(0000) GS:ffff880002200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f74642c81b0 CR3: 0000000078444000 CR4: 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process umount (pid: 22194, threadinfo ffff88007b7ee000, task ffff88007d059500) Stack: ffff88007b7ef5a8 ffffffffa08ee29c ffff8800754f3d98 ffff880073c08400 ffff88007ae2c400 ffff880078d76c38 ffff88007ae2c510 ffff88007ae2c4f0 ffff880076b3a648 0000000100000001 ffff88007ae2c4f8 ffff88007d0449e8 Call Trace: [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b Code: 00 00 00 01 74 05 e8 62 e0 d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 3e 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> 1f 44 00 00 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Call Trace: [] ? osc_cl_page_osc+0x15/0x60 [osc] [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#0 stuck for 67s! [umount:22194] Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] CPU 0 Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] Pid: 22194, comm: umount Not tainted 2.6.32-279.14.1.el6.x86_64 #1 Red Hat KVM RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff88007b7ef508 EFLAGS: 00000206 RAX: 0000000000000001 RBX: ffff88007b7ef508 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88007ae2c400 RDI: ffff880076b3a734 RBP: ffffffff8100bc0e R08: 0000000000000001 R09: 0000000000000001 R10: 0000000000000000 R11: 0000000000000001 R12: ffff8800760154e0 R13: ffff8800760154e0 R14: ffff880024017cf0 R15: ffff88007b7ef488 FS: 00007f7464baa740(0000) GS:ffff880002200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f74642c81b0 CR3: 0000000078444000 CR4: 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process umount (pid: 22194, threadinfo ffff88007b7ee000, task ffff88007d059500) Stack: ffff88007b7ef5a8 ffffffffa08ee29c ffff8800754f3d98 ffff880073c08400 ffff88007ae2c400 ffff880078d76c38 ffff88007ae2c510 ffff88007ae2c4f0 ffff880076b3a648 0000000100000001 ffff88007ae2c4f8 ffff88007d0449e8 Call Trace: [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b Code: 00 00 00 01 74 05 e8 62 e0 d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 3e 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> 1f 44 00 00 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Call Trace: [] ? osc_cl_page_osc+0x15/0x60 [osc] [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#0 stuck for 67s! [umount:22194] Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] CPU 0 Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] Pid: 22194, comm: umount Not tainted 2.6.32-279.14.1.el6.x86_64 #1 Red Hat KVM RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff88007b7ef508 EFLAGS: 00000206 RAX: 0000000000000001 RBX: ffff88007b7ef508 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88007ae2c400 RDI: ffff880076b3a734 RBP: ffffffff8100bc0e R08: 0000000000000001 R09: 0000000000000001 R10: 0000000000000000 R11: 0000000000000001 R12: ffff8800760154e0 R13: ffff8800760154e0 R14: ffff880024017cf0 R15: ffff88007b7ef488 FS: 00007f7464baa740(0000) GS:ffff880002200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f74642c81b0 CR3: 0000000078444000 CR4: 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process umount (pid: 22194, threadinfo ffff88007b7ee000, task ffff88007d059500) Stack: ffff88007b7ef5a8 ffffffffa08ee29c ffff8800754f3d98 ffff880073c08400 ffff88007ae2c400 ffff880078d76c38 ffff88007ae2c510 ffff88007ae2c4f0 ffff880076b3a648 0000000100000001 ffff88007ae2c4f8 ffff88007d0449e8 Call Trace: [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b Code: 00 00 00 01 74 05 e8 62 e0 d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 3e 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> 1f 44 00 00 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Call Trace: [] ? osc_cl_page_osc+0x15/0x60 [osc] [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#0 stuck for 67s! [umount:22194] Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] CPU 0 Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] Pid: 22194, comm: umount Not tainted 2.6.32-279.14.1.el6.x86_64 #1 Red Hat KVM RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff88007b7ef508 EFLAGS: 00000206 RAX: 0000000000000001 RBX: ffff88007b7ef508 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88007ae2c400 RDI: ffff880076b3a734 RBP: ffffffff8100bc0e R08: 0000000000000001 R09: 0000000000000001 R10: 0000000000000000 R11: 0000000000000001 R12: ffff8800760154e0 R13: ffff8800760154e0 R14: ffff880024017cf0 R15: ffff88007b7ef488 FS: 00007f7464baa740(0000) GS:ffff880002200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f74642c81b0 CR3: 0000000078444000 CR4: 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process umount (pid: 22194, threadinfo ffff88007b7ee000, task ffff88007d059500) Stack: ffff88007b7ef5a8 ffffffffa08ee29c ffff8800754f3d98 ffff880073c08400 ffff88007ae2c400 ffff880078d76c38 ffff88007ae2c510 ffff88007ae2c4f0 ffff880076b3a648 0000000100000001 ffff88007ae2c4f8 ffff88007d0449e8 Call Trace: [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b Code: 00 00 00 01 74 05 e8 62 e0 d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 3e 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> 1f 44 00 00 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Call Trace: [] ? osc_cl_page_osc+0x15/0x60 [osc] [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#0 stuck for 67s! [umount:22194] Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] CPU 0 Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] Pid: 22194, comm: umount Not tainted 2.6.32-279.14.1.el6.x86_64 #1 Red Hat KVM RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff88007b7ef508 EFLAGS: 00000206 RAX: 0000000000000001 RBX: ffff88007b7ef508 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88007ae2c400 RDI: ffff880076b3a734 RBP: ffffffff8100bc0e R08: 0000000000000001 R09: 0000000000000001 R10: 0000000000000000 R11: 0000000000000001 R12: ffff8800760154e0 R13: ffff8800760154e0 R14: ffff880024017cf0 R15: ffff88007b7ef488 FS: 00007f7464baa740(0000) GS:ffff880002200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f74642c81b0 CR3: 0000000078444000 CR4: 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process umount (pid: 22194, threadinfo ffff88007b7ee000, task ffff88007d059500) Stack: ffff88007b7ef5a8 ffffffffa08ee29c ffff8800754f3d98 ffff880073c08400 ffff88007ae2c400 ffff880078d76c38 ffff88007ae2c510 ffff88007ae2c4f0 ffff880076b3a648 0000000100000001 ffff88007ae2c4f8 ffff88007d0449e8 Call Trace: [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b Code: 00 00 00 01 74 05 e8 62 e0 d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 3e 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> 1f 44 00 00 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Call Trace: [] ? osc_cl_page_osc+0x15/0x60 [osc] [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#0 stuck for 67s! [umount:22194] Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] CPU 0 Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] Pid: 22194, comm: umount Not tainted 2.6.32-279.14.1.el6.x86_64 #1 Red Hat KVM RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff88007b7ef508 EFLAGS: 00000206 RAX: 0000000000000001 RBX: ffff88007b7ef508 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88007ae2c400 RDI: ffff880076b3a734 RBP: ffffffff8100bc0e R08: 0000000000000001 R09: 0000000000000001 R10: 0000000000000000 R11: 0000000000000001 R12: ffff8800760154e0 R13: ffff8800760154e0 R14: ffff880024017cf0 R15: ffff88007b7ef488 FS: 00007f7464baa740(0000) GS:ffff880002200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f74642c81b0 CR3: 0000000078444000 CR4: 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process umount (pid: 22194, threadinfo ffff88007b7ee000, task ffff88007d059500) Stack: ffff88007b7ef5a8 ffffffffa08ee29c ffff8800754f3d98 ffff880073c08400 ffff88007ae2c400 ffff880078d76c38 ffff88007ae2c510 ffff88007ae2c4f0 ffff880076b3a648 0000000100000001 ffff88007ae2c4f8 ffff88007d0449e8 Call Trace: [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b Code: 00 00 00 01 74 05 e8 62 e0 d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 3e 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> 1f 44 00 00 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Call Trace: [] ? osc_cl_page_osc+0x15/0x60 [osc] [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#0 stuck for 67s! [umount:22194] Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] CPU 0 Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] Pid: 22194, comm: umount Not tainted 2.6.32-279.14.1.el6.x86_64 #1 Red Hat KVM RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff88007b7ef508 EFLAGS: 00000206 RAX: 0000000000000001 RBX: ffff88007b7ef508 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88007ae2c400 RDI: ffff880076b3a734 RBP: ffffffff8100bc0e R08: 0000000000000001 R09: 0000000000000001 R10: 0000000000000000 R11: 0000000000000001 R12: ffff8800760154e0 R13: ffff8800760154e0 R14: ffff880024017cf0 R15: ffff88007b7ef488 FS: 00007f7464baa740(0000) GS:ffff880002200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f74642c81b0 CR3: 0000000078444000 CR4: 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process umount (pid: 22194, threadinfo ffff88007b7ee000, task ffff88007d059500) Stack: ffff88007b7ef5a8 ffffffffa08ee29c ffff8800754f3d98 ffff880073c08400 ffff88007ae2c400 ffff880078d76c38 ffff88007ae2c510 ffff88007ae2c4f0 ffff880076b3a648 0000000100000001 ffff88007ae2c4f8 ffff88007d0449e8 Call Trace: [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b Code: 00 00 00 01 74 05 e8 62 e0 d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 3e 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> 1f 44 00 00 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Call Trace: [] ? osc_cl_page_osc+0x15/0x60 [osc] [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#0 stuck for 67s! [umount:22194] Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] CPU 0 Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] Pid: 22194, comm: umount Not tainted 2.6.32-279.14.1.el6.x86_64 #1 Red Hat KVM RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff88007b7ef508 EFLAGS: 00000206 RAX: 0000000000000001 RBX: ffff88007b7ef508 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88007ae2c400 RDI: ffff880076b3a734 RBP: ffffffff8100bc0e R08: 0000000000000001 R09: 0000000000000001 R10: 0000000000000000 R11: 0000000000000001 R12: ffff8800760154e0 R13: ffff8800760154e0 R14: ffff880024017cf0 R15: ffff88007b7ef488 FS: 00007f7464baa740(0000) GS:ffff880002200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f74642c81b0 CR3: 0000000078444000 CR4: 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process umount (pid: 22194, threadinfo ffff88007b7ee000, task ffff88007d059500) Stack: ffff88007b7ef5a8 ffffffffa08ee29c ffff8800754f3d98 ffff880073c08400 ffff88007ae2c400 ffff880078d76c38 ffff88007ae2c510 ffff88007ae2c4f0 ffff880076b3a648 0000000100000001 ffff88007ae2c4f8 ffff88007d0449e8 Call Trace: [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b Code: 00 00 00 01 74 05 e8 62 e0 d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 3e 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> 1f 44 00 00 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Call Trace: [] ? osc_cl_page_osc+0x15/0x60 [osc] [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#0 stuck for 67s! [umount:22194] Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] CPU 0 Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] Pid: 22194, comm: umount Not tainted 2.6.32-279.14.1.el6.x86_64 #1 Red Hat KVM RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff88007b7ef508 EFLAGS: 00000206 RAX: 0000000000000001 RBX: ffff88007b7ef508 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88007ae2c400 RDI: ffff880076b3a734 RBP: ffffffff8100bc0e R08: 0000000000000001 R09: 0000000000000001 R10: 0000000000000000 R11: 0000000000000001 R12: ffff8800760154e0 R13: ffff8800760154e0 R14: ffff880024017cf0 R15: ffff88007b7ef488 FS: 00007f7464baa740(0000) GS:ffff880002200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f74642c81b0 CR3: 0000000078444000 CR4: 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process umount (pid: 22194, threadinfo ffff88007b7ee000, task ffff88007d059500) Stack: ffff88007b7ef5a8 ffffffffa08ee29c ffff8800754f3d98 ffff880073c08400 ffff88007ae2c400 ffff880078d76c38 ffff88007ae2c510 ffff88007ae2c4f0 ffff880076b3a648 0000000100000001 ffff88007ae2c4f8 ffff88007d0449e8 Call Trace: [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b Code: 00 00 00 01 74 05 e8 62 e0 d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 3e 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> 1f 44 00 00 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Call Trace: [] ? osc_cl_page_osc+0x15/0x60 [osc] [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#0 stuck for 67s! [umount:22194] Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] CPU 0 Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] Pid: 22194, comm: umount Not tainted 2.6.32-279.14.1.el6.x86_64 #1 Red Hat KVM RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff88007b7ef508 EFLAGS: 00000206 RAX: 0000000000000001 RBX: ffff88007b7ef508 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88007ae2c400 RDI: ffff880076b3a734 RBP: ffffffff8100bc0e R08: 0000000000000001 R09: 0000000000000001 R10: 0000000000000000 R11: 0000000000000001 R12: ffff8800760154e0 R13: ffff8800760154e0 R14: ffff880024017cf0 R15: ffff88007b7ef488 FS: 00007f7464baa740(0000) GS:ffff880002200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f74642c81b0 CR3: 0000000078444000 CR4: 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process umount (pid: 22194, threadinfo ffff88007b7ee000, task ffff88007d059500) Stack: ffff88007b7ef5a8 ffffffffa08ee29c ffff8800754f3d98 ffff880073c08400 ffff88007ae2c400 ffff880078d76c38 ffff88007ae2c510 ffff88007ae2c4f0 ffff880076b3a648 0000000100000001 ffff88007ae2c4f8 ffff88007d0449e8 Call Trace: [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b Code: 00 00 00 01 74 05 e8 62 e0 d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 3e 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> 1f 44 00 00 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Call Trace: [] ? osc_cl_page_osc+0x15/0x60 [osc] [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#0 stuck for 68s! [umount:22194] Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] CPU 0 Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] Pid: 22194, comm: umount Not tainted 2.6.32-279.14.1.el6.x86_64 #1 Red Hat KVM RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff88007b7ef508 EFLAGS: 00000206 RAX: 0000000000000001 RBX: ffff88007b7ef508 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88007ae2c400 RDI: ffff880076b3a734 RBP: ffffffff8100bc0e R08: 0000000000000001 R09: 0000000000000001 R10: 0000000000000000 R11: 0000000000000001 R12: ffff8800760154e0 R13: ffff8800760154e0 R14: ffff880024017cf0 R15: ffff88007b7ef488 FS: 00007f7464baa740(0000) GS:ffff880002200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f74642c81b0 CR3: 0000000078444000 CR4: 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process umount (pid: 22194, threadinfo ffff88007b7ee000, task ffff88007d059500) Stack: ffff88007b7ef5a8 ffffffffa08ee29c ffff8800754f3d98 ffff880073c08400 ffff88007ae2c400 ffff880078d76c38 ffff88007ae2c510 ffff88007ae2c4f0 ffff880076b3a648 0000000100000001 ffff88007ae2c4f8 ffff88007d0449e8 Call Trace: [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b Code: 00 00 00 01 74 05 e8 62 e0 d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 3e 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> 1f 44 00 00 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Call Trace: [] ? osc_cl_page_osc+0x15/0x60 [osc] [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#0 stuck for 67s! [umount:22194] Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] CPU 0 Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] Pid: 22194, comm: umount Not tainted 2.6.32-279.14.1.el6.x86_64 #1 Red Hat KVM RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff88007b7ef508 EFLAGS: 00000206 RAX: 0000000000000001 RBX: ffff88007b7ef508 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88007ae2c400 RDI: ffff880076b3a734 RBP: ffffffff8100bc0e R08: 0000000000000001 R09: 0000000000000001 R10: 0000000000000000 R11: 0000000000000001 R12: ffff8800760154e0 R13: ffff8800760154e0 R14: ffff880024017cf0 R15: ffff88007b7ef488 FS: 00007f7464baa740(0000) GS:ffff880002200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f74642c81b0 CR3: 0000000078444000 CR4: 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process umount (pid: 22194, threadinfo ffff88007b7ee000, task ffff88007d059500) Stack: ffff88007b7ef5a8 ffffffffa08ee29c ffff8800754f3d98 ffff880073c08400 ffff88007ae2c400 ffff880078d76c38 ffff88007ae2c510 ffff88007ae2c4f0 ffff880076b3a648 0000000100000001 ffff88007ae2c4f8 ffff88007d0449e8 Call Trace: [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b Code: 00 00 00 01 74 05 e8 62 e0 d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 3e 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> 1f 44 00 00 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Call Trace: [] ? osc_cl_page_osc+0x15/0x60 [osc] [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#0 stuck for 67s! [umount:22194] Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] CPU 0 Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] Pid: 22194, comm: umount Not tainted 2.6.32-279.14.1.el6.x86_64 #1 Red Hat KVM RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff88007b7ef508 EFLAGS: 00000206 RAX: 0000000000000001 RBX: ffff88007b7ef508 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88007ae2c400 RDI: ffff880076b3a734 RBP: ffffffff8100bc0e R08: 0000000000000001 R09: 0000000000000001 R10: 0000000000000000 R11: 0000000000000001 R12: ffff8800760154e0 R13: ffff8800760154e0 R14: ffff880024017cf0 R15: ffff88007b7ef488 FS: 00007f7464baa740(0000) GS:ffff880002200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f74642c81b0 CR3: 0000000078444000 CR4: 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process umount (pid: 22194, threadinfo ffff88007b7ee000, task ffff88007d059500) Stack: ffff88007b7ef5a8 ffffffffa08ee29c ffff8800754f3d98 ffff880073c08400 ffff88007ae2c400 ffff880078d76c38 ffff88007ae2c510 ffff88007ae2c4f0 ffff880076b3a648 0000000100000001 ffff88007ae2c4f8 ffff88007d0449e8 Call Trace: [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b Code: 00 00 00 01 74 05 e8 62 e0 d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 3e 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> 1f 44 00 00 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Call Trace: [] ? osc_cl_page_osc+0x15/0x60 [osc] [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#0 stuck for 67s! [umount:22194] Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] CPU 0 Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] Pid: 22194, comm: umount Not tainted 2.6.32-279.14.1.el6.x86_64 #1 Red Hat KVM RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff88007b7ef508 EFLAGS: 00000206 RAX: 0000000000000001 RBX: ffff88007b7ef508 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88007ae2c400 RDI: ffff880076b3a734 RBP: ffffffff8100bc0e R08: 0000000000000001 R09: 0000000000000001 R10: 0000000000000000 R11: 0000000000000001 R12: ffff8800760154e0 R13: ffff8800760154e0 R14: ffff880024017cf0 R15: ffff88007b7ef488 FS: 00007f7464baa740(0000) GS:ffff880002200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f74642c81b0 CR3: 0000000078444000 CR4: 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process umount (pid: 22194, threadinfo ffff88007b7ee000, task ffff88007d059500) Stack: ffff88007b7ef5a8 ffffffffa08ee29c ffff8800754f3d98 ffff880073c08400 ffff88007ae2c400 ffff880078d76c38 ffff88007ae2c510 ffff88007ae2c4f0 ffff880076b3a648 0000000100000001 ffff88007ae2c4f8 ffff88007d0449e8 Call Trace: [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b Code: 00 00 00 01 74 05 e8 62 e0 d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 3e 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> 1f 44 00 00 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Call Trace: [] ? osc_cl_page_osc+0x15/0x60 [osc] [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#0 stuck for 67s! [umount:22194] Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] CPU 0 Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] Pid: 22194, comm: umount Not tainted 2.6.32-279.14.1.el6.x86_64 #1 Red Hat KVM RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff88007b7ef508 EFLAGS: 00000206 RAX: 0000000000000001 RBX: ffff88007b7ef508 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88007ae2c400 RDI: ffff880076b3a734 RBP: ffffffff8100bc0e R08: 0000000000000001 R09: 0000000000000001 R10: 0000000000000000 R11: 0000000000000001 R12: ffff8800760154e0 R13: ffff8800760154e0 R14: ffff880024017cf0 R15: ffff88007b7ef488 FS: 00007f7464baa740(0000) GS:ffff880002200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f74642c81b0 CR3: 0000000078444000 CR4: 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process umount (pid: 22194, threadinfo ffff88007b7ee000, task ffff88007d059500) Stack: ffff88007b7ef5a8 ffffffffa08ee29c ffff8800754f3d98 ffff880073c08400 ffff88007ae2c400 ffff880078d76c38 ffff88007ae2c510 ffff88007ae2c4f0 ffff880076b3a648 0000000100000001 ffff88007ae2c4f8 ffff88007d0449e8 Call Trace: [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b Code: 00 00 00 01 74 05 e8 62 e0 d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 3e 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> 1f 44 00 00 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Call Trace: [] ? osc_cl_page_osc+0x15/0x60 [osc] [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#0 stuck for 67s! [umount:22194] Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] CPU 0 Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] Pid: 22194, comm: umount Not tainted 2.6.32-279.14.1.el6.x86_64 #1 Red Hat KVM RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff88007b7ef508 EFLAGS: 00000206 RAX: 0000000000000001 RBX: ffff88007b7ef508 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88007ae2c400 RDI: ffff880076b3a734 RBP: ffffffff8100bc0e R08: 0000000000000001 R09: 0000000000000001 R10: 0000000000000000 R11: 0000000000000001 R12: ffff8800760154e0 R13: ffff8800760154e0 R14: ffff880024017cf0 R15: ffff88007b7ef488 FS: 00007f7464baa740(0000) GS:ffff880002200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f74642c81b0 CR3: 0000000078444000 CR4: 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process umount (pid: 22194, threadinfo ffff88007b7ee000, task ffff88007d059500) Stack: ffff88007b7ef5a8 ffffffffa08ee29c ffff8800754f3d98 ffff880073c08400 ffff88007ae2c400 ffff880078d76c38 ffff88007ae2c510 ffff88007ae2c4f0 ffff880076b3a648 0000000100000001 ffff88007ae2c4f8 ffff88007d0449e8 Call Trace: [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b Code: 00 00 00 01 74 05 e8 62 e0 d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 3e 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> 1f 44 00 00 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Call Trace: [] ? osc_cl_page_osc+0x15/0x60 [osc] [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#0 stuck for 67s! [umount:22194] Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] CPU 0 Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] Pid: 22194, comm: umount Not tainted 2.6.32-279.14.1.el6.x86_64 #1 Red Hat KVM RIP: 0010:[] [] _spin_lock+0x26/0x30 RSP: 0018:ffff88007b7ef508 EFLAGS: 00000206 RAX: 0000000000000001 RBX: ffff88007b7ef508 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88007ae2c400 RDI: ffff880076b3a734 RBP: ffffffff8100bc0e R08: 0000000000000001 R09: 0000000000000001 R10: 0000000000000000 R11: 0000000000000001 R12: ffff8800760154e0 R13: ffff8800760154e0 R14: ffff880024017cf0 R15: ffff88007b7ef488 FS: 00007f7464baa740(0000) GS:ffff880002200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f74642c81b0 CR3: 0000000078444000 CR4: 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process umount (pid: 22194, threadinfo ffff88007b7ee000, task ffff88007d059500) Stack: ffff88007b7ef5a8 ffffffffa08ee29c ffff8800754f3d98 ffff880073c08400 ffff88007ae2c400 ffff880078d76c38 ffff88007ae2c510 ffff88007ae2c4f0 ffff880076b3a648 0000000100000001 ffff88007ae2c4f8 ffff88007d0449e8 Call Trace: [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b Code: e0 d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 3e 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f 1f 44 00 00 83 3f 00 <75> f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 44 00 00 3e 81 Call Trace: [] ? osc_cl_page_osc+0x15/0x60 [osc] [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#0 stuck for 67s! [umount:22194] Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] CPU 0 Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] Pid: 22194, comm: umount Not tainted 2.6.32-279.14.1.el6.x86_64 #1 Red Hat KVM RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff88007b7ef508 EFLAGS: 00000206 RAX: 0000000000000001 RBX: ffff88007b7ef508 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88007ae2c400 RDI: ffff880076b3a734 RBP: ffffffff8100bc0e R08: 0000000000000001 R09: 0000000000000001 R10: 0000000000000000 R11: 0000000000000001 R12: ffff8800760154e0 R13: ffff8800760154e0 R14: ffff880024017cf0 R15: ffff88007b7ef488 FS: 00007f7464baa740(0000) GS:ffff880002200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f74642c81b0 CR3: 0000000078444000 CR4: 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process umount (pid: 22194, threadinfo ffff88007b7ee000, task ffff88007d059500) Stack: ffff88007b7ef5a8 ffffffffa08ee29c ffff8800754f3d98 ffff880073c08400 ffff88007ae2c400 ffff880078d76c38 ffff88007ae2c510 ffff88007ae2c4f0 ffff880076b3a648 0000000100000001 ffff88007ae2c4f8 ffff88007d0449e8 Call Trace: [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b Code: 00 00 00 01 74 05 e8 62 e0 d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 3e 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> 1f 44 00 00 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Call Trace: [] ? osc_cl_page_osc+0x15/0x60 [osc] [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#0 stuck for 67s! [umount:22194] Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] CPU 0 Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] Pid: 22194, comm: umount Not tainted 2.6.32-279.14.1.el6.x86_64 #1 Red Hat KVM RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff88007b7ef508 EFLAGS: 00000206 RAX: 0000000000000001 RBX: ffff88007b7ef508 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88007ae2c400 RDI: ffff880076b3a734 RBP: ffffffff8100bc0e R08: 0000000000000001 R09: 0000000000000001 R10: 0000000000000000 R11: 0000000000000001 R12: ffff8800760154e0 R13: ffff8800760154e0 R14: ffff880024017cf0 R15: ffff88007b7ef488 FS: 00007f7464baa740(0000) GS:ffff880002200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f74642c81b0 CR3: 0000000078444000 CR4: 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process umount (pid: 22194, threadinfo ffff88007b7ee000, task ffff88007d059500) Stack: ffff88007b7ef5a8 ffffffffa08ee29c ffff8800754f3d98 ffff880073c08400 ffff88007ae2c400 ffff880078d76c38 ffff88007ae2c510 ffff88007ae2c4f0 ffff880076b3a648 0000000100000001 ffff88007ae2c4f8 ffff88007d0449e8 Call Trace: [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b Code: 00 00 00 01 74 05 e8 62 e0 d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 3e 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> 1f 44 00 00 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Call Trace: [] ? osc_cl_page_osc+0x15/0x60 [osc] [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#0 stuck for 67s! [umount:22194] Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] CPU 0 Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] Pid: 22194, comm: umount Not tainted 2.6.32-279.14.1.el6.x86_64 #1 Red Hat KVM RIP: 0010:[] [] _spin_lock+0x26/0x30 RSP: 0018:ffff88007b7ef508 EFLAGS: 00000206 RAX: 0000000000000001 RBX: ffff88007b7ef508 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88007ae2c400 RDI: ffff880076b3a734 RBP: ffffffff8100bc0e R08: 0000000000000001 R09: 0000000000000001 R10: 0000000000000000 R11: 0000000000000001 R12: ffff8800760154e0 R13: ffff8800760154e0 R14: ffff880024017cf0 R15: ffff88007b7ef488 FS: 00007f7464baa740(0000) GS:ffff880002200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f74642c81b0 CR3: 0000000078444000 CR4: 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process umount (pid: 22194, threadinfo ffff88007b7ee000, task ffff88007d059500) Stack: ffff88007b7ef5a8 ffffffffa08ee29c ffff8800754f3d98 ffff880073c08400 ffff88007ae2c400 ffff880078d76c38 ffff88007ae2c510 ffff88007ae2c4f0 ffff880076b3a648 0000000100000001 ffff88007ae2c4f8 ffff88007d0449e8 Call Trace: [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b Code: e0 d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 3e 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f 1f 44 00 00 83 3f 00 <75> f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 44 00 00 3e 81 Call Trace: [] ? osc_cl_page_osc+0x15/0x60 [osc] [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#0 stuck for 67s! [umount:22194] Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] CPU 0 Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] Pid: 22194, comm: umount Not tainted 2.6.32-279.14.1.el6.x86_64 #1 Red Hat KVM RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff88007b7ef508 EFLAGS: 00000206 RAX: 0000000000000001 RBX: ffff88007b7ef508 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88007ae2c400 RDI: ffff880076b3a734 RBP: ffffffff8100bc0e R08: 0000000000000001 R09: 0000000000000001 R10: 0000000000000000 R11: 0000000000000001 R12: ffff8800760154e0 R13: ffff8800760154e0 R14: ffff880024017cf0 R15: ffff88007b7ef488 FS: 00007f7464baa740(0000) GS:ffff880002200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f74642c81b0 CR3: 0000000078444000 CR4: 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process umount (pid: 22194, threadinfo ffff88007b7ee000, task ffff88007d059500) Stack: ffff88007b7ef5a8 ffffffffa08ee29c ffff8800754f3d98 ffff880073c08400 ffff88007ae2c400 ffff880078d76c38 ffff88007ae2c510 ffff88007ae2c4f0 ffff880076b3a648 0000000100000001 ffff88007ae2c4f8 ffff88007d0449e8 Call Trace: [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b Code: 00 00 00 01 74 05 e8 62 e0 d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 3e 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> 1f 44 00 00 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Call Trace: [] ? osc_cl_page_osc+0x15/0x60 [osc] [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#0 stuck for 67s! [umount:22194] Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] CPU 0 Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] Pid: 22194, comm: umount Not tainted 2.6.32-279.14.1.el6.x86_64 #1 Red Hat KVM RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff88007b7ef508 EFLAGS: 00000206 RAX: 0000000000000001 RBX: ffff88007b7ef508 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88007ae2c400 RDI: ffff880076b3a734 RBP: ffffffff8100bc0e R08: 0000000000000001 R09: 0000000000000001 R10: 0000000000000000 R11: 0000000000000001 R12: ffff8800760154e0 R13: ffff8800760154e0 R14: ffff880024017cf0 R15: ffff88007b7ef488 FS: 00007f7464baa740(0000) GS:ffff880002200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f74642c81b0 CR3: 0000000078444000 CR4: 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process umount (pid: 22194, threadinfo ffff88007b7ee000, task ffff88007d059500) Stack: ffff88007b7ef5a8 ffffffffa08ee29c ffff8800754f3d98 ffff880073c08400 ffff88007ae2c400 ffff880078d76c38 ffff88007ae2c510 ffff88007ae2c4f0 ffff880076b3a648 0000000100000001 ffff88007ae2c4f8 ffff88007d0449e8 Call Trace: [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b Code: 00 00 00 01 74 05 e8 62 e0 d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 3e 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> 1f 44 00 00 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Call Trace: [] ? osc_cl_page_osc+0x15/0x60 [osc] [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#0 stuck for 67s! [umount:22194] Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] CPU 0 Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] Pid: 22194, comm: umount Not tainted 2.6.32-279.14.1.el6.x86_64 #1 Red Hat KVM RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff88007b7ef508 EFLAGS: 00000206 RAX: 0000000000000001 RBX: ffff88007b7ef508 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88007ae2c400 RDI: ffff880076b3a734 RBP: ffffffff8100bc0e R08: 0000000000000001 R09: 0000000000000001 R10: 0000000000000000 R11: 0000000000000001 R12: ffff8800760154e0 R13: ffff8800760154e0 R14: ffff880024017cf0 R15: ffff88007b7ef488 FS: 00007f7464baa740(0000) GS:ffff880002200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f74642c81b0 CR3: 0000000078444000 CR4: 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process umount (pid: 22194, threadinfo ffff88007b7ee000, task ffff88007d059500) Stack: ffff88007b7ef5a8 ffffffffa08ee29c ffff8800754f3d98 ffff880073c08400 ffff88007ae2c400 ffff880078d76c38 ffff88007ae2c510 ffff88007ae2c4f0 ffff880076b3a648 0000000100000001 ffff88007ae2c4f8 ffff88007d0449e8 Call Trace: [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b Code: 00 00 00 01 74 05 e8 62 e0 d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 3e 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> 1f 44 00 00 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Call Trace: [] ? osc_cl_page_osc+0x15/0x60 [osc] [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#0 stuck for 68s! [umount:22194] Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] CPU 0 Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] Pid: 22194, comm: umount Not tainted 2.6.32-279.14.1.el6.x86_64 #1 Red Hat KVM RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff88007b7ef508 EFLAGS: 00000206 RAX: 0000000000000001 RBX: ffff88007b7ef508 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88007ae2c400 RDI: ffff880076b3a734 RBP: ffffffff8100bc0e R08: 0000000000000001 R09: 0000000000000001 R10: 0000000000000000 R11: 0000000000000001 R12: ffff8800760154e0 R13: ffff8800760154e0 R14: ffff880024017cf0 R15: ffff88007b7ef488 FS: 00007f7464baa740(0000) GS:ffff880002200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f74642c81b0 CR3: 0000000078444000 CR4: 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process umount (pid: 22194, threadinfo ffff88007b7ee000, task ffff88007d059500) Stack: ffff88007b7ef5a8 ffffffffa08ee29c ffff8800754f3d98 ffff880073c08400 ffff88007ae2c400 ffff880078d76c38 ffff88007ae2c510 ffff88007ae2c4f0 ffff880076b3a648 0000000100000001 ffff88007ae2c4f8 ffff88007d0449e8 Call Trace: [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b Code: 00 00 00 01 74 05 e8 62 e0 d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 3e 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> 1f 44 00 00 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Call Trace: [] ? osc_cl_page_osc+0x15/0x60 [osc] [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#0 stuck for 67s! [umount:22194] Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] CPU 0 Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] Pid: 22194, comm: umount Not tainted 2.6.32-279.14.1.el6.x86_64 #1 Red Hat KVM RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff88007b7ef508 EFLAGS: 00000206 RAX: 0000000000000001 RBX: ffff88007b7ef508 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88007ae2c400 RDI: ffff880076b3a734 RBP: ffffffff8100bc0e R08: 0000000000000001 R09: 0000000000000001 R10: 0000000000000000 R11: 0000000000000001 R12: ffff8800760154e0 R13: ffff8800760154e0 R14: ffff880024017cf0 R15: ffff88007b7ef488 FS: 00007f7464baa740(0000) GS:ffff880002200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f74642c81b0 CR3: 0000000078444000 CR4: 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process umount (pid: 22194, threadinfo ffff88007b7ee000, task ffff88007d059500) Stack: ffff88007b7ef5a8 ffffffffa08ee29c ffff8800754f3d98 ffff880073c08400 ffff88007ae2c400 ffff880078d76c38 ffff88007ae2c510 ffff88007ae2c4f0 ffff880076b3a648 0000000100000001 ffff88007ae2c4f8 ffff88007d0449e8 Call Trace: [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b Code: 00 00 00 01 74 05 e8 62 e0 d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 3e 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> 1f 44 00 00 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Call Trace: [] ? osc_cl_page_osc+0x15/0x60 [osc] [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#0 stuck for 67s! [umount:22194] Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] CPU 0 Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] Pid: 22194, comm: umount Not tainted 2.6.32-279.14.1.el6.x86_64 #1 Red Hat KVM RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff88007b7ef508 EFLAGS: 00000206 RAX: 0000000000000001 RBX: ffff88007b7ef508 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88007ae2c400 RDI: ffff880076b3a734 RBP: ffffffff8100bc0e R08: 0000000000000001 R09: 0000000000000001 R10: 0000000000000000 R11: 0000000000000001 R12: ffff8800760154e0 R13: ffff8800760154e0 R14: ffff880024017cf0 R15: ffff88007b7ef488 FS: 00007f7464baa740(0000) GS:ffff880002200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f74642c81b0 CR3: 0000000078444000 CR4: 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process umount (pid: 22194, threadinfo ffff88007b7ee000, task ffff88007d059500) Stack: ffff88007b7ef5a8 ffffffffa08ee29c ffff8800754f3d98 ffff880073c08400 ffff88007ae2c400 ffff880078d76c38 ffff88007ae2c510 ffff88007ae2c4f0 ffff880076b3a648 0000000100000001 ffff88007ae2c4f8 ffff88007d0449e8 Call Trace: [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b Code: 00 00 00 01 74 05 e8 62 e0 d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 3e 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> 1f 44 00 00 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Call Trace: [] ? osc_cl_page_osc+0x15/0x60 [osc] [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#0 stuck for 67s! [umount:22194] Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] CPU 0 Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] Pid: 22194, comm: umount Not tainted 2.6.32-279.14.1.el6.x86_64 #1 Red Hat KVM RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff88007b7ef508 EFLAGS: 00000206 RAX: 0000000000000001 RBX: ffff88007b7ef508 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88007ae2c400 RDI: ffff880076b3a734 RBP: ffffffff8100bc0e R08: 0000000000000001 R09: 0000000000000001 R10: 0000000000000000 R11: 0000000000000001 R12: ffff8800760154e0 R13: ffff8800760154e0 R14: ffff880024017cf0 R15: ffff88007b7ef488 FS: 00007f7464baa740(0000) GS:ffff880002200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f74642c81b0 CR3: 0000000078444000 CR4: 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process umount (pid: 22194, threadinfo ffff88007b7ee000, task ffff88007d059500) Stack: ffff88007b7ef5a8 ffffffffa08ee29c ffff8800754f3d98 ffff880073c08400 ffff88007ae2c400 ffff880078d76c38 ffff88007ae2c510 ffff88007ae2c4f0 ffff880076b3a648 0000000100000001 ffff88007ae2c4f8 ffff88007d0449e8 Call Trace: [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b Code: 00 00 00 01 74 05 e8 62 e0 d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 3e 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> 1f 44 00 00 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Call Trace: [] ? osc_cl_page_osc+0x15/0x60 [osc] [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#0 stuck for 67s! [umount:22194] Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] CPU 0 Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] Pid: 22194, comm: umount Not tainted 2.6.32-279.14.1.el6.x86_64 #1 Red Hat KVM RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff88007b7ef508 EFLAGS: 00000206 RAX: 0000000000000001 RBX: ffff88007b7ef508 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88007ae2c400 RDI: ffff880076b3a734 RBP: ffffffff8100bc0e R08: 0000000000000001 R09: 0000000000000001 R10: 0000000000000000 R11: 0000000000000001 R12: ffff8800760154e0 R13: ffff8800760154e0 R14: ffff880024017cf0 R15: ffff88007b7ef488 FS: 00007f7464baa740(0000) GS:ffff880002200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f74642c81b0 CR3: 0000000078444000 CR4: 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process umount (pid: 22194, threadinfo ffff88007b7ee000, task ffff88007d059500) Stack: ffff88007b7ef5a8 ffffffffa08ee29c ffff8800754f3d98 ffff880073c08400 ffff88007ae2c400 ffff880078d76c38 ffff88007ae2c510 ffff88007ae2c4f0 ffff880076b3a648 0000000100000001 ffff88007ae2c4f8 ffff88007d0449e8 Call Trace: [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b Code: 00 00 00 01 74 05 e8 62 e0 d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 3e 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> 1f 44 00 00 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Call Trace: [] ? osc_cl_page_osc+0x15/0x60 [osc] [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#0 stuck for 67s! [umount:22194] Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] CPU 0 Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] Pid: 22194, comm: umount Not tainted 2.6.32-279.14.1.el6.x86_64 #1 Red Hat KVM RIP: 0010:[] [] _spin_lock+0x26/0x30 RSP: 0018:ffff88007b7ef508 EFLAGS: 00000206 RAX: 0000000000000001 RBX: ffff88007b7ef508 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88007ae2c400 RDI: ffff880076b3a734 RBP: ffffffff8100bc0e R08: 0000000000000001 R09: 0000000000000001 R10: 0000000000000000 R11: 0000000000000001 R12: ffff8800760154e0 R13: ffff8800760154e0 R14: ffff880024017cf0 R15: ffff88007b7ef488 FS: 00007f7464baa740(0000) GS:ffff880002200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f74642c81b0 CR3: 0000000078444000 CR4: 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process umount (pid: 22194, threadinfo ffff88007b7ee000, task ffff88007d059500) Stack: ffff88007b7ef5a8 ffffffffa08ee29c ffff8800754f3d98 ffff880073c08400 ffff88007ae2c400 ffff880078d76c38 ffff88007ae2c510 ffff88007ae2c4f0 ffff880076b3a648 0000000100000001 ffff88007ae2c4f8 ffff88007d0449e8 Call Trace: [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b Code: e0 d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 3e 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f 1f 44 00 00 83 3f 00 <75> f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 44 00 00 3e 81 Call Trace: [] ? osc_cl_page_osc+0x15/0x60 [osc] [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#0 stuck for 67s! [umount:22194] Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] CPU 0 Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] Pid: 22194, comm: umount Not tainted 2.6.32-279.14.1.el6.x86_64 #1 Red Hat KVM RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff88007b7ef508 EFLAGS: 00000206 RAX: 0000000000000001 RBX: ffff88007b7ef508 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88007ae2c400 RDI: ffff880076b3a734 RBP: ffffffff8100bc0e R08: 0000000000000001 R09: 0000000000000001 R10: 0000000000000000 R11: 0000000000000001 R12: ffff8800760154e0 R13: ffff8800760154e0 R14: ffff880024017cf0 R15: ffff88007b7ef488 FS: 00007f7464baa740(0000) GS:ffff880002200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f74642c81b0 CR3: 0000000078444000 CR4: 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process umount (pid: 22194, threadinfo ffff88007b7ee000, task ffff88007d059500) Stack: ffff88007b7ef5a8 ffffffffa08ee29c ffff8800754f3d98 ffff880073c08400 ffff88007ae2c400 ffff880078d76c38 ffff88007ae2c510 ffff88007ae2c4f0 ffff880076b3a648 0000000100000001 ffff88007ae2c4f8 ffff88007d0449e8 Call Trace: [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b Code: 00 00 00 01 74 05 e8 62 e0 d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 3e 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> 1f 44 00 00 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Call Trace: [] ? osc_cl_page_osc+0x15/0x60 [osc] [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#0 stuck for 67s! [umount:22194] Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] CPU 0 Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] Pid: 22194, comm: umount Not tainted 2.6.32-279.14.1.el6.x86_64 #1 Red Hat KVM RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff88007b7ef508 EFLAGS: 00000206 RAX: 0000000000000001 RBX: ffff88007b7ef508 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88007ae2c400 RDI: ffff880076b3a734 RBP: ffffffff8100bc0e R08: 0000000000000001 R09: 0000000000000001 R10: 0000000000000000 R11: 0000000000000001 R12: ffff8800760154e0 R13: ffff8800760154e0 R14: ffff880024017cf0 R15: ffff88007b7ef488 FS: 00007f7464baa740(0000) GS:ffff880002200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f74642c81b0 CR3: 0000000078444000 CR4: 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process umount (pid: 22194, threadinfo ffff88007b7ee000, task ffff88007d059500) Stack: ffff88007b7ef5a8 ffffffffa08ee29c ffff8800754f3d98 ffff880073c08400 ffff88007ae2c400 ffff880078d76c38 ffff88007ae2c510 ffff88007ae2c4f0 ffff880076b3a648 0000000100000001 ffff88007ae2c4f8 ffff88007d0449e8 Call Trace: [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b Code: 00 00 00 01 74 05 e8 62 e0 d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 3e 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> 1f 44 00 00 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Call Trace: [] ? osc_cl_page_osc+0x15/0x60 [osc] [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#0 stuck for 67s! [umount:22194] Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] CPU 0 Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] Pid: 22194, comm: umount Not tainted 2.6.32-279.14.1.el6.x86_64 #1 Red Hat KVM RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff88007b7ef508 EFLAGS: 00000206 RAX: 0000000000000001 RBX: ffff88007b7ef508 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88007ae2c400 RDI: ffff880076b3a734 RBP: ffffffff8100bc0e R08: 0000000000000001 R09: 0000000000000001 R10: 0000000000000000 R11: 0000000000000001 R12: ffff8800760154e0 R13: ffff8800760154e0 R14: ffff880024017cf0 R15: ffff88007b7ef488 FS: 00007f7464baa740(0000) GS:ffff880002200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f74642c81b0 CR3: 0000000078444000 CR4: 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process umount (pid: 22194, threadinfo ffff88007b7ee000, task ffff88007d059500) Stack: ffff88007b7ef5a8 ffffffffa08ee29c ffff8800754f3d98 ffff880073c08400 ffff88007ae2c400 ffff880078d76c38 ffff88007ae2c510 ffff88007ae2c4f0 ffff880076b3a648 0000000100000001 ffff88007ae2c4f8 ffff88007d0449e8 Call Trace: [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b Code: 00 00 00 01 74 05 e8 62 e0 d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 3e 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> 1f 44 00 00 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Call Trace: [] ? osc_cl_page_osc+0x15/0x60 [osc] [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#0 stuck for 67s! [umount:22194] Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] CPU 0 Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] Pid: 22194, comm: umount Not tainted 2.6.32-279.14.1.el6.x86_64 #1 Red Hat KVM RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff88007b7ef508 EFLAGS: 00000206 RAX: 0000000000000001 RBX: ffff88007b7ef508 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88007ae2c400 RDI: ffff880076b3a734 RBP: ffffffff8100bc0e R08: 0000000000000001 R09: 0000000000000001 R10: 0000000000000000 R11: 0000000000000001 R12: ffff8800760154e0 R13: ffff8800760154e0 R14: ffff880024017cf0 R15: ffff88007b7ef488 FS: 00007f7464baa740(0000) GS:ffff880002200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f74642c81b0 CR3: 0000000078444000 CR4: 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process umount (pid: 22194, threadinfo ffff88007b7ee000, task ffff88007d059500) Stack: ffff88007b7ef5a8 ffffffffa08ee29c ffff8800754f3d98 ffff880073c08400 ffff88007ae2c400 ffff880078d76c38 ffff88007ae2c510 ffff88007ae2c4f0 ffff880076b3a648 0000000100000001 ffff88007ae2c4f8 ffff88007d0449e8 Call Trace: [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b Code: 00 00 00 01 74 05 e8 62 e0 d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 3e 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> 1f 44 00 00 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Call Trace: [] ? osc_cl_page_osc+0x15/0x60 [osc] [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#0 stuck for 67s! [umount:22194] Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] CPU 0 Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] Pid: 22194, comm: umount Not tainted 2.6.32-279.14.1.el6.x86_64 #1 Red Hat KVM RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff88007b7ef508 EFLAGS: 00000206 RAX: 0000000000000001 RBX: ffff88007b7ef508 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88007ae2c400 RDI: ffff880076b3a734 RBP: ffffffff8100bc0e R08: 0000000000000001 R09: 0000000000000001 R10: 0000000000000000 R11: 0000000000000001 R12: ffff8800760154e0 R13: ffff8800760154e0 R14: ffff880024017cf0 R15: ffff88007b7ef488 FS: 00007f7464baa740(0000) GS:ffff880002200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f74642c81b0 CR3: 0000000078444000 CR4: 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process umount (pid: 22194, threadinfo ffff88007b7ee000, task ffff88007d059500) Stack: ffff88007b7ef5a8 ffffffffa08ee29c ffff8800754f3d98 ffff880073c08400 ffff88007ae2c400 ffff880078d76c38 ffff88007ae2c510 ffff88007ae2c4f0 ffff880076b3a648 0000000100000001 ffff88007ae2c4f8 ffff88007d0449e8 Call Trace: [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b Code: 00 00 00 01 74 05 e8 62 e0 d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 3e 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> 1f 44 00 00 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Call Trace: [] ? osc_cl_page_osc+0x15/0x60 [osc] [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#0 stuck for 67s! [umount:22194] Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] CPU 0 Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] Pid: 22194, comm: umount Not tainted 2.6.32-279.14.1.el6.x86_64 #1 Red Hat KVM RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff88007b7ef508 EFLAGS: 00000206 RAX: 0000000000000001 RBX: ffff88007b7ef508 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88007ae2c400 RDI: ffff880076b3a734 RBP: ffffffff8100bc0e R08: 0000000000000001 R09: 0000000000000001 R10: 0000000000000000 R11: 0000000000000001 R12: ffff8800760154e0 R13: ffff8800760154e0 R14: ffff880024017cf0 R15: ffff88007b7ef488 FS: 00007f7464baa740(0000) GS:ffff880002200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f74642c81b0 CR3: 0000000078444000 CR4: 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process umount (pid: 22194, threadinfo ffff88007b7ee000, task ffff88007d059500) Stack: ffff88007b7ef5a8 ffffffffa08ee29c ffff8800754f3d98 ffff880073c08400 ffff88007ae2c400 ffff880078d76c38 ffff88007ae2c510 ffff88007ae2c4f0 ffff880076b3a648 0000000100000001 ffff88007ae2c4f8 ffff88007d0449e8 Call Trace: [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b Code: 00 00 00 01 74 05 e8 62 e0 d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 3e 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> 1f 44 00 00 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Call Trace: [] ? osc_cl_page_osc+0x15/0x60 [osc] [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#0 stuck for 67s! [umount:22194] Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] CPU 0 Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] Pid: 22194, comm: umount Not tainted 2.6.32-279.14.1.el6.x86_64 #1 Red Hat KVM RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff88007b7ef508 EFLAGS: 00000206 RAX: 0000000000000001 RBX: ffff88007b7ef508 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88007ae2c400 RDI: ffff880076b3a734 RBP: ffffffff8100bc0e R08: 0000000000000001 R09: 0000000000000001 R10: 0000000000000000 R11: 0000000000000001 R12: ffff8800760154e0 R13: ffff8800760154e0 R14: ffff880024017cf0 R15: ffff88007b7ef488 FS: 00007f7464baa740(0000) GS:ffff880002200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f74642c81b0 CR3: 0000000078444000 CR4: 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process umount (pid: 22194, threadinfo ffff88007b7ee000, task ffff88007d059500) Stack: ffff88007b7ef5a8 ffffffffa08ee29c ffff8800754f3d98 ffff880073c08400 ffff88007ae2c400 ffff880078d76c38 ffff88007ae2c510 ffff88007ae2c4f0 ffff880076b3a648 0000000100000001 ffff88007ae2c4f8 ffff88007d0449e8 Call Trace: [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b Code: 00 00 00 01 74 05 e8 62 e0 d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 3e 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> 1f 44 00 00 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Call Trace: [] ? osc_cl_page_osc+0x15/0x60 [osc] [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#0 stuck for 68s! [umount:22194] Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] CPU 0 Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] Pid: 22194, comm: umount Not tainted 2.6.32-279.14.1.el6.x86_64 #1 Red Hat KVM RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff88007b7ef508 EFLAGS: 00000206 RAX: 0000000000000001 RBX: ffff88007b7ef508 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88007ae2c400 RDI: ffff880076b3a734 RBP: ffffffff8100bc0e R08: 0000000000000001 R09: 0000000000000001 R10: 0000000000000000 R11: 0000000000000001 R12: ffff8800760154e0 R13: ffff8800760154e0 R14: ffff880024017cf0 R15: ffff88007b7ef488 FS: 00007f7464baa740(0000) GS:ffff880002200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f74642c81b0 CR3: 0000000078444000 CR4: 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process umount (pid: 22194, threadinfo ffff88007b7ee000, task ffff88007d059500) Stack: ffff88007b7ef5a8 ffffffffa08ee29c ffff8800754f3d98 ffff880073c08400 ffff88007ae2c400 ffff880078d76c38 ffff88007ae2c510 ffff88007ae2c4f0 ffff880076b3a648 0000000100000001 ffff88007ae2c4f8 ffff88007d0449e8 Call Trace: [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b Code: 00 00 00 01 74 05 e8 62 e0 d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 3e 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> 1f 44 00 00 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Call Trace: [] ? osc_cl_page_osc+0x15/0x60 [osc] [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#0 stuck for 67s! [umount:22194] Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] CPU 0 Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] Pid: 22194, comm: umount Not tainted 2.6.32-279.14.1.el6.x86_64 #1 Red Hat KVM RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff88007b7ef508 EFLAGS: 00000206 RAX: 0000000000000001 RBX: ffff88007b7ef508 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88007ae2c400 RDI: ffff880076b3a734 RBP: ffffffff8100bc0e R08: 0000000000000001 R09: 0000000000000001 R10: 0000000000000000 R11: 0000000000000001 R12: ffff8800760154e0 R13: ffff8800760154e0 R14: ffff880024017cf0 R15: ffff88007b7ef488 FS: 00007f7464baa740(0000) GS:ffff880002200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f74642c81b0 CR3: 0000000078444000 CR4: 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process umount (pid: 22194, threadinfo ffff88007b7ee000, task ffff88007d059500) Stack: ffff88007b7ef5a8 ffffffffa08ee29c ffff8800754f3d98 ffff880073c08400 ffff88007ae2c400 ffff880078d76c38 ffff88007ae2c510 ffff88007ae2c4f0 ffff880076b3a648 0000000100000001 ffff88007ae2c4f8 ffff88007d0449e8 Call Trace: [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b Code: 00 00 00 01 74 05 e8 62 e0 d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 3e 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> 1f 44 00 00 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Call Trace: [] ? osc_cl_page_osc+0x15/0x60 [osc] [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#0 stuck for 67s! [umount:22194] Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] CPU 0 Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] Pid: 22194, comm: umount Not tainted 2.6.32-279.14.1.el6.x86_64 #1 Red Hat KVM RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff88007b7ef508 EFLAGS: 00000206 RAX: 0000000000000001 RBX: ffff88007b7ef508 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88007ae2c400 RDI: ffff880076b3a734 RBP: ffffffff8100bc0e R08: 0000000000000001 R09: 0000000000000001 R10: 0000000000000000 R11: 0000000000000001 R12: ffff8800760154e0 R13: ffff8800760154e0 R14: ffff880024017cf0 R15: ffff88007b7ef488 FS: 00007f7464baa740(0000) GS:ffff880002200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f74642c81b0 CR3: 0000000078444000 CR4: 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process umount (pid: 22194, threadinfo ffff88007b7ee000, task ffff88007d059500) Stack: ffff88007b7ef5a8 ffffffffa08ee29c ffff8800754f3d98 ffff880073c08400 ffff88007ae2c400 ffff880078d76c38 ffff88007ae2c510 ffff88007ae2c4f0 ffff880076b3a648 0000000100000001 ffff88007ae2c4f8 ffff88007d0449e8 Call Trace: [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b Code: 00 00 00 01 74 05 e8 62 e0 d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 3e 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> 1f 44 00 00 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Call Trace: [] ? osc_cl_page_osc+0x15/0x60 [osc] [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#0 stuck for 67s! [umount:22194] Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] CPU 0 Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] Pid: 22194, comm: umount Not tainted 2.6.32-279.14.1.el6.x86_64 #1 Red Hat KVM RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff88007b7ef508 EFLAGS: 00000206 RAX: 0000000000000001 RBX: ffff88007b7ef508 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88007ae2c400 RDI: ffff880076b3a734 RBP: ffffffff8100bc0e R08: 0000000000000001 R09: 0000000000000001 R10: 0000000000000000 R11: 0000000000000001 R12: ffff8800760154e0 R13: ffff8800760154e0 R14: ffff880024017cf0 R15: ffff88007b7ef488 FS: 00007f7464baa740(0000) GS:ffff880002200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f74642c81b0 CR3: 0000000078444000 CR4: 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process umount (pid: 22194, threadinfo ffff88007b7ee000, task ffff88007d059500) Stack: ffff88007b7ef5a8 ffffffffa08ee29c ffff8800754f3d98 ffff880073c08400 ffff88007ae2c400 ffff880078d76c38 ffff88007ae2c510 ffff88007ae2c4f0 ffff880076b3a648 0000000100000001 ffff88007ae2c4f8 ffff88007d0449e8 Call Trace: [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b Code: 00 00 00 01 74 05 e8 62 e0 d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 3e 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> 1f 44 00 00 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Call Trace: [] ? osc_cl_page_osc+0x15/0x60 [osc] [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#0 stuck for 67s! [umount:22194] Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] CPU 0 Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] Pid: 22194, comm: umount Not tainted 2.6.32-279.14.1.el6.x86_64 #1 Red Hat KVM RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff88007b7ef508 EFLAGS: 00000206 RAX: 0000000000000001 RBX: ffff88007b7ef508 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88007ae2c400 RDI: ffff880076b3a734 RBP: ffffffff8100bc0e R08: 0000000000000001 R09: 0000000000000001 R10: 0000000000000000 R11: 0000000000000001 R12: ffff8800760154e0 R13: ffff8800760154e0 R14: ffff880024017cf0 R15: ffff88007b7ef488 FS: 00007f7464baa740(0000) GS:ffff880002200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f74642c81b0 CR3: 0000000078444000 CR4: 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process umount (pid: 22194, threadinfo ffff88007b7ee000, task ffff88007d059500) Stack: ffff88007b7ef5a8 ffffffffa08ee29c ffff8800754f3d98 ffff880073c08400 ffff88007ae2c400 ffff880078d76c38 ffff88007ae2c510 ffff88007ae2c4f0 ffff880076b3a648 0000000100000001 ffff88007ae2c4f8 ffff88007d0449e8 Call Trace: [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b Code: 00 00 00 01 74 05 e8 62 e0 d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 3e 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> 1f 44 00 00 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Call Trace: [] ? osc_cl_page_osc+0x15/0x60 [osc] [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#0 stuck for 67s! [umount:22194] Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] CPU 0 Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] Pid: 22194, comm: umount Not tainted 2.6.32-279.14.1.el6.x86_64 #1 Red Hat KVM RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff88007b7ef508 EFLAGS: 00000206 RAX: 0000000000000001 RBX: ffff88007b7ef508 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88007ae2c400 RDI: ffff880076b3a734 RBP: ffffffff8100bc0e R08: 0000000000000001 R09: 0000000000000001 R10: 0000000000000000 R11: 0000000000000001 R12: ffff8800760154e0 R13: ffff8800760154e0 R14: ffff880024017cf0 R15: ffff88007b7ef488 FS: 00007f7464baa740(0000) GS:ffff880002200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f74642c81b0 CR3: 0000000078444000 CR4: 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process umount (pid: 22194, threadinfo ffff88007b7ee000, task ffff88007d059500) Stack: ffff88007b7ef5a8 ffffffffa08ee29c ffff8800754f3d98 ffff880073c08400 ffff88007ae2c400 ffff880078d76c38 ffff88007ae2c510 ffff88007ae2c4f0 ffff880076b3a648 0000000100000001 ffff88007ae2c4f8 ffff88007d0449e8 Call Trace: [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b Code: 00 00 00 01 74 05 e8 62 e0 d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 3e 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> 1f 44 00 00 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Call Trace: [] ? osc_cl_page_osc+0x15/0x60 [osc] [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#0 stuck for 67s! [umount:22194] Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] CPU 0 Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] Pid: 22194, comm: umount Not tainted 2.6.32-279.14.1.el6.x86_64 #1 Red Hat KVM RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff88007b7ef508 EFLAGS: 00000206 RAX: 0000000000000001 RBX: ffff88007b7ef508 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88007ae2c400 RDI: ffff880076b3a734 RBP: ffffffff8100bc0e R08: 0000000000000001 R09: 0000000000000001 R10: 0000000000000000 R11: 0000000000000001 R12: ffff8800760154e0 R13: ffff8800760154e0 R14: ffff880024017cf0 R15: ffff88007b7ef488 FS: 00007f7464baa740(0000) GS:ffff880002200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f74642c81b0 CR3: 0000000078444000 CR4: 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process umount (pid: 22194, threadinfo ffff88007b7ee000, task ffff88007d059500) Stack: ffff88007b7ef5a8 ffffffffa08ee29c ffff8800754f3d98 ffff880073c08400 ffff88007ae2c400 ffff880078d76c38 ffff88007ae2c510 ffff88007ae2c4f0 ffff880076b3a648 0000000100000001 ffff88007ae2c4f8 ffff88007d0449e8 Call Trace: [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b Code: 00 00 00 01 74 05 e8 62 e0 d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 3e 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> 1f 44 00 00 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Call Trace: [] ? osc_cl_page_osc+0x15/0x60 [osc] [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#0 stuck for 67s! [umount:22194] Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] CPU 0 Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] Pid: 22194, comm: umount Not tainted 2.6.32-279.14.1.el6.x86_64 #1 Red Hat KVM RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff88007b7ef508 EFLAGS: 00000206 RAX: 0000000000000001 RBX: ffff88007b7ef508 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88007ae2c400 RDI: ffff880076b3a734 RBP: ffffffff8100bc0e R08: 0000000000000001 R09: 0000000000000001 R10: 0000000000000000 R11: 0000000000000001 R12: ffff8800760154e0 R13: ffff8800760154e0 R14: ffff880024017cf0 R15: ffff88007b7ef488 FS: 00007f7464baa740(0000) GS:ffff880002200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f74642c81b0 CR3: 0000000078444000 CR4: 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process umount (pid: 22194, threadinfo ffff88007b7ee000, task ffff88007d059500) Stack: ffff88007b7ef5a8 ffffffffa08ee29c ffff8800754f3d98 ffff880073c08400 ffff88007ae2c400 ffff880078d76c38 ffff88007ae2c510 ffff88007ae2c4f0 ffff880076b3a648 0000000100000001 ffff88007ae2c4f8 ffff88007d0449e8 Call Trace: [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b Code: 00 00 00 01 74 05 e8 62 e0 d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 3e 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> 1f 44 00 00 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Call Trace: [] ? osc_cl_page_osc+0x15/0x60 [osc] [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#0 stuck for 67s! [umount:22194] Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] CPU 0 Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] Pid: 22194, comm: umount Not tainted 2.6.32-279.14.1.el6.x86_64 #1 Red Hat KVM RIP: 0010:[] [] _spin_lock+0x26/0x30 RSP: 0018:ffff88007b7ef508 EFLAGS: 00000206 RAX: 0000000000000001 RBX: ffff88007b7ef508 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88007ae2c400 RDI: ffff880076b3a734 RBP: ffffffff8100bc0e R08: 0000000000000001 R09: 0000000000000001 R10: 0000000000000000 R11: 0000000000000001 R12: ffff8800760154e0 R13: ffff8800760154e0 R14: ffff880024017cf0 R15: ffff88007b7ef488 FS: 00007f7464baa740(0000) GS:ffff880002200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f74642c81b0 CR3: 0000000078444000 CR4: 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process umount (pid: 22194, threadinfo ffff88007b7ee000, task ffff88007d059500) Stack: ffff88007b7ef5a8 ffffffffa08ee29c ffff8800754f3d98 ffff880073c08400 ffff88007ae2c400 ffff880078d76c38 ffff88007ae2c510 ffff88007ae2c4f0 ffff880076b3a648 0000000100000001 ffff88007ae2c4f8 ffff88007d0449e8 Call Trace: [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b Code: e0 d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 3e 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f 1f 44 00 00 83 3f 00 <75> f4 eb df c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 44 00 00 3e 81 Call Trace: [] ? osc_cl_page_osc+0x15/0x60 [osc] [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#0 stuck for 67s! [umount:22194] Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] CPU 0 Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] Pid: 22194, comm: umount Not tainted 2.6.32-279.14.1.el6.x86_64 #1 Red Hat KVM RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff88007b7ef508 EFLAGS: 00000206 RAX: 0000000000000001 RBX: ffff88007b7ef508 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88007ae2c400 RDI: ffff880076b3a734 RBP: ffffffff8100bc0e R08: 0000000000000001 R09: 0000000000000001 R10: 0000000000000000 R11: 0000000000000001 R12: ffff8800760154e0 R13: ffff8800760154e0 R14: ffff880024017cf0 R15: ffff88007b7ef488 FS: 00007f7464baa740(0000) GS:ffff880002200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f74642c81b0 CR3: 0000000078444000 CR4: 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process umount (pid: 22194, threadinfo ffff88007b7ee000, task ffff88007d059500) Stack: ffff88007b7ef5a8 ffffffffa08ee29c ffff8800754f3d98 ffff880073c08400 ffff88007ae2c400 ffff880078d76c38 ffff88007ae2c510 ffff88007ae2c4f0 ffff880076b3a648 0000000100000001 ffff88007ae2c4f8 ffff88007d0449e8 Call Trace: [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b Code: 00 00 00 01 74 05 e8 62 e0 d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 3e 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> 1f 44 00 00 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Call Trace: [] ? osc_cl_page_osc+0x15/0x60 [osc] [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#0 stuck for 67s! [umount:22194] Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] CPU 0 Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] Pid: 22194, comm: umount Not tainted 2.6.32-279.14.1.el6.x86_64 #1 Red Hat KVM RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff88007b7ef508 EFLAGS: 00000206 RAX: 0000000000000001 RBX: ffff88007b7ef508 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88007ae2c400 RDI: ffff880076b3a734 RBP: ffffffff8100bc0e R08: 0000000000000001 R09: 0000000000000001 R10: 0000000000000000 R11: 0000000000000001 R12: ffff8800760154e0 R13: ffff8800760154e0 R14: ffff880024017cf0 R15: ffff88007b7ef488 FS: 00007f7464baa740(0000) GS:ffff880002200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f74642c81b0 CR3: 0000000078444000 CR4: 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process umount (pid: 22194, threadinfo ffff88007b7ee000, task ffff88007d059500) Stack: ffff88007b7ef5a8 ffffffffa08ee29c ffff8800754f3d98 ffff880073c08400 ffff88007ae2c400 ffff880078d76c38 ffff88007ae2c510 ffff88007ae2c4f0 ffff880076b3a648 0000000100000001 ffff88007ae2c4f8 ffff88007d0449e8 Call Trace: [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b Code: 00 00 00 01 74 05 e8 62 e0 d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 3e 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> 1f 44 00 00 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Call Trace: [] ? osc_cl_page_osc+0x15/0x60 [osc] [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#0 stuck for 67s! [umount:22194] Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] CPU 0 Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] Pid: 22194, comm: umount Not tainted 2.6.32-279.14.1.el6.x86_64 #1 Red Hat KVM RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff88007b7ef508 EFLAGS: 00000206 RAX: 0000000000000001 RBX: ffff88007b7ef508 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88007ae2c400 RDI: ffff880076b3a734 RBP: ffffffff8100bc0e R08: 0000000000000001 R09: 0000000000000001 R10: 0000000000000000 R11: 0000000000000001 R12: ffff8800760154e0 R13: ffff8800760154e0 R14: ffff880024017cf0 R15: ffff88007b7ef488 FS: 00007f7464baa740(0000) GS:ffff880002200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f74642c81b0 CR3: 0000000078444000 CR4: 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process umount (pid: 22194, threadinfo ffff88007b7ee000, task ffff88007d059500) Stack: ffff88007b7ef5a8 ffffffffa08ee29c ffff8800754f3d98 ffff880073c08400 ffff88007ae2c400 ffff880078d76c38 ffff88007ae2c510 ffff88007ae2c4f0 ffff880076b3a648 0000000100000001 ffff88007ae2c4f8 ffff88007d0449e8 Call Trace: [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b Code: 00 00 00 01 74 05 e8 62 e0 d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 3e 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> 1f 44 00 00 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Call Trace: [] ? osc_cl_page_osc+0x15/0x60 [osc] [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#0 stuck for 67s! [umount:22194] Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] CPU 0 Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] Pid: 22194, comm: umount Not tainted 2.6.32-279.14.1.el6.x86_64 #1 Red Hat KVM RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff88007b7ef508 EFLAGS: 00000206 RAX: 0000000000000001 RBX: ffff88007b7ef508 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88007ae2c400 RDI: ffff880076b3a734 RBP: ffffffff8100bc0e R08: 0000000000000001 R09: 0000000000000001 R10: 0000000000000000 R11: 0000000000000001 R12: ffff8800760154e0 R13: ffff8800760154e0 R14: ffff880024017cf0 R15: ffff88007b7ef488 FS: 00007f7464baa740(0000) GS:ffff880002200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f74642c81b0 CR3: 0000000078444000 CR4: 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process umount (pid: 22194, threadinfo ffff88007b7ee000, task ffff88007d059500) Stack: ffff88007b7ef5a8 ffffffffa08ee29c ffff8800754f3d98 ffff880073c08400 ffff88007ae2c400 ffff880078d76c38 ffff88007ae2c510 ffff88007ae2c4f0 ffff880076b3a648 0000000100000001 ffff88007ae2c4f8 ffff88007d0449e8 Call Trace: [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b Code: 00 00 00 01 74 05 e8 62 e0 d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 3e 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> 1f 44 00 00 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Call Trace: [] ? osc_cl_page_osc+0x15/0x60 [osc] [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#0 stuck for 68s! [umount:22194] Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] CPU 0 Modules linked in: lustre(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) lquota(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) ext2 nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] Pid: 22194, comm: umount Not tainted 2.6.32-279.14.1.el6.x86_64 #1 Red Hat KVM RIP: 0010:[] [] _spin_lock+0x1e/0x30 RSP: 0018:ffff88007b7ef508 EFLAGS: 00000206 RAX: 0000000000000001 RBX: ffff88007b7ef508 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88007ae2c400 RDI: ffff880076b3a734 RBP: ffffffff8100bc0e R08: 0000000000000001 R09: 0000000000000001 R10: 0000000000000000 R11: 0000000000000001 R12: ffff8800760154e0 R13: ffff8800760154e0 R14: ffff880024017cf0 R15: ffff88007b7ef488 FS: 00007f7464baa740(0000) GS:ffff880002200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f74642c81b0 CR3: 0000000078444000 CR4: 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process umount (pid: 22194, threadinfo ffff88007b7ee000, task ffff88007d059500) Stack: ffff88007b7ef5a8 ffffffffa08ee29c ffff8800754f3d98 ffff880073c08400 ffff88007ae2c400 ffff880078d76c38 ffff88007ae2c510 ffff88007ae2c4f0 ffff880076b3a648 0000000100000001 ffff88007ae2c4f8 ffff88007d0449e8 Call Trace: [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b Code: 00 00 00 01 74 05 e8 62 e0 d7 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 3e 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> 1f 44 00 00 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 Call Trace: [] ? osc_cl_page_osc+0x15/0x60 [osc] [] ? osc_io_submit+0x31c/0x540 [osc] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? lov_io_submit+0x355/0xc30 [lov] [] ? cl_page_put+0x1e5/0x580 [obdclass] [] ? pgoff_at_lock+0x1a/0x70 [obdclass] [] ? cl_io_submit_rw+0x78/0x180 [obdclass] [] ? cl_io_submit_sync+0x9f/0x210 [obdclass] [] ? cl_lock_page_out+0x207/0x330 [obdclass] [] ? pageout_cb+0x0/0x100 [obdclass] [] ? osc_lock_flush+0x4f/0x90 [osc] [] ? osc_lock_cancel+0xcf/0x1b0 [osc] [] ? cl_lock_cancel0+0x75/0x160 [obdclass] [] ? cl_lock_cancel+0x13b/0x140 [obdclass] [] ? osc_ldlm_blocking_ast+0x13a/0x380 [osc] [] ? ldlm_cancel_callback+0x60/0x100 [ptlrpc] [] ? ldlm_cli_cancel_local+0x7b/0x380 [ptlrpc] [] ? ldlm_cli_cancel+0x58/0x3a0 [ptlrpc] [] ? cleanup_resource+0x168/0x300 [ptlrpc] [] ? cfs_hash_spin_lock+0xe/0x10 [libcfs] [] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [] ? osc_import_event+0x3d2/0x1470 [osc] [] ? ptlrpc_invalidate_import+0x2a9/0x8d0 [ptlrpc] [] ? osc_import_event+0xaec/0x1470 [osc] [] ? ptlrpc_set_import_active+0x33d/0x480 [ptlrpc] [] ? osc_iocontrol+0xc80/0x11b0 [osc] [] ? vsnprintf+0x2b6/0x5f0 [] ? lov_iocontrol+0x27f/0x1e90 [lov] [] ? lmv_iocontrol+0x25f/0x1cf0 [lmv] [] ? path_put+0x31/0x40 [] ? putname+0x35/0x50 [] ? ll_umount_begin+0x2c0/0xc30 [lustre] [] ? sys_umount+0x11a/0x3a0 [] ? audit_syscall_entry+0x272/0x2a0 [] ? do_page_fault+0x3e/0xa0 [] ? system_call_fastpath+0x16/0x1b