[22075.470864] Lustre: DEBUG MARKER: == sanityn test 77b: check CRR-N NRS policy ========================================================== 09:02:25 (1501333345) [22085.359953] BUG: sleeping function called from invalid context at /home/green/git/lustre-release/libcfs/libcfs/hash.c:1120 [22085.362133] in_atomic(): 1, irqs_disabled(): 0, pid: 8995, name: lctl [22085.362674] 3 locks held by lctl/8995: [22085.363514] #0: (sb_writers#3){......}, at: [] vfs_write+0x1ab/0x1e0 [22085.364569] #1: (&nrs_core.nrs_mutex){......}, at: [] ptlrpc_lprocfs_nrs_seq_write+0x38b/0x560 [ptlrpc] [22085.365679] #2: (&(&nrs->nrs_lock)->rlock){......}, at: [] nrs_policy_ctl+0x3a/0x2b0 [ptlrpc] [22085.366790] CPU: 0 PID: 8995 Comm: lctl Tainted: P OE ------------ 3.10.0-debug #2 [22085.367802] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 [22085.368320] ffff8800a9ccc880 00000000bc1ab2ff ffff8800aa97fd18 ffffffff816fd3e4 [22085.369299] ffff8800aa97fd30 ffffffff810b00e9 0000000000000010 ffff8800aa97fd80 [22085.370300] ffffffffa02000a3 ffffffffa0204b66 ffff8803002f7000 0000000000000100 [22085.371272] Call Trace: [22085.371935] [] dump_stack+0x19/0x1b [22085.372436] [] __might_sleep+0xe9/0x110 [22085.373032] [] cfs_hash_putref+0x233/0x500 [libcfs] [22085.373555] [] ? cfs_binheap_destroy+0x396/0x830 [libcfs] [22085.374190] [] nrs_crrn_stop+0x5a/0x220 [ptlrpc] [22085.374809] [] nrs_policy_stop0+0x39/0x1c0 [ptlrpc] [22085.375394] [] nrs_policy_stop_primary.isra.9+0x78/0x1f0 [ptlrpc] [22085.383477] [] nrs_policy_start_locked+0x624/0x670 [ptlrpc] [22085.384411] [] nrs_policy_ctl+0x12b/0x2b0 [ptlrpc] [22085.384932] [] ptlrpc_nrs_policy_control+0x10f/0x2d0 [ptlrpc] [22085.386016] [] ptlrpc_lprocfs_nrs_seq_write+0x3a5/0x560 [ptlrpc] [22085.400516] [] proc_reg_write+0x3d/0x80 [22085.401315] [] vfs_write+0xbd/0x1e0 [22085.402161] [] ? putname+0x3d/0x60 [22085.402955] [] SyS_write+0x84/0xf0 [22085.403620] [] system_call_fastpath+0x16/0x1b [22085.404305] BUG: scheduling while atomic: lctl/8995/0x10000002 [22085.404990] 3 locks held by lctl/8995: [22085.405616] #0: (sb_writers#3){......}, at: [] vfs_write+0x1ab/0x1e0 [22085.408460] #1: (&nrs_core.nrs_mutex){......}, at: [] ptlrpc_lprocfs_nrs_seq_write+0x38b/0x560 [ptlrpc] [22085.409775] #2: (&(&nrs->nrs_lock)->rlock){......}, at: [] nrs_policy_ctl+0x3a/0x2b0 [ptlrpc] [22085.410760] Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) lov(OE) osc(OE) mdc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) brd ext4 mbcache loop zfs(PO) zunicode(PO) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) zlib_deflate jbd2 syscopyarea sysfillrect sysimgblt ttm ata_generic drm_kms_helper pata_acpi drm i2c_piix4 ata_piix serio_raw i2c_core virtio_balloon libata virtio_console virtio_blk pcspkr floppy nfsd ip_tables rpcsec_gss_krb5 [last unloaded: libcfs] [22085.417102] CPU: 0 PID: 8995 Comm: lctl Tainted: P OE ------------ 3.10.0-debug #2 [22085.418669] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 [22085.419524] ffff8800a9ccc880 00000000bc1ab2ff ffff8800aa97fc90 ffffffff816fd3e4 [22085.420915] ffff8800aa97fca8 ffffffff816f9218 ffff8800aa97ffd8 ffff8800aa97fd08 [22085.422310] ffffffff817041a7 ffff8800a9ccc880 ffff8800aa97ffd8 ffff8800aa97ffd8 [22085.423608] Call Trace: [22085.424219] [] dump_stack+0x19/0x1b [22085.424897] [] __schedule_bug+0x59/0x68 [22085.425565] [] __schedule+0x807/0x950 [22085.427363] [] __cond_resched+0x26/0x30 [22085.428194] [] _cond_resched+0x3a/0x50 [22085.429076] [] cfs_hash_putref+0x238/0x500 [libcfs] [22085.429989] [] ? cfs_binheap_destroy+0x396/0x830 [libcfs] [22085.430976] [] nrs_crrn_stop+0x5a/0x220 [ptlrpc] [22085.444327] [] nrs_policy_stop0+0x39/0x1c0 [ptlrpc] [22085.445332] [] nrs_policy_stop_primary.isra.9+0x78/0x1f0 [ptlrpc] [22085.446637] [] nrs_policy_start_locked+0x624/0x670 [ptlrpc] [22085.448168] [] nrs_policy_ctl+0x12b/0x2b0 [ptlrpc] [22085.448924] [] ptlrpc_nrs_policy_control+0x10f/0x2d0 [ptlrpc] [22085.450396] [] ptlrpc_lprocfs_nrs_seq_write+0x3a5/0x560 [ptlrpc] [22085.451662] [] proc_reg_write+0x3d/0x80 [22085.476864] [] vfs_write+0xbd/0x1e0 [22085.478416] [] ? putname+0x3d/0x60 [22085.479075] [] SyS_write+0x84/0xf0 [22085.479723] [] system_call_fastpath+0x16/0x1b [22086.494380] Lustre: DEBUG MARKER: == sanityn test 77c: check ORR NRS policy ============================================================ 09:02:36 (1501333356) [22098.500372] BUG: sleeping function called from invalid context at /home/green/git/lustre-release/libcfs/libcfs/hash.c:1120 [22098.502643] in_atomic(): 1, irqs_disabled(): 0, pid: 10516, name: lctl [22098.503369] 3 locks held by lctl/10516: [22098.503982] #0: (sb_writers#3){......}, at: [] vfs_write+0x1ab/0x1e0 [22098.505533] #1: (&nrs_core.nrs_mutex){......}, at: [] ptlrpc_lprocfs_nrs_seq_write+0x38b/0x560 [ptlrpc] [22098.507153] #2: (&(&nrs->nrs_lock)->rlock){......}, at: [] nrs_policy_ctl+0x3a/0x2b0 [ptlrpc] [22098.512530] CPU: 7 PID: 10516 Comm: lctl Tainted: P W OE ------------ 3.10.0-debug #2 [22098.513513] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 [22098.513979] ffff8800806f2140 000000000396889c ffff88009c13bd18 ffffffff816fd3e4 [22098.515055] ffff88009c13bd30 ffffffff810b00e9 0000000000000018 ffff88009c13bd80 [22098.516160] ffffffffa02000a3 ffffffffa0204b66 ffffc90007882000 0000000000001000 [22098.517472] Call Trace: [22098.518088] [] dump_stack+0x19/0x1b [22098.518561] [] __might_sleep+0xe9/0x110 [22098.519407] [] cfs_hash_putref+0x233/0x500 [libcfs] [22098.520064] [] ? cfs_binheap_destroy+0x396/0x830 [libcfs] [22098.520683] [] nrs_orr_stop+0x69/0x270 [ptlrpc] [22098.521388] [] nrs_policy_stop0+0x39/0x1c0 [ptlrpc] [22098.522069] [] nrs_policy_stop_primary.isra.9+0x78/0x1f0 [ptlrpc] [22098.523021] [] nrs_policy_start_locked+0x624/0x670 [ptlrpc] [22098.524248] [] nrs_policy_ctl+0x12b/0x2b0 [ptlrpc] [22098.524955] [] ptlrpc_nrs_policy_control+0x10f/0x2d0 [ptlrpc] [22098.525962] [] ptlrpc_lprocfs_nrs_seq_write+0x3a5/0x560 [ptlrpc] [22098.540950] [] proc_reg_write+0x3d/0x80 [22098.541677] [] vfs_write+0xbd/0x1e0 [22098.563217] [] ? putname+0x3d/0x60 [22098.563715] [] SyS_write+0x84/0xf0 [22098.564225] [] system_call_fastpath+0x16/0x1b [22098.564889] BUG: scheduling while atomic: lctl/10516/0x10000002 [22098.565713] 3 locks held by lctl/10516: [22098.566367] #0: (sb_writers#3){......}, at: [] vfs_write+0x1ab/0x1e0 [22098.567376] #1: (&nrs_core.nrs_mutex){......}, at: [] ptlrpc_lprocfs_nrs_seq_write+0x38b/0x560 [ptlrpc] [22098.568656] #2: (&(&nrs->nrs_lock)->rlock){......}, at: [] nrs_policy_ctl+0x3a/0x2b0 [ptlrpc] [22098.570482] Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) lov(OE) osc(OE) mdc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) brd ext4 mbcache loop zfs(PO) zunicode(PO) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) zlib_deflate jbd2 syscopyarea sysfillrect sysimgblt ttm ata_generic drm_kms_helper pata_acpi drm i2c_piix4 ata_piix serio_raw i2c_core virtio_balloon libata virtio_console virtio_blk pcspkr floppy nfsd ip_tables rpcsec_gss_krb5 [last unloaded: libcfs] [22098.576645] CPU: 7 PID: 10516 Comm: lctl Tainted: P W OE ------------ 3.10.0-debug #2 [22098.577954] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 [22098.578649] ffff8800806f2140 000000000396889c ffff88009c13bc90 ffffffff816fd3e4 [22098.609723] ffff88009c13bca8 ffffffff816f9218 ffff88009c13bfd8 ffff88009c13bd08 [22098.621068] ffffffff817041a7 ffff8800806f2140 ffff88009c13bfd8 ffff88009c13bfd8 [22098.622066] Call Trace: [22098.622597] [] dump_stack+0x19/0x1b [22098.626701] [] __schedule_bug+0x59/0x68 [22098.627445] [] __schedule+0x807/0x950 [22098.628195] [] __cond_resched+0x26/0x30 [22098.628947] [] _cond_resched+0x3a/0x50 [22098.629622] [] cfs_hash_putref+0x238/0x500 [libcfs] [22098.630912] [] ? cfs_binheap_destroy+0x396/0x830 [libcfs] [22098.631705] [] nrs_orr_stop+0x69/0x270 [ptlrpc] [22098.632445] [] nrs_policy_stop0+0x39/0x1c0 [ptlrpc] [22098.633175] [] nrs_policy_stop_primary.isra.9+0x78/0x1f0 [ptlrpc] [22098.635590] [] nrs_policy_start_locked+0x624/0x670 [ptlrpc] [22098.636861] [] nrs_policy_ctl+0x12b/0x2b0 [ptlrpc] [22098.637590] [] ptlrpc_nrs_policy_control+0x10f/0x2d0 [ptlrpc] [22098.638864] [] ptlrpc_lprocfs_nrs_seq_write+0x3a5/0x560 [ptlrpc] [22098.657741] [] proc_reg_write+0x3d/0x80 [22098.666612] [] vfs_write+0xbd/0x1e0 [22098.667353] [] ? putname+0x3d/0x60 [22098.668022] [] SyS_write+0x84/0xf0 [22098.668697] [] system_call_fastpath+0x16/0x1b [22098.692356] BUG: scheduling while atomic: lctl/10516/0x10000002 [22098.693196] 3 locks held by lctl/10516: [22098.693834] #0: (sb_writers#3){......}, at: [] vfs_write+0x1ab/0x1e0 [22098.695176] #1: (&nrs_core.nrs_mutex){......}, at: [] ptlrpc_lprocfs_nrs_seq_write+0x38b/0x560 [ptlrpc] [22098.698634] #2: (&(&nrs->nrs_lock)->rlock){......}, at: [] nrs_policy_ctl+0x3a/0x2b0 [ptlrpc] [22098.700009] Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) lov(OE) osc(OE) mdc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) brd ext4 mbcache loop zfs(PO) zunicode(PO) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) zlib_deflate jbd2 syscopyarea sysfillrect sysimgblt ttm ata_generic drm_kms_helper pata_acpi drm i2c_piix4 ata_piix serio_raw i2c_core virtio_balloon libata virtio_console virtio_blk pcspkr floppy nfsd ip_tables rpcsec_gss_krb5 [last unloaded: libcfs] [22098.709485] CPU: 7 PID: 10516 Comm: lctl Tainted: P W OE ------------ 3.10.0-debug #2 [22098.712041] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 [22098.712549] ffff8800806f2140 000000000396889c ffff88009c13bc90 ffffffff816fd3e4 [22098.713626] ffff88009c13bca8 ffffffff816f9218 ffff88009c13bfd8 ffff88009c13bd08 [22098.714685] ffffffff817041a7 ffff8800806f2140 ffff88009c13bfd8 ffff88009c13bfd8 [22098.715722] Call Trace: [22098.716312] [] dump_stack+0x19/0x1b [22098.716909] [] __schedule_bug+0x59/0x68 [22098.717487] [] __schedule+0x807/0x950 [22098.718091] [] __cond_resched+0x26/0x30 [22098.718607] [] _cond_resched+0x3a/0x50 [22098.719291] [] cfs_hash_putref+0x238/0x500 [libcfs] [22098.719884] [] ? cfs_binheap_destroy+0x396/0x830 [libcfs] [22098.720537] [] nrs_orr_stop+0x69/0x270 [ptlrpc] [22098.737429] [] nrs_policy_stop0+0x39/0x1c0 [ptlrpc] [22098.737997] [] nrs_policy_stop_primary.isra.9+0x78/0x1f0 [ptlrpc] [22098.738937] [] nrs_policy_start_locked+0x624/0x670 [ptlrpc] [22098.740169] [] nrs_policy_ctl+0x12b/0x2b0 [ptlrpc] [22098.740930] [] ptlrpc_nrs_policy_control+0x10f/0x2d0 [ptlrpc] [22098.741850] [] ptlrpc_lprocfs_nrs_seq_write+0x3a5/0x560 [ptlrpc] [22098.742789] [] proc_reg_write+0x3d/0x80 [22098.743820] [] vfs_write+0xbd/0x1e0 [22098.744372] [] ? putname+0x3d/0x60 [22098.744910] [] SyS_write+0x84/0xf0 [22098.745482] [] system_call_fastpath+0x16/0x1b [22101.664375] BUG: sleeping function called from invalid context at kernel/cpu.c:87 [22101.671449] in_atomic(): 1, irqs_disabled(): 0, pid: 10516, name: lctl [22101.672531] 3 locks held by lctl/10516: [22101.673287] #0: (sb_writers#3){......}, at: [] vfs_write+0x1ab/0x1e0 [22101.674772] #1: (&nrs_core.nrs_mutex){......}, at: [] ptlrpc_lprocfs_nrs_seq_write+0x38b/0x560 [ptlrpc] [22101.676364] #2: (&(&nrs->nrs_lock)->rlock){......}, at: [] nrs_policy_ctl+0x3a/0x2b0 [ptlrpc] [22101.677989] CPU: 7 PID: 10516 Comm: lctl Tainted: P W OE ------------ 3.10.0-debug #2 [22101.679308] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 [22101.680038] ffff8800806f2140 000000000396889c ffff88009c13bd30 ffffffff816fd3e4 [22101.681685] ffff88009c13bd48 ffffffff810b00e9 ffff8802ff68d480 ffff88009c13bd60 [22101.683114] ffffffff81079fc0 0000000000000000 ffff88009c13bd80 ffffffff81196509 [22101.684615] Call Trace: [22101.685334] [] dump_stack+0x19/0x1b [22101.686030] [] __might_sleep+0xe9/0x110 [22101.686858] [] get_online_cpus+0x20/0x80 [22101.687743] [] kmem_cache_destroy+0x19/0xf0 [22101.689915] [] nrs_orr_stop+0x72/0x270 [ptlrpc] [22101.690707] [] nrs_policy_stop0+0x39/0x1c0 [ptlrpc] [22101.691687] [] nrs_policy_stop_primary.isra.9+0x78/0x1f0 [ptlrpc] [22101.692776] [] nrs_policy_start_locked+0x624/0x670 [ptlrpc] [22101.693887] [] nrs_policy_ctl+0x12b/0x2b0 [ptlrpc] [22101.694584] [] ptlrpc_nrs_policy_control+0x10f/0x2d0 [ptlrpc] [22101.695829] [] ptlrpc_lprocfs_nrs_seq_write+0x3a5/0x560 [ptlrpc] [22101.696927] [] proc_reg_write+0x3d/0x80 [22101.697577] [] vfs_write+0xbd/0x1e0 [22101.698431] [] ? putname+0x3d/0x60 [22101.699123] [] SyS_write+0x84/0xf0 [22101.699890] [] system_call_fastpath+0x16/0x1b [22101.700689] BUG: scheduling while atomic: lctl/10516/0x10000002 [22101.701538] 3 locks held by lctl/10516: [22101.702120] #0: (sb_writers#3){......}, at: [] vfs_write+0x1ab/0x1e0 [22101.703038] #1: (&nrs_core.nrs_mutex){......}, at: [] ptlrpc_lprocfs_nrs_seq_write+0x38b/0x560 [ptlrpc] [22101.704859] #2: (&(&nrs->nrs_lock)->rlock){......}, at: [] nrs_policy_ctl+0x3a/0x2b0 [ptlrpc] [22101.706006] Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) lov(OE) osc(OE) mdc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) brd ext4 mbcache loop zfs(PO) zunicode(PO) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) zlib_deflate jbd2 syscopyarea sysfillrect sysimgblt ttm ata_generic drm_kms_helper pata_acpi drm i2c_piix4 ata_piix serio_raw i2c_core virtio_balloon libata virtio_console virtio_blk pcspkr floppy nfsd ip_tables rpcsec_gss_krb5 [last unloaded: libcfs] [22101.710406] CPU: 7 PID: 10516 Comm: lctl Tainted: P W OE ------------ 3.10.0-debug #2 [22101.712061] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 [22101.712712] ffff8800806f2140 000000000396889c ffff88009c13bca8 ffffffff816fd3e4 [22101.713973] ffff88009c13bcc0 ffffffff816f9218 ffff88009c13bfd8 ffff88009c13bd20 [22101.715981] ffffffff817041a7 ffff8800806f2140 ffff88009c13bfd8 ffff88009c13bfd8 [22101.717056] Call Trace: [22101.717844] [] dump_stack+0x19/0x1b [22101.718746] [] __schedule_bug+0x59/0x68 [22101.719506] [] __schedule+0x807/0x950 [22101.720246] [] __cond_resched+0x26/0x30 [22101.720901] [] _cond_resched+0x3a/0x50 [22101.721556] [] get_online_cpus+0x25/0x80 [22101.723405] [] kmem_cache_destroy+0x19/0xf0 [22101.724158] [] nrs_orr_stop+0x72/0x270 [ptlrpc] [22101.724856] [] nrs_policy_stop0+0x39/0x1c0 [ptlrpc] [22101.725577] [] nrs_policy_stop_primary.isra.9+0x78/0x1f0 [ptlrpc] [22101.726831] [] nrs_policy_start_locked+0x624/0x670 [ptlrpc] [22101.728108] [] nrs_policy_ctl+0x12b/0x2b0 [ptlrpc] [22101.728810] [] ptlrpc_nrs_policy_control+0x10f/0x2d0 [ptlrpc] [22101.730108] [] ptlrpc_lprocfs_nrs_seq_write+0x3a5/0x560 [ptlrpc] [22101.731424] [] proc_reg_write+0x3d/0x80 [22101.732090] [] vfs_write+0xbd/0x1e0 [22101.732750] [] ? putname+0x3d/0x60 [22101.733428] [] SyS_write+0x84/0xf0 [22101.734087] [] system_call_fastpath+0x16/0x1b [22101.754061] BUG: scheduling while atomic: lctl/10516/0x10000002 [22101.755444] 4 locks held by lctl/10516: [22101.756076] #0: (sb_writers#3){......}, at: [] vfs_write+0x1ab/0x1e0 [22101.757374] #1: (&nrs_core.nrs_mutex){......}, at: [] ptlrpc_lprocfs_nrs_seq_write+0x38b/0x560 [ptlrpc] [22101.758910] #2: (&(&nrs->nrs_lock)->rlock){......}, at: [] nrs_policy_ctl+0x3a/0x2b0 [ptlrpc] [22101.760366] #3: (cpu_hotplug.lock){......}, at: [] get_online_cpus+0x37/0x80 [22101.761834] Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) lov(OE) osc(OE) mdc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) brd ext4 mbcache loop zfs(PO) zunicode(PO) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) zlib_deflate jbd2 syscopyarea sysfillrect sysimgblt ttm ata_generic drm_kms_helper pata_acpi drm i2c_piix4 ata_piix serio_raw i2c_core virtio_balloon libata virtio_console virtio_blk pcspkr floppy nfsd ip_tables rpcsec_gss_krb5 [last unloaded: libcfs] [22101.804771] CPU: 7 PID: 10516 Comm: lctl Tainted: P W OE ------------ 3.10.0-debug #2 [22101.806078] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 [22101.806918] ffff8800806f2140 000000000396889c ffff88009c13bc28 ffffffff816fd3e4 [22101.808247] ffff88009c13bc40 ffffffff816f9218 ffff88009c13bfd8 ffff88009c13bca0 [22101.809580] ffffffff817041a7 ffff8800806f2140 ffff88009c13bfd8 ffff88009c13bfd8 [22101.811773] Call Trace: [22101.812543] [] dump_stack+0x19/0x1b [22101.813890] [] __schedule_bug+0x59/0x68 [22101.814562] [] __schedule+0x807/0x950 [22101.815239] [] __cond_resched+0x26/0x30 [22101.815912] [] _cond_resched+0x3a/0x50 [22101.816759] [] mutex_lock_nested+0x5a/0x3b0 [22101.817509] [] ? kfree_debugcheck+0x12/0x30 [22101.819529] [] put_online_cpus+0x29/0xa0 [22101.820231] [] kmem_cache_destroy+0x81/0xf0 [22101.821015] [] nrs_orr_stop+0x72/0x270 [ptlrpc] [22101.821739] [] nrs_policy_stop0+0x39/0x1c0 [ptlrpc] [22101.822480] [] nrs_policy_stop_primary.isra.9+0x78/0x1f0 [ptlrpc] [22101.823786] [] nrs_policy_start_locked+0x624/0x670 [ptlrpc] [22101.825075] [] nrs_policy_ctl+0x12b/0x2b0 [ptlrpc] [22101.825798] [] ptlrpc_nrs_policy_control+0x10f/0x2d0 [ptlrpc] [22101.827102] [] ptlrpc_lprocfs_nrs_seq_write+0x3a5/0x560 [ptlrpc] [22101.832387] [] proc_reg_write+0x3d/0x80 [22101.833065] [] vfs_write+0xbd/0x1e0 [22101.833737] [] ? putname+0x3d/0x60 [22101.834408] [] SyS_write+0x84/0xf0 [22101.835065] [] system_call_fastpath+0x16/0x1b [22106.466529] BUG: sleeping function called from invalid context at kernel/cpu.c:87 [22106.468169] in_atomic(): 1, irqs_disabled(): 0, pid: 10516, name: lctl [22106.469179] 3 locks held by lctl/10516: [22106.469693] #0: (sb_writers#3){......}, at: [] vfs_write+0x1ab/0x1e0 [22106.470711] #1: (&nrs_core.nrs_mutex){......}, at: [] ptlrpc_lprocfs_nrs_seq_write+0x38b/0x560 [ptlrpc] [22106.471859] #2: (&(&nrs->nrs_lock)->rlock){......}, at: [] nrs_policy_ctl+0x3a/0x2b0 [ptlrpc] [22106.473182] CPU: 7 PID: 10516 Comm: lctl Tainted: P W OE ------------ 3.10.0-debug #2 [22106.474447] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 [22106.475120] ffff8800806f2140 000000000396889c ffff88009c13bd30 ffffffff816fd3e4 [22106.476459] ffff88009c13bd48 ffffffff810b00e9 ffff88008b6014c0 ffff88009c13bd60 [22106.477727] ffffffff81079fc0 0000000000000000 ffff88009c13bd80 ffffffff81196509 [22106.479016] Call Trace: [22106.479612] [] dump_stack+0x19/0x1b [22106.480273] [] __might_sleep+0xe9/0x110 [22106.480943] [] get_online_cpus+0x20/0x80 [22106.497896] [] kmem_cache_destroy+0x19/0xf0 [22106.498653] [] nrs_orr_stop+0x72/0x270 [ptlrpc] [22106.499327] [] nrs_policy_stop0+0x39/0x1c0 [ptlrpc] [22106.499866] [] nrs_policy_stop_primary.isra.9+0x78/0x1f0 [ptlrpc] [22106.501042] [] nrs_policy_start_locked+0x624/0x670 [ptlrpc] [22106.502117] [] nrs_policy_ctl+0x12b/0x2b0 [ptlrpc] [22106.502796] [] ptlrpc_nrs_policy_control+0xbd/0x2d0 [ptlrpc] [22106.504144] [] ptlrpc_lprocfs_nrs_seq_write+0x3a5/0x560 [ptlrpc] [22106.505134] [] proc_reg_write+0x3d/0x80 [22106.505719] [] vfs_write+0xbd/0x1e0 [22106.506240] [] ? putname+0x3d/0x60 [22106.506722] [] SyS_write+0x84/0xf0 [22106.507258] [] system_call_fastpath+0x16/0x1b [22106.507783] BUG: scheduling while atomic: lctl/10516/0x10000002 [22106.508375] 3 locks held by lctl/10516: [22106.509171] #0: (sb_writers#3){......}, at: [] vfs_write+0x1ab/0x1e0 [22106.510117] #1: (&nrs_core.nrs_mutex){......}, at: [] ptlrpc_lprocfs_nrs_seq_write+0x38b/0x560 [ptlrpc] [22106.511345] #2: (&(&nrs->nrs_lock)->rlock){......}, at: [] nrs_policy_ctl+0x3a/0x2b0 [ptlrpc] [22106.512792] Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) lov(OE) osc(OE) mdc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) brd ext4 mbcache loop zfs(PO) zunicode(PO) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) zlib_deflate jbd2 syscopyarea sysfillrect sysimgblt ttm ata_generic drm_kms_helper pata_acpi drm i2c_piix4 ata_piix serio_raw i2c_core virtio_balloon libata virtio_console virtio_blk pcspkr floppy nfsd ip_tables rpcsec_gss_krb5 [last unloaded: libcfs] [22106.519166] CPU: 7 PID: 10516 Comm: lctl Tainted: P W OE ------------ 3.10.0-debug #2 [22106.520180] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 [22106.521299] ffff8800806f2140 000000000396889c ffff88009c13bca8 ffffffff816fd3e4 [22106.522241] ffff88009c13bcc0 ffffffff816f9218 ffff88009c13bfd8 ffff88009c13bd20 [22106.523352] ffffffff817041a7 ffff8800806f2140 ffff88009c13bfd8 ffff88009c13bfd8 [22106.524451] Call Trace: [22106.525377] [] dump_stack+0x19/0x1b [22106.526204] [] __schedule_bug+0x59/0x68 [22106.527053] [] __schedule+0x807/0x950 [22106.527762] [] __cond_resched+0x26/0x30 [22106.529048] [] _cond_resched+0x3a/0x50 [22106.530056] [] get_online_cpus+0x25/0x80 [22106.530921] [] kmem_cache_destroy+0x19/0xf0 [22106.531818] [] nrs_orr_stop+0x72/0x270 [ptlrpc] [22106.532686] [] nrs_policy_stop0+0x39/0x1c0 [ptlrpc] [22106.533551] [] nrs_policy_stop_primary.isra.9+0x78/0x1f0 [ptlrpc] [22106.534987] [] nrs_policy_start_locked+0x624/0x670 [ptlrpc] [22106.536409] [] nrs_policy_ctl+0x12b/0x2b0 [ptlrpc] [22106.537582] [] ptlrpc_nrs_policy_control+0xbd/0x2d0 [ptlrpc] [22106.539022] [] ptlrpc_lprocfs_nrs_seq_write+0x3a5/0x560 [ptlrpc] [22106.540403] [] proc_reg_write+0x3d/0x80 [22106.541200] [] vfs_write+0xbd/0x1e0 [22106.541881] [] ? putname+0x3d/0x60 [22106.542744] [] SyS_write+0x84/0xf0 [22106.543572] [] system_call_fastpath+0x16/0x1b [22106.544545] BUG: scheduling while atomic: lctl/10516/0x00000002 [22106.545250] 5 locks held by lctl/10516: [22106.545938] #0: (sb_writers#3){......}, at: [] vfs_write+0x1ab/0x1e0 [22106.550917] #1: (&nrs_core.nrs_mutex){......}, at: [] ptlrpc_lprocfs_nrs_seq_write+0x38b/0x560 [ptlrpc] [22106.552458] #2: (&(&nrs->nrs_lock)->rlock){......}, at: [] nrs_policy_ctl+0x3a/0x2b0 [ptlrpc] [22106.553979] #3: (cpu_hotplug.lock){......}, at: [] get_online_cpus+0x37/0x80 [22106.555301] #4: (slab_mutex){......}, at: [] kmem_cache_destroy+0x27/0xf0 [22106.556618] Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) lov(OE) osc(OE) mdc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) brd ext4 mbcache loop zfs(PO) zunicode(PO) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) zlib_deflate jbd2 syscopyarea sysfillrect sysimgblt ttm ata_generic drm_kms_helper pata_acpi drm i2c_piix4 ata_piix serio_raw i2c_core virtio_balloon libata virtio_console virtio_blk pcspkr floppy nfsd ip_tables rpcsec_gss_krb5 [last unloaded: libcfs] [22106.562895] CPU: 13 PID: 10516 Comm: lctl Tainted: P W OE ------------ 3.10.0-debug #2 [22106.571958] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 [22106.572821] ffff8800806f2140 000000000396889c ffff88009c13bc50 ffffffff816fd3e4 [22106.574523] ffff88009c13bc68 ffffffff816f9218 ffff88009c13bfd8 ffff88009c13bcc8 [22106.575505] ffffffff817041a7 ffff8800806f2140 ffff88009c13bfd8 ffff88009c13bfd8 [22106.577215] Call Trace: [22106.578904] [] dump_stack+0x19/0x1b [22106.579640] [] __schedule_bug+0x59/0x68 [22106.581772] [] __schedule+0x807/0x950 [22106.582524] [] schedule_preempt_disabled+0x39/0x90 [22106.583209] [] mutex_lock_nested+0x17e/0x3b0 [22106.583944] [] ? kmem_cache_destroy+0x27/0xf0 [22106.584723] [] kmem_cache_destroy+0x27/0xf0 [22106.585615] [] nrs_orr_stop+0x72/0x270 [ptlrpc] [22106.586411] [] nrs_policy_stop0+0x39/0x1c0 [ptlrpc] [22106.587141] [] nrs_policy_stop_primary.isra.9+0x78/0x1f0 [ptlrpc] [22106.589472] [] nrs_policy_start_locked+0x624/0x670 [ptlrpc] [22106.590827] [] nrs_policy_ctl+0x12b/0x2b0 [ptlrpc] [22106.591631] [] ptlrpc_nrs_policy_control+0xbd/0x2d0 [ptlrpc] [22106.592974] [] ptlrpc_lprocfs_nrs_seq_write+0x3a5/0x560 [ptlrpc] [22106.594253] [] proc_reg_write+0x3d/0x80 [22106.594939] [] vfs_write+0xbd/0x1e0 [22106.595601] [] ? putname+0x3d/0x60 [22106.603299] [] SyS_write+0x84/0xf0 [22106.604406] [] system_call_fastpath+0x16/0x1b [22106.605372] BUG: scheduling while atomic: lctl/10516/0x00000002 [22106.606128] 5 locks held by lctl/10516: [22106.625201] #0: (sb_writers#3){......}, at: [] vfs_write+0x1ab/0x1e0 [22106.647426] #1: (&nrs_core.nrs_mutex){......}, at: [] ptlrpc_lprocfs_nrs_seq_write+0x38b/0x560 [ptlrpc] [22106.649045] #2: (&(&nrs->nrs_lock)->rlock){......}, at: [] nrs_policy_ctl+0x3a/0x2b0 [ptlrpc] [22106.655996] #3: (cpu_hotplug.lock){......}, at: [] get_online_cpus+0x37/0x80 [22106.656985] #4: (slab_mutex){......}, at: [] kmem_cache_destroy+0x27/0xf0 [22106.658068] Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) lov(OE) osc(OE) mdc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) brd ext4 mbcache loop zfs(PO) zunicode(PO) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) zlib_deflate jbd2 syscopyarea sysfillrect sysimgblt ttm ata_generic drm_kms_helper pata_acpi drm i2c_piix4 ata_piix serio_raw i2c_core virtio_balloon libata virtio_console virtio_blk pcspkr floppy nfsd ip_tables rpcsec_gss_krb5 [last unloaded: libcfs] [22106.672917] CPU: 4 PID: 10516 Comm: lctl Tainted: P W OE ------------ 3.10.0-debug #2 [22106.675673] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 [22106.676451] ffff8800806f2140 000000000396889c ffff88009c13bc50 ffffffff816fd3e4 [22106.677923] ffff88009c13bc68 ffffffff816f9218 ffff88009c13bfd8 ffff88009c13bcc8 [22106.679436] ffffffff817041a7 ffff8800806f2140 ffff88009c13bfd8 ffff88009c13bfd8 [22106.680925] Call Trace: [22106.681623] [] dump_stack+0x19/0x1b [22106.682423] [] __schedule_bug+0x59/0x68 [22106.683993] [] __schedule+0x807/0x950 [22106.684757] [] schedule_preempt_disabled+0x39/0x90 [22106.685612] [] mutex_lock_nested+0x17e/0x3b0 [22106.686412] [] ? kmem_cache_destroy+0x27/0xf0 [22106.687183] [] kmem_cache_destroy+0x27/0xf0 [22106.688069] [] nrs_orr_stop+0x72/0x270 [ptlrpc] [22106.688909] [] nrs_policy_stop0+0x39/0x1c0 [ptlrpc] [22106.689781] [] nrs_policy_stop_primary.isra.9+0x78/0x1f0 [ptlrpc] [22106.691275] [] nrs_policy_start_locked+0x624/0x670 [ptlrpc] [22106.693931] [] nrs_policy_ctl+0x12b/0x2b0 [ptlrpc] [22106.694511] [] ptlrpc_nrs_policy_control+0xbd/0x2d0 [ptlrpc] [22106.695429] [] ptlrpc_lprocfs_nrs_seq_write+0x3a5/0x560 [ptlrpc] [22106.696324] [] proc_reg_write+0x3d/0x80 [22106.696776] [] vfs_write+0xbd/0x1e0 [22106.697240] [] ? putname+0x3d/0x60 [22106.697712] [] SyS_write+0x84/0xf0 [22106.698174] [] system_call_fastpath+0x16/0x1b [22106.698846] BUG: scheduling while atomic: lctl/10516/0x00000002 [22106.699732] 5 locks held by lctl/10516: [22106.700468] #0: (sb_writers#3){......}, at: [] vfs_write+0x1ab/0x1e0 [22106.701937] #1: (&nrs_core.nrs_mutex){......}, at: [] ptlrpc_lprocfs_nrs_seq_write+0x38b/0x560 [ptlrpc] [22106.703475] #2: (&(&nrs->nrs_lock)->rlock){......}, at: [] nrs_policy_ctl+0x3a/0x2b0 [ptlrpc] [22106.721163] #3: (cpu_hotplug.lock){......}, at: [] get_online_cpus+0x37/0x80 [22106.722612] #4: (slab_mutex){......}, at: [] kmem_cache_destroy+0x27/0xf0 [22106.723962] Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) lov(OE) osc(OE) mdc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) brd ext4 mbcache loop zfs(PO) zunicode(PO) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) zlib_deflate jbd2 syscopyarea sysfillrect sysimgblt ttm ata_generic drm_kms_helper pata_acpi drm i2c_piix4 ata_piix serio_raw i2c_core virtio_balloon libata virtio_console virtio_blk pcspkr floppy nfsd ip_tables rpcsec_gss_krb5 [last unloaded: libcfs] [22106.729813] CPU: 15 PID: 10516 Comm: lctl Tainted: P W OE ------------ 3.10.0-debug #2 [22106.742906] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 [22106.743507] ffff8800806f2140 000000000396889c ffff88009c13bc50 ffffffff816fd3e4 [22106.744799] ffff88009c13bc68 ffffffff816f9218 ffff88009c13bfd8 ffff88009c13bcc8 [22106.746086] ffffffff817041a7 ffff8800806f2140 ffff88009c13bfd8 ffff88009c13bfd8 [22106.747362] Call Trace: [22106.747977] [] dump_stack+0x19/0x1b [22106.748629] [] __schedule_bug+0x59/0x68 [22106.749303] [] __schedule+0x807/0x950 [22106.749978] [] schedule_preempt_disabled+0x39/0x90 [22106.750665] [] mutex_lock_nested+0x17e/0x3b0 [22106.751358] [] ? kmem_cache_destroy+0x27/0xf0 [22106.768923] [] kmem_cache_destroy+0x27/0xf0 [22106.769748] [] nrs_orr_stop+0x72/0x270 [ptlrpc] [22106.770568] [] nrs_policy_stop0+0x39/0x1c0 [ptlrpc] [22106.771417] [] nrs_policy_stop_primary.isra.9+0x78/0x1f0 [ptlrpc] [22106.772729] [] nrs_policy_start_locked+0x624/0x670 [ptlrpc] [22106.774052] [] nrs_policy_ctl+0x12b/0x2b0 [ptlrpc] [22106.774784] [] ptlrpc_nrs_policy_control+0xbd/0x2d0 [ptlrpc] [22106.781734] [] ptlrpc_lprocfs_nrs_seq_write+0x3a5/0x560 [ptlrpc] [22106.783742] [] proc_reg_write+0x3d/0x80 [22106.784223] [] vfs_write+0xbd/0x1e0 [22106.784710] [] ? putname+0x3d/0x60 [22106.785240] [] SyS_write+0x84/0xf0 [22106.785902] [] system_call_fastpath+0x16/0x1b [22106.791973] BUG: spinlock wrong CPU on CPU#15, lctl/10516 [22106.792701] lock: 0xffff8802fdbdfcc0, .magic: dead4ead, .owner: lctl/10516, .owner_cpu: 7 [22106.794035] CPU: 15 PID: 10516 Comm: lctl Tainted: P W OE ------------ 3.10.0-debug #2 [22106.795077] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 [22106.807924] ffff8800806f2140 000000000396889c ffff88009c13bd90 ffffffff816fd3e4 [22106.809497] ffff88009c13bdb0 ffffffff816fd472 ffff8802fdbdfcc0 ffffffff81aa1abe [22106.810766] ffff88009c13bdd0 ffffffff816fd498 ffff8802fdbdfcc0 ffff8802fdbdfcc0 [22106.812047] Call Trace: [22106.812652] [] dump_stack+0x19/0x1b [22106.813446] [] spin_dump+0x8c/0x91 [22106.814202] [] spin_bug+0x21/0x26 [22106.814855] [] do_raw_spin_unlock+0x6c/0x90 [22106.815538] [] _raw_spin_unlock+0x27/0x40 [22106.816403] [] nrs_policy_ctl+0x90/0x2b0 [ptlrpc] [22106.817138] [] ptlrpc_nrs_policy_control+0xbd/0x2d0 [ptlrpc] [22106.818537] [] ptlrpc_lprocfs_nrs_seq_write+0x3a5/0x560 [ptlrpc] [22106.819906] [] proc_reg_write+0x3d/0x80 [22106.820570] [] vfs_write+0xbd/0x1e0 [22106.827557] [] ? putname+0x3d/0x60 [22106.828139] [] SyS_write+0x84/0xf0 [22106.828661] [] system_call_fastpath+0x16/0x1b [22108.110582] Lustre: DEBUG MARKER: == sanityn test 77d: check TRR nrs policy ============================================================ 09:02:57 (1501333377) [22120.691292] BUG: sleeping function called from invalid context at /home/green/git/lustre-release/libcfs/libcfs/hash.c:1120 [22120.693434] in_atomic(): 1, irqs_disabled(): 0, pid: 12741, name: lctl [22120.694112] INFO: lockdep is turned off. [22120.694613] CPU: 9 PID: 12741 Comm: lctl Tainted: P W OE ------------ 3.10.0-debug #2 [22120.696642] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 [22120.697247] ffff880073434b80 00000000ca53a485 ffff88008057bd18 ffffffff816fd3e4 [22120.698349] ffff88008057bd30 ffffffff810b00e9 0000000000000004 ffff88008057bd80 [22120.699533] ffffffffa02000a3 ffffffffa0204b66 ffff88028a771f80 0000000000000004 [22120.700896] Call Trace: [22120.701816] [] dump_stack+0x19/0x1b [22120.702438] [] __might_sleep+0xe9/0x110 [22120.702985] [] cfs_hash_putref+0x233/0x500 [libcfs] [22120.703546] [] ? cfs_binheap_destroy+0x396/0x830 [libcfs] [22120.716682] [] nrs_orr_stop+0x69/0x270 [ptlrpc] [22120.717230] [] nrs_policy_stop0+0x39/0x1c0 [ptlrpc] [22120.722067] [] nrs_policy_stop_primary.isra.9+0x78/0x1f0 [ptlrpc] [22120.723684] [] nrs_policy_start_locked+0x624/0x670 [ptlrpc] [22120.725372] [] nrs_policy_ctl+0x12b/0x2b0 [ptlrpc] [22120.726276] [] ptlrpc_nrs_policy_control+0x10f/0x2d0 [ptlrpc] [22120.727896] [] ptlrpc_lprocfs_nrs_seq_write+0x3a5/0x560 [ptlrpc] [22120.729493] [] proc_reg_write+0x3d/0x80 [22120.730477] [] vfs_write+0xbd/0x1e0 [22120.731311] [] ? putname+0x3d/0x60 [22120.732147] [] SyS_write+0x84/0xf0 [22120.732987] [] system_call_fastpath+0x16/0x1b [22120.733831] BUG: scheduling while atomic: lctl/12741/0x10000002 [22120.734746] INFO: lockdep is turned off. [22120.735614] Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) lov(OE) osc(OE) mdc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) brd ext4 mbcache loop zfs(PO) zunicode(PO) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) zlib_deflate jbd2 syscopyarea sysfillrect sysimgblt ttm ata_generic drm_kms_helper pata_acpi drm i2c_piix4 ata_piix serio_raw i2c_core virtio_balloon libata virtio_console virtio_blk pcspkr floppy nfsd ip_tables rpcsec_gss_krb5 [last unloaded: libcfs] [22120.763431] CPU: 9 PID: 12741 Comm: lctl Tainted: P W OE ------------ 3.10.0-debug #2 [22120.769006] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 [22120.770687] ffff880073434b80 00000000ca53a485 ffff88008057bc90 ffffffff816fd3e4 [22120.774839] ffff88008057bca8 ffffffff816f9218 ffff88008057bfd8 ffff88008057bd08 [22120.777191] ffffffff817041a7 ffff880073434b80 ffff88008057bfd8 ffff88008057bfd8 [22120.780385] Call Trace: [22120.781840] [] dump_stack+0x19/0x1b [22120.783524] [] __schedule_bug+0x59/0x68 [22120.785190] [] __schedule+0x807/0x950 [22120.786814] [] __cond_resched+0x26/0x30 [22120.788015] [] _cond_resched+0x3a/0x50 [22120.790636] [] cfs_hash_putref+0x238/0x500 [libcfs] [22120.794345] [] ? cfs_binheap_destroy+0x396/0x830 [libcfs] [22120.795397] [] nrs_orr_stop+0x69/0x270 [ptlrpc] [22120.797393] [] nrs_policy_stop0+0x39/0x1c0 [ptlrpc] [22120.799836] [] nrs_policy_stop_primary.isra.9+0x78/0x1f0 [ptlrpc] [22120.801265] [] nrs_policy_start_locked+0x624/0x670 [ptlrpc] [22120.803191] [] nrs_policy_ctl+0x12b/0x2b0 [ptlrpc] [22120.803997] [] ptlrpc_nrs_policy_control+0x10f/0x2d0 [ptlrpc] [22120.828023] [] ptlrpc_lprocfs_nrs_seq_write+0x3a5/0x560 [ptlrpc] [22120.837729] [] proc_reg_write+0x3d/0x80 [22120.848944] [] vfs_write+0xbd/0x1e0 [22120.849644] [] ? putname+0x3d/0x60 [22120.850441] [] SyS_write+0x84/0xf0 [22120.851186] [] system_call_fastpath+0x16/0x1b [22120.851897] LustreError: 12741:0:(nrs_orr.c:483:nrs_trr_hop_exit()) ASSERTION( orro->oo_ref == 0 ) failed: Busy NRS TRR policy object for OST with index 1, with 1 refs [22120.856255] LustreError: 12741:0:(nrs_orr.c:483:nrs_trr_hop_exit()) LBUG [22120.856969] Pid: 12741, comm: lctl [22120.857575] Call Trace: [22120.858750] [] libcfs_call_trace+0x4e/0x60 [libcfs] [22120.859460] [] lbug_with_loc+0x4c/0xb0 [libcfs] [22120.860231] [] nrs_trr_hop_exit+0xfb/0x130 [ptlrpc] [22120.861629] [] cfs_hash_putref+0x1e2/0x500 [libcfs] [22120.862123] [] ? cfs_binheap_destroy+0x396/0x830 [libcfs] [22120.862821] [] nrs_orr_stop+0x69/0x270 [ptlrpc] [22120.863589] [] nrs_policy_stop0+0x39/0x1c0 [ptlrpc] [22120.864315] [] nrs_policy_stop_primary.isra.9+0x78/0x1f0 [ptlrpc] [22120.865614] [] nrs_policy_start_locked+0x624/0x670 [ptlrpc] [22120.866922] [] nrs_policy_ctl+0x12b/0x2b0 [ptlrpc] [22120.872716] [] ptlrpc_nrs_policy_control+0x10f/0x2d0 [ptlrpc] [22120.874257] [] ptlrpc_lprocfs_nrs_seq_write+0x3a5/0x560 [ptlrpc] [22120.875521] [] proc_reg_write+0x3d/0x80 [22120.876212] [] vfs_write+0xbd/0x1e0 [22120.900486] [] ? putname+0x3d/0x60 [22120.901959] [] SyS_write+0x84/0xf0 [22120.920647] [] system_call_fastpath+0x16/0x1b [22120.921334] [22120.921903] Kernel panic - not syncing: LBUG [22120.922521] CPU: 9 PID: 12741 Comm: lctl Tainted: P W OE ------------ 3.10.0-debug #2 [22120.923760] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 [22120.924463] ffffffffa020bed2 00000000ca53a485 ffff88008057bc70 ffffffff816fd3e4 [22120.925713] ffff88008057bcf0 ffffffff816f8c34 ffffffff00000008 ffff88008057bd00 [22120.927552] ffff88008057bca0 00000000ca53a485 00000000ca53a485 ffff88033e52d948 [22120.929142] Call Trace: [22120.929738] [] dump_stack+0x19/0x1b [22120.930405] [] panic+0xd8/0x1e7 [22120.931066] [] lbug_with_loc+0x64/0xb0 [libcfs] [22120.931861] [] nrs_trr_hop_exit+0xfb/0x130 [ptlrpc] [22120.932566] [] cfs_hash_putref+0x1e2/0x500 [libcfs] [22120.933266] [] ? cfs_binheap_destroy+0x396/0x830 [libcfs] [22120.934002] [] nrs_orr_stop+0x69/0x270 [ptlrpc] [22120.934708] [] nrs_policy_stop0+0x39/0x1c0 [ptlrpc] [22120.935433] [] nrs_policy_stop_primary.isra.9+0x78/0x1f0 [ptlrpc] [22120.936713] [] nrs_policy_start_locked+0x624/0x670 [ptlrpc] [22120.937994] [] nrs_policy_ctl+0x12b/0x2b0 [ptlrpc] [22120.938714] [] ptlrpc_nrs_policy_control+0x10f/0x2d0 [ptlrpc] [22120.939994] [] ptlrpc_lprocfs_nrs_seq_write+0x3a5/0x560 [ptlrpc] [22120.941251] [] proc_reg_write+0x3d/0x80 [22120.941916] [] vfs_write+0xbd/0x1e0 [22120.942587] [] ? putname+0x3d/0x60 [22120.943243] [] SyS_write+0x84/0xf0 [22120.943899] [] system_call_fastpath+0x16/0x1b