[LU-17097] RCU stall caused by osc_quota_cleanup

• Type: Bug
• Resolution: Fixed
• Priority: Minor
• Severity: 3
• Assignee: James A Simmons
• Reporter: Sergey Cheremencev

Description

sanity-quota@ldiskfs+DNE failed with Timeout(8009s)(Client: RCU stall).
https://testing.whamcloud.com/gerrit-janitor/34572/testresults/sanity-quota-ldiskfs-DNE-centos7_x86_64-centos7_x86_64/

Sep  6 06:54:52 oleg310-client kernel: Lustre: DEBUG MARKER: == sanity-quota test complete, duration 4313 sec ========= 06:54:51 (1693997691)
Sep  6 06:54:52 oleg310-client sshd[20583]: Received disconnect from 192.168.203.10 port 42586:11: disconnected by user
Sep  6 06:54:52 oleg310-client sshd[20583]: Disconnected from 192.168.203.10 port 42586
Sep  6 06:54:52 oleg310-client sshd[20583]: pam_unix(sshd:session): session closed for user root
Sep  6 06:54:52 oleg310-client systemd-logind: Removed session 600.
Sep  6 06:55:07 oleg310-client kernel: ------------[ cut here ]------------
Sep  6 06:55:07 oleg310-client kernel: WARNING: CPU: 2 PID: 20679 at lib/debugobjects.c:286 debug_print_object+0x83/0xa0
Sep  6 06:55:07 oleg310-client kernel: ODEBUG: activate active (active state 1) object type: rcu_head hint:           (null)
Sep  6 06:55:07 oleg310-client kernel: Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) crc32_generic libcfs(OE) crc_t10dif crct10dif_generic crct10dif_common rpcsec_gss_krb5 squashfs i2c_piix4 i2c_core pcspkr binfmt_misc ip_tables ext4 mbcache jbd2 ata_generic pata_acpi ata_piix serio_raw libata
Sep  6 06:55:07 oleg310-client kernel: CPU: 2 PID: 20679 Comm: umount Kdump: loaded Tainted: G           OE  ------------   3.10.0-7.9-debug #1
Sep  6 06:55:07 oleg310-client kernel: Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
Sep  6 06:55:07 oleg310-client kernel: Call Trace:
Sep  6 06:55:07 oleg310-client kernel: [<ffffffff817ded29>] dump_stack+0x19/0x1b
Sep  6 06:55:07 oleg310-client kernel: [<ffffffff8108d558>] __warn+0xd8/0x100
Sep  6 06:55:07 oleg310-client kernel: [<ffffffff8108d5df>] warn_slowpath_fmt+0x5f/0x80
Sep  6 06:55:07 oleg310-client kernel: [<ffffffff81414723>] debug_print_object+0x83/0xa0
Sep  6 06:55:07 oleg310-client kernel: [<ffffffff814150af>] debug_object_activate+0x1af/0x210
Sep  6 06:55:07 oleg310-client kernel: [<ffffffff8114dc8f>] __call_rcu+0x3f/0x2d0
Sep  6 06:55:07 oleg310-client kernel: [<ffffffff8114df3d>] call_rcu_sched+0x1d/0x20
Sep  6 06:55:07 oleg310-client kernel: [<ffffffffa0179f44>] xas_free_nodes+0xa4/0xf0 [libcfs]
Sep  6 06:55:07 oleg310-client kernel: [<ffffffffa017b26f>] xa_destroy+0xdf/0xf0 [libcfs]
Sep  6 06:55:07 oleg310-client kernel: [<ffffffffa088d4d5>] osc_quota_cleanup+0x15/0x20 [osc]
Sep  6 06:55:07 oleg310-client kernel: [<ffffffffa086ef1f>] osc_cleanup_common+0xbf/0x1b0 [osc]
Sep  6 06:55:07 oleg310-client kernel: [<ffffffffa02f48f9>] class_free_dev+0x219/0x730 [obdclass]
Sep  6 06:55:07 oleg310-client kernel: [<ffffffffa02f4ff0>] class_export_put+0x1e0/0x2e0 [obdclass]
Sep  6 06:55:07 oleg310-client kernel: [<ffffffffa02f6c15>] class_unlink_export+0x125/0x160 [obdclass]
Sep  6 06:55:07 oleg310-client kernel: [<ffffffffa030ca30>] class_decref+0x80/0x160 [obdclass]
Sep  6 06:55:07 oleg310-client kernel: [<ffffffffa030ce71>] class_detach+0x1c1/0x310 [obdclass]
Sep  6 06:55:07 oleg310-client kernel: [<ffffffffa0314b2b>] class_process_config+0x163b/0x27c0 [obdclass]
Sep  6 06:55:07 oleg310-client kernel: [<ffffffffa0315e90>] class_manual_cleanup+0x1e0/0x770 [obdclass]
Sep  6 06:55:07 oleg310-client kernel: [<ffffffffa0903955>] lov_tgts_putref+0x385/0xad0 [lov]
Sep  6 06:55:07 oleg310-client kernel: [<ffffffffa0908927>] lov_disconnect+0x237/0x280 [lov]
Sep  6 06:55:07 oleg310-client kernel: [<ffffffffa0e8fa96>] obd_disconnect+0x56/0x300 [lustre]
Sep  6 06:55:07 oleg310-client kernel: [<ffffffffa0e98ebc>] ll_put_super+0x81c/0xf30 [lustre]
Sep  6 06:55:07 oleg310-client kernel: [<ffffffff812476ca>] generic_shutdown_super+0x6a/0xf0
Sep  6 06:55:07 oleg310-client kernel: [<ffffffff81247ac2>] kill_anon_super+0x12/0x20 Sep  6 06:55:07 oleg310-client kernel: [<ffffffffa0ec711b>] lustre_kill_super+0x2b/0x30 [lustre]
Sep  6 06:55:07 oleg310-client kernel: [<ffffffff81247ec9>] deactivate_locked_super+0x49/0x60
Sep  6 06:55:07 oleg310-client kernel: [<ffffffff81248616>] deactivate_super+0x46/0x60
Sep  6 06:55:07 oleg310-client kernel: [<ffffffff81268b1f>] cleanup_mnt+0x3f/0x80
Sep  6 06:55:07 oleg310-client kernel: [<ffffffff81268bb2>] __cleanup_mnt+0x12/0x20
Sep  6 06:55:07 oleg310-client kernel: [<ffffffff810b69b5>] task_work_run+0xb5/0xf0
Sep  6 06:55:07 oleg310-client kernel: [<ffffffff8102ccb2>] do_notify_resume+0x92/0xb0
Sep  6 06:55:07 oleg310-client kernel: [<ffffffff817f4363>] int_signal+0x12/0x17
Sep  6 06:55:07 oleg310-client kernel: ---[ end trace 37e266df8306097f ]---
Sep  6 06:55:07 oleg310-client kernel: ------------[ cut here ]------------
Sep  6 06:55:07 oleg310-client kernel: WARNING: CPU: 2 PID: 20679 at kernel/rcupdate.c:311 rcuhead_fixup_activate+0x5a/0x70
Sep  6 06:55:07 oleg310-client kernel: Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) crc32_generic libcfs(OE) crc_t10dif crct10dif_generic crct10dif_common rpcsec_gss_krb5 squashfs i2c_piix4 i2c_core pcspkr binfmt_misc ip_tables ext4 mbcache jbd2 ata_generic pata_acpi ata_piix serio_raw libata
Sep  6 06:55:07 oleg310-client kernel: CPU: 2 PID: 20679 Comm: umount Kdump: loaded Tainted: G        W  OE  ------------   3.10.0-7.9-debug #1
Sep  6 06:55:07 oleg310-client kernel: Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
Sep  6 06:55:07 oleg310-client kernel: Call Trace: Sep  6 06:55:07 oleg310-client kernel: [<ffffffff817ded29>] dump_stack+0x19/0x1b
Sep  6 06:55:07 oleg310-client kernel: [<ffffffff8108d558>] __warn+0xd8/0x100
Sep  6 06:55:07 oleg310-client kernel: [<ffffffff8108d69d>] warn_slowpath_null+0x1d/0x20
Sep  6 06:55:07 oleg310-client kernel: [<ffffffff810b6e8a>] rcuhead_fixup_activate+0x5a/0x70
Sep  6 06:55:07 oleg310-client kernel: [<ffffffff814150cf>] debug_object_activate+0x1cf/0x210
Sep  6 06:55:07 oleg310-client kernel: [<ffffffff8114dc8f>] __call_rcu+0x3f/0x2d0
Sep  6 06:55:07 oleg310-client kernel: [<ffffffff8114df3d>] call_rcu_sched+0x1d/0x20
Sep  6 06:55:07 oleg310-client kernel: [<ffffffffa0179f44>] xas_free_nodes+0xa4/0xf0 [libcfs]
Sep  6 06:55:07 oleg310-client kernel: [<ffffffffa017b26f>] xa_destroy+0xdf/0xf0 [libcfs]
Sep  6 06:55:07 oleg310-client kernel: [<ffffffffa088d4d5>] osc_quota_cleanup+0x15/0x20 [osc]
Sep  6 06:55:07 oleg310-client kernel: [<ffffffffa086ef1f>] osc_cleanup_common+0xbf/0x1b0 [osc]
Sep  6 06:55:07 oleg310-client kernel: [<ffffffffa02f48f9>] class_free_dev+0x219/0x730 [obdclass]
Sep  6 06:55:07 oleg310-client kernel: [<ffffffffa02f4ff0>] class_export_put+0x1e0/0x2e0 [obdclass]
Sep  6 06:55:07 oleg310-client kernel: [<ffffffffa02f6c15>] class_unlink_export+0x125/0x160 [obdclass]
Sep  6 06:55:07 oleg310-client kernel: [<ffffffffa030ca30>] class_decref+0x80/0x160 [obdclass]
Sep  6 06:55:07 oleg310-client kernel: [<ffffffffa030ce71>] class_detach+0x1c1/0x310 [obdclass]
Sep  6 06:55:07 oleg310-client kernel: [<ffffffffa0314b2b>] class_process_config+0x163b/0x27c0 [obdclass]
Sep  6 06:55:07 oleg310-client kernel: [<ffffffffa0315e90>] class_manual_cleanup+0x1e0/0x770 [obdclass]
Sep  6 06:55:07 oleg310-client kernel: [<ffffffffa0903955>] lov_tgts_putref+0x385/0xad0 [lov]
Sep  6 06:55:07 oleg310-client kernel: [<ffffffffa0908927>] lov_disconnect+0x237/0x280 [lov]
Sep  6 06:55:07 oleg310-client kernel: [<ffffffffa0e8fa96>] obd_disconnect+0x56/0x300 [lustre]
Sep  6 06:55:07 oleg310-client kernel: [<ffffffffa0e98ebc>] ll_put_super+0x81c/0xf30 [lustre]
Sep  6 06:55:07 oleg310-client kernel: [<ffffffff812476ca>] generic_shutdown_super+0x6a/0xf0
Sep  6 06:55:07 oleg310-client kernel: [<ffffffff81247ac2>] kill_anon_super+0x12/0x20
Sep  6 06:55:07 oleg310-client kernel: [<ffffffffa0ec711b>] lustre_kill_super+0x2b/0x30 [lustre]
Sep  6 06:55:07 oleg310-client kernel: [<ffffffff81247ec9>] deactivate_locked_super+0x49/0x60
Sep  6 06:55:07 oleg310-client kernel: [<ffffffff81248616>] deactivate_super+0x46/0x60
Sep  6 06:55:07 oleg310-client kernel: [<ffffffff81268b1f>] cleanup_mnt+0x3f/0x80
Sep  6 06:55:07 oleg310-client kernel: [<ffffffff81268bb2>] __cleanup_mnt+0x12/0x20
Sep  6 06:55:07 oleg310-client kernel: [<ffffffff810b69b5>] task_work_run+0xb5/0xf0
Sep  6 06:55:07 oleg310-client kernel: [<ffffffff8102ccb2>] do_notify_resume+0x92/0xb0
Sep  6 06:55:07 oleg310-client kernel: [<ffffffff817f4363>] int_signal+0x12/0x17
Sep  6 06:55:07 oleg310-client kernel: ---[ end trace 37e266df83060980 ]---
 

 

Issue Links

• is related to: LU-11063 RHEL7.[345] RCU breakage