Details
-
Bug
-
Resolution: Fixed
-
Minor
-
None
-
None
-
3
-
9223372036854775807
Description
sanity-quota@ldiskfs+DNE failed with Timeout(8009s)(Client: RCU stall).
https://testing.whamcloud.com/gerrit-janitor/34572/testresults/sanity-quota-ldiskfs-DNE-centos7_x86_64-centos7_x86_64/
Sep 6 06:54:52 oleg310-client kernel: Lustre: DEBUG MARKER: == sanity-quota test complete, duration 4313 sec ========= 06:54:51 (1693997691) Sep 6 06:54:52 oleg310-client sshd[20583]: Received disconnect from 192.168.203.10 port 42586:11: disconnected by user Sep 6 06:54:52 oleg310-client sshd[20583]: Disconnected from 192.168.203.10 port 42586 Sep 6 06:54:52 oleg310-client sshd[20583]: pam_unix(sshd:session): session closed for user root Sep 6 06:54:52 oleg310-client systemd-logind: Removed session 600. Sep 6 06:55:07 oleg310-client kernel: ------------[ cut here ]------------ Sep 6 06:55:07 oleg310-client kernel: WARNING: CPU: 2 PID: 20679 at lib/debugobjects.c:286 debug_print_object+0x83/0xa0 Sep 6 06:55:07 oleg310-client kernel: ODEBUG: activate active (active state 1) object type: rcu_head hint: (null) Sep 6 06:55:07 oleg310-client kernel: Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) crc32_generic libcfs(OE) crc_t10dif crct10dif_generic crct10dif_common rpcsec_gss_krb5 squashfs i2c_piix4 i2c_core pcspkr binfmt_misc ip_tables ext4 mbcache jbd2 ata_generic pata_acpi ata_piix serio_raw libata Sep 6 06:55:07 oleg310-client kernel: CPU: 2 PID: 20679 Comm: umount Kdump: loaded Tainted: G OE ------------ 3.10.0-7.9-debug #1 Sep 6 06:55:07 oleg310-client kernel: Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 Sep 6 06:55:07 oleg310-client kernel: Call Trace: Sep 6 06:55:07 oleg310-client kernel: [<ffffffff817ded29>] dump_stack+0x19/0x1b Sep 6 06:55:07 oleg310-client kernel: [<ffffffff8108d558>] __warn+0xd8/0x100 Sep 6 06:55:07 oleg310-client kernel: [<ffffffff8108d5df>] warn_slowpath_fmt+0x5f/0x80 Sep 6 06:55:07 oleg310-client kernel: [<ffffffff81414723>] debug_print_object+0x83/0xa0 Sep 6 06:55:07 oleg310-client kernel: [<ffffffff814150af>] debug_object_activate+0x1af/0x210 Sep 6 06:55:07 oleg310-client kernel: [<ffffffff8114dc8f>] __call_rcu+0x3f/0x2d0 Sep 6 06:55:07 oleg310-client kernel: [<ffffffff8114df3d>] call_rcu_sched+0x1d/0x20 Sep 6 06:55:07 oleg310-client kernel: [<ffffffffa0179f44>] xas_free_nodes+0xa4/0xf0 [libcfs] Sep 6 06:55:07 oleg310-client kernel: [<ffffffffa017b26f>] xa_destroy+0xdf/0xf0 [libcfs] Sep 6 06:55:07 oleg310-client kernel: [<ffffffffa088d4d5>] osc_quota_cleanup+0x15/0x20 [osc] Sep 6 06:55:07 oleg310-client kernel: [<ffffffffa086ef1f>] osc_cleanup_common+0xbf/0x1b0 [osc] Sep 6 06:55:07 oleg310-client kernel: [<ffffffffa02f48f9>] class_free_dev+0x219/0x730 [obdclass] Sep 6 06:55:07 oleg310-client kernel: [<ffffffffa02f4ff0>] class_export_put+0x1e0/0x2e0 [obdclass] Sep 6 06:55:07 oleg310-client kernel: [<ffffffffa02f6c15>] class_unlink_export+0x125/0x160 [obdclass] Sep 6 06:55:07 oleg310-client kernel: [<ffffffffa030ca30>] class_decref+0x80/0x160 [obdclass] Sep 6 06:55:07 oleg310-client kernel: [<ffffffffa030ce71>] class_detach+0x1c1/0x310 [obdclass] Sep 6 06:55:07 oleg310-client kernel: [<ffffffffa0314b2b>] class_process_config+0x163b/0x27c0 [obdclass] Sep 6 06:55:07 oleg310-client kernel: [<ffffffffa0315e90>] class_manual_cleanup+0x1e0/0x770 [obdclass] Sep 6 06:55:07 oleg310-client kernel: [<ffffffffa0903955>] lov_tgts_putref+0x385/0xad0 [lov] Sep 6 06:55:07 oleg310-client kernel: [<ffffffffa0908927>] lov_disconnect+0x237/0x280 [lov] Sep 6 06:55:07 oleg310-client kernel: [<ffffffffa0e8fa96>] obd_disconnect+0x56/0x300 [lustre] Sep 6 06:55:07 oleg310-client kernel: [<ffffffffa0e98ebc>] ll_put_super+0x81c/0xf30 [lustre] Sep 6 06:55:07 oleg310-client kernel: [<ffffffff812476ca>] generic_shutdown_super+0x6a/0xf0 Sep 6 06:55:07 oleg310-client kernel: [<ffffffff81247ac2>] kill_anon_super+0x12/0x20 Sep 6 06:55:07 oleg310-client kernel: [<ffffffffa0ec711b>] lustre_kill_super+0x2b/0x30 [lustre] Sep 6 06:55:07 oleg310-client kernel: [<ffffffff81247ec9>] deactivate_locked_super+0x49/0x60 Sep 6 06:55:07 oleg310-client kernel: [<ffffffff81248616>] deactivate_super+0x46/0x60 Sep 6 06:55:07 oleg310-client kernel: [<ffffffff81268b1f>] cleanup_mnt+0x3f/0x80 Sep 6 06:55:07 oleg310-client kernel: [<ffffffff81268bb2>] __cleanup_mnt+0x12/0x20 Sep 6 06:55:07 oleg310-client kernel: [<ffffffff810b69b5>] task_work_run+0xb5/0xf0 Sep 6 06:55:07 oleg310-client kernel: [<ffffffff8102ccb2>] do_notify_resume+0x92/0xb0 Sep 6 06:55:07 oleg310-client kernel: [<ffffffff817f4363>] int_signal+0x12/0x17 Sep 6 06:55:07 oleg310-client kernel: ---[ end trace 37e266df8306097f ]--- Sep 6 06:55:07 oleg310-client kernel: ------------[ cut here ]------------ Sep 6 06:55:07 oleg310-client kernel: WARNING: CPU: 2 PID: 20679 at kernel/rcupdate.c:311 rcuhead_fixup_activate+0x5a/0x70 Sep 6 06:55:07 oleg310-client kernel: Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) crc32_generic libcfs(OE) crc_t10dif crct10dif_generic crct10dif_common rpcsec_gss_krb5 squashfs i2c_piix4 i2c_core pcspkr binfmt_misc ip_tables ext4 mbcache jbd2 ata_generic pata_acpi ata_piix serio_raw libata Sep 6 06:55:07 oleg310-client kernel: CPU: 2 PID: 20679 Comm: umount Kdump: loaded Tainted: G W OE ------------ 3.10.0-7.9-debug #1 Sep 6 06:55:07 oleg310-client kernel: Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 Sep 6 06:55:07 oleg310-client kernel: Call Trace: Sep 6 06:55:07 oleg310-client kernel: [<ffffffff817ded29>] dump_stack+0x19/0x1b Sep 6 06:55:07 oleg310-client kernel: [<ffffffff8108d558>] __warn+0xd8/0x100 Sep 6 06:55:07 oleg310-client kernel: [<ffffffff8108d69d>] warn_slowpath_null+0x1d/0x20 Sep 6 06:55:07 oleg310-client kernel: [<ffffffff810b6e8a>] rcuhead_fixup_activate+0x5a/0x70 Sep 6 06:55:07 oleg310-client kernel: [<ffffffff814150cf>] debug_object_activate+0x1cf/0x210 Sep 6 06:55:07 oleg310-client kernel: [<ffffffff8114dc8f>] __call_rcu+0x3f/0x2d0 Sep 6 06:55:07 oleg310-client kernel: [<ffffffff8114df3d>] call_rcu_sched+0x1d/0x20 Sep 6 06:55:07 oleg310-client kernel: [<ffffffffa0179f44>] xas_free_nodes+0xa4/0xf0 [libcfs] Sep 6 06:55:07 oleg310-client kernel: [<ffffffffa017b26f>] xa_destroy+0xdf/0xf0 [libcfs] Sep 6 06:55:07 oleg310-client kernel: [<ffffffffa088d4d5>] osc_quota_cleanup+0x15/0x20 [osc] Sep 6 06:55:07 oleg310-client kernel: [<ffffffffa086ef1f>] osc_cleanup_common+0xbf/0x1b0 [osc] Sep 6 06:55:07 oleg310-client kernel: [<ffffffffa02f48f9>] class_free_dev+0x219/0x730 [obdclass] Sep 6 06:55:07 oleg310-client kernel: [<ffffffffa02f4ff0>] class_export_put+0x1e0/0x2e0 [obdclass] Sep 6 06:55:07 oleg310-client kernel: [<ffffffffa02f6c15>] class_unlink_export+0x125/0x160 [obdclass] Sep 6 06:55:07 oleg310-client kernel: [<ffffffffa030ca30>] class_decref+0x80/0x160 [obdclass] Sep 6 06:55:07 oleg310-client kernel: [<ffffffffa030ce71>] class_detach+0x1c1/0x310 [obdclass] Sep 6 06:55:07 oleg310-client kernel: [<ffffffffa0314b2b>] class_process_config+0x163b/0x27c0 [obdclass] Sep 6 06:55:07 oleg310-client kernel: [<ffffffffa0315e90>] class_manual_cleanup+0x1e0/0x770 [obdclass] Sep 6 06:55:07 oleg310-client kernel: [<ffffffffa0903955>] lov_tgts_putref+0x385/0xad0 [lov] Sep 6 06:55:07 oleg310-client kernel: [<ffffffffa0908927>] lov_disconnect+0x237/0x280 [lov] Sep 6 06:55:07 oleg310-client kernel: [<ffffffffa0e8fa96>] obd_disconnect+0x56/0x300 [lustre] Sep 6 06:55:07 oleg310-client kernel: [<ffffffffa0e98ebc>] ll_put_super+0x81c/0xf30 [lustre] Sep 6 06:55:07 oleg310-client kernel: [<ffffffff812476ca>] generic_shutdown_super+0x6a/0xf0 Sep 6 06:55:07 oleg310-client kernel: [<ffffffff81247ac2>] kill_anon_super+0x12/0x20 Sep 6 06:55:07 oleg310-client kernel: [<ffffffffa0ec711b>] lustre_kill_super+0x2b/0x30 [lustre] Sep 6 06:55:07 oleg310-client kernel: [<ffffffff81247ec9>] deactivate_locked_super+0x49/0x60 Sep 6 06:55:07 oleg310-client kernel: [<ffffffff81248616>] deactivate_super+0x46/0x60 Sep 6 06:55:07 oleg310-client kernel: [<ffffffff81268b1f>] cleanup_mnt+0x3f/0x80 Sep 6 06:55:07 oleg310-client kernel: [<ffffffff81268bb2>] __cleanup_mnt+0x12/0x20 Sep 6 06:55:07 oleg310-client kernel: [<ffffffff810b69b5>] task_work_run+0xb5/0xf0 Sep 6 06:55:07 oleg310-client kernel: [<ffffffff8102ccb2>] do_notify_resume+0x92/0xb0 Sep 6 06:55:07 oleg310-client kernel: [<ffffffff817f4363>] int_signal+0x12/0x17 Sep 6 06:55:07 oleg310-client kernel: ---[ end trace 37e266df83060980 ]---
Attachments
Issue Links
- is related to
-
-
- Resolved
-
Activity
close to this issue
[ 5407.737299] BUG: unable to handle kernel NULL pointer dereference at (null) [ 5407.739021] IP: [<ffffffffa016df58>] xas_free_nodes+0xb8/0xf0 [libcfs] [ 5407.740398] PGD 0 [ 5407.740789] Oops: 0000 [#1] SMP DEBUG_PAGEALLOC [ 5407.741671] Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) crc32_generic libcfs(OE) crc_t10dif crct10dif_generic crct10dif_common rpcsec_gss_krb5 squashfs i2c_piix4 i2c_core pcspkr binfmt_misc ip_tables ext4 mbcache jbd2 ata_generic pata_acpi ata_piix serio_raw libata [ 5407.750803] CPU: 0 PID: 9909 Comm: umount Kdump: loaded Tainted: G OE ------------ 3.10.0-7.9-debug #1 [ 5407.753185] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 5407.755316] task: ffff8800b6498000 ti: ffff8800a804c000 task.ti: ffff8800a804c000 [ 5407.756831] RIP: 0010:[<ffffffffa016df58>] [<ffffffffa016df58>] xas_free_nodes+0xb8/0xf0 [libcfs] [ 5407.758521] RSP: 0018:ffff8800a804f950 EFLAGS: 00010083 [ 5407.759476] RAX: 0000000000002710 RBX: ffff880136ef66c8 RCX: 0000000000000023 [ 5407.760702] RDX: ffff880136ef5930 RSI: ffff880136ef66e0 RDI: 0000000000000046 [ 5407.761989] RBP: ffff8800a804f978 R08: 0000000000000000 R09: 77b0000000000000 [ 5407.763203] R10: 00000000af0c2d01 R11: ffff8800af0c2f80 R12: 000000000000002a [ 5407.764433] R13: 0000000000000000 R14: ffff880136ef56d0 R15: ffff8800a804f988 [ 5407.765708] FS: 00007f19ba87f880(0000) GS:ffff88013e200000(0000) knlGS:0000000000000000 [ 5407.767225] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5407.768257] CR2: 0000000000000000 CR3: 00000000a4d64000 CR4: 00000000000006f0 [ 5407.769490] Call Trace: [ 5407.769952] [<ffffffffa016f26f>] xa_destroy+0xdf/0xf0 [libcfs] [ 5407.771357] [<ffffffffa08814c5>] osc_quota_cleanup+0x15/0x20 [osc] [ 5407.773200] [<ffffffffa0862f1f>] osc_cleanup_common+0xbf/0x1b0 [osc] [ 5407.774839] [<ffffffffa02e88f9>] class_free_dev+0x219/0x730 [obdclass] [ 5407.776139] [<ffffffffa02e8ff0>] class_export_put+0x1e0/0x2e0 [obdclass] [ 5407.777489] [<ffffffffa02eac15>] class_unlink_export+0x125/0x160 [obdclass] [ 5407.778876] [<ffffffffa030016e>] class_decref_free+0x4e/0x90 [obdclass] [ 5407.780371] [<ffffffffa0300ad8>] class_decref+0x48/0xf0 [obdclass] [ 5407.781595] [<ffffffffa0300ee1>] class_detach+0x1c1/0x310 [obdclass] [ 5407.782914] [<ffffffffa0308b9b>] class_process_config+0x163b/0x27c0 [obdclass] [ 5407.784281] [<ffffffff81220310>] ? __kmalloc+0x1e0/0x340 [ 5407.785303] [<ffffffffa0309f00>] class_manual_cleanup+0x1e0/0x770 [obdclass] [ 5407.786693] [<ffffffffa08f7955>] lov_tgts_putref+0x385/0xad0 [lov] [ 5407.787835] [<ffffffffa08fc927>] lov_disconnect+0x237/0x280 [lov] [ 5407.789099] [<ffffffffa0e82b46>] obd_disconnect+0x56/0x300 [lustre]
Sorry it took awhile to run the patch. As you can see with a non-debug normal kernel sanity-quota passes.
https://testing.whamcloud.com/test_sets/921b8bf1-b5ba-4664-9d6c-1c4c164e543c
I just pushed a patch to validate the xarray work with a normal RHEL7 kernel.
"James Simmons <jsimmons@infradead.org>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/52381
Subject: LU-17097 tests: validate xarray on RHEL7 non debug kernels
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: f615ad49ca9d4673acdbdafdbea595b911c6269e
Hi simmonsja
'm working with Oleg to move to RHLE8 for debug kernel testing. This should go away then.
Thanks for update. Does it mean there is no issue with your patch and it is caused by some problem in RHEL7 debug kernel?
I'm working with Oleg to move to RHLE8 for debug kernel testing. This should go away then.
No, I haven't seen such messages somewhere besides RHEL7. I saw these warnings just once during testing LU-17034. I'll send https://review.whamcloud.com/c/fs/lustre-release/+/52094 without revert "LU-8130 osc: convert osc_quota hash to xarray" to see whether it is reproducible.
I'm looking. Can this be reproduced only on RHEL7? If this is a RHEL7 only problem I don't think its worth reverting.
"Sergey Cheremencev <scherementsev@ddn.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/52304
Subject: LU-17097 quota: Revert "LU-8130 osc: convert osc_quota hash to xarray"
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: bb8306699f0f63ce5ee27c42d5ef08e61ee8f2ac
Hit the same panic as Alex reported above:
simmonsja , could you comment? Do we need a separate ticket for that? Can we expect that https://review.whamcloud.com/c/fs/lustre-release/+/52381 could solve the problem with the panic also?
https://testing.whamcloud.com/gerrit-janitor/37062/testresults/sanity-quota-zfs-centos7_x86_64-centos7_x86_64/