-
Bug
-
Resolution: Fixed
-
Medium
-
None
-
None
-
3
-
9223372036854775807
This issue was created by maloo for Bruno Faccini <bfaccini62@gmail.com>
This issue relates to the following test suite run: https://testing.whamcloud.com/test_sets/2760a645-98d8-4215-b569-c9e0357e5dac
sanity-krb5/test_11 failed with the following error:
remount /mnt/lustre failed
Test session details:
clients: https://build.whamcloud.com/job/lustre-reviews/128362 - 5.14.0-611.55.1.el9_7.x86_64
servers: https://build.whamcloud.com/job/lustre-reviews/128362 - 5.14.0-611.55.1.el9_7_lustre.x86_64
Problem has occurred during https://review.whamcloud.com/c/fs/lustre-release/+/66081 testing with the following parameters to force KRB5 testing and also skip sanity-krb5/test{_}[10,13,152,200,201] because of their solid failures with latest master (see LU-20537) :
Test-Parameters: kerberos=true testlist=sanity-krb5 env=SANITY_GSS_EXCEPT="10 13 152 200 201"
The dmesg log is really interesting and giving details for this problem, but also shows 2 others warnings, for a counter overflow/underflow, which look as direct consequences that should have occurred in the original error handling path .... :
........................
[ 687.939011] Lustre: DEBUG MARKER: == sanity-krb5 test 9: Do not trust primary and supp gids from client ========================================================== 11:59:43 (1785412783)
[ 691.268832] Lustre: lustre-OST0000-osc-ff2eb5f1843b3000: disconnect after 24s idle
[ 691.269049] Lustre: Skipped 15 previous similar messages
[ 694.115459] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity-krb5 test_10 skipping ALWAYS excluded test 10
[ 694.363097] Lustre: DEBUG MARKER: SKIP: sanity-krb5 test_10 skipping ALWAYS excluded test 10
[ 694.671562] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-krb5 test 11: KCM ccache ======================= 11:59:50 \(1785412790\)
[ 694.900245] Lustre: DEBUG MARKER: == sanity-krb5 test 11: KCM ccache ======================= 11:59:50 (1785412790)
[ 694.917024] Autotest: Test running for 10 minutes (lustre-reviews_custom_128362.1001)
[ 696.243811] Lustre: DEBUG MARKER: grep -c /mnt/lustre' ' /proc/mounts
[ 696.260276] Lustre: DEBUG MARKER: lsof -t /mnt/lustre
[ 696.399932] Lustre: DEBUG MARKER: umount /mnt/lustre 2>&1
[ 725.298315] Lustre: Unmounted lustre-client
[ 725.307911] Lustre: DEBUG MARKER: grep -c /mnt/lustre2' ' /proc/mounts
[ 725.323697] Lustre: DEBUG MARKER: lsof -t /mnt/lustre2
[ 725.463003] Lustre: DEBUG MARKER: umount /mnt/lustre2 2>&1
[ 725.590564] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre
[ 725.607231] Lustre: DEBUG MARKER: mount -t lustre -o user_xattr,flock 10.240.28.70@tcp:/lustre /mnt/lustre
[ 725.633509] sysfs: cannot create duplicate filename '/fs/lustre/ldlm/namespaces/lustre-MDT0000-mdc-ff2eb5f1a1d0d000'
[ 725.633616] CPU: 0 PID: 41737 Comm: llog_process_th Kdump: loaded Tainted: G OE ------ --- 5.14.0-611.55.1.el9_7.x86_64 #1
[ 725.633672] Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014
[ 725.633731] Call Trace:
[ 725.633794] <TASK>
[ 725.633853] dump_stack_lvl+0x34/0x48
[ 725.633931] sysfs_warn_dup.cold+0x17/0x24
[ 725.633996] sysfs_create_dir_ns+0xba/0xd0
[ 725.634060] create_dir+0x25/0x170
[ 725.634122] kobject_add_internal+0xa8/0x180
[ 725.634181] kobject_init_and_add+0x71/0xa0
[ 725.634239] ? srso_alias_return_thunk+0x5/0xfbef5
[ 725.634300] ? lprocfs_counter_init_units+0x72/0x390 [obdclass]
[ 725.634446] ldlm_namespace_new+0x7e2/0xc00 [ptlrpc]
[ 725.634646] client_obd_setup+0x89d/0x1950 [ptlrpc]
[ 725.634786] osc_setup_common+0xe1/0x2b0 [osc]
[ 725.634868] mdc_setup+0x2b/0x350 [mdc]
[ 725.634955] mdc_device_alloc+0x101/0x320 [mdc]
[ 725.635027] obd_setup+0x21d/0x3f0 [obdclass]
[ 725.635158] class_setup+0x4ba/0x5f0 [obdclass]
[ 725.635273] class_process_config+0x17a9/0x1dc0 [obdclass]
[ 725.635388] ? srso_alias_return_thunk+0x5/0xfbef5
[ 725.635446] ? __kmalloc+0x1d8/0x440
[ 725.635504] ? srso_alias_return_thunk+0x5/0xfbef5
[ 725.635560] ? class_config_llog_handler+0x5ea/0x1150 [obdclass]
[ 725.635682] class_config_llog_handler+0x738/0x1150 [obdclass]
[ 725.635805] llog_process_thread+0xd9f/0x1b10 [obdclass]
[ 725.635943] llog_process_thread_daemonize+0x6c/0x90 [obdclass]
[ 725.636058] ? __pfx_llog_process_thread_daemonize+0x10/0x10 [obdclass]
[ 725.636173] kthread+0x101/0x110
[ 725.636238] ? __pfx_kthread+0x10/0x10
[ 725.636297] ret_from_fork+0x28/0x50
[ 725.636362] </TASK>
[ 725.636475] kobject_add_internal failed for lustre-MDT0000-mdc-ff2eb5f1a1d0d000 with -EEXIST, don't try to register things with the same name in the same directory.
[ 725.636548] LustreError: 41737:0:(ldlm_resource.c:1066:ldlm_namespace_new()) lustre-MDT0000-mdc-ff2eb5f1a1d0d000: cannot initialize ns sysfs: rc = -17
[ 725.638671] LustreError: 41737:0:(ldlm_lib.c:597:client_obd_setup()) lustre-MDT0000-mdc-ff2eb5f1a1d0d000: unable to create client namespace: rc = -17
[ 725.638846] ------------[ cut here ]------------
[ 725.639010] refcount_t: addition on 0; use-after-free.
[ 725.639151] WARNING: CPU: 0 PID: 41737 at lib/refcount.c:25 refcount_warn_saturate+0x74/0x110
[ 725.675275] Modules linked in: lustre(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) ec(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) tls rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache netfs rfkill sunrpc intel_rapl_msr intel_rapl_common kvm_amd ccp kvm i2c_i801 iTCO_wdt iTCO_vendor_support pcspkr i2c_smbus lpc_ich virtio_balloon joydev fuse drm dm_mod ext4 mbcache jbd2 crct10dif_pclmul crc32_pclmul crc32c_intel virtio_net ahci ghash_clmulni_intel libahci virtio_blk libata net_failover failover serio_raw
[ 725.684233] CPU: 0 PID: 41737 Comm: llog_process_th Kdump: loaded Tainted: G OE ------ --- 5.14.0-611.55.1.el9_7.x86_64 #1
[ 725.686490] Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014
[ 725.687537] RIP: 0010:refcount_warn_saturate+0x74/0x110
[ 725.688409] Code: 01 01 e8 cf 45 a9 ff 0f 0b e9 73 2a 88 00 80 3d 2a cf cc 01 00 75 cb 48 c7 c7 18 27 32 98 c6 05 1a cf cc 01 01 e8 ac 45 a9 ff <0f> 0b e9 50 2a 88 00 80 3d 09 cf cc 01 00 75 a8 48 c7 c7 f0 26 32
[ 725.691787] RSP: 0018:ff37691042377b40 EFLAGS: 00010282
[ 725.692797] RAX: 0000000000000000 RBX: ff2eb5f1823a0000 RCX: 0000000000000027
[ 725.694135] RDX: 0000000000000027 RSI: 00000000ffff7fff RDI: ff2eb5f1fba20908
[ 725.695428] RBP: ff2eb5f1ccfa8000 R08: 0000000000000000 R09: ff376910423779e8
[ 725.696699] R10: ff376910423779e0 R11: ffffffff98de2348 R12: ff2eb5f1ccfa8000
[ 725.697884] R13: ff2eb5f185e27780 R14: ff37691042377c10 R15: ffffffffc1466420
[ 725.698981] FS: 0000000000000000(0000) GS:ff2eb5f1fba00000(0000) knlGS:0000000000000000
[ 725.700530] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 725.701650] CR2: 00007fb19efaa000 CR3: 00000001087ae004 CR4: 0000000000771ef0
[ 725.702768] PKRU: 55555554
[ 725.703213] Call Trace:
[ 725.703613] <TASK>
[ 725.703991] ? srso_alias_return_thunk+0x5/0xfbef5
[ 725.704828] ? show_trace_log_lvl+0x26e/0x2df
[ 725.705652] ? show_trace_log_lvl+0x26e/0x2df
[ 725.706535] ? class_import_get+0xb9/0xc0 [obdclass]
[ 725.707594] ? refcount_warn_saturate+0x74/0x110
[ 725.708500] ? __warn+0x7d/0xd0
[ 725.709173] ? refcount_warn_saturate+0x74/0x110
[ 725.710077] ? report_bug+0x102/0x140
[ 725.710805] ? prb_read_valid+0x17/0x20
[ 725.711541] ? handle_bug+0x3c/0x70
[ 725.712149] ? exc_invalid_op+0x14/0x70
[ 725.712863] ? asm_exc_invalid_op+0x16/0x20
[ 725.713585] ? refcount_warn_saturate+0x74/0x110
[ 725.714399] class_import_get+0xb9/0xc0 [obdclass]
[ 725.715330] client_obd_cleanup+0x87/0x500 [ptlrpc]
[ 725.716188] osc_cleanup_common+0x110/0x1e0 [osc]
[ 725.716947] mdc_device_free+0x174/0x180 [mdc]
[ 725.717636] mdc_device_alloc+0x177/0x320 [mdc]
[ 725.718360] obd_setup+0x21d/0x3f0 [obdclass]
[ 725.719124] class_setup+0x4ba/0x5f0 [obdclass]
[ 725.719925] class_process_config+0x17a9/0x1dc0 [obdclass]
[ 725.720809] ? srso_alias_return_thunk+0x5/0xfbef5
[ 725.721541] ? __kmalloc+0x1d8/0x440
[ 725.722125] ? srso_alias_return_thunk+0x5/0xfbef5
[ 725.722861] ? class_config_llog_handler+0x5ea/0x1150 [obdclass]
[ 725.723816] class_config_llog_handler+0x738/0x1150 [obdclass]
[ 725.724741] llog_process_thread+0xd9f/0x1b10 [obdclass]
[ 725.725590] llog_process_thread_daemonize+0x6c/0x90 [obdclass]
[ 725.726528] ? __pfx_llog_process_thread_daemonize+0x10/0x10 [obdclass]
[ 725.727549] kthread+0x101/0x110
[ 725.728086] ? __pfx_kthread+0x10/0x10
[ 725.728668] ret_from_fork+0x28/0x50
[ 725.729251] </TASK>
[ 725.729611] ---[ end trace 0000000000000000 ]---
[ 725.730327] ------------[ cut here ]------------
[ 725.730380] refcount_t: underflow; use-after-free.
[ 725.730516] WARNING: CPU: 0 PID: 41737 at lib/refcount.c:28 refcount_warn_saturate+0xba/0x110
[ 725.733251] Modules linked in: lustre(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) ec(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) tls rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache netfs rfkill sunrpc intel_rapl_msr intel_rapl_common kvm_amd ccp kvm i2c_i801 iTCO_wdt iTCO_vendor_support pcspkr i2c_smbus lpc_ich virtio_balloon joydev fuse drm dm_mod ext4 mbcache jbd2 crct10dif_pclmul crc32_pclmul crc32c_intel virtio_net ahci ghash_clmulni_intel libahci virtio_blk libata net_failover failover serio_raw
[ 725.740692] CPU: 0 PID: 41737 Comm: llog_process_th Kdump: loaded Tainted: G W OE ------ --- 5.14.0-611.55.1.el9_7.x86_64 #1
[ 725.742754] Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014
[ 725.744031] RIP: 0010:refcount_warn_saturate+0xba/0x110
[ 725.744868] Code: 01 01 e8 89 45 a9 ff 0f 0b e9 2d 2a 88 00 80 3d e3 ce cc 01 00 75 85 48 c7 c7 48 27 32 98 c6 05 d3 ce cc 01 01 e8 66 45 a9 ff <0f> 0b e9 0a 2a 88 00 80 3d be ce cc 01 00 0f 85 5e ff ff ff 48 c7
[ 725.747667] RSP: 0018:ff37691042377b40 EFLAGS: 00010282
[ 725.748510] RAX: 0000000000000000 RBX: ff2eb5f1823a0000 RCX: 0000000000000027
[ 725.749593] RDX: 0000000000000027 RSI: 00000000ffff7fff RDI: ff2eb5f1fba20908
[ 725.750817] RBP: ff2eb5f1ccfa8000 R08: 0000000000000000 R09: ff376910423779e8
[ 725.752035] R10: ff376910423779e0 R11: ffffffff98de2348 R12: ff2eb5f1823a0618
[ 725.753133] R13: ff2eb5f185e27780 R14: ff37691042377c10 R15: ffffffffc1466420
[ 725.754216] FS: 0000000000000000(0000) GS:ff2eb5f1fba00000(0000) knlGS:0000000000000000
[ 725.755520] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 725.756512] CR2: 00007fb19efaa000 CR3: 00000001087ae004 CR4: 0000000000771ef0
[ 725.757665] PKRU: 55555554
[ 725.758167] Call Trace:
[ 725.758567] <TASK>
[ 725.758972] ? srso_alias_return_thunk+0x5/0xfbef5
[ 725.759744] ? show_trace_log_lvl+0x26e/0x2df
[ 725.760438] ? show_trace_log_lvl+0x26e/0x2df
[ 725.761176] ? class_import_put+0x254/0x290 [obdclass]
[ 725.762068] ? refcount_warn_saturate+0xba/0x110
[ 725.762781] ? __warn+0x7d/0xd0
[ 725.763291] ? refcount_warn_saturate+0xba/0x110
[ 725.764015] ? report_bug+0x102/0x140
[ 725.764582] ? prb_read_valid+0x17/0x20
[ 725.765193] ? handle_bug+0x3c/0x70
[ 725.765820] ? exc_invalid_op+0x14/0x70
[ 725.766425] ? asm_exc_invalid_op+0x16/0x20
[ 725.767091] ? refcount_warn_saturate+0xba/0x110
[ 725.767796] class_import_put+0x254/0x290 [obdclass]
[ 725.768588] client_obd_cleanup+0x8f/0x500 [ptlrpc]
[ 725.769410] osc_cleanup_common+0x110/0x1e0 [osc]
[ 725.770153] mdc_device_free+0x174/0x180 [mdc]
[ 725.770852] mdc_device_alloc+0x177/0x320 [mdc]
[ 725.771551] obd_setup+0x21d/0x3f0 [obdclass]
[ 725.772280] class_setup+0x4ba/0x5f0 [obdclass]
[ 725.773030] class_process_config+0x17a9/0x1dc0 [obdclass]
[ 725.773918] ? srso_alias_return_thunk+0x5/0xfbef5
[ 725.774645] ? __kmalloc+0x1d8/0x440
[ 725.775220] ? srso_alias_return_thunk+0x5/0xfbef5
[ 725.775953] ? class_config_llog_handler+0x5ea/0x1150 [obdclass]
[ 725.776940] class_config_llog_handler+0x738/0x1150 [obdclass]
[ 725.777991] llog_process_thread+0xd9f/0x1b10 [obdclass]
[ 725.779045] llog_process_thread_daemonize+0x6c/0x90 [obdclass]
[ 725.780057] ? __pfx_llog_process_thread_daemonize+0x10/0x10 [obdclass]
[ 725.781188] kthread+0x101/0x110
[ 725.781750] ? __pfx_kthread+0x10/0x10
[ 725.782391] ret_from_fork+0x28/0x50
[ 725.783040] </TASK>
[ 725.783408] ---[ end trace 0000000000000000 ]---
[ 725.784213] LustreError: 41737:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-mdc-ff2eb5f1a1d0d000 failed (-17)
[ 725.784362] LustreError: 41737:0:(obd_config.c:2102:class_config_llog_handler()) MGC10.240.28.70@tcp: cfg command failed: rc = -17
[ 725.784498] Lustre: cmd=cf003 0:lustre-MDT0000-mdc 1:lustre-MDT0000_UUID 2:10.240.28.70@tcp
[ 725.787867] LustreError: MGC10.240.28.70@tcp: Confguration from log lustre-client failed from MGS -17. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info
[ 725.788413] LustreError: 41733:0:(ldlm_resource.c:1205:ldlm_resource_complain()) MGC10.240.28.70@tcp: namespace resource [0x65727473756c:0x0:0x0].0x0 (ff2eb5f1a1457b00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 725.790200] LustreError: 41733:0:(super25.c:177:lustre_fill_super()) llite: Unable to mount <unknown>: rc = -17
[ 726.091573] Lustre: DEBUG MARKER: /usr/sbin/lctl mark sanity-krb5 test_11: @@@@@@ FAIL: remount \/mnt\/lustre failed
[ 726.316022] Lustre: DEBUG MARKER: sanity-krb5 test_11: @@@@@@ FAIL: remount /mnt/lustre failed
[ 726.597506] Lustre: DEBUG MARKER: /usr/sbin/lctl dk > /autotest/autotest-2/2026-07-30/lustre-reviews_custom_128362_1001_e55519fb-af7d-4e7f-b1c0-48049cb43c1d//sanity-krb5.test_11.debug_log.$(hostname -s).1785412822.log;
dmesg > /autotest/autotest-2/2026-07-30/lustre-reviews_custom_128362
So, this seems to indicate that there are possibly 3 improvements to be added into the concerned code...
- is related to
-
-
- Open
-