[LU-17212] sanity-sec test_36: crash - Whamcloud Community JIRA

Details

Type: Bug
Resolution: Fixed
Priority: Minor
Fix Version/s: Lustre 2.16.0
Affects Version/s: Lustre 2.16.0
Labels:
- gss
- patch
- sec

Severity:
3
Rank (Obsolete):
9223372036854775807

Description

Client crashes with the following:

Lustre: DEBUG MARKER: mount -t lustre -o user_xattr,flock,test_dummy_encryption,skpath=/tmp/test-framework-keys trevis-66vm6@tcp:/lustre /mnt/lustre
Lustre: Test dummy encryption mode enabled
Lustre: DEBUG MARKER: grep -c /mnt/lustre' ' /proc/mounts
Lustre: DEBUG MARKER: lsof -t /mnt/lustre
Lustre: DEBUG MARKER: umount  /mnt/lustre 2>&1
LustreError: 588361:0:(gss_cli_upcall.c:296:gss_do_ctx_init_rpc()) import has been deactivated
Lustre: 588408:0:(gss_keyring.c:1039:flush_spec_ctx_cache_kr()) flush busy ctx 00000000ceae75f0(0->lustre-OST0002_UUID, extra ref 1)
LustreError: 588365:0:(gss_cli_upcall.c:259:gss_do_ctx_init_rpc()) no such obd lustre-OST0005-osc-ffff95a585165000
LustreError: 588365:0:(gss_cli_upcall.c:259:gss_do_ctx_init_rpc()) Skipped 6 previous similar messages
BUG: unable to handle kernel NULL pointer dereference at 0000000000000020
CPU: 1 PID: 588359 Comm: lgss_keyring  4.18.0-477.27.1.el8_8.x86_64 #1
RIP: 0010:gss_do_ctx_init_rpc+0x84e/0x1100 [ptlrpc_gss]
Call Trace:
 gss_proc_write_secinit+0x14/0x60 [ptlrpc_gss]
 full_proxy_write+0x53/0x80
 vfs_write+0xa5/0x1b0
 ksys_write+0x4f/0xb0
 do_syscall_64+0x5b/0x1b0

Attachments

Activity

[LU-17212] sanity-sec test_36: crash

Andreas Dilger made changes - 29/Apr/24 11:40 PM

Description

Original: Client crashes with the following:
{noformat}
[12391.117740] Lustre: DEBUG MARKER: mount -t lustre -o user_xattr,flock,test_dummy_encryption,skpath=/tmp/test-framework-keys trevis-66vm6@tcp:/lustre /mnt/lustre
[12392.975436] Lustre: Test dummy encryption mode enabled
[12393.733894] Lustre: DEBUG MARKER: grep -c /mnt/lustre' ' /proc/mounts
[12393.925961] Lustre: DEBUG MARKER: lsof -t /mnt/lustre
[12394.854963] Lustre: DEBUG MARKER: umount /mnt/lustre 2>&1
[12395.289436] LustreError: 588361:0:(gss_cli_upcall.c:296:gss_do_ctx_init_rpc()) import has been deactivated
[12395.301159] Lustre: 588408:0:(gss_keyring.c:1039:flush_spec_ctx_cache_kr()) flush busy ctx 00000000ceae75f0(0->lustre-OST0002_UUID, extra ref 1)
[12395.307688] LustreError: 588365:0:(gss_cli_upcall.c:259:gss_do_ctx_init_rpc()) no such obd lustre-OST0005-osc-ffff95a585165000
[12395.309857] LustreError: 588365:0:(gss_cli_upcall.c:259:gss_do_ctx_init_rpc()) Skipped 6 previous similar messages
[12395.314608] BUG: unable to handle kernel NULL pointer dereference at 0000000000000020
[12395.316095] PGD 8000000040686067 P4D 8000000040686067 PUD 40687067 PMD 0
[12395.317356] Oops: 0000 1 SMP PTI
[12395.318054] CPU: 1 PID: 588359 Comm: lgss_keyring Kdump: loaded Tainted: G OE --------- - - 4.18.0-477.27.1.el8_8.x86_64 #1
[12395.320256] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
[12395.321328] RIP: 0010:gss_do_ctx_init_rpc+0x84e/0x1100 [ptlrpc_gss]
[12395.322576] Code: 8b 44 24 58 89 44 24 08 8b 44 24 54 89 44 24 10 8b 44 24 50 89 44 24 14 e8 cf d4 ec ff 48 89 c1 c6 00 01 48 8b 83 e0 00 00 00 <8b> 40 20 48 c7 41 0c 00 00 00 00 c7 41 20 00 00 00 00 88 41 01 48
[12395.325875] RSP: 0018:ffffb185c2b6fd80 EFLAGS: 00010282
[12395.326853] RAX: 0000000000000000 RBX: ffff95a583476800 RCX: ffff95a5bb4b0030
[12395.328179] RDX: 0000000000000024 RSI: 0000000000000000 RDI: 0000000000000030
[12395.329493] RBP: ffff95a582d4b180 R08: 000000000000002c R09: 0000000000000000
[12395.330796] R10: 0000000000000002 R11: 0000000000000000 R12: 000000000000028f
[12395.332109] R13: 00007ffc8a747680 R14: ffff95a5bb4b0000 R15: 0000000000000002
[12395.333417] FS: 00007f978ac9b840(0000) GS:ffff95a63cd00000(0000) knlGS:0000000000000000
[12395.334879] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[12395.335941] CR2: 0000000000000020 CR3: 0000000039ed4004 CR4: 00000000000606e0
[12395.337259] Call Trace:
[12395.337792] ? __inode_security_revalidate+0x63/0x80
[12395.338762] gss_proc_write_secinit+0x14/0x60 [ptlrpc_gss]
[12395.339834] full_proxy_write+0x53/0x80
[12395.340616] vfs_write+0xa5/0x1b0
[12395.341305] ksys_write+0x4f/0xb0
[12395.341955] do_syscall_64+0x5b/0x1b0
[12395.342707] entry_SYSCALL_64_after_hwframe+0x61/0xc6
[12395.343698] RIP: 0033:0x7f9789d8e9e5
[12395.344408] Code: 00 00 75 05 48 83 c4 58 c3 e8 27 4a ff ff 0f 1f 80 00 00 00 00 f3 0f 1e fa 8b 05 66 da 20 00 85 c0 75 12 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 53 c3 66 90 41 54 49 89 d4 55 48 89 f5 53 89
[12395.347714] RSP: 002b:00007ffc8a7475e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
[12395.349109] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f9789d8e9e5
[12395.350433] RDX: 0000000000000050 RSI: 00007ffc8a747680 RDI: 0000000000000004
[12395.351741] RBP: 00007ffc8a7475f0 R08: 00000000024c9fb0 R09: 00000000024c2018
[12395.353061] R10: 0000000000000018 R11: 0000000000000246 R12: 0000000000000004
[12395.354366] R13: 0000000000613640 R14: 00007ffc8a747680 R15: 00007ffc8a749770
[12395.355668] Modules linked in: lustre(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache sunrpc intel_rapl_msr intel_rapl_common crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcspkr joydev virtio_balloon i2c_piix4 ext4 mbcache jbd2 ata_generic crc32c_intel ata_piix libata serio_raw virtio_net virtio_blk net_failover failover
[12395.363302] CR2: 0000000000000020
{noformat}

New: Client crashes with the following:
{noformat}
Lustre: DEBUG MARKER: mount -t lustre -o user_xattr,flock,test_dummy_encryption,skpath=/tmp/test-framework-keys trevis-66vm6@tcp:/lustre /mnt/lustre
Lustre: Test dummy encryption mode enabled
Lustre: DEBUG MARKER: grep -c /mnt/lustre' ' /proc/mounts
Lustre: DEBUG MARKER: lsof -t /mnt/lustre
Lustre: DEBUG MARKER: umount /mnt/lustre 2>&1
LustreError: 588361:0:(gss_cli_upcall.c:296:gss_do_ctx_init_rpc()) import has been deactivated
Lustre: 588408:0:(gss_keyring.c:1039:flush_spec_ctx_cache_kr()) flush busy ctx 00000000ceae75f0(0->lustre-OST0002_UUID, extra ref 1)
LustreError: 588365:0:(gss_cli_upcall.c:259:gss_do_ctx_init_rpc()) no such obd lustre-OST0005-osc-ffff95a585165000
LustreError: 588365:0:(gss_cli_upcall.c:259:gss_do_ctx_init_rpc()) Skipped 6 previous similar messages
BUG: unable to handle kernel NULL pointer dereference at 0000000000000020
CPU: 1 PID: 588359 Comm: lgss_keyring 4.18.0-477.27.1.el8_8.x86_64 #1
RIP: 0010:gss_do_ctx_init_rpc+0x84e/0x1100 [ptlrpc_gss]
Call Trace:
gss_proc_write_secinit+0x14/0x60 [ptlrpc_gss]
full_proxy_write+0x53/0x80
vfs_write+0xa5/0x1b0
ksys_write+0x4f/0xb0
do_syscall_64+0x5b/0x1b0
{noformat}

Peter Jones made changes - 29/Nov/23 10:47 PM

Fix Version/s		New: Lustre 2.16.0 [ 15190 ]
Resolution		New: Fixed [ 1 ]
Status	Original: Open [ 1 ]	New: Resolved [ 5 ]

Peter Jones added a comment - 29/Nov/23 10:47 PM

Landed for 2.16

Peter Jones added a comment - 29/Nov/23 10:47 PM Landed for 2.16

Gerrit Updater added a comment - 29/Nov/23 9:28 PM

"Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/52755/
Subject: ~~LU-17212~~ gss: survive improper obd or imp at ctx init
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 3fcddf6dcdd92df6557c59913a61944f21d58615

Gerrit Updater added a comment - 29/Nov/23 9:28 PM "Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/52755/ Subject: LU-17212 gss: survive improper obd or imp at ctx init Project: fs/lustre-release Branch: master Current Patch Set: Commit: 3fcddf6dcdd92df6557c59913a61944f21d58615

Sebastien Buisson made changes - 19/Oct/23 4:45 PM

Description

Original: Client crashes with the following:
{noformat}
[12395.289436] LustreError: 588361:0:(gss_cli_upcall.c:296:gss_do_ctx_init_rpc()) import has been deactivated
[12395.301159] Lustre: 588408:0:(gss_keyring.c:1039:flush_spec_ctx_cache_kr()) flush busy ctx 00000000ceae75f0(0->lustre-OST0002_UUID, extra ref 1)
[12395.307688] LustreError: 588365:0:(gss_cli_upcall.c:259:gss_do_ctx_init_rpc()) no such obd lustre-OST0005-osc-ffff95a585165000
[12395.309857] LustreError: 588365:0:(gss_cli_upcall.c:259:gss_do_ctx_init_rpc()) Skipped 6 previous similar messages
[12395.314608] BUG: unable to handle kernel NULL pointer dereference at 0000000000000020
[12395.316095] PGD 8000000040686067 P4D 8000000040686067 PUD 40687067 PMD 0
[12395.317356] Oops: 0000 1 SMP PTI
[12395.318054] CPU: 1 PID: 588359 Comm: lgss_keyring Kdump: loaded Tainted: G OE --------- - - 4.18.0-477.27.1.el8_8.x86_64 #1
[12395.320256] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
[12395.321328] RIP: 0010:gss_do_ctx_init_rpc+0x84e/0x1100 [ptlrpc_gss]
[12395.322576] Code: 8b 44 24 58 89 44 24 08 8b 44 24 54 89 44 24 10 8b 44 24 50 89 44 24 14 e8 cf d4 ec ff 48 89 c1 c6 00 01 48 8b 83 e0 00 00 00 <8b> 40 20 48 c7 41 0c 00 00 00 00 c7 41 20 00 00 00 00 88 41 01 48
[12395.325875] RSP: 0018:ffffb185c2b6fd80 EFLAGS: 00010282
[12395.326853] RAX: 0000000000000000 RBX: ffff95a583476800 RCX: ffff95a5bb4b0030
[12395.328179] RDX: 0000000000000024 RSI: 0000000000000000 RDI: 0000000000000030
[12395.329493] RBP: ffff95a582d4b180 R08: 000000000000002c R09: 0000000000000000
[12395.330796] R10: 0000000000000002 R11: 0000000000000000 R12: 000000000000028f
[12395.332109] R13: 00007ffc8a747680 R14: ffff95a5bb4b0000 R15: 0000000000000002
[12395.333417] FS: 00007f978ac9b840(0000) GS:ffff95a63cd00000(0000) knlGS:0000000000000000
[12395.334879] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[12395.335941] CR2: 0000000000000020 CR3: 0000000039ed4004 CR4: 00000000000606e0
[12395.337259] Call Trace:
[12395.337792] ? __inode_security_revalidate+0x63/0x80
[12395.338762] gss_proc_write_secinit+0x14/0x60 [ptlrpc_gss]
[12395.339834] full_proxy_write+0x53/0x80
[12395.340616] vfs_write+0xa5/0x1b0
[12395.341305] ksys_write+0x4f/0xb0
[12395.341955] do_syscall_64+0x5b/0x1b0
[12395.342707] entry_SYSCALL_64_after_hwframe+0x61/0xc6
[12395.343698] RIP: 0033:0x7f9789d8e9e5
[12395.344408] Code: 00 00 75 05 48 83 c4 58 c3 e8 27 4a ff ff 0f 1f 80 00 00 00 00 f3 0f 1e fa 8b 05 66 da 20 00 85 c0 75 12 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 53 c3 66 90 41 54 49 89 d4 55 48 89 f5 53 89
[12395.347714] RSP: 002b:00007ffc8a7475e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
[12395.349109] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f9789d8e9e5
[12395.350433] RDX: 0000000000000050 RSI: 00007ffc8a747680 RDI: 0000000000000004
[12395.351741] RBP: 00007ffc8a7475f0 R08: 00000000024c9fb0 R09: 00000000024c2018
[12395.353061] R10: 0000000000000018 R11: 0000000000000246 R12: 0000000000000004
[12395.354366] R13: 0000000000613640 R14: 00007ffc8a747680 R15: 00007ffc8a749770
[12395.355668] Modules linked in: lustre(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache sunrpc intel_rapl_msr intel_rapl_common crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcspkr joydev virtio_balloon i2c_piix4 ext4 mbcache jbd2 ata_generic crc32c_intel ata_piix libata serio_raw virtio_net virtio_blk net_failover failover
[12395.363302] CR2: 0000000000000020
{noformat}

New: Client crashes with the following:
{noformat}
[12391.117740] Lustre: DEBUG MARKER: mount -t lustre -o user_xattr,flock,test_dummy_encryption,skpath=/tmp/test-framework-keys trevis-66vm6@tcp:/lustre /mnt/lustre
[12392.975436] Lustre: Test dummy encryption mode enabled
[12393.733894] Lustre: DEBUG MARKER: grep -c /mnt/lustre' ' /proc/mounts
[12393.925961] Lustre: DEBUG MARKER: lsof -t /mnt/lustre
[12394.854963] Lustre: DEBUG MARKER: umount /mnt/lustre 2>&1
[12395.289436] LustreError: 588361:0:(gss_cli_upcall.c:296:gss_do_ctx_init_rpc()) import has been deactivated
[12395.301159] Lustre: 588408:0:(gss_keyring.c:1039:flush_spec_ctx_cache_kr()) flush busy ctx 00000000ceae75f0(0->lustre-OST0002_UUID, extra ref 1)
[12395.307688] LustreError: 588365:0:(gss_cli_upcall.c:259:gss_do_ctx_init_rpc()) no such obd lustre-OST0005-osc-ffff95a585165000
[12395.309857] LustreError: 588365:0:(gss_cli_upcall.c:259:gss_do_ctx_init_rpc()) Skipped 6 previous similar messages
[12395.314608] BUG: unable to handle kernel NULL pointer dereference at 0000000000000020
[12395.316095] PGD 8000000040686067 P4D 8000000040686067 PUD 40687067 PMD 0
[12395.317356] Oops: 0000 1 SMP PTI
[12395.318054] CPU: 1 PID: 588359 Comm: lgss_keyring Kdump: loaded Tainted: G OE --------- - - 4.18.0-477.27.1.el8_8.x86_64 #1
[12395.320256] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
[12395.321328] RIP: 0010:gss_do_ctx_init_rpc+0x84e/0x1100 [ptlrpc_gss]
[12395.322576] Code: 8b 44 24 58 89 44 24 08 8b 44 24 54 89 44 24 10 8b 44 24 50 89 44 24 14 e8 cf d4 ec ff 48 89 c1 c6 00 01 48 8b 83 e0 00 00 00 <8b> 40 20 48 c7 41 0c 00 00 00 00 c7 41 20 00 00 00 00 88 41 01 48
[12395.325875] RSP: 0018:ffffb185c2b6fd80 EFLAGS: 00010282
[12395.326853] RAX: 0000000000000000 RBX: ffff95a583476800 RCX: ffff95a5bb4b0030
[12395.328179] RDX: 0000000000000024 RSI: 0000000000000000 RDI: 0000000000000030
[12395.329493] RBP: ffff95a582d4b180 R08: 000000000000002c R09: 0000000000000000
[12395.330796] R10: 0000000000000002 R11: 0000000000000000 R12: 000000000000028f
[12395.332109] R13: 00007ffc8a747680 R14: ffff95a5bb4b0000 R15: 0000000000000002
[12395.333417] FS: 00007f978ac9b840(0000) GS:ffff95a63cd00000(0000) knlGS:0000000000000000
[12395.334879] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[12395.335941] CR2: 0000000000000020 CR3: 0000000039ed4004 CR4: 00000000000606e0
[12395.337259] Call Trace:
[12395.337792] ? __inode_security_revalidate+0x63/0x80
[12395.338762] gss_proc_write_secinit+0x14/0x60 [ptlrpc_gss]
[12395.339834] full_proxy_write+0x53/0x80
[12395.340616] vfs_write+0xa5/0x1b0
[12395.341305] ksys_write+0x4f/0xb0
[12395.341955] do_syscall_64+0x5b/0x1b0
[12395.342707] entry_SYSCALL_64_after_hwframe+0x61/0xc6
[12395.343698] RIP: 0033:0x7f9789d8e9e5
[12395.344408] Code: 00 00 75 05 48 83 c4 58 c3 e8 27 4a ff ff 0f 1f 80 00 00 00 00 f3 0f 1e fa 8b 05 66 da 20 00 85 c0 75 12 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 53 c3 66 90 41 54 49 89 d4 55 48 89 f5 53 89
[12395.347714] RSP: 002b:00007ffc8a7475e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
[12395.349109] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f9789d8e9e5
[12395.350433] RDX: 0000000000000050 RSI: 00007ffc8a747680 RDI: 0000000000000004
[12395.351741] RBP: 00007ffc8a7475f0 R08: 00000000024c9fb0 R09: 00000000024c2018
[12395.353061] R10: 0000000000000018 R11: 0000000000000246 R12: 0000000000000004
[12395.354366] R13: 0000000000613640 R14: 00007ffc8a747680 R15: 00007ffc8a749770
[12395.355668] Modules linked in: lustre(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache sunrpc intel_rapl_msr intel_rapl_common crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcspkr joydev virtio_balloon i2c_piix4 ext4 mbcache jbd2 ata_generic crc32c_intel ata_piix libata serio_raw virtio_net virtio_blk net_failover failover
[12395.363302] CR2: 0000000000000020
{noformat}

Gerrit Updater added a comment - 19/Oct/23 9:18 AM

"Sebastien Buisson <sbuisson@ddn.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/52755
Subject: ~~LU-17212~~ gss: survive improper obd or imp at ctx init
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 22b4d4f8942ce9131962021c7351c1ff96c4b7e0

Gerrit Updater added a comment - 19/Oct/23 9:18 AM "Sebastien Buisson <sbuisson@ddn.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/52755 Subject: LU-17212 gss: survive improper obd or imp at ctx init Project: fs/lustre-release Branch: master Current Patch Set: 1 Commit: 22b4d4f8942ce9131962021c7351c1ff96c4b7e0

Sebastien Buisson made changes - 19/Oct/23 9:12 AM

Link

New: This issue is related to EX-8416 [ EX-8416 ]

Sebastien Buisson created issue - 19/Oct/23 9:11 AM

People

Assignee:: Sebastien Buisson

Reporter:: Sebastien Buisson

Votes:: 0 Vote for this issue

Watchers:: 4 Start watching this issue

Dates

Created:: 19/Oct/23 9:11 AM

Updated:: 29/Apr/24 11:40 PM

Resolved:: 29/Nov/23 10:47 PM