Whamcloud Community JIRA

Whamcloud Community JIRA https://jira.whamcloud.com This file is an XML representation of an issue en-us

9.4.14

940014

05-12-2023

[LU-3289] IU Shared Secret Key authentication and encryption https://jira.whamcloud.com/browse/LU-3289 Lustre <p>Tracking bug for Indiana University's Shared Secret Key authentication and encryption security feature.</p> LU-3289

IU Shared Secret Key authentication and encryption

New Feature Minor Resolved Fixed Jeremy Filizetti Andreas Dilger SSK gssapi patch Tue, 7 May 2013 19:24:10 +0000 Mon, 24 Oct 2016 00:22:20 +0000 Fri, 12 Aug 2016 14:30:21 +0000 Lustre 2.9.0 0 19 <p><a href="http://review.whamcloud.com/8475" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/8475</a></p> <p>Also <a href="http://review.whamcloud.com/#/c/8525/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/8525/</a></p> <p>Scaffolding for shared key security flavors & GSSAPI mechanism: <a href="http://review.whamcloud.com/#/c/8629/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/8629/</a></p> <p>No need for reviewers yet; there's not much here.</p> <p>Shared key test script (also just more scaffolding): <a href="http://review.whamcloud.com/#/c/8649/1" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/8649/1</a></p> <p>Shared key work in progress (for test only): <a href="http://review.whamcloud.com/#/c/9960/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/9960/</a></p> <p>One question that was posed in a past code review of these changes: can the GSSAPI mechanisms refer to the OIDs defined in lustre/utils/gss/gss_oids.c instead of using literal constants? My response is I'm not sure. The GSSAPI mechanisms are in kernel space, whereas that gss_oids.c file is for user-space utilities. I assume it would be difficult to link that file into both. Should I define a macro somewhere that can be included by both?</p> <p>Jeremy Filizetti (jeremy.filizetti@gmail.com) uploaded a new patch: <a href="http://review.whamcloud.com/16728" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/16728</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-3289" title="IU Shared Secret Key authentication and encryption" class="issue-link" data-issue-key="LU-3289"><del>LU-3289</del></a> gss: Interface and code changes for shared key<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 3151d1648fe91f93ef203eb15cdfdc144818e01b</p> <p>Jeremy Filizetti (jeremy.filizetti@gmail.com) uploaded a new patch: <a href="http://review.whamcloud.com/17505" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/17505</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-3289" title="IU Shared Secret Key authentication and encryption" class="issue-link" data-issue-key="LU-3289"><del>LU-3289</del></a> gss: Implment Shared key and GSS Null functionality<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: d74de1f4360a93d7b90fc5764f77a4236b50d40b</p> <p>Jeremy Filizetti (jeremy.filizetti@gmail.com) uploaded a new patch: <a href="http://review.whamcloud.com/17597" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/17597</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-3289" title="IU Shared Secret Key authentication and encryption" class="issue-link" data-issue-key="LU-3289"><del>LU-3289</del></a> gss: Add Shared key and GSS Null functionality<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: ae348ce309a5df6ebfff4007807ca16ab73d0d4c</p> <p>Jeremy Filizetti (jeremy.filizetti@gmail.com) uploaded a new patch: <a href="http://review.whamcloud.com/17598" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/17598</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-3289" title="IU Shared Secret Key authentication and encryption" class="issue-link" data-issue-key="LU-3289"><del>LU-3289</del></a> osp: osp_precreate_thread does not check for errors<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 3c9f66bcd1cfc7c0615589846e3f71ccbe09ed5f</p> <p>Jeremy Filizetti (jeremy.filizetti@gmail.com) uploaded a new patch: <a href="http://review.whamcloud.com/17599" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/17599</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-3289" title="IU Shared Secret Key authentication and encryption" class="issue-link" data-issue-key="LU-3289"><del>LU-3289</del></a> gss: Return GSS major and minor status<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 42630a0a973df10d08c7b0c7ee213af786797ef1</p> <p>Jeremy Filizetti (jeremy.filizetti@gmail.com) uploaded a new patch: <a href="http://review.whamcloud.com/17600" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/17600</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-3289" title="IU Shared Secret Key authentication and encryption" class="issue-link" data-issue-key="LU-3289"><del>LU-3289</del></a> gss: Add userspace support for GSS null and sk<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: f01fd7d7342ef4264ccc7555468684fb8f026065</p> <p>Jeremy Filizetti (jeremy.filizetti@gmail.com) uploaded a new patch: <a href="http://review.whamcloud.com/17601" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/17601</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-3289" title="IU Shared Secret Key authentication and encryption" class="issue-link" data-issue-key="LU-3289"><del>LU-3289</del></a> gss: Add option for loading keys during mount<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 163ff2cb99149ba0c58958db8db14e76bd99fcfe</p> <p>Jeremy Filizetti (jeremy.filizetti@gmail.com) uploaded a new patch: <a href="http://review.whamcloud.com/17602" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/17602</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-3289" title="IU Shared Secret Key authentication and encryption" class="issue-link" data-issue-key="LU-3289"><del>LU-3289</del></a> gss: Cleanup gss print statements and comments<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: c0bd1a87e17db439b2399ed5173c1770d4b9ec73</p> <p>Jeremy, if I pull the latest of your shared key patches then should I be able to setup a shared key FS. I have tried to follow/interpret the directions attached to <a href="https://jira.whamcloud.com/browse/LUDOC-197" title="Complete Lustre Manual updates for IU Shared Secret Key authentication and encryption" class="issue-link" data-issue-key="LUDOC-197"><del>LUDOC-197</del></a> but have not been successful. Can you post a script?</p> <p>I don't have a particular script but here is what I am doing for testing on my local systems:</p> <p>Generate a key and distribute to client and servers:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">lgss_sk -f lustre -w /root/lustre.key </pre> </div></div> <p>MDS/MGS/OSS Server:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">#!/bin/bash # Load key before mount lgss_sk -t mgs -l /root/lustre.key lgss_sk -t server -l /root/lustre.key lgss_sk -t client -l /root/lustre.key modprobe lustre /usr/lib64/lustre/tests/llmount.sh umount /mnt/lustre lctl conf_param lustre.srpc.flavor.<span class="code-keyword">default</span>=skpi sysctl lnet.debug=-2 sysctl lnet.debug_mb=102 sysctl lnet.panic_on_lbug=0 # Run the daemon in foreground /usr/sbin/lsvcgssd -f -vvvvvvv -z -s </pre> </div></div> <p>And on the client(s) after the default security flavor has been set:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">#!/bin/bash modprobe lustre modprobe ptlrpc_gss lctl set_param sptlrpc.gss.lgss_keyring.debug_level=4 sysctl lnet.debug=-2 # Load using lgss_sk and mount lgss_sk -t client -l /root/lustre.key mount -t lustre 192.168.1.107:/lustre /lustre # or load using mount command with skpath #mount -t lustre -o skpath=/root/lustre.key 192.168.1.107:/lustre /lustre # or with mgssec also #mount -t lustre -o mgssec=skpi,skpath=/root/lustre.key 192.168.1.107:/lustre /lustre </pre> </div></div> <p>Note that there is a <tt>MGS_SET_INFO</tt> RPC that an unkeyed client can use to turn skpi off.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">diff --git a/lustre/llite/dir.c b/lustre/llite/dir.c index 142b48d..17109a2 100644 --- a/lustre/llite/dir.c +++ b/lustre/llite/dir.c @@ -413,7 +413,7 @@ out: RETURN(rc); } -<span class="code-keyword">static</span> <span class="code-object">int</span> ll_send_mgc_param(struct obd_export *mgc, <span class="code-object">char</span> *string) +<span class="code-object">int</span> ll_send_mgc_param(struct obd_export *mgc, <span class="code-object">char</span> *string) { struct mgs_send_param *msp; <span class="code-object">int</span> rc = 0; diff --git a/lustre/llite/llite_lib.c b/lustre/llite/llite_lib.c index 050d0e6..88ddb20 100644 --- a/lustre/llite/llite_lib.c +++ b/lustre/llite/llite_lib.c @@ -974,6 +974,8 @@ <span class="code-keyword">static</span> inline <span class="code-object">int</span> ll_bdi_register(struct backing_dev_info *bdi) atomic_inc_return(&ll_bdi_num)); } +<span class="code-object">int</span> ll_send_mgc_param(struct obd_export *mgc, <span class="code-object">char</span> *string); + <span class="code-object">int</span> ll_fill_super(struct super_block *sb, struct vfsmount *mnt) { struct lustre_profile *lprof = NULL; @@ -1003,6 +1005,13 @@ <span class="code-object">int</span> ll_fill_super(struct super_block *sb, struct vfsmount *mnt) RETURN(-ENOMEM); } + err = ll_send_mgc_param(lsi->lsi_mgc->u.cli.cl_mgc_mgsexp, + <span class="code-quote">"lustre.srpc.flavor.<span class="code-keyword">default</span>=<span class="code-keyword">null</span>"</span>); + <span class="code-keyword">if</span> (err < 0) + CERROR(<span class="code-quote">"cannot send MGC param: err = %d\n"</span>, err); + + ssleep(10); + err = ll_options(lsi->lsi_lmd->lmd_opts, &sbi->ll_flags); <span class="code-keyword">if</span> (err) GOTO(out_free, err); </pre> </div></div> <p>Jeremy Filizetti (jeremy.filizetti@gmail.com) uploaded a new patch: <a href="http://review.whamcloud.com/18773" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/18773</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-3289" title="IU Shared Secret Key authentication and encryption" class="issue-link" data-issue-key="LU-3289"><del>LU-3289</del></a> gss: Add two additional security flavors for sk<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 25840598e11424bf5ab2988981b53ed9cd05cb03</p> <p>I got a kernel panic while running the latest series of patches:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">[64306.291569] Lustre: 15188:0:(sec_gss.c:2086:gss_svc_handle_init()) create svc ctx ffff8800007b7840: user from 192.168.122.35@tcp authenticated as root [64306.294194] BUG: unable to handle kernel NULL pointer dereference at (<span class="code-keyword">null</span>) [64306.294332] IP: [<ffffffff812b0cad>] hash_walk_new_entry+0xd/0x50 [64306.295049] PGD 0 [64306.295049] Oops: 0000 [#1] SMP [64306.295049] Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) lov(OE) osc(OE) mdc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) sunrpc mbcache jbd2 sha512_generic crypto_null snd_hda_codec_generic snd_hda_intel crc32_pclmul crc32c_intel snd_hda_codec snd_hda_core ghash_clmulni_intel snd_hwdep snd_seq snd_seq_device snd_pcm ppdev aesni_intel lrw gf128mul glue_helper ablk_helper cryptd snd_timer virtio_balloon pcspkr serio_raw parport_pc snd parport soundcore i2c_piix4 9pnet_virtio(OE) 9p(OE) 9pnet(OE) xfs libcrc32c sd_mod crc_t10dif crct10dif_generic sr_mod cdrom virtio_scsi ata_generic virtio_net virtio_console pata_acpi qxl syscopyarea sysfillrect [64306.295049] sysimgblt drm_kms_helper ttm crct10dif_pclmul crct10dif_common ata_piix drm i2c_core libata virtio_pci virtio_ring virtio floppy dm_mirror dm_region_hash dm_log dm_mod loop [last unloaded: libcfs] [64306.295049] CPU: 0 PID: 15188 Comm: mdt00_002 Tainted: G OE ------------ 3.10.0-327.13.1.el7_lustre.x86_64 #1 [64306.295049] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.8.2-20150714_191134- 04/01/2014 [64306.295049] task: ffff88003dbbdc00 ti: ffff880005d6c000 task.ti: ffff880005d6c000 [64306.295049] RIP: 0010:[<ffffffff812b0cad>] [<ffffffff812b0cad>] hash_walk_new_entry+0xd/0x50 [64306.295049] RSP: 0018:ffff880005d6f9f0 EFLAGS: 00010202 [64306.295049] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 00000000000001dc [64306.295049] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff880005d6fa28 [64306.295049] RBP: ffff880005d6f9f0 R08: 0000000000000000 R09: 00000000e0168ddd [64306.295049] R10: 00000000789f2d9a R11: 00000000ea5dacd6 R12: 0000000000000000 [64306.295049] R13: ffff880005d6fc40 R14: 0000000000000000 R15: 0000000000000000 [64306.295049] FS: 0000000000000000(0000) GS:ffff88003fc00000(0000) knlGS:0000000000000000 [64306.295049] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [64306.295049] CR2: 0000000000000000 CR3: 000000003c5a8000 CR4: 00000000000406f0 [64306.295049] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [64306.295049] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [64306.295049] Stack: [64306.295049] ffff880005d6fa18 ffffffff812b0d64 ffff880005d6fc40 0000000000000000 [64306.295049] ffff88000caf46c0 ffff880005d6fa68 ffffffff812b1e99 ffff88001bfeb000 [64306.295049] 0000000000000f38 ffffea00006ffac0 000001dc00000000 0000000000000000 [64306.295049] Call Trace: [64306.295049] [<ffffffff812b0d64>] crypto_hash_walk_done+0x74/0x110 [64306.295049] [<ffffffff812b1e99>] shash_compat_update+0x59/0x80 [64306.295049] [<ffffffffa0a62ab1>] gss_digest_hmac+0xe1/0x200 [ptlrpc_gss] [64306.295049] [<ffffffffa0a5fb9f>] sk_make_checksum+0x6f/0xe0 [ptlrpc_gss] [64306.295049] [<ffffffffa0a61127>] sk_verify_checksum+0xf7/0x6b0 [ptlrpc_gss] [64306.295049] [<ffffffffa0a6171c>] gss_verify_mic_sk+0x3c/0x40 [ptlrpc_gss] [64306.295049] [<ffffffffa0a59cfe>] lgss_verify_mic+0x2e/0x100 [ptlrpc_gss] [64306.295049] [<ffffffffa0a4590a>] gss_verify_msg+0xda/0x1c0 [ptlrpc_gss] [64306.295049] [<ffffffffa0a49f24>] gss_svc_verify_request+0x124/0x710 [ptlrpc_gss] [64306.295049] [<ffffffffa0a4e1b2>] gss_svc_handle_data+0x3a2/0xa30 [ptlrpc_gss] [64306.295049] [<ffffffff811c11ee>] ? kmem_cache_alloc_trace+0x1ce/0x1f0 [64306.295049] [<ffffffffa0a4eb94>] gss_svc_accept+0x354/0xb00 [ptlrpc_gss] [64306.295049] [<ffffffffa0a635e8>] gss_svc_accept_kr+0x18/0x20 [ptlrpc_gss] [64306.295049] [<ffffffffa08452ae>] sptlrpc_svc_unwrap_request+0xee/0x610 [ptlrpc] [64306.295049] [<ffffffffa08262c4>] ptlrpc_main+0x954/0x1db0 [ptlrpc] [64306.295049] [<ffffffffa0825970>] ? ptlrpc_register_service+0xe40/0xe40 [ptlrpc] [64306.295049] [<ffffffff810a5acf>] kthread+0xcf/0xe0 [64306.295049] [<ffffffff810a5a00>] ? kthread_create_on_node+0x140/0x140 [64306.295049] [<ffffffff81646018>] ret_from_fork+0x58/0x90 [64306.295049] [<ffffffff810a5a00>] ? kthread_create_on_node+0x140/0x140 [64306.295049] Code: 8b 7d d8 4c 01 f0 48 c1 f8 06 48 c1 e0 0c 48 01 d0 eb 93 66 66 2e 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 8b 47 20 48 89 e5 <48> 8b 10 48 83 e2 fc 48 89 57 10 8b 50 08 89 57 08 8b 57 1c 8b [64306.295049] RIP [<ffffffff812b0cad>] hash_walk_new_entry+0xd/0x50 </pre> </div></div> <p>Running that through gdb, gets:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">(gdb) list *(hash_walk_new_entry+0xd) 0xffffffff812b0cad is in hash_walk_new_entry (include/linux/scatterlist.h:101). 96 { 97 #ifdef CONFIG_DEBUG_SG 98 BUG_ON(sg->sg_magic != SG_MAGIC); 99 BUG_ON(sg_is_chain(sg)); 100 #endif 101 <span class="code-keyword">return</span> (struct page *)((sg)->page_link & ~0x3); 102 } 103 104 /** 105 * sg_set_buf - Set sg entry to point at given data </pre> </div></div> <p>I haven't dug into why sg might be null, but I thought I'd post it here to document it.</p> <p>Nathaniel Clark (nathaniel.l.clark@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/20727" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/20727</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-3289" title="IU Shared Secret Key authentication and encryption" class="issue-link" data-issue-key="LU-3289"><del>LU-3289</del></a> gss: Cleanup gss code<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 37b3fa2032b74c1f0eac1121563dca852fc1a31d</p> <p>Got a list_del corruption message after changing the shared key:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">[ 987.235627] Lustre: 2444:0:(tgt_handler.c:894:tgt_init_sec_level()) client 192.168.122.32@tcp -> target test-MDT0000 uses old version, run under security level 0. [ 987.237530] Lustre: 2444:0:(sec_gss.c:2346:gss_svc_handle_destroy()) destroy svc ctx ffff8800112d2e40 idx 0xa62e511956836924 (0->192.168.122.32@tcp) [ 987.700506] Lustre: test-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 1010.991408] Lustre: 2500:0:(sec_gss.c:394:gss_cli_ctx_uptodate()) server installed reverse ctx ffff8800112cd3c0 idx 0xa62e511956836921, expiry 1466276335(+604750s) [ 1010.991421] Lustre: 2500:0:(sec_gss.c:394:gss_cli_ctx_uptodate()) Skipped 2 previous similar messages [ 1010.991438] Lustre: test-MDT0000: Connection restored to 155649eb-9bd7-edb3-7b65-66a22f0de548 (at 0@lo) [ 1010.991443] Lustre: Skipped 2 previous similar messages [ 1012.247047] Lustre: 2500:0:(sec_gss.c:2086:gss_svc_handle_init()) create svc ctx ffff880011a8c240: user from 192.168.122.32@tcp authenticated as oss [ 4560.355291] Lustre: 2416:0:(gss_keyring.c:1038:gss_sec_gc_ctx_kr()) running gc [ 8160.355220] Lustre: 2416:0:(gss_keyring.c:1038:gss_sec_gc_ctx_kr()) running gc [ 9968.634019] Lustre: 2444:0:(sec_gss.c:2346:gss_svc_handle_destroy()) destroy svc ctx ffff8800364b0040 idx 0xa62e51195683692a (0->192.168.122.35@tcp) [ 9968.634023] Lustre: 2444:0:(sec_gss.c:2346:gss_svc_handle_destroy()) Skipped 1 previous similar message [ 9968.802451] Lustre: 2416:0:(sec_gss.c:1245:gss_cli_ctx_fini_common()) reverse sec ffff88003bbc6000: destroy ctx ffff8800112cd3c0 [ 9968.802624] Lustre: Failing over test-MDT0000 [ 9968.803134] Lustre: 2416:0:(gss_cli_upcall.c:400:gss_do_ctx_fini_rpc()) client finishing forward ctx ffff88001129ff00 idx 0xa62e511956836922 (0->test-MDT0000_UUID) [ 9968.803200] Lustre: 2416:0:(sec_gss.c:1249:gss_cli_ctx_fini_common()) gss.keyring@ffff88003db67300: destroy ctx ffff88001129ff00(0->test-MDT0000_UUID) [ 9968.866112] ------------[ cut here ]------------ [ 9968.866121] WARNING: at lib/list_debug.c:59 __list_del_entry+0xa1/0xd0() [ 9968.866123] list_del corruption. prev->next should be ffff88003c2e7338, but was ffff88003db67338 [ 9968.866124] Modules linked in: ptlrpc_gss(OE) osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) sunrpc mbcache jbd2 sha512_generic crypto_null snd_hda_codec_generic crc32_pclmul crc32c_intel ppdev ghash_clmulni_intel snd_hda_intel snd_hda_codec snd_hda_core snd_hwdep aesni_intel lrw gf128mul glue_helper ablk_helper cryptd snd_seq snd_seq_device serio_raw pcspkr virtio_balloon snd_pcm parport_pc parport snd_timer snd soundcore i2c_piix4 9pnet_virtio(OE) 9p(OE) 9pnet(OE) xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi virtio_net virtio_console virtio_scsi qxl syscopyarea sysfillrect sysimgblt drm_kms_helper ttm ata_piix crct10dif_pclmul crct10dif_common [ 9968.866177] virtio_pci virtio_ring virtio drm i2c_core libata floppy dm_mirror dm_region_hash dm_log dm_mod loop [last unloaded: libcfs] [ 9968.866189] CPU: 0 PID: 2717 Comm: umount Tainted: G W OE ------------ 3.10.0-327.13.1.el7_lustre.x86_64 #1 [ 9968.866191] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.8.2-20150714_191134- 04/01/2014 [ 9968.866193] ffff88003633f520 00000000e3385889 ffff88003633f4d8 ffffffff81635924 [ 9968.866197] ffff88003633f510 ffffffff8107b1e0 ffff88003c2e7300 ffff88003c2e7338 [ 9968.866200] ffff88003af7a338 ffff88003af79eb0 ffff88001dc56000 ffff88003633f578 [ 9968.866204] Call Trace: [ 9968.866211] [<ffffffff81635924>] dump_stack+0x19/0x1b [ 9968.866216] [<ffffffff8107b1e0>] warn_slowpath_common+0x70/0xb0 [ 9968.866219] [<ffffffff8107b27c>] warn_slowpath_fmt+0x5c/0x80 [ 9968.866276] [<ffffffffa081bf3c>] ? ptlrpc_invalidate_import+0x23c/0x8e0 [ptlrpc] [ 9968.866280] [<ffffffff8130c721>] __list_del_entry+0xa1/0xd0 [ 9968.866323] [<ffffffffa0837579>] sptlrpc_gc_del_sec+0x39/0xf0 [ptlrpc] [ 9968.866362] [<ffffffffa082c3e9>] sptlrpc_sec_put+0x29/0x70 [ptlrpc] [ 9968.866402] [<ffffffffa083020a>] sptlrpc_import_sec_put+0x2a/0x40 [ptlrpc] [ 9968.866435] [<ffffffffa07bdc52>] client_destroy_import+0x22/0x30 [ptlrpc] [ 9968.866468] [<ffffffffa07befcc>] client_obd_cleanup+0x8c/0x3c0 [ptlrpc] [ 9968.866479] [<ffffffffa0d902f4>] osp_device_fini+0x244/0x540 [osp] [ 9968.866515] [<ffffffffa05a638c>] class_cleanup+0x8dc/0xd70 [obdclass] [ 9968.866541] [<ffffffffa05a8d64>] class_process_config+0x1e24/0x2f60 [obdclass] [ 9968.866544] [<ffffffff811c1f53>] ? __kmalloc+0x1f3/0x230 [ 9968.866567] [<ffffffffa05a228b>] ? lustre_cfg_new+0x8b/0x400 [obdclass] [ 9968.866590] [<ffffffffa05a9f8f>] class_manual_cleanup+0xef/0x810 [obdclass] [ 9968.866599] [<ffffffffa0d8f268>] osp_obd_disconnect+0x178/0x210 [osp] [ 9968.866611] [<ffffffffa0cdaf48>] lod_putref+0x258/0x960 [lod] [ 9968.866620] [<ffffffffa0cdd0dd>] lod_fini_tgt+0xdd/0x3a0 [lod] [ 9968.866630] [<ffffffffa0cd1c70>] lod_device_fini+0x60/0x1f0 [lod] [ 9968.866652] [<ffffffffa05a638c>] class_cleanup+0x8dc/0xd70 [obdclass] [ 9968.866672] [<ffffffffa05a8d64>] class_process_config+0x1e24/0x2f60 [obdclass] [ 9968.866676] [<ffffffff811c1f53>] ? __kmalloc+0x1f3/0x230 [ 9968.866697] [<ffffffffa05a228b>] ? lustre_cfg_new+0x8b/0x400 [obdclass] [ 9968.866718] [<ffffffffa05a9f8f>] class_manual_cleanup+0xef/0x810 [obdclass] [ 9968.866730] [<ffffffffa0cd1bff>] lod_obd_disconnect+0x1bf/0x1d0 [lod] [ 9968.866740] [<ffffffffa0d3a0ba>] mdd_process_config+0x2ba/0x5c0 [mdd] [ 9968.866754] [<ffffffffa0bfa743>] mdt_stack_fini+0x1b3/0xe00 [mdt] [ 9968.866766] [<ffffffffa0bfbab6>] mdt_device_fini+0x726/0xfc0 [mdt] [ 9968.866788] [<ffffffffa05a638c>] class_cleanup+0x8dc/0xd70 [obdclass] [ 9968.866821] [<ffffffffa05a8d64>] class_process_config+0x1e24/0x2f60 [obdclass] [ 9968.866825] [<ffffffff811c1f53>] ? __kmalloc+0x1f3/0x230 [ 9968.866851] [<ffffffffa05a228b>] ? lustre_cfg_new+0x8b/0x400 [obdclass] [ 9968.866872] [<ffffffffa05a9f8f>] class_manual_cleanup+0xef/0x810 [obdclass] [ 9968.866900] [<ffffffffa05da60e>] server_put_super+0x8de/0xcd0 [obdclass] [ 9968.866904] [<ffffffff811e0ad6>] generic_shutdown_super+0x56/0xe0 [ 9968.866907] [<ffffffff811e0eb2>] kill_anon_super+0x12/0x20 [ 9968.866929] [<ffffffffa05ad6f2>] lustre_kill_super+0x32/0x50 [obdclass] [ 9968.866933] [<ffffffff811e1269>] deactivate_locked_super+0x49/0x60 [ 9968.866936] [<ffffffff811e1866>] deactivate_super+0x46/0x60 [ 9968.866939] [<ffffffff811fe7d5>] mntput_no_expire+0xc5/0x120 [ 9968.866943] [<ffffffff811ff90f>] SyS_umount+0x9f/0x3c0 [ 9968.866947] [<ffffffff816460c9>] system_call_fastpath+0x16/0x1b [ 9968.866950] ---[ end trace 8136914c4f134ff8 ]--- [ 9968.867903] ------------[ cut here ]------------ </pre> </div></div> <p>Kit, can you please file those problems you hit as separate issues linked to this one, so that they can be tracked separately. </p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/16728/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/16728/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-3289" title="IU Shared Secret Key authentication and encryption" class="issue-link" data-issue-key="LU-3289"><del>LU-3289</del></a> gss: Interface and code changes for shared key<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 9a163f389e96f8c3682391f8aa967a48e3daaaf4</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/17597/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/17597/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-3289" title="IU Shared Secret Key authentication and encryption" class="issue-link" data-issue-key="LU-3289"><del>LU-3289</del></a> gss: Add Shared key and GSS Null functionality<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 49fd10ec8a2a85bbc56618d70f2f71d323d90a4c</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/17598/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/17598/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-3289" title="IU Shared Secret Key authentication and encryption" class="issue-link" data-issue-key="LU-3289"><del>LU-3289</del></a> osp: osp_precreate_thread does not check for errors<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 07146e09be0478b4f8293ed7065b2722c85b1659</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/17599/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/17599/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-3289" title="IU Shared Secret Key authentication and encryption" class="issue-link" data-issue-key="LU-3289"><del>LU-3289</del></a> gss: Return GSS major and minor status<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: cc45b429c83c0506169dbdcdbe7d917900981664</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/17600/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/17600/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-3289" title="IU Shared Secret Key authentication and encryption" class="issue-link" data-issue-key="LU-3289"><del>LU-3289</del></a> gss: Add userspace support for GSS null and sk<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 3565394baa9589e0db190184e00f43689d6a4150</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/17601/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/17601/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-3289" title="IU Shared Secret Key authentication and encryption" class="issue-link" data-issue-key="LU-3289"><del>LU-3289</del></a> gss: Add option for loading keys during mount<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 09898fdea91045c7414a7b326811b819c670263c</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/17602/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/17602/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-3289" title="IU Shared Secret Key authentication and encryption" class="issue-link" data-issue-key="LU-3289"><del>LU-3289</del></a> gss: Cleanup gss print statements and comments<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 61d492b8bab829e401c32583ee9d86b0655a95bc</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/18773/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/18773/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-3289" title="IU Shared Secret Key authentication and encryption" class="issue-link" data-issue-key="LU-3289"><del>LU-3289</del></a> gss: Add two additional security flavors for sk<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 7dd6c394161a62b229ba65f50044a1d8f2da2d03</p> <p>I created <a href="https://jira.whamcloud.com/browse/LU-8269" title="Kernel panic with shared secret key" class="issue-link" data-issue-key="LU-8269"><del>LU-8269</del></a> and <a href="https://jira.whamcloud.com/browse/LU-8270" title="List corruption with Shared Secret Key enabled" class="issue-link" data-issue-key="LU-8270"><del>LU-8270</del></a> to track the two issues reported</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/20727/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/20727/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-3289" title="IU Shared Secret Key authentication and encryption" class="issue-link" data-issue-key="LU-3289"><del>LU-3289</del></a> gss: Cleanup gss code<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: ff4357229efe87781e65382c20d3d718ecc3114d</p> <p>Jeremy Filizetti (jeremy.filizetti@gmail.com) uploaded a new patch: <a href="http://review.whamcloud.com/21491" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/21491</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-3289" title="IU Shared Secret Key authentication and encryption" class="issue-link" data-issue-key="LU-3289"><del>LU-3289</del></a> gss: Fix for SK bulk HMACs<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 6fd7f577bcdaca05554c7496d9cae7f2e69de3fc</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/21491/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/21491/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-3289" title="IU Shared Secret Key authentication and encryption" class="issue-link" data-issue-key="LU-3289"><del>LU-3289</del></a> gss: Fix for SK bulk HMACs<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: f0e2fc455895e6fd780be34fe79a2a198fd1651f</p> <p>Landed for 2.9</p> <p>Jeremy Filizetti (jeremy.filizetti@gmail.com) uploaded a new patch: <a href="http://review.whamcloud.com/21922" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/21922</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-3289" title="IU Shared Secret Key authentication and encryption" class="issue-link" data-issue-key="LU-3289"><del>LU-3289</del></a> gss: Fix issues with SK privacy and integrity mode<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: c1f9e81eefb7e62d3e6c07af8d9636351d5377b2</p> <p>Ugh. While I was adding support for the 4.6 version of linux this gss work totally breaks due to struct crypto_hash being removed. Now this can be approached in two ways. One is move to struct crypto_ahash or instead use the libcfs crypto api instead. Well will need to pick one to move forward. </p> <p>The scope of Lustre 2.9 does not include support for such new kernels; we will defer any work for supporting 4.6 until Lustre 2.10.</p> <p>I can do the work <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/> The question is what approach should be done? Do I use the libcfs crypto API's or the kernel internal APIs. Their are pros and cons to both approaches. The main difference being that libcfs crypto is limited to the number of algo's it supports and the digest size. If I remembering right libcfs crypto API's limits the digest to 64 bytes whereas the kernel's api can support up to 2K if I remember right. </p> <p>Since this affects the network protocol compatibility, I don't think we should allow arbitrary crypto algorithms for SSK. I argued in the past that we shouldn't even allow old algorithms like 3DES for Kerberos, but this might be needed for backward compatibility. For SSK the current plan is to allow only AES with varying key sizes. We'll need more input from Jeremy before this moves forward.</p> <p>I just noticed that this ticket is closed, since the SSK code is landed for 2.9.0.</p> <p>James, can you please file a separate ticket for the crypto API changes for 4.6 so that this can be tracked separately.</p> <p>It really sounds to me like sometihng to spin out into a separate ticket</p> <p>I created ticket <a href="https://jira.whamcloud.com/browse/LU-8602" title="Support GSS crypto code with linux 4.6 kernels" class="issue-link" data-issue-key="LU-8602"><del>LU-8602</del></a> for the crypto API changes in 4.6+ kernels.</p> <p>Jeremy Filizetti (jeremy.filizetti@gmail.com) uploaded a new patch: <a href="http://review.whamcloud.com/22626" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/22626</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-3289" title="IU Shared Secret Key authentication and encryption" class="issue-link" data-issue-key="LU-3289"><del>LU-3289</del></a> gss: Change the handling of keys for SK<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 8ae0caabad096181b21ba7d7013662f928327738</p> <p>Jeremy Filizetti (jeremy.filizetti@gmail.com) uploaded a new patch: <a href="http://review.whamcloud.com/22801" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/22801</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-3289" title="IU Shared Secret Key authentication and encryption" class="issue-link" data-issue-key="LU-3289"><del>LU-3289</del></a> gss: Add autoconf check to prevent GSS with SLES11<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 2807d42f2dcead6092d4ffc58c1aaf9631e41253</p> <p>Andreas Dilger (andreas.dilger@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/22806" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/22806</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-3289" title="IU Shared Secret Key authentication and encryption" class="issue-link" data-issue-key="LU-3289"><del>LU-3289</del></a> gss: don't build SSK if libssl-1.0+ not available<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 3d750b41a6892e5b19f3bab4de75ca4dd0b6c033</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/22806/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/22806/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-3289" title="IU Shared Secret Key authentication and encryption" class="issue-link" data-issue-key="LU-3289"><del>LU-3289</del></a> gss: don't build SSK if libssl-1.0+ not available<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: bbee5d1ae941a208d7a07d0348e835ab58ca90ce</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/22626/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/22626/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-3289" title="IU Shared Secret Key authentication and encryption" class="issue-link" data-issue-key="LU-3289"><del>LU-3289</del></a> gss: Change the handling of keys for SK<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 7b596ad0b36ec5f0281be368b67b7e624457de18</p> <p>Andreas Dilger (andreas.dilger@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/23089" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/23089</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-3289" title="IU Shared Secret Key authentication and encryption" class="issue-link" data-issue-key="LU-3289"><del>LU-3289</del></a> ssk: fix SK_IV_REV_START on 32-bit systems<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 118a53cf2f0d6aa2fc9afbbe65d2361f8e371676</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/21922/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/21922/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-3289" title="IU Shared Secret Key authentication and encryption" class="issue-link" data-issue-key="LU-3289"><del>LU-3289</del></a> gss: Fix issues with SK privacy and integrity mode<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 462037d5d302f408b840d2543de89b22f8eb7b73</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/23089/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/23089/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-3289" title="IU Shared Secret Key authentication and encryption" class="issue-link" data-issue-key="LU-3289"><del>LU-3289</del></a> ssk: fix SK_IV_REV_START on 32-bit systems<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: bdb254df1060f1ad4b21d53396b5b4014c926758</p> <p>Jeremy Filizetti (jeremy.filizetti@gmail.com) uploaded a new patch: <a href="http://review.whamcloud.com/23322" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/23322</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-3289" title="IU Shared Secret Key authentication and encryption" class="issue-link" data-issue-key="LU-3289"><del>LU-3289</del></a> gss: Move DH parameter generation out of upcall<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 9fbbcc2448661ed944d1a06ab5184bcdbca5b964</p> Blocker LU-5951 LU-7508 Related LU-8590 LU-6356 LU-4371 LU-4764 LU-8275 LU-3290 LU-4625 LU-7661 LU-7182 LU-7184 LU-7185 LU-7183 LUDOC-197 LU-3778 LU-8229 LU-8602 LU-3288 LU-3490 LU-3778 Development Rank 1|hzvqcv: Rank (Obsolete) 8147