Whamcloud Community JIRA https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 [LU-8270] List corruption with Shared Secret Key enabled https://jira.whamcloud.com/browse/LU-8270 Lustre <p>Got a list_del corruption message after changing the shared key:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">[ 987.235627] Lustre: 2444:0:(tgt_handler.c:894:tgt_init_sec_level()) client 192.168.122.32@tcp -&gt; target test-MDT0000 uses old version, run under security level 0. [ 987.237530] Lustre: 2444:0:(sec_gss.c:2346:gss_svc_handle_destroy()) destroy svc ctx ffff8800112d2e40 idx 0xa62e511956836924 (0-&gt;192.168.122.32@tcp) [ 987.700506] Lustre: test-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 1010.991408] Lustre: 2500:0:(sec_gss.c:394:gss_cli_ctx_uptodate()) server installed reverse ctx ffff8800112cd3c0 idx 0xa62e511956836921, expiry 1466276335(+604750s) [ 1010.991421] Lustre: 2500:0:(sec_gss.c:394:gss_cli_ctx_uptodate()) Skipped 2 previous similar messages [ 1010.991438] Lustre: test-MDT0000: Connection restored to 155649eb-9bd7-edb3-7b65-66a22f0de548 (at 0@lo) [ 1010.991443] Lustre: Skipped 2 previous similar messages [ 1012.247047] Lustre: 2500:0:(sec_gss.c:2086:gss_svc_handle_init()) create svc ctx ffff880011a8c240: user from 192.168.122.32@tcp authenticated as oss [ 4560.355291] Lustre: 2416:0:(gss_keyring.c:1038:gss_sec_gc_ctx_kr()) running gc [ 8160.355220] Lustre: 2416:0:(gss_keyring.c:1038:gss_sec_gc_ctx_kr()) running gc [ 9968.634019] Lustre: 2444:0:(sec_gss.c:2346:gss_svc_handle_destroy()) destroy svc ctx ffff8800364b0040 idx 0xa62e51195683692a (0-&gt;192.168.122.35@tcp) [ 9968.634023] Lustre: 2444:0:(sec_gss.c:2346:gss_svc_handle_destroy()) Skipped 1 previous similar message [ 9968.802451] Lustre: 2416:0:(sec_gss.c:1245:gss_cli_ctx_fini_common()) reverse sec ffff88003bbc6000: destroy ctx ffff8800112cd3c0 [ 9968.802624] Lustre: Failing over test-MDT0000 [ 9968.803134] Lustre: 2416:0:(gss_cli_upcall.c:400:gss_do_ctx_fini_rpc()) client finishing forward ctx ffff88001129ff00 idx 0xa62e511956836922 (0-&gt;test-MDT0000_UUID) [ 9968.803200] Lustre: 2416:0:(sec_gss.c:1249:gss_cli_ctx_fini_common()) gss.keyring@ffff88003db67300: destroy ctx ffff88001129ff00(0-&gt;test-MDT0000_UUID) [ 9968.866112] ------------[ cut here ]------------ [ 9968.866121] WARNING: at lib/list_debug.c:59 __list_del_entry+0xa1/0xd0() [ 9968.866123] list_del corruption. prev-&gt;next should be ffff88003c2e7338, but was ffff88003db67338 [ 9968.866124] Modules linked in: ptlrpc_gss(OE) osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) sunrpc mbcache jbd2 sha512_generic crypto_null snd_hda_codec_generic crc32_pclmul crc32c_intel ppdev ghash_clmulni_intel snd_hda_intel snd_hda_codec snd_hda_core snd_hwdep aesni_intel lrw gf128mul glue_helper ablk_helper cryptd snd_seq snd_seq_device serio_raw pcspkr virtio_balloon snd_pcm parport_pc parport snd_timer snd soundcore i2c_piix4 9pnet_virtio(OE) 9p(OE) 9pnet(OE) xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi virtio_net virtio_console virtio_scsi qxl syscopyarea sysfillrect sysimgblt drm_kms_helper ttm ata_piix crct10dif_pclmul crct10dif_common [ 9968.866177] virtio_pci virtio_ring virtio drm i2c_core libata floppy dm_mirror dm_region_hash dm_log dm_mod loop [last unloaded: libcfs] [ 9968.866189] CPU: 0 PID: 2717 Comm: umount Tainted: G W OE ------------ 3.10.0-327.13.1.el7_lustre.x86_64 #1 [ 9968.866191] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.8.2-20150714_191134- 04/01/2014 [ 9968.866193] ffff88003633f520 00000000e3385889 ffff88003633f4d8 ffffffff81635924 [ 9968.866197] ffff88003633f510 ffffffff8107b1e0 ffff88003c2e7300 ffff88003c2e7338 [ 9968.866200] ffff88003af7a338 ffff88003af79eb0 ffff88001dc56000 ffff88003633f578 [ 9968.866204] Call Trace: [ 9968.866211] [&lt;ffffffff81635924&gt;] dump_stack+0x19/0x1b [ 9968.866216] [&lt;ffffffff8107b1e0&gt;] warn_slowpath_common+0x70/0xb0 [ 9968.866219] [&lt;ffffffff8107b27c&gt;] warn_slowpath_fmt+0x5c/0x80 [ 9968.866276] [&lt;ffffffffa081bf3c&gt;] ? ptlrpc_invalidate_import+0x23c/0x8e0 [ptlrpc] [ 9968.866280] [&lt;ffffffff8130c721&gt;] __list_del_entry+0xa1/0xd0 [ 9968.866323] [&lt;ffffffffa0837579&gt;] sptlrpc_gc_del_sec+0x39/0xf0 [ptlrpc] [ 9968.866362] [&lt;ffffffffa082c3e9&gt;] sptlrpc_sec_put+0x29/0x70 [ptlrpc] [ 9968.866402] [&lt;ffffffffa083020a&gt;] sptlrpc_import_sec_put+0x2a/0x40 [ptlrpc] [ 9968.866435] [&lt;ffffffffa07bdc52&gt;] client_destroy_import+0x22/0x30 [ptlrpc] [ 9968.866468] [&lt;ffffffffa07befcc&gt;] client_obd_cleanup+0x8c/0x3c0 [ptlrpc] [ 9968.866479] [&lt;ffffffffa0d902f4&gt;] osp_device_fini+0x244/0x540 [osp] [ 9968.866515] [&lt;ffffffffa05a638c&gt;] class_cleanup+0x8dc/0xd70 [obdclass] [ 9968.866541] [&lt;ffffffffa05a8d64&gt;] class_process_config+0x1e24/0x2f60 [obdclass] [ 9968.866544] [&lt;ffffffff811c1f53&gt;] ? __kmalloc+0x1f3/0x230 [ 9968.866567] [&lt;ffffffffa05a228b&gt;] ? lustre_cfg_new+0x8b/0x400 [obdclass] [ 9968.866590] [&lt;ffffffffa05a9f8f&gt;] class_manual_cleanup+0xef/0x810 [obdclass] [ 9968.866599] [&lt;ffffffffa0d8f268&gt;] osp_obd_disconnect+0x178/0x210 [osp] [ 9968.866611] [&lt;ffffffffa0cdaf48&gt;] lod_putref+0x258/0x960 [lod] [ 9968.866620] [&lt;ffffffffa0cdd0dd&gt;] lod_fini_tgt+0xdd/0x3a0 [lod] [ 9968.866630] [&lt;ffffffffa0cd1c70&gt;] lod_device_fini+0x60/0x1f0 [lod] [ 9968.866652] [&lt;ffffffffa05a638c&gt;] class_cleanup+0x8dc/0xd70 [obdclass] [ 9968.866672] [&lt;ffffffffa05a8d64&gt;] class_process_config+0x1e24/0x2f60 [obdclass] [ 9968.866676] [&lt;ffffffff811c1f53&gt;] ? __kmalloc+0x1f3/0x230 [ 9968.866697] [&lt;ffffffffa05a228b&gt;] ? lustre_cfg_new+0x8b/0x400 [obdclass] [ 9968.866718] [&lt;ffffffffa05a9f8f&gt;] class_manual_cleanup+0xef/0x810 [obdclass] [ 9968.866730] [&lt;ffffffffa0cd1bff&gt;] lod_obd_disconnect+0x1bf/0x1d0 [lod] [ 9968.866740] [&lt;ffffffffa0d3a0ba&gt;] mdd_process_config+0x2ba/0x5c0 [mdd] [ 9968.866754] [&lt;ffffffffa0bfa743&gt;] mdt_stack_fini+0x1b3/0xe00 [mdt] [ 9968.866766] [&lt;ffffffffa0bfbab6&gt;] mdt_device_fini+0x726/0xfc0 [mdt] [ 9968.866788] [&lt;ffffffffa05a638c&gt;] class_cleanup+0x8dc/0xd70 [obdclass] [ 9968.866821] [&lt;ffffffffa05a8d64&gt;] class_process_config+0x1e24/0x2f60 [obdclass] [ 9968.866825] [&lt;ffffffff811c1f53&gt;] ? __kmalloc+0x1f3/0x230 [ 9968.866851] [&lt;ffffffffa05a228b&gt;] ? lustre_cfg_new+0x8b/0x400 [obdclass] [ 9968.866872] [&lt;ffffffffa05a9f8f&gt;] class_manual_cleanup+0xef/0x810 [obdclass] [ 9968.866900] [&lt;ffffffffa05da60e&gt;] server_put_super+0x8de/0xcd0 [obdclass] [ 9968.866904] [&lt;ffffffff811e0ad6&gt;] generic_shutdown_super+0x56/0xe0 [ 9968.866907] [&lt;ffffffff811e0eb2&gt;] kill_anon_super+0x12/0x20 [ 9968.866929] [&lt;ffffffffa05ad6f2&gt;] lustre_kill_super+0x32/0x50 [obdclass] [ 9968.866933] [&lt;ffffffff811e1269&gt;] deactivate_locked_super+0x49/0x60 [ 9968.866936] [&lt;ffffffff811e1866&gt;] deactivate_super+0x46/0x60 [ 9968.866939] [&lt;ffffffff811fe7d5&gt;] mntput_no_expire+0xc5/0x120 [ 9968.866943] [&lt;ffffffff811ff90f&gt;] SyS_umount+0x9f/0x3c0 [ 9968.866947] [&lt;ffffffff816460c9&gt;] system_call_fastpath+0x16/0x1b [ 9968.866950] ---[ end trace 8136914c4f134ff8 ]--- [ 9968.867903] ------------[ cut here ]------------ </pre> </div></div> LU-8270 List corruption with Shared Secret Key enabled Bug Minor Resolved Fixed John Hammond Kit Westneat Tue, 14 Jun 2016 12:22:04 +0000 Sat, 1 Oct 2016 00:39:40 +0000 Fri, 8 Jul 2016 20:31:59 +0000 Lustre 2.9.0 Lustre 2.9.0 0 4 <p>This is because we use the wrong parameter order when calling <tt>list_add_tail()</tt> from <tt>sptlrpc_gc_add_sec()</tt>:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">void sptlrpc_gc_add_sec(struct ptlrpc_sec *sec) { LASSERT(sec-&gt;ps_policy-&gt;sp_cops-&gt;gc_ctx); LASSERT(sec-&gt;ps_gc_interval &gt; 0); LASSERT(list_empty(&amp;sec-&gt;ps_gc_list)); sec-&gt;ps_gc_next = cfs_time_current_sec() + sec-&gt;ps_gc_interval; spin_lock(&amp;sec_gc_list_lock); list_add_tail(&amp;sec_gc_list, &amp;sec-&gt;ps_gc_list); spin_unlock(&amp;sec_gc_list_lock); </pre> </div></div> <p>John L. Hammond (john.hammond@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/20784" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/20784</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8270" title="List corruption with Shared Secret Key enabled" class="issue-link" data-issue-key="LU-8270"><del>LU-8270</del></a> ptlrpc: correct use of list_add_tail()<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: bc9886a836f07266c7f6d62cd0d801276a58b639</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/20784/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/20784/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8270" title="List corruption with Shared Secret Key enabled" class="issue-link" data-issue-key="LU-8270"><del>LU-8270</del></a> ptlrpc: correct use of list_add_tail()<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 0d7022d7eee3f36dfda0e0e58a531954de3ba335</p> <p>Landed to master for 2.9.0</p> Related Development Rank 1|hzyelj: Rank (Obsolete) 9223372036854775807 Severity