https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-11986 Lustre <p>While investigating <a href="https://jira.whamcloud.com/browse/LU-9525" title="&quot;lctl network down&quot; won&#39;t work if network brought up by a mount" class="issue-link" data-issue-key="LU-9525"><del>LU-9525</del></a> I ran across this behavior.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># modprobe osd-zfs # lctl list_nids 192.168.56.20@tcp # lustre_rmmod lnet </pre> </div></div> <p>Wait a minute for (I assume) iml-agent to try to get lnet status and the following panic results:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[ 68.383777] BUG: unable to handle kernel paging request at ffffffffc0959ca0 [ 68.388689] IP: [&lt;ffffffffc089b654&gt;] lnet_debugfs_read+0x24/0x40 [libcfs] [ 68.391817] PGD 11c14067 PUD 11c16067 PMD 16472067 PTE 0 [ 68.402787] Oops: 0000 [#1] SMP [ 68.404489] Modules linked in: libcfs(OE) nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi sunrpc zfs(POE) zunicode(POE) zavl(POE) icp(POE) ppdev iosf_mbi crc32_pclmul zcommon(POE) znvpair(POE) ghash_clmulni_intel spl(OE) aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr sg parport_pc parport video i2c_piix4 ip_tables ext4 mbcache jbd2 sr_mod cdrom ata_generic sd_mod crc_t10dif crct10dif_generic pata_acpi crct10dif_pclmul crct10dif_common crc32c_intel serio_raw ahci ata_piix libahci libata e1000 dm_mirror dm_region_hash dm_log dm_mod [last unloaded: lnet] [ 68.438921] CPU: 1 PID: 4853 Comm: lctl Tainted: P OE ------------ 3.10.0-957.el7_lustre.x86_64 #1 [ 68.444438] Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006 [ 68.448776] task: ffff893597914100 ti: ffff893596f44000 task.ti: ffff893596f44000 [ 68.452887] RIP: 0010:[&lt;ffffffffc089b654&gt;] [&lt;ffffffffc089b654&gt;] lnet_debugfs_read+0x24/0x40 [libcfs] [ 68.458006] RSP: 0018:ffff893596f47ec8 EFLAGS: 00010246 [ 68.460928] RAX: ffffffffc089b630 RBX: ffff89359ead7900 RCX: ffff893596f47ec8 [ 68.465870] RDX: 0000000000cd50c0 RSI: 0000000000000000 RDI: ffffffffc0959c80 [ 68.475879] RBP: ffff893596f47ed0 R08: ffff893596f47f18 R09: 0000000000000000 [ 68.479744] R10: 00007fff4e9a0f60 R11: 0000000000000246 R12: 0000000000cd50c0 [ 68.483282] R13: ffff893596f47f18 R14: 0000000000001000 R15: 0000000000000000 [ 68.487155] FS: 00007fd435e89740(0000) GS:ffff89359fd00000(0000) knlGS:0000000000000000 [ 68.492060] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 68.495707] CR2: ffffffffc0959ca0 CR3: 0000000017940000 CR4: 00000000000606e0 [ 68.499746] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 68.503180] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 68.507283] Call Trace: [ 68.508720] [&lt;ffffffff94640f0f&gt;] vfs_read+0x9f/0x170 [ 68.511351] [&lt;ffffffff94641dcf&gt;] SyS_read+0x7f/0xf0 [ 68.513924] [&lt;ffffffff94b74d21&gt;] ? system_call_after_swapgs+0xae/0x146 [ 68.519936] [&lt;ffffffff94b74ddb&gt;] system_call_fastpath+0x22/0x27 [ 68.524797] [&lt;ffffffff94b74d21&gt;] ? system_call_after_swapgs+0xae/0x146 [ 68.529162] Code: 5b 41 5c 41 5d 5d c3 0f 1f 44 00 00 55 49 89 c8 48 89 e5 48 83 ec 08 48 8b bf a8 00 00 00 48 89 55 f8 48 8d 4d f8 48 89 f2 31 f6 &lt;48&gt; 8b 47 20 e8 23 b5 ee d3 48 98 48 85 c0 48 0f 44 45 f8 c9 c3 [ 68.545905] RIP [&lt;ffffffffc089b654&gt;] lnet_debugfs_read+0x24/0x40 [libcfs] [ 68.549423] RSP &lt;ffff893596f47ec8&gt; [ 68.551714] CR2: ffffffffc0959ca0 [ 68.553976] ---[ end trace 31c5ad0e3a22fb28 ]--- [ 68.556862] Kernel panic - not syncing: Fatal exception [ 68.559764] Kernel Offset: 0x13400000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff) </pre> </div></div> IML 5-devel on VirtualBox <br/> Lustre 2.12.0 LU-11986

After partial lustre_rmmod, lnet panics on debugfs read

Bug Major Resolved Fixed James A Simmons Nathaniel Clark Thu, 21 Feb 2019 16:50:10 +0000 Tue, 8 Sep 2020 18:09:41 +0000 Tue, 8 Sep 2020 18:09:41 +0000 Lustre 2.12.0 Lustre 2.14.0 0 11 <p>Running <tt>lustre_rmmod</tt> with no arguments does not exhibit this behavior.</p> <p>I can also cause a panic by doing the following:</p> <p><tt>lustre_rmmod</tt></p> <p><tt>lctl network down</tt></p> <p> </p> <p><tt><span class="image-wrap" style=""><img src="https://jira.whamcloud.com/secure/attachment/32274/32274_Screen+Shot+2019-03-20+at+12.46.35+PM.png" style="border: 0px solid black" /></span></tt></p> <p>Attached a full console dump if it's useful: <span class="nobr"><a href="https://jira.whamcloud.com/secure/attachment/32275/32275_console-dump.txt" title="console-dump.txt attached to LU-11986">console-dump.txt^{<img class="rendericon" src="https://jira.whamcloud.com/images/icons/link_attachment_7.gif" height="7" width="7" align="absmiddle" alt="" border="0"/>}</a></span></p> <p> </p> <p>Full trace I'm seeing:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> [ 257.222895] CPU: 0 PID: 7331 Comm: lctl Tainted: P OE ------------ 3.10.0-957.el7_lustre.x86_64 #1 [ 257.229312] Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006 [ 257.233659] task: ffff9c9fbaf15140 ti: ffff9c9fbabcc000 task.ti: ffff9c9fbabcc000 [ 257.238388] RIP: 0010:[&lt;ffffffffc0bb95c6&gt;] [&lt;ffffffffc0bb95c6&gt;] cfs_percpt_number+0x6/0x10 [libcfs] [ 257.243851] RSP: 0018:ffff9c9fbabcfdb0 EFLAGS: 00010296 [ 257.246400] RAX: 0000000000000000 RBX: ffff9c9fba2a5200 RCX: 0000000000000000 [ 257.250304] RDX: 0000000000000001 RSI: 00000000ffffffff RDI: 0000000000000000 [ 257.253677] RBP: ffff9c9fbabcfdd0 R08: 000000000001f120 R09: ffff9c9fbe001700 [ 257.257073] R10: ffffffffc0c376db R11: 0000000000000246 R12: 0000000000000000 [ 257.260339] R13: 0000000000000000 R14: 0000000000001000 R15: ffff9c9fba2a5200 [ 257.263204] FS: 00007fbdc89c6740(0000) GS:ffff9c9fbfc00000(0000) knlGS:0000000000000000 [ 257.266409] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 257.269105] CR2: fffffffffffffff0 CR3: 0000000022e36000 CR4: 00000000000606f0 [ 257.272529] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 257.275209] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 257.277936] Call Trace: [ 257.279245] [&lt;ffffffffc0c0a88b&gt;] ? lnet_counters_get_common+0xeb/0x150 [lnet] [ 257.283071] [&lt;ffffffffc0c0a95c&gt;] lnet_counters_get+0x6c/0x150 [lnet] [ 257.286224] [&lt;ffffffffc0c3771b&gt;] __proc_lnet_stats+0xfb/0x810 [lnet] [ 257.288975] [&lt;ffffffffc0ba6602&gt;] lprocfs_call_handler+0x22/0x50 [libcfs] [ 257.292387] [&lt;ffffffffc0c36bf5&gt;] proc_lnet_stats+0x25/0x30 [lnet] [ 257.295184] [&lt;ffffffffc0ba665d&gt;] lnet_debugfs_read+0x2d/0x40 [libcfs] [ 257.298195] [&lt;ffffffffba440f0f&gt;] vfs_read+0x9f/0x170 [ 257.300211] [&lt;ffffffffba441dcf&gt;] SyS_read+0x7f/0xf0 [ 257.302775] [&lt;ffffffffba974d21&gt;] ? system_call_after_swapgs+0xae/0x146 [ 257.305742] [&lt;ffffffffba974ddb&gt;] system_call_fastpath+0x22/0x27 [ 257.308191] [&lt;ffffffffba974d21&gt;] ? system_call_after_swapgs+0xae/0x146 [ 257.310673] Code: 98 03 00 10 2e bf c0 c7 05 8c 98 03 00 00 00 02 00 e8 ff f8 fe ff 8b 1d 49 7a 01 00 e9 e8 f9 ff ff 0f 1f 40 00 0f 1f 44 00 00 55 &lt;8b&gt; 47 f0 48 89 e5 5d c3 66 90 0f 1f 44 00 00 55 48 89 e5 41 57 [ 257.322785] RIP [&lt;ffffffffc0bb95c6&gt;] cfs_percpt_number+0x6/0x10 [libcfs] [ 257.325230] RSP &lt;ffff9c9fbabcfdb0&gt; [ 257.326493] CR2: fffffffffffffff0 [ 257.328279] ---[ end trace 6e448abd7d16753c ]--- [ 257.330671] Kernel panic - not syncing: Fatal exception [ 257.333186] Kernel Offset: 0x39200000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff) </pre> </div></div> <p>Sonia</p> <p>Could you please investigate?</p> <p>Thanks</p> <p>Peter</p> <p>I could not replicate this behavior on lustre-master </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> [root@client udsp]# modprobe lustre [root@client udsp]# lctl list_nids 10.0.2.15@tcp [root@client udsp]# lustre_rmmod lnet [root@client udsp]# </pre> </div></div> <p> </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> [root@oss ~]# modprobe lustre [root@oss ~]# lctl list_nids 10.0.2.15@tcp [root@oss ~]# lustre_rmmod [root@oss ~]# lctl network down opening /dev/lnet failed: No such file or directory hint: the kernel modules may not be loaded LNET unconfigure error 2: No such file or directory [root@oss ~]# </pre> </div></div> <p>Is it something specific that iml code is trying to do which is causing this?</p> <p>You are correct, there was one more step the IML agent was doing. I am able to reproduce this without the IML agent running by doing:</p> <p> </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> lustre_rmmod lctl network down lctl get_param -n stats </pre> </div></div> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=sharmaso" class="user-hover" rel="sharmaso">sharmaso</a>,</p> <p>Where you able to reproduce with Joe's steps above?  We're hitting this in our automated testing for IML pretty regularly.</p> <p>I believe this is due to struct ctl_table disappearing before the debugfs files deregister. I think the solution is to check if table still exist in both lnet_debugfs_read() and lnet_debugfs_write().</p> <p>Sonia Sharma (sharmaso@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/34622" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34622</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11986" title="After partial lustre_rmmod, lnet panics on debugfs read" class="issue-link" data-issue-key="LU-11986"><del>LU-11986</del></a> lnet: Avoid lnet debugfs read/write if ctl_table does not exist<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: dd38cf7edb0d7ecb852c03fdc69db34ff26fe649</p> <p>Sonia Sharma (sharmaso@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/34634" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34634</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11986" title="After partial lustre_rmmod, lnet panics on debugfs read" class="issue-link" data-issue-key="LU-11986"><del>LU-11986</del></a> lnet: Avoid lnet debugfs read/write if ctl_table does not exist<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: b0c272bc69b077f48ab59e67f543b377c0639b7e</p> <p>I may be missing something, but I appear to be getting a panic using the review on the b2_12 branch.</p> <p>repo I am using:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> [lustre] name=Lustre Server baseurl=https:<span class="code-comment">//build.whamcloud.com/jobs-pub/lustre-reviews/configurations/axis-arch/x86_64/axis-build_type/server/axis-distro/el7.6/axis-ib_stack/inkernel/builds/63380/archive/artifacts/ </span>enabled=1 gpgcheck=0 repo_gpgcheck=0 </pre> </div></div> <p>Installed lustre:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> Name : lustre Arch : x86_64 Version : 2.12.0_79_gb0c272b Release : 1.el7 Size : 766 k Repo : lustre Summary : Lustre File <span class="code-object">System</span> URL : https:<span class="code-comment">//wiki.whamcloud.com/ </span>License : GPL Description : Userspace tools and files <span class="code-keyword">for</span> the Lustre file system. </pre> </div></div> <p>panic:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> [ 140.614378] BUG: unable to handle kernel paging request at fffffffffffffff0 [ 140.616028] IP: [&lt;ffffffffc09825f6&gt;] cfs_percpt_number+0x6/0x10 [libcfs] [ 140.617528] PGD 1a614067 PUD 1a616067 PMD 0 [ 140.618546] Oops: 0000 [#1] SMP [ 140.619337] Modules linked in: ksocklnd(OE) lnet(OE) libcfs(OE) ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter sunrpc zfs(POE) zunicode(POE) zavl(POE) icp(POE) ppdev zcommon(POE) znvpair(POE) spl(OE) sb_edac iosf_mbi crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr sg joydev virtio_net cirrus virtio_balloon ttm drm_kms_helper parport_pc parport syscopyarea sysfillrect sysimgblt fb_sys_fops drm i2c_piix4 drm_panel_orientation_quirks ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic virtio_blk ata_generic pata_acpi virtio_scsi crct10dif_pclmul crct10dif_common serio_raw ata_piix libata virtio_pci virtio_ring virtio [ 140.641386] CPU: 0 PID: 4913 Comm: lctl Tainted: P OE ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [ 140.643553] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 [ 140.644779] task: ffff9382edb1c100 ti: ffff9382fb748000 task.ti: ffff9382fb748000 [ 140.646357] RIP: 0010:[&lt;ffffffffc09825f6&gt;] [&lt;ffffffffc09825f6&gt;] cfs_percpt_number+0x6/0x10 [libcfs] [ 140.648304] RSP: 0018:ffff9382fb74bdb0 EFLAGS: 00010296 [ 140.649441] RAX: 0000000000000000 RBX: ffff9382f8903580 RCX: 0000000000000000 [ 140.650952] RDX: 0000000000000001 RSI: 00000000ffffffff RDI: 0000000000000000 [ 140.652454] RBP: ffff9382fb74bdd0 R08: 000000000001f120 R09: ffff9382fd001700 [ 140.653968] R10: ffffffffc0a179cb R11: 0000000000000246 R12: 0000000000000000 [ 140.655469] R13: 0000000000000000 R14: 0000000000001000 R15: ffff9382f8903580 [ 140.656984] FS: 00007fcf084be740(0000) GS:ffff9382ffc00000(0000) knlGS:0000000000000000 [ 140.658685] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 140.659908] CR2: fffffffffffffff0 CR3: 000000006c568000 CR4: 00000000000607f0 [ 140.661412] Call Trace: [ 140.661989] [&lt;ffffffffc09ea96b&gt;] ? lnet_counters_get_common+0xeb/0x150 [lnet] [ 140.663520] [&lt;ffffffffc09eaa3c&gt;] lnet_counters_get+0x6c/0x150 [lnet] [ 140.664907] [&lt;ffffffffc0a17a0b&gt;] __proc_lnet_stats+0xfb/0x810 [lnet] [ 140.666278] [&lt;ffffffffc096f602&gt;] lprocfs_call_handler+0x22/0x50 [libcfs] [ 140.667727] [&lt;ffffffffc0a16ee5&gt;] proc_lnet_stats+0x25/0x30 [lnet] [ 140.669050] [&lt;ffffffffc096f662&gt;] lnet_debugfs_read+0x32/0x50 [libcfs] [ 140.670466] [&lt;ffffffffb904117f&gt;] vfs_read+0x9f/0x170 [ 140.671554] [&lt;ffffffffb904203f&gt;] SyS_read+0x7f/0xf0 [ 140.672655] [&lt;ffffffffb9574d21&gt;] ? system_call_after_swapgs+0xae/0x146 [ 140.674061] [&lt;ffffffffb9574ddb&gt;] system_call_fastpath+0x22/0x27 [ 140.675342] [&lt;ffffffffb9574d21&gt;] ? system_call_after_swapgs+0xae/0x146 [ 140.676747] Code: 85 03 00 50 ab 9b c0 c7 05 9c 85 03 00 00 00 02 00 e8 ff f8 fe ff 8b 1d 59 67 01 00 e9 e8 f9 ff ff 0f 1f 40 00 66 66 66 66 90 55 &lt;8b&gt; 47 f0 48 89 e5 5d c3 66 90 66 66 66 66 90 55 48 89 e5 41 57 [ 140.685523] RIP [&lt;ffffffffc09825f6&gt;] cfs_percpt_number+0x6/0x10 [libcfs] [ 140.688462] RSP &lt;ffff9382fb74bdb0&gt; [ 140.690693] CR2: fffffffffffffff0 [ 140.692874] ---[ end trace 3fe411e27d033edb ]--- [ 140.695279] Kernel panic - not syncing: Fatal exception [ 140.697872] Kernel Offset: 0x37e00000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff) </pre> </div></div> <p>I just tried on my local VM (building b2_12 locally) and this is how I get - </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> [root@oss lustre-release]# lctl list_nids 192.168.57.102@tcp [root@oss lustre-release]# lnetctl lnet unconfigure [root@oss lustre-release]# lustre_rmmod [root@oss lustre-release]# lctl network down opening /dev/lnet failed: No such file or directory hint: the kernel modules may not be loaded LNET unconfigure error 2: No such file or directory [root@oss lustre-release]# lctl get_param -n stats error: get_param: param_path <span class="code-quote">'stats'</span>: No such file or directory [root@oss lustre-release]# </pre> </div></div> <p>Let me try installing the rpms from the above link and see why it is still giving panic.</p> <p>I tried the above and that seems to work. My teardown steps are now a little different, I am able to get a panic by doing:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> [root@vm8 ~]# lnetctl net show net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.121.128@tcp status: up [root@vm8 ~]# lustre_rmmod ptlrpc [root@vm8 ~]# lnetctl lnet unconfigure [root@vm8 ~]# lctl get_param -n stats </pre> </div></div> <p>This is basically what <tt>lnet.service</tt> does as well:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> ExecStop=/usr/sbin/lustre_rmmod ptlrpc ExecStop=/usr/sbin/lnetctl lnet unconfigure ExecStop=/usr/sbin/lustre_rmmod libcfs ldiskfs </pre> </div></div> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/34634/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34634/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11986" title="After partial lustre_rmmod, lnet panics on debugfs read" class="issue-link" data-issue-key="LU-11986"><del>LU-11986</del></a> lnet: Avoid lnet debugfs read/write if ctl_table does not exist<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: <br/> Commit: 54ca5e471d9f1caf63c57dfe2655e177d26dca0f</p> <p>Oh my. LNetNIFini() calls lnet_router_debugfs_init() instead of lnet_router_debugfs_fini(). Joe try changing that code and I bet the problem goes away. Other wise what happens is that debugfs still exist but the lnet counters are free which is causing your crash.</p> <p> </p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=joe.grund" class="user-hover" rel="joe.grund">joe.grund</a> I do see crash when I follow the steps you shared -</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> [root@vm8 ~]# lnetctl net show net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.57.102@tcp status: up [root@vm8 ~]# lustre_rmmod ptlrpc [root@vm8 ~]# lnetctl lnet unconfigure [root@vm8 ~]# lctl get_param -n stats </pre> </div></div> <p>But if I unload the libcfs module too before running <tt>lctl get_param -n stats</tt> then I don't see the crash. I understand it should not crash but just to understand the use-case, may I know in what scenario, would you unload ptlrpc module and unconfigure lnet but not unload libcfs?</p> <p> </p> <p>James Simmons (uja.ornl@yahoo.com) uploaded a new patch: <a href="https://review.whamcloud.com/34669" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34669</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11986" title="After partial lustre_rmmod, lnet panics on debugfs read" class="issue-link" data-issue-key="LU-11986"><del>LU-11986</del></a> lnet: properly cleanup lnet debugfs files<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 5e01a964ce352bf85c8cfb5e1c6b8846d816679b</p> <blockquote><p>but just to understand the use-case, may I know in what scenario, would you unload ptlrpc module and unconfigure LNet but not unload libcfs?</p></blockquote> <p>The IML state machine splits LNet teardown into two steps, "stopping" and "unloading". The stopping step looks like this currently:</p> <p><a href="https://github.com/whamcloud/iml-agent/blob/b2236376f91448cd50c5a74f63a0d3b2c16d1121/chroma_agent/action_plugins/manage_lnet.py#L43-L54" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/whamcloud/iml-agent/blob/b2236376f91448cd50c5a74f63a0d3b2c16d1121/chroma_agent/action_plugins/manage_lnet.py#L43-L54</a></p> <p>and the unloading step looks like this currently:</p> <p><a href="https://github.com/whamcloud/iml-agent/blob/b2236376f91448cd50c5a74f63a0d3b2c16d1121/chroma_agent/action_plugins/manage_lnet.py#L64-L71" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/whamcloud/iml-agent/blob/b2236376f91448cd50c5a74f63a0d3b2c16d1121/chroma_agent/action_plugins/manage_lnet.py#L64-L71</a></p> <p>Stopping and unloading are usually performed together, but there is no hard and fast guarantee that will always be the case (user can stop without unloading)</p> <p>Path 34669 should resolve your problem. The reason is that libcfs registers the root debugfs file for libcfs/LNet. When lnetctl lnet configure is called additional debugfs files are registered. The problem was nothing was cleaned up until libcfs module was unloaded. What we needed to do was just remove the LNet specific debugfs files when lnetctl lnet unconfigure was called. libcfs module unload would do the rest of the cleanup like it does now.</p> <p>I would recommend both patches to land!!!</p> <p>Minh Diep (mdiep@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/34692" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34692</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11986" title="After partial lustre_rmmod, lnet panics on debugfs read" class="issue-link" data-issue-key="LU-11986"><del>LU-11986</del></a> lnet: properly cleanup lnet debugfs files<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: 3db5bf21974a59de88b44a49848f1c76844a6ec2</p> <p>I've run patch 34669 (review 63547) through IML automated integration testing, it has passed all checks: <a href="https://build.whamcloud.com/job/integration-tests-shared-storage-configuration/1571/" class="external-link" target="_blank" rel="nofollow noopener">https://build.whamcloud.com/job/integration-tests-shared-storage-configuration/1571/</a></p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/34622/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34622/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11986" title="After partial lustre_rmmod, lnet panics on debugfs read" class="issue-link" data-issue-key="LU-11986"><del>LU-11986</del></a> lnet: Avoid lnet debugfs read/write if ctl_table does not exist<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 013dafd857d1c8f045a704c865f99a53b777c5e3</p> <p>Landed for 2.13</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/34669/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34669/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11986" title="After partial lustre_rmmod, lnet panics on debugfs read" class="issue-link" data-issue-key="LU-11986"><del>LU-11986</del></a> lnet: properly cleanup lnet debugfs files<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 8cb7ccf54e2d34010bea9a71fe0eff708545ab78</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/34692/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34692/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11986" title="After partial lustre_rmmod, lnet panics on debugfs read" class="issue-link" data-issue-key="LU-11986"><del>LU-11986</del></a> lnet: properly cleanup lnet debugfs files<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: <br/> Commit: 9d42660e173e5b9a2683f412fc8a756094c6f1b4</p> <p>afaics, it happened again:<br/> <a href="https://testing.whamcloud.com/test_sets/4b204efc-bb02-11e9-9fc9-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/4b204efc-bb02-11e9-9fc9-52540065bddc</a></p> <p>So we reduce the chance of this bug <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/sad.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>Andreas Dilger (adilger@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/38090" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38090</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11986" title="After partial lustre_rmmod, lnet panics on debugfs read" class="issue-link" data-issue-key="LU-11986"><del>LU-11986</del></a> libcfs: provide QSTR_INIT compat macro<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: c765cb2c59b39ce0706e88d7b4c6cec6ef75e313</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/38090/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38090/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11986" title="After partial lustre_rmmod, lnet panics on debugfs read" class="issue-link" data-issue-key="LU-11986"><del>LU-11986</del></a> libcfs: provide QSTR_INIT compat macro<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: <br/> Commit: fd50d435119f0993ee05a4653315e1a55627817b</p> <p>Andreas Dilger (adilger@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/38529" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38529</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11986" title="After partial lustre_rmmod, lnet panics on debugfs read" class="issue-link" data-issue-key="LU-11986"><del>LU-11986</del></a> libcfs: add compat for d_hash_and_lookup()<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: e57f7dab3c9017a9ef979db0d4d2f685a150e0c5</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/38529/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38529/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11986" title="After partial lustre_rmmod, lnet panics on debugfs read" class="issue-link" data-issue-key="LU-11986"><del>LU-11986</del></a> libcfs: add compat for d_hash_and_lookup()<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: <br/> Commit: fa72fe50b9b4ee8ca2165607e32360a6bebd86e4</p> <p>Andreas Dilger (adilger@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/38716" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38716</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11986" title="After partial lustre_rmmod, lnet panics on debugfs read" class="issue-link" data-issue-key="LU-11986"><del>LU-11986</del></a> libcfs: lnet_remove_debugfs() compat for RHEL6<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: d38b26e84bccbb2c87a948e46408e67c884eeb76</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/38716/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38716/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11986" title="After partial lustre_rmmod, lnet panics on debugfs read" class="issue-link" data-issue-key="LU-11986"><del>LU-11986</del></a> libcfs: lnet_remove_debugfs() compat for RHEL6<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: <br/> Commit: 7f821c9382c39fd16156593569737df27dfb0467</p> <p>James Simmons (jsimmons@infradead.org) uploaded a new patch: <a href="https://review.whamcloud.com/39404" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39404</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11986" title="After partial lustre_rmmod, lnet panics on debugfs read" class="issue-link" data-issue-key="LU-11986"><del>LU-11986</del></a> lnet: don't read debugfs lnet stats when shutting down<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: b6587af8415064ec8c305cea03abfccddaef1341</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/39404/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39404/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11986" title="After partial lustre_rmmod, lnet panics on debugfs read" class="issue-link" data-issue-key="LU-11986"><del>LU-11986</del></a> lnet: don't read debugfs lnet stats when shutting down<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: f53eea15d470c9bb29a7b867b733db9249aec95b</p> Related LU-9673 LU-8066 Development Rank 1|i00c0f: Rank (Obsolete) 9223372036854775807 Severity