https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-10804 Lustre <p>This issue was created by maloo for elena &lt;c17455@cray.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://testing.hpdd.intel.com/test_sets/5d126b54-2233-11e8-9ec4-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/5d126b54-2233-11e8-9ec4-52540065bddc</a></p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">[ 3072.347530] LustreError: 30713:0:(lprocfs_status_server.c:647:lprocfs_alloc_obd_stats()) ASSERTION( obd-&gt;obd_proc_entry != ((void *)0) ) failed: [ 3072.351782] LustreError: 30713:0:(lprocfs_status_server.c:647:lprocfs_alloc_obd_stats()) LBUG [ 3072.354022] Pid: 30713, comm: lctl [ 3072.355821] [ 3072.355821] Call Trace: [ 3072.359137] [&lt;ffffffffc06917ae&gt;] libcfs_call_trace+0x4e/0x60 [libcfs] [ 3072.361177] [&lt;ffffffffc069183c&gt;] lbug_with_loc+0x4c/0xb0 [libcfs] [ 3072.363336] [&lt;ffffffffc0cbe06f&gt;] lprocfs_alloc_obd_stats+0x18f/0x200 [obdclass] [ 3072.365454] [&lt;ffffffffc0c81730&gt;] ? lprocfs_obd_setup+0x130/0x320 [obdclass] [ 3072.367602] [&lt;ffffffffc12b13ad&gt;] echo_srv_device_alloc+0x48d/0x540 [obdecho] [ 3072.369738] [&lt;ffffffffc0c91cb4&gt;] obd_setup+0x114/0x2a0 [obdclass] [ 3072.371892] [&lt;ffffffffc0c920e8&gt;] class_setup+0x2a8/0x840 [obdclass] [ 3072.373960] [&lt;ffffffffc0c9651f&gt;] class_process_config+0x1b3f/0x27e0 [obdclass] [ 3072.376145] [&lt;ffffffffc0c86ff9&gt;] ? lprocfs_counter_add+0xf9/0x160 [obdclass] [ 3072.378240] [&lt;ffffffffc0c7053a&gt;] class_handle_ioctl+0x56a/0x1df0 [obdclass] [ 3072.380342] [&lt;ffffffff8121f7b8&gt;] ? destroy_inode+0x38/0x60 [ 3072.382250] [&lt;ffffffff812b3ea8&gt;] ? security_capable+0x18/0x20 [ 3072.384178] [&lt;ffffffffc0c567f2&gt;] obd_class_ioctl+0xd2/0x170 [obdclass] [ 3072.386089] [&lt;ffffffff8121730d&gt;] do_vfs_ioctl+0x33d/0x540 [ 3072.387929] [&lt;ffffffff81224a04&gt;] ? mntput+0x24/0x40 [ 3072.389676] [&lt;ffffffff81205013&gt;] ? __fput+0x183/0x260 [ 3072.391419] [&lt;ffffffff812175b1&gt;] SyS_ioctl+0xa1/0xc0 [ 3072.393128] [&lt;ffffffff816b8930&gt;] ? system_call_after_swapgs+0x15d/0x214 [ 3072.394862] [&lt;ffffffff816b89fd&gt;] system_call_fastpath+0x16/0x1b [ 3072.396554] [&lt;ffffffff816b889d&gt;] ? system_call_after_swapgs+0xca/0x214 [ 3072.398245] [ 3072.399550] Kernel panic - not syncing: LBUG </pre> </div></div> LU-10804

obdfilter-survey test_3a: lprocfs_alloc_obd_stats()) ASSERTION( obd->obd_proc_entry != ((void *)0) ) failed; LBUG

Bug Major Resolved Fixed Mikhail Pershin Maloo Mon, 12 Mar 2018 13:55:26 +0000 Sat, 17 Mar 2018 13:20:02 +0000 Sat, 17 Mar 2018 13:20:02 +0000 Lustre 2.11.0 Lustre 2.11.0 0 12 <p>Obdfilter-survey ought to work very reliably.  We should consider this a major issue (or greater).</p> <p>The report referenced is for a gerrit patch that is based on another patch that seems to change relevant functionality. We looked and this sort of failure was not observed in the latest master test results, so the primary suspicion is it's the patch in question that broke things, esp. since if you look, the baseline patch is "move procfs stuff to debugfs" and the assertion you hit is in procfs-related code.</p> <p>I've reviewed all obdfilter-survey test 3a hangs over all branches starting from January 1, 2018 to today and I've found the following test sessions that have the same assertion as in this ticket:</p> <p>2018-03-09 22:59:29 UTC full-patchless - <a href="https://testing.hpdd.intel.com/test_sets/67d21086-240d-11e8-b74b-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/67d21086-240d-11e8-b74b-52540065bddc</a></p> <p>2018-03-09 23:53:56 UTC full-patchless - <a href="https://testing.hpdd.intel.com/test_sets/6d7aa550-240f-11e8-b3c6-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/6d7aa550-240f-11e8-b3c6-52540065bddc</a></p> <p>2018-03-10 02:13:04 UTC full-patchless - <a href="https://testing.hpdd.intel.com/test_sets/8d7b15dc-242e-11e8-b74b-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/8d7b15dc-242e-11e8-b74b-52540065bddc</a></p> <p>2018-03-09 00:54:41 UTC full-patchless - <a href="https://testing.hpdd.intel.com/test_sets/7c8f22cc-2348-11e8-9852-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/7c8f22cc-2348-11e8-9852-52540065bddc</a></p> <p>2018-03-08 18:53:42 UTC full-patchless - <a href="https://testing.hpdd.intel.com/test_sets/965a62b4-231d-11e8-9852-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/965a62b4-231d-11e8-9852-52540065bddc</a></p> <p>2018-03-08 21:41:26 UTC full-patchless - <a href="https://testing.hpdd.intel.com/test_sets/15d7653a-233e-11e8-8d2f-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/15d7653a-233e-11e8-8d2f-52540065bddc</a></p> <p>2018-03-08 22:23:08 UTC full-patchless – <a href="https://testing.hpdd.intel.com/test_sets/0f8acdfa-2340-11e8-88de-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/0f8acdfa-2340-11e8-88de-52540065bddc</a></p> <p>All are from full test sessions for our patchless Lustre builds/testing.</p> <p>hm, so that's all after Mar 6th round of landings, I guess.</p> <p>This was not observed in the latest master test results because of test_3a was not functional, see <a href="https://jira.whamcloud.com/browse/LU-7420" title=" obdfilter-survey.sh test_3a: tgt_request_handle()) @@@ echo_srv: No target for connected export" class="issue-link" data-issue-key="LU-7420"><del>LU-7420</del></a> :</p> <p>&gt; this test passes at Maloo, seems it does just nothing.</p> <p><a href="https://jira.whamcloud.com/browse/LU-7420" title=" obdfilter-survey.sh test_3a: tgt_request_handle()) @@@ echo_srv: No target for connected export" class="issue-link" data-issue-key="LU-7420"><del>LU-7420</del></a> patch <a href="https://review.whamcloud.com/#/c/18443/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/18443/</a> fixes the test to be executed. The reported issue hit with this patch.</p> <p>Mike</p> <p>Can you please investigate?</p> <p>Thanks</p> <p>Peter</p> <p>Looking at the assertion, this appears like it is related to changes in /proc, possibly related to /sysfs changes? Possibly James already has a patch to fix this part of the code?</p> <p>Hit this in interop testing</p> <p><a href="https://testing.hpdd.intel.com/test_sets/baa94546-2709-11e8-b3c6-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/baa94546-2709-11e8-b3c6-52540065bddc</a></p> <p> </p> <p>client: 2.10.3</p> <p>server: lustre-master tag-2.10.59</p> <p>OSS console</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> [75396.029755] Lustre: DEBUG MARKER: == obdfilter-survey test 3a: Network survey ========================================================== 18:48:29 (1520966909) [75401.745879] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 10.9.4.73@tcp) was lost; in progress operations using this service will wait for recovery to complete [75401.749951] Lustre: Skipped 9 previous similar messages [75410.250117] Lustre: DEBUG MARKER: grep -c /mnt/lustre-ost1' ' /proc/mounts [75410.567342] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-ost1 [75410.784095] Lustre: server umount lustre-OST0000 complete [75410.950964] Lustre: DEBUG MARKER: lsmod | grep lnet &gt; /dev/null &amp;&amp; lctl dl | grep ' ST ' [75411.276952] Lustre: DEBUG MARKER: grep -c /mnt/lustre-ost2' ' /proc/mounts [75411.590521] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-ost2 [75411.965347] Lustre: DEBUG MARKER: lsmod | grep lnet &gt; /dev/null &amp;&amp; lctl dl | grep ' ST ' [75412.289128] Lustre: DEBUG MARKER: grep -c /mnt/lustre-ost3' ' /proc/mounts [75412.601354] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-ost3 [75412.988541] Lustre: DEBUG MARKER: lsmod | grep lnet &gt; /dev/null &amp;&amp; lctl dl | grep ' ST ' [75413.310862] Lustre: DEBUG MARKER: grep -c /mnt/lustre-ost4' ' /proc/mounts [75413.620037] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-ost4 [75413.745052] Lustre: 26996:0:(client.c:2100:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1520966920/real 1520966920] req@ffff880037402400 x1594796080200464/t0(0) o400-&gt;MGC10.9.4.73@tcp@10.9.4.73@tcp:26/25 lens 224/224 e 0 to 1 dl 1520966927 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [75413.751282] Lustre: 26996:0:(client.c:2100:ptlrpc_expire_one_request()) Skipped 1 previous similar message [75413.753466] LustreError: 166-1: MGC10.9.4.73@tcp: Connection to MGS (at 10.9.4.73@tcp) was lost; in progress operations using this service will fail [75413.977797] Lustre: DEBUG MARKER: lsmod | grep lnet &gt; /dev/null &amp;&amp; lctl dl | grep ' ST ' [75414.299315] Lustre: DEBUG MARKER: grep -c /mnt/lustre-ost5' ' /proc/mounts [75414.606635] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-ost5 [75414.967344] Lustre: DEBUG MARKER: lsmod | grep lnet &gt; /dev/null &amp;&amp; lctl dl | grep ' ST ' [75415.289338] Lustre: DEBUG MARKER: grep -c /mnt/lustre-ost6' ' /proc/mounts [75415.598284] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-ost6 [75415.793961] Lustre: server umount lustre-OST0005 complete [75415.795939] Lustre: Skipped 4 previous similar messages [75415.954904] Lustre: DEBUG MARKER: lsmod | grep lnet &gt; /dev/null &amp;&amp; lctl dl | grep ' ST ' [75416.274434] Lustre: DEBUG MARKER: grep -c /mnt/lustre-ost7' ' /proc/mounts [75416.583237] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-ost7 [75416.965393] Lustre: DEBUG MARKER: lsmod | grep lnet &gt; /dev/null &amp;&amp; lctl dl | grep ' ST ' [75419.722863] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d @ [75421.172060] LustreError: 14767:0:(lprocfs_status_server.c:647:lprocfs_alloc_obd_stats()) ASSERTION( obd-&gt;obd_proc_entry != ((void *)0) ) failed: [75421.176024] LustreError: 14767:0:(lprocfs_status_server.c:647:lprocfs_alloc_obd_stats()) LBUG [75421.178166] Pid: 14767, comm: lctl [75421.179941] [75421.179941] Call Trace: [75421.207941] [&lt;ffffffffc06ae7ae&gt;] libcfs_call_trace+0x4e/0x60 [libcfs] [75421.210086] [&lt;ffffffffc06ae83c&gt;] lbug_with_loc+0x4c/0xb0 [libcfs] [75421.212244] [&lt;ffffffffc08c867f&gt;] lprocfs_alloc_obd_stats+0x18f/0x200 [obdclass] [75421.214387] [&lt;ffffffffc088b920&gt;] ? lprocfs_obd_setup+0x130/0x320 [obdclass] [75421.237295] [&lt;ffffffffc10a8ced&gt;] echo_setup+0x1bd/0x2c0 [obdecho] [75421.273090] [&lt;ffffffffc0ab6810&gt;] ? ldlm_completion_ast+0x0/0x920 [ptlrpc] [75421.275072] [&lt;ffffffffc089bfda&gt;] obd_setup+0x18a/0x2b0 [obdclass] [75421.276950] [&lt;ffffffffc06c0a66&gt;] ? cfs_hash_create+0x366/0xa20 [libcfs] [75421.278899] [&lt;ffffffffc089c3a8&gt;] class_setup+0x2a8/0x840 [obdclass] [75421.280725] [&lt;ffffffffc08a081c&gt;] class_process_config+0x1b5c/0x2810 [obdclass] [75421.282654] [&lt;ffffffffc0891219&gt;] ? lprocfs_counter_add+0xf9/0x160 [obdclass] [75421.284527] [&lt;ffffffffc087a71a&gt;] class_handle_ioctl+0x56a/0x1de0 [obdclass] [75421.286438] [&lt;ffffffff8122239b&gt;] ? destroy_inode+0x3b/0x60 [75421.288202] [&lt;ffffffff812b72be&gt;] ? security_capable+0x1e/0x20 [75421.290020] [&lt;ffffffffc08607f2&gt;] obd_class_ioctl+0xd2/0x170 [obdclass] [75421.291824] [&lt;ffffffff81219e90&gt;] do_vfs_ioctl+0x350/0x560 [75421.293554] [&lt;ffffffff812275e4&gt;] ? mntput+0x24/0x40 [75421.295289] [&lt;ffffffff812079b6&gt;] ? __fput+0x186/0x260 [75421.296973] [&lt;ffffffff8121a141&gt;] SyS_ioctl+0xa1/0xc0 [75421.298680] [&lt;ffffffff816c0655&gt;] ? system_call_after_swapgs+0xa2/0x146 [75421.300454] [&lt;ffffffff816c0715&gt;] system_call_fastpath+0x1c/0x21 [75421.302229] [&lt;ffffffff816c0661&gt;] ? system_call_after_swapgs+0xae/0x146 [75421.304033] [75421.305435] Kernel panic - not syncing: LBUG [75421.306428] CPU: 1 PID: 14767 Comm: lctl Tainted: G OE ------------ 3.10.0-693.21.1.el7_lustre.x86_64 #1 [75421.306428] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2007 [75421.306428] Call Trace: [75421.306428] [&lt;ffffffff816ae7c8&gt;] dump_stack+0x19/0x1b [75421.306428] [&lt;ffffffff816a8634&gt;] panic+0xe8/0x21f [75421.306428] [&lt;ffffffffc06ae854&gt;] lbug_with_loc+0x64/0xb0 [libcfs] [75421.306428] [&lt;ffffffffc08c867f&gt;] lprocfs_alloc_obd_stats+0x18f/0x200 [obdclass] [75421.306428] [&lt;ffffffffc088b920&gt;] ? lprocfs_obd_setup+0x130/0x320 [obdclass] [75421.306428] [&lt;ffffffffc10a8ced&gt;] echo_setup+0x1bd/0x2c0 [obdecho] [75421.306428] [&lt;ffffffffc0ab6810&gt;] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc] [75421.306428] [&lt;ffffffffc089bfda&gt;] obd_setup+0x18a/0x2b0 [obdclass] [75421.306428] [&lt;ffffffffc06c0a66&gt;] ? cfs_hash_create+0x366/0xa20 [libcfs] [75421.306428] [&lt;ffffffffc089c3a8&gt;] class_setup+0x2a8/0x840 [obdclass] [75421.306428] [&lt;ffffffffc08a081c&gt;] class_process_config+0x1b5c/0x2810 [obdclass] [75421.306428] [&lt;ffffffffc0891219&gt;] ? lprocfs_counter_add+0xf9/0x160 [obdclass] [75421.306428] [&lt;ffffffffc087a71a&gt;] class_handle_ioctl+0x56a/0x1de0 [obdclass] [75421.306428] [&lt;ffffffff8122239b&gt;] ? destroy_inode+0x3b/0x60 [75421.306428] [&lt;ffffffff812b72be&gt;] ? security_capable+0x1e/0x20 [75421.306428] [&lt;ffffffffc08607f2&gt;] obd_class_ioctl+0xd2/0x170 [obdclass] [75421.306428] [&lt;ffffffff81219e90&gt;] do_vfs_ioctl+0x350/0x560 [75421.306428] [&lt;ffffffff812275e4&gt;] ? mntput+0x24/0x40 [75421.306428] [&lt;ffffffff812079b6&gt;] ? __fput+0x186/0x260 [75421.306428] [&lt;ffffffff8121a141&gt;] SyS_ioctl+0xa1/0xc0 [75421.306428] [&lt;ffffffff816c0655&gt;] ? system_call_after_swapgs+0xa2/0x146 [75421.306428] [&lt;ffffffff816c0715&gt;] system_call_fastpath+0x1c/0x21 [75421.306428] [&lt;ffffffff816c0661&gt;] ? system_call_after_swapgs+0xae/0x146 [ 0.000000] Initializing cgroup subsys cpuset </pre> </div></div> <p>This is <a href="https://jira.whamcloud.com/browse/LU-8066" title="Move lustre procfs handling to sysfs and debugfs." class="issue-link" data-issue-key="LU-8066">LU-8066</a>, commit 0100ab268c and this particular change:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">diff --git a/lustre/obdecho/echo_client.c b/lustre/obdecho/echo_client.c index 081a339..6e25a02 100644 --- a/lustre/obdecho/echo_client.c +++ b/lustre/obdecho/echo_client.c @@ -3071,7 +3071,7 @@ <span class="code-keyword">static</span> <span class="code-object">int</span> __init obdecho_init(void) <span class="code-keyword">if</span> (rc != 0) <span class="code-keyword">goto</span> failed_0; - rc = class_register_type(&amp;echo_obd_ops, NULL, <span class="code-keyword">true</span>, NULL, + rc = class_register_type(&amp;echo_obd_ops, NULL, <span class="code-keyword">false</span>, NULL, LUSTRE_ECHO_NAME, NULL); <span class="code-keyword">if</span> (rc != 0) <span class="code-keyword">goto</span> failed_1; </pre> </div></div> <p>that change causes empty lprocfs root for echo server device and later it asserts on it in lprocfs_alloc_obd_stats() as I can see.</p> <p>Mike Pershin (mike.pershin@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/31664" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/31664</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10804" title="obdfilter-survey test_3a: lprocfs_alloc_obd_stats()) ASSERTION( obd-&gt;obd_proc_entry != ((void *)0) ) failed; LBUG" class="issue-link" data-issue-key="LU-10804"><del>LU-10804</del></a> echo: allow echo server to setup procfs<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 8685e8db0b55651fab21170ed75bc9df454c0e58</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/31664/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/31664/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10804" title="obdfilter-survey test_3a: lprocfs_alloc_obd_stats()) ASSERTION( obd-&gt;obd_proc_entry != ((void *)0) ) failed; LBUG" class="issue-link" data-issue-key="LU-10804"><del>LU-10804</del></a> echo: allow echo server to setup procfs<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: af94e4b7afd3fa724d161a5127dc9a89d898e73c</p> <p>Landed for 2.11</p> Related LU-8066 Development Rank 1|hzzu9z: Rank (Obsolete) 9223372036854775807 Severity