https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-9843 Lustre <p>Client hit LBUG.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">[1498857336.755629] LNetError: 57600:0:(linux-cpu.c:572:cfs_cpt_spread_node()) LBUG [1498857336.779630] Pid: 57600, comm: lfs [1498857336.779630] LNetError: 57594:0:(linux-cpu.c:572:cfs_cpt_spread_node()) LBUG [1498857336.779630] LNetError: 57588:0:(linux-cpu.c:572:cfs_cpt_spread_node()) LBUG [1498857336.779630] Pid: 57594, comm: lfs [1498857336.779630] [1498857336.779630] Call Trace: [1498857336.779630] Pid: 57588, comm: lfs [1498857336.779630] [14988573 3 67.7 o7u96t 3o0]f 8Ca cllp usT rainc e:kd^Mb , waiting <span class="code-keyword">for</span> the <span class="code-keyword">rest</span>, timeout in 10 second(s) [1498857336.779630] [&lt;ffffffff81004b35&gt;] dump_trace+0x75/0x300 [1498857336.779630] [&lt;ffffffff81004b35&gt;] dump_trace+0x75/0x300 [1498857336.779630] [&lt;ffffffffa09df82a&gt;] libcfs_debug_dumpstack+0x4a/0x70 [libcfs] [1498857336.779630] [&lt;ffffffffa09df82a&gt;] libcfs_debug_dumpstack+0x4a/0x70 [libcfs] [1498857336.779630] [&lt;ffffffffa09dfd5e&gt;] lbug_with_loc+0x3e/0xb0 [libcfs] [1498857336.779630] [&lt;ffffffffa09dfd5e&gt;] lbug_with_loc+0x3e/0xb0 [libcfs] [1498857336.779630] [&lt;ffffffffa09e18a6&gt;] cfs_cpt_spread_node+0xf6/0x130 [libcfs] [1498857336.779630] [&lt;ffffffffa09e18a6&gt;] cfs_cpt_spread_node+0xf6/0x130 [libcfs] [1498857336.779630] [&lt;ffffffffa09e0488&gt;] cfs_cpt_malloc+0x18/0x40 [libcfs] [1498857336.779630] [&lt;ffffffffa09e0488&gt;] cfs_cpt_malloc+0x18/0x40 [libcfs] [1498857336.779630] [&lt;ffffffffa0d1d60b&gt;] ptlrpc_prep_set+0x4b/0x310 [ptlrpc] [1498857336.779630] [&lt;ffffffffa0d1d60b&gt;] ptlrpc_prep_set+0x4b/0x310 [ptlrpc] [1498857336.779630] [&lt;ffffffffa0d238ac&gt;] ptlrpc_queue_wait+0x3c/0x220 [ptlrpc] [1498857336.779630] [&lt;ffffffffa0d238ac&gt;] ptlrpc_queue_wait+0x3c/0x220 [ptlrpc] [1498857336.779630] [&lt;ffffffffa1086993&gt;] osc_quotactl+0xf3/0x360 [osc] [1498857336.779630] [&lt;ffffffffa1086993&gt;] osc_quotactl+0xf3/0x360 [osc] [1498857336.779630] [&lt;ffffffffa0ecb8ea&gt;] lov_quotactl+0x38a/0x930 [lov] [1498857336.779630] [&lt;ffffffffa0ecb8ea&gt;] lov_quotactl+0x38a/0x930 [lov] [1498857336.779630] [&lt;ffffffffa0f2f4d9&gt;] obd_quotactl+0xb9/0x340 [lustre] [1498857336.779630] [&lt;ffffffffa0f2f4d9&gt;] obd_quotactl+0xb9/0x340 [lustre] [1498857336.779630] [&lt;ffffffffa0f3552b&gt;] quotactl_ioctl+0x100b/0x15b0 [lustre] [1498857336.779630] [&lt;ffffffffa0f3552b&gt;] quotactl_ioctl+0x100b/0x15b0 [lustre] [1498857336.779630] [&lt;ffffffffa0f37a48&gt;] ll_dir_ioctl+0x1908/0x62f0 [lustre] [1498857336.779630] [&lt;ffffffffa0f37a48&gt;] ll_dir_ioctl+0x1908/0x62f0 [lustre] [1498857336.779630] [&lt;ffffffff8117117b&gt;] do_vfs_ioctl+0x8b/0x3b0 [1498857336.779630] [&lt;ffffffff8117117b&gt;] do_vfs_ioctl+0x8b/0x3b0 [1498857336.779630] [&lt;ffffffff81171541&gt;] sys_ioctl+0xa1/0xb0 [1498857336.779630] [&lt;ffffffff81171541&gt;] sys_ioctl+0xa1/0xb0 [1498857336.779630] [&lt;ffffffff81483972&gt;] system_call_fastpath+0x16/0x1b [1498857336.779630] [&lt;ffffffff81483972&gt;] system_call_fastpath+0x16/0x1b [1498857336.779630] [&lt;00007fffed63f9a7&gt;] 0x7fffed63f9a7 [1498857336.779630] [&lt;00007fffed63f9a7&gt;] 0x7fffed63f9a7 [1498857336.779630] [1498857336.779630] [1498857336.779630] Kernel panic - not syncing: LBUG [1498857336.779630] Pid: 57594, comm: lfs Tainted: P ENX 3.0.101-100.1.20170523-nasa #1 [1498857336.779630] Call Trace: [1498857336.779630] [&lt;ffffffff81004b35&gt;] dump_trace+0x75/0x300 [1498857336.779630] [&lt;ffffffff814786b3&gt;] dump_stack+0x69/0x6f [1498857336.779630] [&lt;ffffffff8147876f&gt;] panic+0xb6/0x224 [1498857336.779630] [&lt;ffffffffa09dfdc3&gt;] lbug_with_loc+0xa3/0xb0 [libcfs] [1498857336.779630] [&lt;ffffffffa09e18a6&gt;] cfs_cpt_spread_node+0xf6/0x130 [libcfs] [1498857336.779630] [&lt;ffffffffa09e0488&gt;] cfs_cpt_malloc+0x18/0x40 [libcfs] [1498857336.779630] [&lt;ffffffffa0d1d60b&gt;] ptlrpc_prep_set+0x4b/0x310 [ptlrpc] [1498857336.779630] [&lt;ffffffffa0d238ac&gt;] ptlrpc_queue_wait+0x3c/0x220 [ptlrpc] [1498857336.779630] [&lt;ffffffffa1086993&gt;] osc_quotactl+0xf3/0x360 [osc] [1498857336.779630] [&lt;ffffffffa0ecb8ea&gt;] lov_quotactl+0x38a/0x930 [lov] [1498857336.779630] [&lt;ffffffffa0f2f4d9&gt;] obd_quotactl+0xb9/0x340 [lustre] [1498857336.779630] [&lt;ffffffffa0f3552b&gt;] quotactl_ioctl+0x100b/0x15b0 [lustre] [1498857336.779630] [&lt;ffffffffa0f37a48&gt;] ll_dir_ioctl+0x1908/0x62f0 [lustre] [1498857336.779630] [&lt;ffffffff8117117b&gt;] do_vfs_ioctl+0x8b/0x3b0 [1498857336.779630] [&lt;ffffffff81171541&gt;] sys_ioctl+0xa1/0xb0 [1498857336.779630] [&lt;ffffffff81483972&gt;] system_call_fastpath+0x16/0x1b [1498857336.779630] [&lt;0000.7fffed63f9a7&gt;] 0x7fffed63f9a6 All cpus are now in kdb </pre> </div></div> SLES11SP4 with lustre 2.7.3 LU-9843

LNetError: 57600:0:(linux-cpu.c:572:cfs_cpt_spread_node()) LBUG

Bug Minor Resolved Cannot Reproduce Sonia Sharma Mahmoud Hanafi Mon, 7 Aug 2017 21:31:26 +0000 Thu, 10 Oct 2019 21:35:18 +0000 Thu, 10 Oct 2019 21:35:18 +0000 Lustre 2.7.0 0 7 <p>Sonia</p> <p>Can you please advise?</p> <p>Thanks</p> <p>Peter</p> <p>Hi Mahmoud,<br/> Can we have access to vmcore and vmlinux for this?</p> <p>Thanks!</p> <p>Unfortunately we were not able to get a crash dump. Only other clue I have, I believe this occurred when one of the OST had hit a bitmap error and was remounted Read-only.</p> <p>Can you please give details on from where you got the build for SLES11SP4 with lustre 2.7.3 or did you build yourself?<br/> I could not find a branch with tag 2.7.3. Is it the tag in a fe branch? </p> <p>Sonia</p> <p>NASA have their own distribution based on the 2.7 FE branch. They will need to grant you access to it on github</p> <p>Peter</p> <p>Hi Sonia,</p> <p>If you give me your login ID at github.com I can add you to the list with access permission to our FE git repo.</p> <p>Hi Jay<br/> My login-id for github.com is <span class="nobr"><a href="mailto:soniash24@gmail.com" class="external-link" target="_blank" rel="nofollow noopener">soniash24@gmail.com^{<img class="rendericon" src="https://jira.whamcloud.com/images/icons/mail_small.gif" height="12" width="13" align="absmiddle" alt="" border="0"/>}</a></span> (username-Sonia241087)</p> <p>Hi Sonia,<br/> Have you received invitation from github to join "NASA Earth Exchange (NEX)?" The lustre FE (lustre-nas-fe) repo is housed under NEX. You will join as a member to Lustre team.</p> <p>Hi Jay<br/> Yes I am able to access the repo. Thanks</p> <p>Can we get information on how many NUMA nodes and CPU cores are there? Also what is the value of MAX_NUMNODES ( from /proc/self/status, field Mems_allowed)?</p> <p>Hi Mahmoud,</p> <p>Remounting of ost might have exposed a bug in the cfs_cpt_spread_node() code which is the reason it hit LBUG. Here is what happened we think - when ost was remounted, it called function ost_setup() which further called cfs_cpt_nodemask() and cfs_cpt_set_node() that might change the mask. This would then expose a race condition, where the mask changes after the rotor has been calculated but before the rotor is checked, causing the LBUG to be hit.<br/> I will push a patch to branch b2_7_fe to fix this issue in the code. <br/> Though, the 2.10 release has the fix and a series of many other patches related to CPT rework code. So it might be a good idea to update to the 2.10 release.</p> <p>Thanks</p> <p>2.10 release is a longer term option. For now we will need a 2.7fe patch.</p> <p> </p> <p>I have pushed this patch but it still needs to be reviewed and landed.<br/> <a href="https://review.whamcloud.com/28538" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/28538</a></p> <p>Hi Sonia, there was comments on #28538 from your reviewer. Should I ignore the comments and pick you your patchset #1?</p> <p>Hi Jay,</p> <p>I need to revise the patch but I was waiting for one more reviewer's (Dmitry's) feedback on the comments as he had done the major changes related to this fix on master. <br/> I will revise it soon and upload a new patch which should anyways fix this particular issue.</p> <p>Sonia</p> <p>Dmitry is out of the office this week so I recommend refreshing with Amir's comments and then getting Dmitry's input upon his return</p> <p>Peter</p> <p>Sure. I refreshed the patch per Amir's comments.</p> <p>Thanks</p> <p>Does 2.10.0 need this patch? Thanks!</p> <p>2.10.0 doesn't need this patch as it had a series of patches related to cpt rework which already incorporated this fix.</p> <p>Sorry I forgot to ask if the patch is needed for 2.9.0. We have lustre clients running 2.7.3, 2.9.0 and 2.10.0. Please advise. Thanks!</p> <p>Yes 2.9.0 would also require this patch.</p> <p>please close we are no longer running 2.7</p> <p>ok - thanks</p> Related Development Rank 1|hzzi0f: Rank (Obsolete) 9223372036854775807 Severity