https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-6285 Lustre <p>Enabling CONFIG_CPUMASK_OFFSTACK in stock kernel 3.18.0 causes the staging ptlrpc module to emit the message</p> <p>LustreError: 1203:0:(service.c:2796:ptlrpc_hr_init()) ASSERTION( hrp-&gt;hrp_nthrs &gt; 0 ) failed: </p> <p>followed by a backtrace and kernel lockup upon loading. I'll attach my dmesg dump and the .config file I used. I picked version 2.4.0 above as there doesn't seem to be anyway to indicate the staging client version.</p> LU-6285

Assert fails in staging client module crashes kernel if CPUMASK_OFFSTACK set

Bug Minor Resolved Fixed Oleg Drokin Tyson Whitehead Wed, 25 Feb 2015 21:35:50 +0000 Wed, 10 Aug 2016 13:58:06 +0000 Wed, 27 Jul 2016 22:04:53 +0000 Lustre 2.4.0 Lustre 2.8.0 Lustre 2.9.0 0 10 <p>Thank you for the report.<br/> I think I traced this to a bug in the kernel where cpumask copy is not copying enough bits and cpu_weight is counting too many creating a problem where thousands of "phantom" cpus are detected due to garbage in variables.</p> <p>I submitted a bugreport upstream with a couple of proposed patches and hopefully that would be taken care of: <a href="https://lkml.org/lkml/2015/2/26/29" class="external-link" target="_blank" rel="nofollow noopener">https://lkml.org/lkml/2015/2/26/29</a></p> <p>Wow. That's great! Thanks for the very quick turn around.</p> <p>We are really looking forward to being able to use the latest Fedora and Ubuntu releases as lustre clients.</p> <p>You can also use this as a workaround (and a minor performance optimization):</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>diff --git a/drivers/staging/lustre/lustre/ptlrpc/service.c b/drivers/staging/lustre/lustre/ptlrpc/service.c index 635b12b..4a27c79 100644 --- a/drivers/staging/lustre/lustre/ptlrpc/service.c +++ b/drivers/staging/lustre/lustre/ptlrpc/service.c @@ -2752,7 +2752,6 @@ int ptlrpc_start_thread(struct ptlrpc_service_part *svcpt, int wait) int ptlrpc_hr_init(void) { - cpumask_t mask; struct ptlrpc_hr_partition *hrp; struct ptlrpc_hr_thread *hrt; int rc; @@ -2770,8 +2769,7 @@ int ptlrpc_hr_init(void) init_waitqueue_head(&amp;ptlrpc_hr.hr_waitq); - cpumask_copy(&amp;mask, topology_thread_cpumask(0)); - weight = cpus_weight(mask); + weight = cpus_weight(*topology_thread_cpumask(0)); cfs_percpt_for_each(hrp, i, ptlrpc_hr.hr_partitions) { hrp-&gt;hrp_cpt = i; </pre> </div></div> <p>I'll send a separate patch for this to Greg, but who knows when it'll actually make it to Fedora.</p> <p>Oleg Drokin (oleg.drokin@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/13904" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/13904</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6285" title="Assert fails in staging client module crashes kernel if CPUMASK_OFFSTACK set" class="issue-link" data-issue-key="LU-6285"><del>LU-6285</del></a> libcfs: Do not unnecessarily copy cpumask<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 2651de49d3e54db84f50285eb55a900b4a96ca1e</p> <p>Oleg Drokin (oleg.drokin@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/13905" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/13905</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6285" title="Assert fails in staging client module crashes kernel if CPUMASK_OFFSTACK set" class="issue-link" data-issue-key="LU-6285"><del>LU-6285</del></a> ptlrpc: Do not recalculate siblings of CPU 0 in a loop<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: ff0c3c553b42e699a2960cf2788732fbedc3f485</p> <p>Oleg Drokin (oleg.drokin@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/13925" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/13925</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6285" title="Assert fails in staging client module crashes kernel if CPUMASK_OFFSTACK set" class="issue-link" data-issue-key="LU-6285"><del>LU-6285</del></a> ptlrpc: Get rid of cpus_* calls as deprecated<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: c79914c41d71d95b95bdd237537d450984ab8894</p> <p>Oleg Drokin (oleg.drokin@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/13926" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/13926</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6285" title="Assert fails in staging client module crashes kernel if CPUMASK_OFFSTACK set" class="issue-link" data-issue-key="LU-6285"><del>LU-6285</del></a> libcfs: get rid of deprecated cpumask function usage<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 14f60e7f68c969f0a9aa0cf35510216b1019bef3</p> <p>Oleg Drokin (oleg.drokin@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/13954" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/13954</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6285" title="Assert fails in staging client module crashes kernel if CPUMASK_OFFSTACK set" class="issue-link" data-issue-key="LU-6285"><del>LU-6285</del></a>: o2iblnd: Do not use cpus_weight, it's deprecated<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: e0da4f48095531ca651cdcbc8bca9f22d6cc5860</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/13904/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/13904/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6285" title="Assert fails in staging client module crashes kernel if CPUMASK_OFFSTACK set" class="issue-link" data-issue-key="LU-6285"><del>LU-6285</del></a> libcfs: Do not unnecessarily copy cpumask<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: b4f41e5fef3ff644f9adb95921329ef59e1e3e74</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/13905/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/13905/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6285" title="Assert fails in staging client module crashes kernel if CPUMASK_OFFSTACK set" class="issue-link" data-issue-key="LU-6285"><del>LU-6285</del></a> ptlrpc: Do not recalculate siblings of CPU 0 in a loop<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 0eb4582d87e32dd3e5491e13ba659e625624bfe7</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/13954/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/13954/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6285" title="Assert fails in staging client module crashes kernel if CPUMASK_OFFSTACK set" class="issue-link" data-issue-key="LU-6285"><del>LU-6285</del></a>: o2iblnd: Do not use cpus_weight, it's deprecated<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: b95db057d0501fb19f807cddf3a8ba3f7f47cb1a</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/13925/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/13925/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6285" title="Assert fails in staging client module crashes kernel if CPUMASK_OFFSTACK set" class="issue-link" data-issue-key="LU-6285"><del>LU-6285</del></a> ptlrpc: Get rid of cpus_* calls as deprecated<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 61787e1cea610ba38ba917b73db0d43589c029df</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/13926/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/13926/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6285" title="Assert fails in staging client module crashes kernel if CPUMASK_OFFSTACK set" class="issue-link" data-issue-key="LU-6285"><del>LU-6285</del></a> libcfs: get rid of deprecated cpumask function usage<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 3b3233792869e706fe1ebfb6605d93fbc0d0d63c</p> <p>All the patches have landed. We can close this ticket.</p> <p>Excellent! Thanks everyone. <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>Please close this ticket</p> <p>There is still an issue which could cause the assert.</p> <p>cpu_pattern can sepcify exactly 1 cpu in a partition:<br/> "0<span class="error">&#91;0&#93;</span>". That means CPT0 will have CPU 0. CPU 0 can have<br/> hyperthreading enabled. This combination would result in</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">weight = cfs_cpu_ht_nsiblings(0); hrp-&gt;hrp_nthrs = cfs_cpt_weight(ptlrpc_hr.hr_cpt_table, i); hrp-&gt;hrp_nthrs /= weight; </pre> </div></div> <p>evaluating to 0. Where</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">cfs_cpt_weight(ptlrpc_hr.hr_cpt_table, i) == 1 weight == 2 </pre> </div></div> <p>Therefore only divide out with weight if</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">hrp-&gt;hrp_nthrs &gt;= weight </pre> </div></div> <p>This will avoid the assert:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">LASSERT(hrp-&gt;hrp_nthrs &gt; 0); </pre> </div></div> <p>Amir Shehata (amir.shehata@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/19106" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/19106</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6285" title="Assert fails in staging client module crashes kernel if CPUMASK_OFFSTACK set" class="issue-link" data-issue-key="LU-6285"><del>LU-6285</del></a> ptlrpc: Correctly calculate hrp-&gt;hrp_nthrs<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 2d6210842775e4fa7b0c7a6bce1dde8e948e56c9</p> <p>Bulk of work landed for 2.9. Amir, please open a new ticket to track the landing of <a href="http://review.whamcloud.com/#/c/19106/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/19106/</a></p> Blocker Related LU-4011 LU-8492 LU-6215 Development Epic/Theme staging Rank 1|hzx72v: Rank (Obsolete) 17617 Severity