https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-8715 Lustre <p>Since our nas-2.7.2-2nas rebased to b2_7_fe to nas-2.7.2-3nas, we found loading lustre module fld.ko hanged. Modprobe took 100% cpu time and could not be killed.</p> <p>I identified the culprit of the problem using git bisect:<br/> commit f23e22da88f07e95071ec76807aaa42ecd39e8ca<br/> Author: Amitoj Kaur Chawla &lt;amitoj1606@gmail.com&gt;<br/> Date: Thu Jun 16 23:12:03 2016 +0800</p> <p> <a href="https://jira.whamcloud.com/browse/LU-8057" title="o2iblnd driver is causing memory corruption due to improper handling of scatter list." class="issue-link" data-issue-key="LU-8057"><del>LU-8057</del></a> ko2iblnd: Replace sg++ with sg = sg_next(sg)</p> <p>It was a b2_7_fe back port from the following one:<br/> Lustre-commit: d226464acaacccd240da43dcc22372fbf8cb04a6<br/> Lustre-change: <a href="http://review.whamcloud.com/19342" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/19342</a></p> lustre server nas-2.7.2-3nasS running in centos 6.7. LU-8715

Regression from LU-8057 causes loading of fld.ko hung in 2.7.2

Bug Critical Resolved Cannot Reproduce Amir Shehata Jay Lan Tue, 18 Oct 2016 01:34:35 +0000 Wed, 18 Apr 2018 18:07:37 +0000 Wed, 18 Apr 2018 18:07:37 +0000 Lustre 2.7.0 0 9 <p>Well, both the failure and suspected cause look surprising.<br/> Do you mean that the fld.ko module load simply hangs on a fresh system when running "modprobre lustre"?</p> <p>Module load time before was about 2-5mins, because we have large ntx values.<br/> (options ko2iblnd ntx=125536 credits=62768 fmr_pool_size=31385)<br/> But after the patch it takes &gt;20mins</p> <p>The fix is correct and it fixes a real bug. What this change did is exposed another problem in the ko2iblnd driver. I have to ask is your system really consuming all those credits? I don't think the IB driver queue pair depth is big enough to handle all those credits.</p> <p>Hi Doug,</p> <p>Can you please have a look into the issue since it relates to the <a href="https://jira.whamcloud.com/browse/LU-8057" title="o2iblnd driver is causing memory corruption due to improper handling of scatter list." class="issue-link" data-issue-key="LU-8057"><del>LU-8057</del></a> change?</p> <p>Thanks.<br/> Joe</p> <p>@Bruno Faccini: Yes, I can reproduce the problem on our freshly rebooted lustre servers by doing 'modprobe fld.' </p> <p>we have &gt;12,000 clients. We do see some servers consume all the credits.</p> <p>perf top showed during module load all the time is spent in __vmalloc_node.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">Samples: 748K of event <span class="code-quote">'cycles'</span>, Event count (approx.): 53812402443 Overhead Shared <span class="code-object">Object</span> Symbol 96.21% [kernel] [k] __vmalloc_node 0.91% [kernel] [k] read_hpet 0.28% [kernel] [k] get_vmalloc_info 0.26% [kernel] [k] __write_lock_failed 0.25% [kernel] [k] __read_lock_failed 0.05% [kernel] [k] apic_timer_interrupt 0.05% [kernel] [k] _spin_lock 0.04% perf [.] dso__find_symbol 0.03% [kernel] [k] find_busiest_group 0.03% [kernel] [k] clear_page_c 0.03% [kernel] [k] page_fault 0.03% [kernel] [k] memset 0.02% [kernel] [k] rcu_process_gp_end 0.02% perf [.] perf_evsel__parse_sample 0.02% [kernel] [k] sha_transform 0.02% [kernel] [k] native_write_msr_safe </pre> </div></div> <p>I know exactly what your problem is. We saw this problem in the lustre core some time ago and changed the OBD_ALLOC macros. The libcfs/LNet layer uses it own LIBCFS_ALLOC macros which means when the allocations are more than 2 pages in size they hit the vmalloc spinlock serialization issue. We need a fix for libcfs much like lustre had.</p> <p>James, can we do that fix under this ticket?</p> <p>Why not. The problem is the LIBCFS_ALLOC and FREE macros. Looking at the macros gave me a headache so no patch from me. I need to get into the right mental state to tackle it <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>Any updates?</p> <p>As best we can figure out, the change in <a href="https://jira.whamcloud.com/browse/LU-8057" title="o2iblnd driver is causing memory corruption due to improper handling of scatter list." class="issue-link" data-issue-key="LU-8057"><del>LU-8057</del></a> has causes a little more memory to be used per connection. That is pushing your system over the edge.</p> <p>As James has indicated, a proper fix will be to change how to allocate memory in LNet. That is going to take some time to get right as the potential to break all of LNet is pretty good.</p> <p>I don't believe the fix for <a href="https://jira.whamcloud.com/browse/LU-8057" title="o2iblnd driver is causing memory corruption due to improper handling of scatter list." class="issue-link" data-issue-key="LU-8057"><del>LU-8057</del></a> is critical for your setup. If this needs to be fixed quickly, then removing <a href="https://jira.whamcloud.com/browse/LU-8085" title="Inode Iteration should increase the cursor to skip unused inodes" class="issue-link" data-issue-key="LU-8085"><del>LU-8085</del></a> from your build will be the best approach. As ORNL needs <a href="https://jira.whamcloud.com/browse/LU-8085" title="Inode Iteration should increase the cursor to skip unused inodes" class="issue-link" data-issue-key="LU-8085"><del>LU-8085</del></a>, we cannot remove it from master.</p> <p>Doug,</p> <p>You wrote in previous commet:<br/> " If this needs to be fixed quickly, then removing <a href="https://jira.whamcloud.com/browse/LU-8085" title="Inode Iteration should increase the cursor to skip unused inodes" class="issue-link" data-issue-key="LU-8085"><del>LU-8085</del></a> from your build will be the best approach. As ORNL needs <a href="https://jira.whamcloud.com/browse/LU-8085" title="Inode Iteration should increase the cursor to skip unused inodes" class="issue-link" data-issue-key="LU-8085"><del>LU-8085</del></a>, we cannot remove it from master."</p> <p>Did you actually mean to write "<a href="https://jira.whamcloud.com/browse/LU-8057" title="o2iblnd driver is causing memory corruption due to improper handling of scatter list." class="issue-link" data-issue-key="LU-8057"><del>LU-8057</del></a>"? We do not have <a href="https://jira.whamcloud.com/browse/LU-8057" title="o2iblnd driver is causing memory corruption due to improper handling of scatter list." class="issue-link" data-issue-key="LU-8057"><del>LU-8057</del></a> in our git repo...</p> <p>Yes, sorry about that. <a href="https://jira.whamcloud.com/browse/LU-8057" title="o2iblnd driver is causing memory corruption due to improper handling of scatter list." class="issue-link" data-issue-key="LU-8057"><del>LU-8057</del></a> is what I was referring to.</p> <p>Is this still a problem</p> <p> </p> <p>This case can be closed. Thanks.</p> <p>ok thanks!</p> Related LU-8057 Development Rank 1|hzys8n: Rank (Obsolete) 9223372036854775807 Severity