https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-11693 Lustre <p>We get quite a few soft lockups on our Lustre gateways (Lustre clients that export Lustre filesystems over NFS). Example:</p> <p>Nov 13 00:26:06 foxtrot2 kernel: NMI watchdog: BUG: soft lockup - CPU#0 stuck for 23s! <span class="error">&#91;nfsd:11973&#93;</span><br/> Nov 13 00:26:06 foxtrot2 kernel: NMI watchdog: BUG: soft lockup - CPU#1 stuck for 23s! <span class="error">&#91;rsync:36079&#93;</span><br/> Nov 13 00:26:06 foxtrot2 kernel: Modules linked in: vfat fat dm_service_time mpt3sas mpt2sas raid_class scsi_transport_sas mptctl mptb<br/> ase nfsv3 nfs fscache osc(OE) mgc(OE) lustre(OE) lmv(OE) mdc(OE) lov(OE) fid(OE) fld(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE)<br/> dell_rbu libcfs(OE) bonding sb_edac edac_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel iTCO_wdt iTCO_vendor_support kv<br/> m joydev dcdbas irqbypass sg shpchp ipmi_si ipmi_devintf ipmi_msghandler lpc_ich mei_me mei acpi_power_meter acpi_pad nfsd auth_rpcgss<br/> nfs_acl lockd grace binfmt_misc ip_tables xfs sd_mod crc_t10dif crct10dif_generic 8021q garp stp llc mrp mgag200 i2c_algo_bit drm_kms<br/> _helper scsi_transport_iscsi bnx2x syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul crct10dif_common crc32_pclmul cr<br/> c32c_intel ahci drm ghash_clmulni_intel<br/> Nov 13 00:26:06 foxtrot2 kernel: libahci aesni_intel dm_multipath libata lrw gf128mul glue_helper ablk_helper cryptd megaraid_sas i2c_<br/> core ptp pps_core mdio libcrc32c wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod <span class="error">&#91;last unloaded: usb_storage&#93;</span><br/> Nov 13 00:26:06 foxtrot2 kernel: CPU: 1 PID: 36079 Comm: rsync Tainted: G W OE ------------ 3.10.0-693.5.2.el7_lustre.x86_6<br/> 4 #1<br/> Nov 13 00:26:06 foxtrot2 kernel: Hardware name: Dell Inc. PowerEdge R620/01W23F, BIOS 2.5.4 01/22/2016<br/> Nov 13 00:26:06 foxtrot2 kernel: task: ffff883ff8a04f10 ti: ffff8815a1200000 task.ti: ffff8815a1200000<br/> Nov 13 00:26:06 foxtrot2 kernel: RIP: 0010:<span class="error">&#91;&lt;ffffffff810fa332&gt;&#93;</span> <span class="error">&#91;&lt;ffffffff810fa332&gt;&#93;</span> native_queued_spin_lock_slowpath+0x112/0x1e0<br/> Nov 13 00:26:06 foxtrot2 kernel: RSP: 0018:ffff8815a1203700 EFLAGS: 00000246<br/> Nov 13 00:26:06 foxtrot2 kernel: RAX: 0000000000000000 RBX: ffff883fff017880 RCX: 0000000000090000<br/> Nov 13 00:26:06 foxtrot2 kernel: RDX: ffff883fff4d7880 RSI: 0000000001390101 RDI: ffff881ff99da818<br/> Nov 13 00:26:06 foxtrot2 kernel: RBP: ffff8815a1203700 R08: ffff883fff017880 R09: 0000000000000000<br/> Nov 13 00:26:06 foxtrot2 kernel: R10: 0004c5dab524ba0b R11: 0000000000000000 R12: 0004c5dab524ba0b<br/> Nov 13 00:26:06 foxtrot2 kernel: R13: 0000000000000000 R14: 0004c5dab39dc857 R15: ffff8815a12036e8<br/> Nov 13 00:26:06 foxtrot2 kernel: FS: 00007f0ff1094740(0000) GS:ffff883fff000000(0000) knlGS:0000000000000000<br/> Nov 13 00:26:06 foxtrot2 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033<br/> Nov 13 00:26:06 foxtrot2 kernel: CR2: 00007fd6cb1e9000 CR3: 000000163eff9000 CR4: 00000000001407e0<br/> Nov 13 00:26:06 foxtrot2 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000<br/> Nov 13 00:26:06 foxtrot2 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400<br/> Nov 13 00:26:06 foxtrot2 kernel: Stack:<br/> Nov 13 00:26:06 foxtrot2 kernel: ffff8815a1203710 ffffffff8169e6bf ffff8815a1203720 ffffffff816abbf0<br/> Nov 13 00:26:06 foxtrot2 kernel: ffff8815a12037a0 ffffffffc0c2d421 ffff8815a12037e0 ffffffffc0c2ba60<br/> Nov 13 00:26:06 foxtrot2 kernel: 0000000000000000 00000161000ab602 0004c5dab524ba0b ffff88130fb65c00<br/> Nov 13 00:26:06 foxtrot2 kernel: Call Trace:<br/> Nov 13 00:26:06 foxtrot2 kernel: <span class="error">&#91;&lt;ffffffff8169e6bf&gt;&#93;</span> queued_spin_lock_slowpath+0xb/0xf<br/> Nov 13 00:26:06 foxtrot2 kernel: <span class="error">&#91;&lt;ffffffff816abbf0&gt;&#93;</span> _raw_spin_lock+0x20/0x30<br/> Nov 13 00:26:06 foxtrot2 kernel: <span class="error">&#91;&lt;ffffffffc0c2d421&gt;&#93;</span> ldlm_prepare_lru_list+0x361/0x4e0 <span class="error">&#91;ptlrpc&#93;</span><br/> Nov 13 00:26:06 foxtrot2 kernel: <span class="error">&#91;&lt;ffffffffc0c2ba60&gt;&#93;</span> ? ldlm_cancel_aged_no_wait_policy+0x70/0x70 <span class="error">&#91;ptlrpc&#93;</span><br/> Nov 13 00:26:06 foxtrot2 kernel: <span class="error">&#91;&lt;ffffffffc0c30c5a&gt;&#93;</span> ldlm_cancel_lru_local+0x1a/0x30 <span class="error">&#91;ptlrpc&#93;</span><br/> Nov 13 00:26:06 foxtrot2 kernel: <span class="error">&#91;&lt;ffffffffc0c30e8e&gt;&#93;</span> ldlm_prep_elc_req+0x21e/0x490 <span class="error">&#91;ptlrpc&#93;</span><br/> Nov 13 00:26:06 foxtrot2 kernel: <span class="error">&#91;&lt;ffffffffc0c31128&gt;&#93;</span> ldlm_prep_enqueue_req+0x28/0x30 <span class="error">&#91;ptlrpc&#93;</span><br/> Nov 13 00:26:06 foxtrot2 kernel: <span class="error">&#91;&lt;ffffffffc07c67a3&gt;&#93;</span> mdc_intent_getattr_pack.isra.15+0x93/0x280 <span class="error">&#91;mdc&#93;</span><br/> Nov 13 00:26:06 foxtrot2 kernel: <span class="error">&#91;&lt;ffffffffc07c8f3b&gt;&#93;</span> mdc_enqueue_base+0x9fb/0x18f0 <span class="error">&#91;mdc&#93;</span><br/> Nov 13 00:26:06 foxtrot2 kernel: <span class="error">&#91;&lt;ffffffff810c45a3&gt;&#93;</span> ? try_to_wake_up+0x183/0x340<br/> Nov 13 00:26:06 foxtrot2 kernel: <span class="error">&#91;&lt;ffffffff810ba598&gt;&#93;</span> ? __wake_up_common+0x58/0x90<br/> Nov 13 00:26:06 foxtrot2 kernel: <span class="error">&#91;&lt;ffffffffc07ca6cb&gt;&#93;</span> mdc_intent_lock+0x26b/0x520 <span class="error">&#91;mdc&#93;</span><br/> Nov 13 00:26:06 foxtrot2 kernel: <span class="error">&#91;&lt;ffffffffc0c66243&gt;&#93;</span> ? reply_in_callback+0x143/0x5e0 <span class="error">&#91;ptlrpc&#93;</span><br/> Nov 13 00:26:06 foxtrot2 kernel: <span class="error">&#91;&lt;ffffffffc0972e30&gt;&#93;</span> ? ll_invalidate_negative_children+0x1d0/0x1d0 <span class="error">&#91;lustre&#93;</span><br/> Nov 13 00:26:06 foxtrot2 kernel: <span class="error">&#91;&lt;ffffffffc0c2c7a0&gt;&#93;</span> ? ldlm_expired_completion_wait+0x240/0x240 <span class="error">&#91;ptlrpc&#93;</span><br/> Nov 13 00:26:06 foxtrot2 kernel: <span class="error">&#91;&lt;ffffffffc0910e4f&gt;&#93;</span> lmv_intent_lock+0x5cf/0x1b50 <span class="error">&#91;lmv&#93;</span><br/> Nov 13 00:26:06 foxtrot2 kernel: <span class="error">&#91;&lt;ffffffff810b8a01&gt;&#93;</span> ? in_group_p+0x31/0x40<br/> Nov 13 00:26:06 foxtrot2 kernel: <span class="error">&#91;&lt;ffffffffc09738c5&gt;&#93;</span> ? ll_i2suppgid+0x15/0x40 <span class="error">&#91;lustre&#93;</span><br/> Nov 13 00:26:06 foxtrot2 kernel: <span class="error">&#91;&lt;ffffffffc0973914&gt;&#93;</span> ? ll_i2gids+0x24/0xb0 <span class="error">&#91;lustre&#93;</span><br/> Nov 13 00:26:06 foxtrot2 kernel: <span class="error">&#91;&lt;ffffffff81114b02&gt;&#93;</span> ? from_kgid+0x12/0x20<br/> Nov 13 00:26:06 foxtrot2 kernel: <span class="error">&#91;&lt;ffffffffc0972e30&gt;&#93;</span> ? ll_invalidate_negative_children+0x1d0/0x1d0 <span class="error">&#91;lustre&#93;</span><br/> Nov 13 00:26:06 foxtrot2 kernel: <span class="error">&#91;&lt;ffffffffc0974feb&gt;&#93;</span> ll_lookup_it+0x29b/0xee0 <span class="error">&#91;lustre&#93;</span><br/> Nov 13 00:26:06 foxtrot2 kernel: <span class="error">&#91;&lt;ffffffff810c8f28&gt;&#93;</span> ? __enqueue_entity+0x78/0x80<br/> Nov 13 00:26:06 foxtrot2 kernel: <span class="error">&#91;&lt;ffffffffc0976fbb&gt;&#93;</span> ll_lookup_nd+0xbb/0x190 <span class="error">&#91;lustre&#93;</span><br/> Nov 13 00:26:06 foxtrot2 kernel: <span class="error">&#91;&lt;ffffffff8120b3dd&gt;&#93;</span> lookup_real+0x1d/0x50<br/> Nov 13 00:26:06 foxtrot2 kernel: <span class="error">&#91;&lt;ffffffff8120bcb2&gt;&#93;</span> __lookup_hash+0x42/0x60<br/> Nov 13 00:26:06 foxtrot2 kernel: <span class="error">&#91;&lt;ffffffff816a13e2&gt;&#93;</span> lookup_slow+0x42/0xa7<br/> Nov 13 00:26:06 foxtrot2 kernel: <span class="error">&#91;&lt;ffffffff8120f25b&gt;&#93;</span> path_lookupat+0x77b/0x7b0<br/> Nov 13 00:26:06 foxtrot2 kernel: <span class="error">&#91;&lt;ffffffff811df623&gt;&#93;</span> ? kmem_cache_alloc+0x193/0x1e0<br/> Nov 13 00:26:06 foxtrot2 kernel: <span class="error">&#91;&lt;ffffffff81211c9f&gt;&#93;</span> ? getname_flags+0x4f/0x1a0<br/> Nov 13 00:26:06 foxtrot2 kernel: <span class="error">&#91;&lt;ffffffff8120f2bb&gt;&#93;</span> filename_lookup+0x2b/0xc0<br/> Nov 13 00:26:06 foxtrot2 kernel: <span class="error">&#91;&lt;ffffffff81212e37&gt;&#93;</span> user_path_at_empty+0x67/0xc0<br/> Nov 13 00:26:06 foxtrot2 kernel: <span class="error">&#91;&lt;ffffffff81212ea1&gt;&#93;</span> user_path_at+0x11/0x20<br/> Nov 13 00:26:06 foxtrot2 kernel: <span class="error">&#91;&lt;ffffffff812063e3&gt;&#93;</span> vfs_fstatat+0x63/0xc0<br/> Nov 13 00:26:06 foxtrot2 kernel: <span class="error">&#91;&lt;ffffffff812069b1&gt;&#93;</span> SYSC_newlstat+0x31/0x60<br/> Nov 13 00:26:06 foxtrot2 kernel: <span class="error">&#91;&lt;ffffffff81206c3e&gt;&#93;</span> SyS_newlstat+0xe/0x10<br/> Nov 13 00:26:06 foxtrot2 kernel: <span class="error">&#91;&lt;ffffffff816b5089&gt;&#93;</span> system_call_fastpath+0x16/0x1b</p> <p> </p> LU-11693

Soft lockups on Lustre clients

Bug Minor Open Unresolved Jian Yu Campbell Mcleay Thu, 22 Nov 2018 14:59:50 +0000 Thu, 28 Feb 2019 10:44:32 +0000 Lustre 2.10.2 0 5 <p>Versions are:</p> <p>lustre-2.10.2-1.el7</p> <p>kernel-3.10.0-693.5.2.el7_lustre</p> <p>This looks like a duplicate of a previously-reported issue. Please try:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># lctl set_param ldlm.namespaces.*.lru_size=10000 </pre> </div></div> <p>On these clients to see if this avoids the issue?</p> <p>Hi Andreas,</p> <p>I think we set this already, when I run:</p> <p>lctl get_param 'ldlm.namespaces.*.lru_size'</p> <p>I get:</p> <p>ldlm.namespaces.MGC10.21.22.10@tcp.lru_size=10000</p> <p>where 10.21.22.10 is our MDS</p> <p>Thanks,</p> <p>Campbell</p> <p>I should mention it prints out a list of values for all the OSTs and the MDT which are larger than this value, e.g., </p> <p>ldlm.namespaces.foxtrot-MDT0000-mdc-ffff883ff9b89000.lru_size=246531<br/> ldlm.namespaces.foxtrot-OST0000-osc-ffff883ff9b89000.lru_size=17716<br/> ldlm.namespaces.foxtrot-OST0001-osc-ffff883ff9b89000.lru_size=17472<br/> ldlm.namespaces.foxtrot-OST0002-osc-ffff883ff9b89000.lru_size=17561<br/> ldlm.namespaces.foxtrot-OST0003-osc-ffff883ff9b89000.lru_size=17628<br/> ldlm.namespaces.foxtrot-OST0004-osc-ffff883ff9b89000.lru_size=17492<br/> ldlm.namespaces.foxtrot-OST0005-osc-ffff883ff9b89000.lru_size=17555<br/> ldlm.namespaces.foxtrot-OST0006-osc-ffff883ff9b89000.lru_size=17334<br/> ldlm.namespaces.foxtrot-OST0007-osc-ffff883ff9b89000.lru_size=17511<br/> ldlm.namespaces.foxtrot-OST0008-osc-ffff883ff9b89000.lru_size=17534<br/> ldlm.namespaces.foxtrot-OST0009-osc-ffff883ff9b89000.lru_size=17689<br/> ldlm.namespaces.foxtrot-OST000a-osc-ffff883ff9b89000.lru_size=17609<br/> ldlm.namespaces.foxtrot-OST000b-osc-ffff883ff9b89000.lru_size=17144<br/> ldlm.namespaces.foxtrot-OST000c-osc-ffff883ff9b89000.lru_size=17438</p> <p>etc</p> <p>in case that is important</p> <p>The patch <a href="https://review.whamcloud.com/33130" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33130</a> "<a href="https://jira.whamcloud.com/browse/LU-9230" title="soft lockup on v2.9 Lustre clients (ldlm?)" class="issue-link" data-issue-key="LU-9230"><del>LU-9230</del></a> ldlm: speed up preparation for list of lock cancel" should resolve the CPU contention in <tt>ldlm_prepare_lru_list()</tt> that you are seeing here.</p> <p>This patch has been landed to the master branch (for 2.12) for over 6 months and has seen a lot of testing already. It is in the process of landing to the b2_10 branch for the next 2.10.x release, so there is not yet a release package available with this patch included. It is a client-only patch, so could be installed on the affected nodes without taking down the whole system.</p> <p>Thanks Andreas. I was looking for a compatibility matrix to see whether 2.12 on the client is compatible with 2.10 on the server. Is there something available online that shows compatibility of releases? </p> <p>regards,</p> <p>Campbell</p> <p>Actually, I see that 2.12 is not listed as supported by Whamcloud. I'll patch it then.</p> <p>Campbell</p> <p>2.12 is very close to release - we tagged the first RC yesterday. So, upon GA, another option to patching will be to use a 2.12 client as this interoperates with 2.10.x servers ok.</p> <p>Peter</p> <p>Thanks Peter. I applied the patches for both the crashes and the lockups to 2.10.2-1 source and it fails to build. Can you tell me what I need to do here? Attached is the build log <span class="nobr"><a href="https://jira.whamcloud.com/secure/attachment/31577/31577_build.log" title="build.log attached to LU-11693">build.log^{<img class="rendericon" src="https://jira.whamcloud.com/images/icons/link_attachment_7.gif" height="7" width="7" align="absmiddle" alt="" border="0"/>}</a></span></p> <p>Jian</p> <p>Could you please assist Campbell in porting the fix for <a href="https://jira.whamcloud.com/browse/LU-9230" title="soft lockup on v2.9 Lustre clients (ldlm?)" class="issue-link" data-issue-key="LU-9230"><del>LU-9230</del></a> to 2.10.2?</p> <p>thanks</p> <p>Peter</p> <p>I'm also happy to apply the patches to a later supported release if that is easier...</p> <p>Jian</p> <p>The port already exists to b2_10 - <a href="https://review.whamcloud.com/#/c/33130/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/33130/</a> - but does it need refreshing to apply to the tip of b2_10 or else to the 2.10.5 release?</p> <p>Peter</p> <p>Hi Peter,<br/> I just checked that the patch can be applied cleanly to both the tip of b2_10 and 2.10.5 release.</p> <p>Hi Campbell,<br/> Patch <a href="https://review.whamcloud.com/33130" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33130</a> is now on the tip of Lustre b2_10 branch. Please find the el7 builds in <a href="https://build.whamcloud.com/job/lustre-reviews/60456/" class="external-link" target="_blank" rel="nofollow noopener">https://build.whamcloud.com/job/lustre-reviews/60456/</a>.</p> <p>Thanks Jian. I still have to add a patch for a kernel panic issue (<a href="https://jira.whamcloud.com/browse/LU-11692" title="lustre kernel panic - (niobuf.c:330:ptlrpc_register_bulk()) LBUG" class="issue-link" data-issue-key="LU-11692"><del>LU-11692</del></a>), so might grab the src rpm for 2.10.5 and try to patch that.</p> <p>-Campbell</p> <p>2.10.5 fails to build. Should I send the build log?</p> <p>Campbell, I'm not sure what build problem you are seeing (we build this branch daily), but I've cherry-picked the <a href="https://jira.whamcloud.com/browse/LU-11692" title="lustre kernel panic - (niobuf.c:330:ptlrpc_register_bulk()) LBUG" class="issue-link" data-issue-key="LU-11692"><del>LU-11692</del></a> patch on top of 33130. It looks like the builders are a bit backed up, but there should be a link to a build reported in <a href="https://review.whamcloud.com/33798" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33798</a> in a couple of hours. Feel free to attach your build logs here, in case it is a trivial problem to fix.</p> <p>Hi Andreas,</p> <p>I'm doing something wrong here, I cloned git://git.whamcloud.com/fs/lustre-release.git and checked out the b2_10 branch, but the files are unpatched and I'm not quite sure how to add that patch via git. I can't find it to cherry-pick it. Or can I just add the patches manually via diff and patch? I was doing it this way before but the build fails (whereas an unpatched tree compiles fine). Sorry for my ignorance here.</p> <p>regards,</p> <p>Campbell</p> <p> </p> <p>Hi Campbell,<br/> Build <a href="https://build.whamcloud.com/job/lustre-reviews/60480/" class="external-link" target="_blank" rel="nofollow noopener">https://build.whamcloud.com/job/lustre-reviews/60480/</a> in <a href="https://review.whamcloud.com/33798" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33798</a> is ready. It contains both the patches for <a href="https://jira.whamcloud.com/browse/LU-11693" title="Soft lockups on Lustre clients" class="issue-link" data-issue-key="LU-11693">LU-11693</a>/<a href="https://jira.whamcloud.com/browse/LU-9230" title="soft lockup on v2.9 Lustre clients (ldlm?)" class="issue-link" data-issue-key="LU-9230"><del>LU-9230</del></a> and <a href="https://jira.whamcloud.com/browse/LU-11692" title="lustre kernel panic - (niobuf.c:330:ptlrpc_register_bulk()) LBUG" class="issue-link" data-issue-key="LU-11692"><del>LU-11692</del></a>/<a href="https://jira.whamcloud.com/browse/LU-11647" title="niobuf.c:330:ptlrpc_register_bulk()) ASSERTION( desc-&gt;bd_md_count == 0 ) failed:" class="issue-link" data-issue-key="LU-11647"><del>LU-11647</del></a> applied on the tip of Lustre b2_10 branch (tag 2.10.6-RC3).</p> <p>Campbell, what process are you using to build, and what files are "unpatched"? I'd recommend to follow e.g. <a href="https://wiki.whamcloud.com/pages/viewpage.action?pageId=52104622" class="external-link" target="_blank" rel="nofollow noopener">https://wiki.whamcloud.com/pages/viewpage.action?pageId=52104622</a> or <a href="http://wiki.lustre.org/Compiling_Lustre" class="external-link" target="_blank" rel="nofollow noopener">http://wiki.lustre.org/Compiling_Lustre</a> if you've never done this before. At its simplest, doing "<tt>sh autogen.sh; ./configure; make rpms</tt>" is all that is needed, once you have the kernel source RPMs but it can become more complex if you are using OFED, ZFS, etc.</p> <p>As Jian wrote, it is a lot easier to use a pre-built package if that has the features you need.</p> <p>Hi Andreas,</p> <p>I cloned the lustre repo and then checked the b2_10 branch. I then ran an autogen, copied the spec file to my rpmbuild tree and tarred the source up and copied it to rpmbuild/SOURCES. I was expecting the b2_10 to already be patched but a comparison showed it hadn't been. I created a patch file from a recursive diff and then modify the spec file to apply that patch. I then built a source rpm and tried an rpm rebuild. I was getting build errors, e.g., </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> /u/cmcl/rpmbuild/BUILD/lustre-2.10.2/lustre/include/lustre_lib.h:357:9: error: implicit declaration of function <span class="code-quote">'is_bl_done'</span> [-Werror=implicit-function-declaration] struct l_wait_info *__info = (info); \ ^ /u/cmcl/rpmbuild/BUILD/lustre-2.10.2/lustre/ptlrpc/../../lustre/ldlm/ldlm_lock.c:2330:3: note: in expansion of macro <span class="code-quote">'l_wait_event'</span> l_wait_event(lock-&gt;l_waitq, is_bl_done(lock), &amp;lwi) </pre> </div></div> <p>I'm doing something wrong and/or in an overly complicated way. I thought the b2_10 branch would have already been patched.<br/> I'd seen the whamcloud wiki page you'd mentioned but thought that was for server rather than client. The wiki.lustre.org I hadn't seen.<br/> Anyway, I found a build on <a href="https://review.whamcloud.com/33798" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33798</a> linked by Jian which has the patches in it, so I'll build from that. Sorry for wasting your time with this but hopefully I'll be on the right track from here on in.</p> <p>Cheers,</p> <p>Campbell</p> <p>Glad to hear that you've got this sorted out. Let us know whether the fix works as expected.</p> <p>I've built the rpms fine but I have another question: the client has the lustre kernel package installed (I am told it was installed as the lustre kernel has better performance than a vanilla kernel), which provides the fs and net kernel modules. The kmod-lustre-client package provides the kernel modules, though it installs them in /lib/modules/`uname -r`/extra/lustre-client rather than /lib/modules/`uname -r`/extra/lustre. Will this cause any kind of issue if both are installed, or is it better to install e.g., a vanilla kernel and rebuild the packages against this?</p> <p>Thanks,</p> <p>Campbell</p> <p>Hi Campbell,<br/> Lustre client is patchless, which means while building Lustre codes for client, we do not need to patch Linux vendor or vanilla kernel. All of the regression testings were performed on patchless Lustre clients, so we suggest to use vendor kernel.</p> <p>Campbell</p> <p>Even the servers only need to be patched if you are using the project quotas feature. The patches that gave performance improvements in past versions have now been upstreamed and many customers prefer the simplified admin over project quotas...</p> <p>Peter</p> <p>Just some feedback: got some soft lockups on one of our clients, though it only happened once however. The other clients have been fine.</p> Duplicate LU-11391 LU-9230 Development Rank 1|i006sf: Rank (Obsolete) 9223372036854775807 Severity