https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-9551 Lustre <p>When the lustre servers have OST configured with multipath devices, there are I/O errors that can lead to a server crash.</p> <p>The following error appears in the system log:<br/> Mar 31 00:02:44 oss01 kernel: blk_cloned_rq_check_limits: over max size limit.<br/> Mar 31 00:02:44 oss01 kernel: device-mapper: multipath: Failing path 8:160.</p> <p>Followed by several I/O errors<br/> Mar 31 00:17:30 oss01 kernel: blk_update_request: I/O error, dev dm-17, sector 1182279680<br/> Mar 31 00:17:30 oss01 kernel: blk_update_request: I/O error, dev dm-17, sector 1182291968<br/> Mar 31 00:17:30 oss01 kernel: blk_update_request: I/O error, dev dm-17, sector 1182267392<br/> Mar 31 00:17:30 oss01 kernel: blk_update_request: I/O error, dev dm-17, sector 1182304256<br/> Mar 30 21:04:22 oss01 kernel: LDISKFS-fs (dm-17): Remounting filesystem read-only</p> CentOS Linux release 7.3.1611 (Core),OFED.3.4.2.0.0.1,lustre-2.7.19.8,Mellanox Technologies MT27500 Family LU-9551

I/O errors when lustre uses multipath devices

Bug Critical Resolved Fixed Nathaniel Clark xiangmin shen Wed, 24 May 2017 07:03:51 +0000 Fri, 3 Dec 2021 22:40:56 +0000 Fri, 13 Apr 2018 18:54:19 +0000 Lustre 2.7.0 Lustre 2.12.0 Lustre 2.10.4 0 11 <p>Message " blk_cloned_rq_check_limits" seen on non-lustre filesystems, believed caused by upstream commit to 4.3 kernel<br/> <a href="https://patchwork.kernel.org/patch/8307491/" class="external-link" target="_blank" rel="nofollow noopener">https://patchwork.kernel.org/patch/8307491/</a></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Feb. 14, 2016, 10:20 p.m. From: Hannes Reinecke &lt;hare@suse.de&gt; commit bf4e6b4e757488dee1b6a581f49c7ac34cd217f8 upstream. When a cloned request is retried on other queues it always needs to be checked against the queue limits of that queue. Otherwise the calculations for nr_phys_segments might be wrong, leading to a crash in scsi_init_sgtable(). </pre> </div></div> <p>We just hit this at ANU. The fix is to ensure that max_sectors_kb is 'large enough'.</p> <p>We had an issue where multipath was generating 1MB I/Os (as that's what lustre was configured for) but the underlying /dev block devices had max_sectors_kb = 512</p> <p>I'm not sure how that is possible, but naturally it was resolved by adding a udev rule to set max_sectors_kb=&gt; 1024 but &lt; max_hw_sectors_kb</p> <p>I'm not sure if this is actually a lustre error or a multipath error. Based on my reading of <a href="https://patchwork.kernel.org/patch/9140337/" class="external-link" target="_blank" rel="nofollow noopener">https://patchwork.kernel.org/patch/9140337/</a><br/> this is resolved in a new enough kernel but it seems that there might be some patches that require backporting into Centos/RHEL</p> <p>EDIT: Interestingly this was only seen months after the filesystem went into production.</p> <p>EDIT: yes I know that patch is for ppc.. The conversation was relevant. </p> <p>One possible workaround is described in <a href="https://jira.whamcloud.com/browse/LU-9132" title="Tuning max_sectors_kb on mount" class="issue-link" data-issue-key="LU-9132"><del>LU-9132</del></a> setting env variable "MOUNT_LUSTRE_MAX_SECTORS_KB=0", which will stop mount.lustre from changing max_sectors_kb when mounting OSTs. The OSTs would retain the max_sectors_kb value set by your udev rules.</p> <p>Has that been backported into 2.7/IEEL3?</p> <p>I can see that it exists in 2.10 and Master.</p> <p>Also It doesn't explain why we would get issues months after going live. The OST's were mounted and were not remounted.</p> <p>Also this might not fix it. Our issue seemed to come from the fact that the backing devices behind multipath had been reset to the default 512 value. Not the multipath devices that lustre was mounted on. </p> <p>Our udev rules only change the backing devices/paths not the resulting dm-X devices lustre is mounted on</p> <p>Reading some of the discussions on the kernel.org threads it seems that also during failover between paths multipath can do the wrong thing and not check against max_sectors_kb and only check max_hw_sectors_kb. </p> <p>Previously, this would not have been an issue. But with the extra checks, this is clearly an issue.</p> <p>The exact cause of our issues was discovered:</p> <p>Lustre had increased the values at mount, some paths went away and came back. They were set to default values upon return.</p> <p>Prior to the patch to the kernel this would not have been an issue, so for us the udev rule enforcing max on probe will resolve the issue </p> <p>This is fixed in more current releases</p> <p>Nathaniel Clark (nathaniel.l.clark@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/31464" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/31464</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9551" title="I/O errors when lustre uses multipath devices" class="issue-link" data-issue-key="LU-9551"><del>LU-9551</del></a> utils: add l_tunedisk to fix disk tunings<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 681f208b5ec25a12eeac5a7c1cea238154ffd6ff</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/31464/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/31464/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9551" title="I/O errors when lustre uses multipath devices" class="issue-link" data-issue-key="LU-9551"><del>LU-9551</del></a> utils: add l_tunedisk to fix disk tunings<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 892280742a2b6347df1464379b3ed223b2961ed4</p> <p>Landed for 2.12</p> <p>Minh Diep (minh.diep@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/31951" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/31951</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9551" title="I/O errors when lustre uses multipath devices" class="issue-link" data-issue-key="LU-9551"><del>LU-9551</del></a> utils: add l_tunedisk to fix disk tunings<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: 1<br/> Commit: b30fb047c12c6354df2d81e2a0cd5dd21852f6b3</p> <p>The old mount method in <a href="https://jira.whamcloud.com/browse/LU-275" title="I/O errors when lustre uses multipath devices" class="issue-link" data-issue-key="LU-275"><del>LU-275</del></a> sets value from sysfs/block parameter max_hw_sectors_kb. </p> <p>However due to bugs in the transport protocol this value can be wrong (<a href="https://patchwork.kernel.org/patch/7614871/" class="external-link" target="_blank" rel="nofollow noopener">https://patchwork.kernel.org/patch/7614871/</a>; <a href="https://patchwork.kernel.org/patch/6662311/" class="external-link" target="_blank" rel="nofollow noopener">https://patchwork.kernel.org/patch/6662311/</a>) and produce an error when used by lustre mount command. <br/> Feature in <a href="https://jira.whamcloud.com/browse/LU-9132" title="Tuning max_sectors_kb on mount" class="issue-link" data-issue-key="LU-9132"><del>LU-9132</del></a> to adjust mount behaviour would be useful in this scenario.</p> <p>John L. Hammond (john.hammond@intel.com) merged in patch <a href="https://review.whamcloud.com/31951/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/31951/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9551" title="I/O errors when lustre uses multipath devices" class="issue-link" data-issue-key="LU-9551"><del>LU-9551</del></a> utils: add l_tunedisk to fix disk tunings<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: <br/> Commit: 3281d5d57cec9d6deaa50cb4d9ec9509e3d03507</p> <p>This patch caused <a href="https://jira.whamcloud.com/browse/LU-10898" title="conf-sanity test 32a and 32d fail with ‘rmmod: ERROR: Module zfs is in use’" class="issue-link" data-issue-key="LU-10898"><del>LU-10898</del></a></p> <p>It looks like it is going to be fixed under <a href="https://jira.whamcloud.com/browse/LU-10898" title="conf-sanity test 32a and 32d fail with ‘rmmod: ERROR: Module zfs is in use’" class="issue-link" data-issue-key="LU-10898"><del>LU-10898</del></a> rather than reverted so keeping as resolved</p> <p>This got reverted on b2_10, but it didn't actually cause <a href="https://jira.whamcloud.com/browse/LU-10898" title="conf-sanity test 32a and 32d fail with ‘rmmod: ERROR: Module zfs is in use’" class="issue-link" data-issue-key="LU-10898"><del>LU-10898</del></a> (afaik).  ZED holds zfs open if it's running.  Can we re-land this?  Should I resubmit?</p> <p>Yes we want to resubmit it</p> <p>Nathaniel Clark (nathaniel.l.clark@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/32583" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/32583</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9551" title="I/O errors when lustre uses multipath devices" class="issue-link" data-issue-key="LU-9551"><del>LU-9551</del></a> utils: add l_tunedisk to fix disk tunings<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: 1<br/> Commit: 1743fa638e8fdbe16e6cfd33dd91c24fa5047492</p> <p>John L. Hammond (jhammond@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/32583/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/32583/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9551" title="I/O errors when lustre uses multipath devices" class="issue-link" data-issue-key="LU-9551"><del>LU-9551</del></a> utils: add l_tunedisk to fix disk tunings<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: <br/> Commit: 703d418908fa32f60decc3bd535e77784d2721c6</p> Duplicate LU-9132 LU-9552 Related LU-10898 LU-12029 LU-12387 LU-11563 LU-12530 LU-11736 Development Epic mount server Epic/Theme centos7.3 lustre-2.7.19.8 Rank 1|hzzdin: Rank (Obsolete) 9223372036854775807 Severity