https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-10510 Lustre <p>Some sites have encountered issues with messages like:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>May 10 13:54:50 oss02 kernel: blk_cloned_rq_check_limits: over max size limit. May 10 13:54:50 oss02 kernel: blk_cloned_rq_check_limits: over max size limit. May 10 13:54:50 oss02 kernel: device-mapper: multipath: Failing path 8:48. May 10 13:54:50 oss02 kernel: blk_cloned_rq_check_limits: over max size limit. May 10 13:54:50 oss02 kernel: blk_cloned_rq_check_limits: over max size limit. May 10 13:54:50 oss02 kernel: device-mapper: multipath: Failing path 8:240. </pre> </div></div> <p>This issue can cause corruption on storage side. It can be fixed by reverting an upstream patch or adding a udev script (but we will change max_sectors_kb value while mounting time, so it still can be triggered even with this script). The better way is to handle this value in mount.lustre tool. I'll make up a patch for master first.</p> LU-10510

Fix 'over max size limit' issue

Bug Minor Closed Duplicate Yang Sheng Yang Sheng Sun, 14 Jan 2018 17:50:12 +0000 Fri, 15 Nov 2019 05:05:19 +0000 Tue, 29 May 2018 05:08:21 +0000 0 12 <p>Cliff hit this on the soak test cluster. He will collect <tt>grep . /sys/block/{dm&#42;,sd&#42;}/queue/max*sectors_kb</tt> for the affected MDT multipath device and the underlying SCSI devices (we don't need all of the others). My suspicion is that one of the underlying devices was reset for some reason, and <tt>max_sectors_kb</tt> is the default (maybe 128) but the multipath device is set larger (maybe 1024 or 16384) and this is causing the failures.</p> <p>Using <tt>mount.lustre -o max_sectors_kb=0</tt> will prevent Lustre from changing these tunables in the first place, which may hurt OST performance, but likely has a lesser effect on MDT performance. However, if the system is already in this state (underlying devices have lower <tt>max_sectors_kb</tt> than parent), this needs to be fixed manually to change the underlying settings (or a reboot would work).</p> <p>Yang Sheng, can you please look at what could be done for creating a udev script to handle this? I guess the tricky part is that we don't want to install a udev script to cover all devices from the RPM package, we only want to affect the Lustre target devices.</p> <p>One option would be to generate udev rules like <tt>/etc/udev/rules.d/99-lustre-&lt;device&gt;.rules</tt> at <tt>mkfs.lustre</tt> time for the multipath and children devices. That wouldn't help existing filesystems, and means users could see a performance regression when they upgraded to a version of <tt>mkfs.lustre</tt> that doesn't include this tuning, if users don't know the tunable is no longer applied by <tt>mount.lustre</tt>.</p> <p>A second option would be to change <tt>tune_max_sectors_kb()</tt> to generate and install udev tuning rules for each mounted device and children if they do not already exist. This would be least complex, but possibly a surprising action for <tt>mount.lustre</tt> to take. However, it is not worse than our current practice of changing the block device tunables at mount time</p> <p>A third option (possibly in addition to the first) would be to change <tt>tune_max_sectors_kb()</tt> to complain about the lack of udev tuning rules for the device and children, or if the multipath child device settings do not match the parent, and indicate how to create the udev rules, but not actually install the rules. Then, the admin can create such a rule and add any tunables desired to quiet <tt>mount.lustre</tt>, and this should avoid having it do the tuning itself.</p> <p>Known issue certain kernels, related to mismatch in <b>max_sectors_kb</b> setting between dm devices (eg. mutipath) and underlying block devices.<br/> If you have redhat account you can read their KB articles:</p> <p><a href="https://access.redhat.com/solutions/247991" class="external-link" target="_blank" rel="nofollow noopener">https://access.redhat.com/solutions/247991</a></p> <p><a href="https://access.redhat.com/solutions/145163" class="external-link" target="_blank" rel="nofollow noopener">https://access.redhat.com/solutions/145163</a><br/>  </p> <p>This issue should be fixed after <a href="https://review.whamcloud.com/31951/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/31951/</a> landed.</p> Duplicate Related LU-9132 Development Rank 1|hzzr3z: Rank (Obsolete) 9223372036854775807 Severity