[LU-9551] I/O errors when lustre uses multipath devices

• Type: Bug
• Resolution: Fixed
• Priority: Critical
• Fix Version/s: Lustre 2.12.0, Lustre 2.10.4
• Affects Version/s: Lustre 2.7.0
• Environment: CentOS Linux release 7.3.1611 (Core),OFED.3.4.2.0.0.1,lustre-2.7.19.8,Mellanox Technologies MT27500 Family

Epic/Theme

centos7.3 lustre-2.7.19.8

• Severity: 3

Epic

mount server

• Assignee: Nathaniel Clark
• Reporter: xiangmin shen (Inactive)

Description

When the lustre servers have OST configured with multipath devices, there are I/O errors that can lead to a server crash.

The following error appears in the system log:
Mar 31 00:02:44 oss01 kernel: blk_cloned_rq_check_limits: over max size limit.
Mar 31 00:02:44 oss01 kernel: device-mapper: multipath: Failing path 8:160.

Followed by several I/O errors
Mar 31 00:17:30 oss01 kernel: blk_update_request: I/O error, dev dm-17, sector 1182279680
Mar 31 00:17:30 oss01 kernel: blk_update_request: I/O error, dev dm-17, sector 1182291968
Mar 31 00:17:30 oss01 kernel: blk_update_request: I/O error, dev dm-17, sector 1182267392
Mar 31 00:17:30 oss01 kernel: blk_update_request: I/O error, dev dm-17, sector 1182304256
Mar 30 21:04:22 oss01 kernel: LDISKFS-fs (dm-17): Remounting filesystem read-only

Issue Links

• duplicates: LU-9132 Tuning max_sectors_kb on mount
• is duplicated by: LU-9552 lustre uses multipath devices I/O errors
• is related to: LU-12029 do not try to muck with max_sectors_kb on multipath configurations
• is related to: LU-12387 l_tunedisk does not properly handle multipath devices
• is related to: LU-11563 99-lustre.rules on clients tries to execute non-existent /usr/sbin/l_tunedisk
• is related to: LU-12530 udev add/change rule loads zfs module on clients
• is related to: LU-11736 Do not apply bulk IO tuning on MDT or MGT devices
• is related to: LU-10898 conf-sanity test 32a and 32d fail with ‘rmmod: ERROR: Module zfs is in use’

Comments

[Gerrit Updater]

John L. Hammond ([e-mail]) merged in patch https://review.whamcloud.com/32583/
Subject: LU-9551 utils: add l_tunedisk to fix disk tunings
Project: fs/lustre-release
Branch: b2_10
Current Patch Set:
Commit: 703d418908fa32f60decc3bd535e77784d2721c6

[Gerrit Updater]

Nathaniel Clark ([e-mail]) uploaded a new patch: https://review.whamcloud.com/32583
Subject: LU-9551 utils: add l_tunedisk to fix disk tunings
Project: fs/lustre-release
Branch: b2_10
Current Patch Set: 1
Commit: 1743fa638e8fdbe16e6cfd33dd91c24fa5047492

[Peter Jones]

Yes we want to resubmit it

[Nathaniel Clark]

This got reverted on b2_10, but it didn't actually cause LU-10898 (afaik).  ZED holds zfs open if it's running.  Can we re-land this?  Should I resubmit?

[Peter Jones]

It looks like it is going to be fixed under LU-10898 rather than reverted so keeping as resolved

[Minh Diep]

This patch caused LU-10898

[Gerrit Updater]

John L. Hammond ([e-mail]) merged in patch https://review.whamcloud.com/31951/
Subject: LU-9551 utils: add l_tunedisk to fix disk tunings
Project: fs/lustre-release
Branch: b2_10
Current Patch Set:
Commit: 3281d5d57cec9d6deaa50cb4d9ec9509e3d03507

[Chris Hunter (Inactive)]

The old mount method in LU-275 sets value from sysfs/block parameter max_hw_sectors_kb.

However due to bugs in the transport protocol this value can be wrong (https://patchwork.kernel.org/patch/7614871/; https://patchwork.kernel.org/patch/6662311/) and produce an error when used by lustre mount command.
Feature in LU-9132 to adjust mount behaviour would be useful in this scenario.

[Gerrit Updater]

Minh Diep ([e-mail]) uploaded a new patch: https://review.whamcloud.com/31951
Subject: LU-9551 utils: add l_tunedisk to fix disk tunings
Project: fs/lustre-release
Branch: b2_10
Current Patch Set: 1
Commit: b30fb047c12c6354df2d81e2a0cd5dd21852f6b3

[Peter Jones]

Landed for 2.12