[LU-7851] ZFS rolling upgrade: cannot mount MDS: mdt: lustre-MDT0000 unknown param som=disabled

• Type: Bug
• Resolution: Fixed
• Priority: Critical
• Fix Version/s: Lustre 2.9.0
• Affects Version/s: Lustre 2.8.0
• Environment: before upgrade, system was formatted as 2.7.1 RHEL6.7 zfs
• Severity: 3
• Assignee: Hongchao Zhang
• Reporter: Sarah Liu

Description

Before upgrade, system was formatted as 2.7.1 RHEL6.7 zfs
1. upgrade OSS from 2.7.1 RHEL6.7 zfs to 2.8.0 RHEL7.1 zfs, MDS and clients were remained 2.7.1, ran sanity, hit known issues;
2. then upgrade MDS from 2.7.1 RHEL6.7 zfs to 2.8.0 RHEL7.1 zfs, cannot mount

[root@onyx-25 ~]# mount -t lustre -o acl,user_xattr lustre-mdt1/mdt1 /mnt/mds1
[ 1753.734535] Lustre: MGS: Connection restored to 3f133170-911b-365a-d7e8-6a31eadc485c (at 0@lo)
[ 1755.859175] LustreError: 22512:0:(obd_config.c:1387:class_process_proc_param()) mdt: lustre-MDT0000 unknown param som=disabled
[ 1755.878206] LustreError: 22512:0:(obd_config.c:1666:class_config_llog_handler()) MGC10.2.4.47@tcp: cfg command failed: rc = -38
[ 1755.896907] Lustre:    cmd=cf00f 0:lustre-MDT0000  1:mdt.som=disabled  
[ 1755.896907] 
[ 1755.911461] LustreError: 15c-8: MGC10.2.4.47@tcp: The configuration from log 'lustre-MDT0000' failed (-38). This may be the result of communication errors between this node and the MGS, a bad configuration, or other errors. See the syslog for more information.
[ 1755.946877] LustreError: 22417:0:(obd_mount_server.c:1309:server_start_targets()) failed to start server lustre-MDT0000: -38
[ 1755.962800] LustreError: 22417:0:(obd_mount_server.c:1798:server_fill_super()) Unable to start targets: -38
[ 1755.976801] Lustre: Failing over lustre-MDT0000
[ 1763.422747] Lustre: 22417:0:(client.c:2063:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1457156520/real 1457156520]  req@ffff8807f1960000 x1527938522546332/t0(0) o251->MGC10.2.4.47@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1457156526 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
[ 1765.473299] Lustre: server umount lustre-MDT0000 complete
[ 1765.481424] LustreError: 22417:0:(obd_mount.c:1426:lustre_fill_super()) Unable to mount  (-38)
mount.lustre: mount lustre-mdt1/mdt1 at /mnt/mds1 failed: Function not implemented

Issue Links

• is related to: LU-6047 remove client Size on MDS support

Comments

[Joseph Gmitter (Inactive)]

Landed to master for 2.9.0

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch http://review.whamcloud.com/18834/
Subject: LU-7851 mdt: skip SOM related configuration
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 109cdb83d77844e00ebc6d5df6bf73845cf9d45e

[Sarah Liu]

hi Hongchao,

sorry for the late reply. So here is the scenario will hit the problem
1. format and setup system as 2.5.5/2.7.1 zfs
2. upgrade OSS only to 2.8 zfs
*3. run sanity with 2.8 OSS, 2.5.5/2.7.1 MDS and clients
4. now upgrade MDS to 2.8 zfs, hit the problem

Step 3 is the key, without running sanity with such config won't hit this problem.

[Gerrit Updater]

Hongchao Zhang ([e-mail]) uploaded a new patch: http://review.whamcloud.com/18834
Subject: LU-7851 mdt: skip SOM related configuration
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 1555ccad216290befa4f32d93b366b6440bcc9d2

[Hongchao Zhang]

Oleg: Okay, I'll create a patch to ignore the configuration.

[Oleg Drokin]

Hongchao: even when users of the actual option might be rare, we still need the patch to ignore it since it was a valid one.

That said making sure it does not hit by default is still good.