[LU-8044] class_process_config() no device for: lustre-MDT0021-mdtlov

• Type: Bug
• Resolution: Fixed
• Priority: Major
• Fix Version/s: Lustre 2.9.0
• Affects Version/s: Lustre 2.8.0
• Labels: llnl

Environment

TOSS 2 (RHEL 6.7 based)
kernel 2.6.32-573.22.1.1chaos.ch5.4.x86_64
Lustre 2.8.0+patches 2.8-llnl-preview1
zfs-0.6.5.4-1.ch5.4.x86_64
1 MGS - separate server
40 MDTs - each on separate server
10 OSTs - each on separate server
Filesystem name is "lustre"

• Severity: 3
• Assignee: Di Wang
• Reporter: Olaf Faaland

Description

On startup for the first time after formatting, the MDT fails to process the config provided by the MGS. The MDT then fails to start.
The config log on the MGS appears to be invalid, with more than one setup and modify_mdc_tgt record for one of the other MDTs.

The MDT which fails to start reports:

Lustre: Lustre: Build Version: 2.8.0
LustreError: 11797:0:(obd_config.c:1262:class_process_config()) no device for: lustre-MDT0021-mdtlov
LustreError: 11797:0:(obd_config.c:1666:class_config_llog_handler()) MGC192.168.112.240@o2ib15: cfg command failed: rc = -22
Lustre:    cmd=cf014 0:lustre-MDT0021-mdtlov  1:lustre-MDT0014_UUID  2:20  3:1

LustreError: 15b-f: MGC192.168.112.240@o2ib15: The configuration from log 'lustre-MDT0021'failed from the MGS (-22).  Make sure this client and the MGS are running compatible versions of Lustre.
LustreError: 11667:0:(obd_mount_server.c:1309:server_start_targets()) failed to start server lustre-MDT0021: -22
LustreError: 11667:0:(obd_mount_server.c:1798:server_fill_super()) Unable to start targets: -22
LustreError: 11667:0:(obd_mount_server.c:1512:server_put_super()) no obd lustre-MDT0021
Lustre: server umount lustre-MDT0021 complete
LustreError: 11667:0:(obd_mount.c:1426:lustre_fill_super()) Unable to mount  (-22)

The config logs CONFIGS/lustre-MDT* do not all have the same number of records. lustre-MDT0021 has 2 more records than the other 29 MDTs.

The suspicious llog records are:

#04 (152)setup     0:lustre-MDT0014-osp-MDT0021  1:lustre-MDT0014_UUID  2:192.168.113.6@o2ib15
#05 (136)modify_mdc_tgts add 0:lustre-MDT0021-mdtlov  1:lustre-MDT0014_UUID  2:20  3:1
#179 (152)setup     0:lustre-MDT0014-osp-MDT0021  1:lustre-MDT0014_UUID  2:192.168.113.6@o2ib15
#180 (136)modify_mdc_tgts add 0:lustre-MDT0021-mdtlov  1:lustre-MDT0014_UUID  2:20  3:1