[LU-7668] permanently remove deactivated OSTs from configuration log

• Type: Improvement
• Resolution: Fixed
• Priority: Minor
• Fix Version/s: Lustre 2.16.0
• Assignee: Stephane Thiell
• Reporter: Andreas Dilger

Description

When an OST is permanently removed from the filesystem, the current process is to store a conf_param that marks the OSC permanently inactive in the configuration log, but it doesn't remove the actual OSC records from the config llog. That is needed for clients already mounting the filesystem so that they don't wait for the OST to be recovered, since only new records added to the end of config llog are processed by already-mounted clients.

However, for clients newly mounting the filesystem it may be desirable, if the OST is permanently deleted, to remove the OST record from the configuration log completely so that new client mounts don't even try to connect to it. This can be done relatively easily by cancelling the llog record(s) for the removed OST(s) so that the client doesn't process them at all.

The other area that may need fixing is lfs df, since it currently iterates over OST and MDT indices sequentially until it gets a -ENODEV return code that indicates no more OST/MDT devices are available. This will result in temporarily inactive OSTs to be printed, since the admin should know when there are offline OSTs, but it should not result in unconfigured OSTs being printed. It looks like -EAGAIN being returned from IOC_OBD_STATFS will result in the OST/MDT being silently skipped as we would want in this case. This needs to be verified.

It looks like lov_iocontrol() is returning -EAGAIN correctly, but lmv_iocontrol() is incorrectly returning -ENODATA for tgt == NULL instead of -EAGAIN, and not handling the OBD_STATFS_NODELAY flag in uarg at all.

Issue Links

• is blocked by: LU-15000 MDS crashes with (osp_dev.c:1404:osp_obd_connect()) ASSERTION( osp->opd_connects == 1 ) failed
• is related to: LU-7731 lctl dl - command don't report correct 'active' - state
• is related to: LU-14090 lctl replace_nids and starting target with local copy of logs
• is related to: LU-14403 lctl dl UP and lfs df problem with conf_param osc.active=0 after client remount
• is related to: LUDOC-352 document lctl llog_print
• is related to: LU-17299 DNE3: disable new regular file creation on MDTs mounted with 'no_create'
• is related to: LU-16475 Reusing OST indexes after lctl del_ost
• is related to: LU-12998 DNE3: tunable to disable directory creation on MDT
• is related to: LU-4295 removing files on deactivated OST doesn't free up space
• is related to: LU-4397 Permanently disabled OST causes clients to hang on df (statfs)
• is related to: LU-6601 deactivated OSTs do not appear to be inactive according to 'lctl dl'
• is related to: LU-16024 Allow permanently removing an MDT from config
• is related to: LU-6818 quiet permanently deactivated OSTs in "lfs df"
• is related to: LU-8920 don't print permanently deactivated OSTs in lfs df output

Comments

[Gerrit Updater]

"Andreas Dilger <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/48237
Subject: LU-7668 misc: add support for versions 2.17/2.18
Project: doc/manual
Branch: master
Current Patch Set: 1
Commit: 6632007c3a4b9ce912593133df6546fc128433ff

[Gerrit Updater]

"Andreas Dilger <[e-mail]>" merged in patch https://review.whamcloud.com/41454/
Subject: LU-7668 admin: reference the del_ost command
Project: doc/manual
Branch: master
Current Patch Set:
Commit: 805e8a34afc5002bb1d843faeb067dfc4d70444b

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/41449/
Subject: LU-7668 utils: add lctl del_ost
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 1121816c4a4e1bb2ef097c4a9802362181c43800

[Etienne Aujames]

It appears that the method describes in https://review.whamcloud.com/41453/ is not safe (for now) .

lctl llog_cancel remove/cancel directly the record inside llog bitmap header. This creates index gaps in the llog config. llog_backup() is used to create a local copy of the MGS config when mounting a target. This function does not keep the index gaps in llog copy.
So indexes mismatch between the copy and the original, it mess up the config update mechanism (e.g: adding a new target).

For more information see LU-15000.

[Gerrit Updater]

Andreas Dilger ([e-mail]) merged in patch https://review.whamcloud.com/41453/
Subject: LU-7668 admin: remove OST from config logs
Project: doc/manual
Branch: master
Current Patch Set:
Commit: c7a2886198b7273165d90d5054e7da669ba97843

[Etienne Aujames]

Hello Stephane,

I think that the LU-14090 describes your issue.

The patch https://review.whamcloud.com/40448 ("LU-14090 mgs: no local logs flag") add a tunefs.lustre option to ignore local log.

[Stephane Thiell]

I'm still working on this, but we hit an unusual problem with llog_cancel / del_ost prototype. It worked fine at first, but after a few days, we restarted a MDS while the MGS was down, and two MDTs used a cached version of their config to start, trying to connect to the old OSTs again and we hit a MDS crash same as LU-9699. This cached version didn't even have the latest OSTs that we added to the filesystem. I'm wondering if there is a way to make sure the MDTs have up-to-date versions of their config, especially after adding or removing OSTs.

[Andreas Dilger]

Etienne, I haven't looked into the details, but it would leave an unused network connection to that OSS, which may cause spurious connection attempts, though it may be the connection is never used if there are no targets on it. Definitely worthwhile to test out at least.

[Etienne Aujames]

Hello Andreas,

What are the consequences to keep the "add_uuid" records when removing all the OSTs of a node?

Should we add an option to lctl del_ost" to remove "add_uuid" records if all the OSTs of a node have been deleted?

[Andreas Dilger]

Essentially, yes. The llog_cancel command doesn't actually remove the records from the config log, but they are marked as "already processed" so they are skipped by the client and MDS during mount.

[Cedric Castagnede (Inactive)]

Hello Andreas,

Instead of writing the Lustre Configuration Logs using "tunefs.lustre --writeconf", we could proceed using llog_cancel to completely remove OST from the configuration and get the same results for our case.

Am I understanding properly here ?