[LU-11227] client process hangs when lod_sync accesses deactivated OSTs

• Type: Bug
• Resolution: Fixed
• Priority: Minor
• Fix Version/s: Lustre 2.12.0, Lustre 2.10.6
• Affects Version/s: Lustre 2.11.0, Lustre 2.10.4
• Environment: x64_64, ldiskfs, not DNE
• Severity: 3
• Assignee: Robin Humble (Inactive)
• Reporter: Robin Humble (Inactive)

Description

Hi,

we have a ldiskfs Lustre install where one OST is permanently deactivated with

lctl conf_param lustre-OST005a.osc.active=0

we were in IEEL and saw no issues there, but are now in 2.10.4 to try and get better compatibility for file transfers to the new cluster.

the problem is that in 2.10.4 a chgrp on the client hangs forever as it re-tries infinitely. MDS load is significant too.

message from the MDS when the chgrp hangs is 1000's of these per second

Aug  7 19:45:41 metadata01 kernel: LustreError: 4502:0:(lod_dev.c:1400:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 90: -107

looking at that lod_sync() code, it doesn't check for OSTs being deactivated. the below seems to work as a quick fix so that we don't have to reboot back into IEEL.

diff --git a/lustre/lod/lod_dev.c b/lustre/lod/lod_dev.c
index d61ad2d..2194110 100644
--- a/lustre/lod/lod_dev.c
+++ b/lustre/lod/lod_dev.c
@@ -1409,6 +1409,8 @@ static int lod_sync(const struct lu_env *env, struct dt_device *dev)
        lod_foreach_ost(lod, i) {
                ost = OST_TGT(lod, i);
                LASSERT(ost && ost->ltd_ost);
+               if (!ost->ltd_active)
+                       continue;
                rc = dt_sync(env, ost->ltd_ost);
                if (rc) {
                        CERROR("%s: can't sync ost %u: %d\n",

the same fix would also seem to be appropriate for a deactivated MDT a few lines lower down.

please let me know if this is totally the wrong thing to do, or if alternatively if it's useful and you'd like me to upload a patch to Gerrit.

also dry-run of lfsck in 2.10.4 wants to correct layout for 15% of mdt files, and namespace (linkea_inconsistent) for about 15k files out of 225M. presumably because of this fs being old and/or a bit damaged from bugs and hardware failures and/or created under IEEL. OSTs all look ok in lfsck. we have not run a lfsck for real because this all looks too intrusive.

cheers,
robin

Issue Links

• is related to: LU-11236 client MDT OST ENOTCONN loops
• is related to: LU-11119 A 'mv' of a file from a local file system to a lustre file system hangs
• is related to: LU-5152 Can't enforce block quota when unprivileged user change group
• is related to: LU-11303 slow chgrp as user when quotas are enabled

Comments

[Gerrit Updater]

John L. Hammond ([e-mail]) merged in patch https://review.whamcloud.com/32991/
Subject: LU-11227 lod: lod_sync: don't attempt sync to inactive targets
Project: fs/lustre-release
Branch: b2_10
Current Patch Set:
Commit: 2898c1765adfd6d966be6e5c24511519dede0188

[Peter Jones]

Landed for 2.12

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/32964/
Subject: LU-11227 lod: lod_sync: don't attempt sync to inactive targets
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 7c099467eab7b64c00e6a14c9cab7f09153571c1

[Gerrit Updater]

John L. Hammond ([e-mail]) uploaded a new patch: https://review.whamcloud.com/32991
Subject: LU-11227 lod: lod_sync: don't attempt sync to inactive targets
Project: fs/lustre-release
Branch: b2_10
Current Patch Set: 1
Commit: 576f1ee0cfb6f56028b82e28cea289576d0472fc

[Gerrit Updater]

Robin Humble ([e-mail]) uploaded a new patch: https://review.whamcloud.com/32964
Subject: LU-11227 lod: lod_sync: don't attempt sync to inactive targets
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 4d06197738c6a29b98082949478837041f83a14c

[Andreas Dilger]

I've deleted the MGS dirdata part of your comment, since this is already fixed in 2.11 via patch https://review.whamcloud.com/29274 "LU-4923 osd-ldiskfs: dirdata is not needed on MGS".

Could you please submit a patch to Gerrit with your changes so that we can get it reviewed and landed.

The LFSCK layout and linkea_inconsistent issue may be holdovers from older versions of Lustre if you have migrated files between OSTs, but the parent FID did not get updated. If you have a list of FIDs that LFSCK would repair, you could examine some of them manually to see what kind of issues there are using "lfs fid2path" to see if the reported pathname matches the actual pathname of the file, "lfs getstripe $MOUNT/.lustre/fid/<fid>" to see that the FID stored in the layout is matches that in the LMA and directory, and "debugfs -c -R 'stat O/d$(($objid % 32))/$objid' /dev/<ostdev>" to see if the objects have the correct parent (MDT) FID.

Whether you fix these with LFSCK or not is up to you. You should to consider making an MDT device-level backup, whether you run the LFSCK or not. Not repairing these issues reported by LFSCK means that if you do have some kind of corruption that LFSCK may not be able to recover as much, or may recover the filesystem incorrectly based on stale data.

[Robin Humble (Inactive)]

hmm, I can't figure out how to edit the above, but ignore the Lustre-1.x stuff - I've realised that's just the mgt. mdt is dirdata.

cheers,
robin