[LU-11227] client process hangs when lod_sync accesses deactivated OSTs Created: 08/Aug/18 Updated: 11/Sep/18 Resolved: 23/Aug/18 |
|
| Status: | Resolved |
| Project: | Lustre |
| Component/s: | None |
| Affects Version/s: | Lustre 2.11.0, Lustre 2.10.4 |
| Fix Version/s: | Lustre 2.12.0, Lustre 2.10.6 |
| Type: | Bug | Priority: | Minor |
| Reporter: | Robin Humble | Assignee: | Robin Humble |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | None | ||
| Environment: |
x64_64, ldiskfs, not DNE |
||
| Issue Links: |
|
||||||||||||||||||||
| Severity: | 3 | ||||||||||||||||||||
| Rank (Obsolete): | 9223372036854775807 | ||||||||||||||||||||
| Description |
|
Hi, we have a ldiskfs Lustre install where one OST is permanently deactivated with lctl conf_param lustre-OST005a.osc.active=0 we were in IEEL and saw no issues there, but are now in 2.10.4 to try and get better compatibility for file transfers to the new cluster. the problem is that in 2.10.4 a chgrp on the client hangs forever as it re-tries infinitely. MDS load is significant too. message from the MDS when the chgrp hangs is 1000's of these per second Aug 7 19:45:41 metadata01 kernel: LustreError: 4502:0:(lod_dev.c:1400:lod_sync()) lustre-MDT0000-mdtlov: can't sync ost 90: -107 looking at that lod_sync() code, it doesn't check for OSTs being deactivated. the below seems to work as a quick fix so that we don't have to reboot back into IEEL. diff --git a/lustre/lod/lod_dev.c b/lustre/lod/lod_dev.c
index d61ad2d..2194110 100644
--- a/lustre/lod/lod_dev.c
+++ b/lustre/lod/lod_dev.c
@@ -1409,6 +1409,8 @@ static int lod_sync(const struct lu_env *env, struct dt_device *dev)
lod_foreach_ost(lod, i) {
ost = OST_TGT(lod, i);
LASSERT(ost && ost->ltd_ost);
+ if (!ost->ltd_active)
+ continue;
rc = dt_sync(env, ost->ltd_ost);
if (rc) {
CERROR("%s: can't sync ost %u: %d\n",
the same fix would also seem to be appropriate for a deactivated MDT a few lines lower down. please let me know if this is totally the wrong thing to do, or if alternatively if it's useful and you'd like me to upload a patch to Gerrit. also dry-run of lfsck in 2.10.4 wants to correct layout for 15% of mdt files, and namespace (linkea_inconsistent) for about 15k files out of 225M. presumably because of this fs being old and/or a bit damaged from bugs and hardware failures and/or created under IEEL. OSTs all look ok in lfsck. we have not run a lfsck for real because this all looks too intrusive. cheers, |
| Comments |
| Comment by Robin Humble [ 08/Aug/18 ] |
|
hmm, I can't figure out how to edit the above, but ignore the Lustre-1.x stuff - I've realised that's just the mgt. mdt is dirdata. cheers, |
| Comment by Andreas Dilger [ 08/Aug/18 ] |
|
I've deleted the MGS dirdata part of your comment, since this is already fixed in 2.11 via patch https://review.whamcloud.com/29274 " Could you please submit a patch to Gerrit with your changes so that we can get it reviewed and landed. The LFSCK layout and linkea_inconsistent issue may be holdovers from older versions of Lustre if you have migrated files between OSTs, but the parent FID did not get updated. If you have a list of FIDs that LFSCK would repair, you could examine some of them manually to see what kind of issues there are using "lfs fid2path" to see if the reported pathname matches the actual pathname of the file, "lfs getstripe $MOUNT/.lustre/fid/<fid>" to see that the FID stored in the layout is matches that in the LMA and directory, and "debugfs -c -R 'stat O/d$(($objid % 32))/$objid' /dev/<ostdev>" to see if the objects have the correct parent (MDT) FID. Whether you fix these with LFSCK or not is up to you. You should to consider making an MDT device-level backup, whether you run the LFSCK or not. Not repairing these issues reported by LFSCK means that if you do have some kind of corruption that LFSCK may not be able to recover as much, or may recover the filesystem incorrectly based on stale data. |
| Comment by Gerrit Updater [ 09/Aug/18 ] |
|
Robin Humble (plaguedbypenguins@gmail.com) uploaded a new patch: https://review.whamcloud.com/32964 |
| Comment by Gerrit Updater [ 13/Aug/18 ] |
|
John L. Hammond (jhammond@whamcloud.com) uploaded a new patch: https://review.whamcloud.com/32991 |
| Comment by Gerrit Updater [ 23/Aug/18 ] |
|
Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/32964/ |
| Comment by Peter Jones [ 23/Aug/18 ] |
|
Landed for 2.12 |
| Comment by Gerrit Updater [ 11/Sep/18 ] |
|
John L. Hammond (jhammond@whamcloud.com) merged in patch https://review.whamcloud.com/32991/ |