|
Di is looking into this issue
|
|
It looks like the FID sequences for your objects are very strange, and the node cannot figure out on which server those objects are located. The FID sequence of [0x607000002000000:0x8a0:0x0] is way outside the range of IGIF FIDs (0x0000000c-0xffffffff) reserved for 1.8 objects, and also way outside the range of objects that would normally be allocated for 2.x MDT objects (starting at 0x200000400). It almost looks like some kind of endian bug? If that was swabbed it would be 0x200000706, which would be a very reasonable FID sequence.
What does the FLDB (FID->server location mapping table) look like on your MDS? The following will dump out the FID sequence allocation tables on the node:
In particular, the "fldb" entry will show the global mapping table of sequence numbers (first part of the FID) to the server:
In this case (single MDT filesystem) all of the allocated sequences map to MDT0000. It also shows that the rest of the unallocated "space" is reserved by the sequence controller (always on MDT0) for future assignment:
|
|
I don't necessarily suspect LFSCK as the culprit here. There are only about 15M inodes that were created with a 1.x MDS, while the remaining 185M were created with a 2.x MDS (either 1.8 or 2.1 clients).
It seems more likely that the DNE code is refusing to process these strange FIDs because it thinks that they belong to a remote MDT. That probably wasn't being checked in the 2.1 code, since it only could handle objects on MDT0.
Could you please provide a sample of FIDs that are reporting errors (e.g. grep for "Cannot find sequence" in syslog) and attach here? I suspect that they are all byte-swapped FID sequences.
The other important question is whether this problem is due to legacy objects (e.g. files created with 1.8 clients on PPC nodes), or if they are still actively being created? Could you please try on a PPC client node and on an x86 client node:
If this shows that 2.4 PPC clients (or 2.1 PPC clients, if you still have them) are still creating these objects, then this would be the first problem to find and fix.
It might be possible to make a workaround for this by adding an FLDB entry to cover the sequence range [0x0000N0002000000-0xffffN0002000000):0:mdt], but I'm not sure how confused the FLDB code would get if the unallocated "space" didn't extend to 0xffffffffffffffff. It also depends on how many FID sequences were allocated in swabbed order. If there were a large number of these sequences allocated (i.e. "0x0000002000N0000" is very large, and hence "0x0000N0002000000" is) then this could encroach on the unallocated sequence space and potentially cause problems in the future. That said, with the FLDB workaround entry in place, and if no new ones were being created, it would be possible to find and migrate those inodes to have "normal" FIDs, and then remove the FLDB workaround entry to avoid issues in the future.
|
|
It might also be useful to get the FLDB and sequence allocation information from a PPC client (both 2.4 and 2.1 if you are still running both):
|
|
Andreas, I am not aware of any PPC systems mounting this filesystem. Perhaps at some time in the past, I don't really know. But not now to the best of my knowledge.
We are also having trouble with top-level directories in lustre, which were certainly never created from PPC nodes.
Here are some of the problem fids reports on the MDS console:
EDIT: fixed sequence list
|
|
More requested info:
|
|
Andreas, the only fldb file is under the fld tree, not seq like you demonstrated above. Does that mean anything?
|
|
ah, so [0x607000002000000:0x8a0:0x0] is out of allocated sequence space, but I do not understand how top-level directories can be attached with some wrong sequence FIDs if not with PPC client. I do not recall any bugs which can trigger the problem. but I might miss sth.
|
|
Location of fldb file isn't important. The "fldb" entry looks consistent with the "space" entry for MDT0.
It does show is that the MDS thinks the FID sequences are being allocated somewhat normally, from 0x200000400 onward on MDT0. It is somewhat unusual in that it appears to have allocated 0x8c7da139f01-0x200000400 = 9645860297473 (~= 2^43 or 9 trillion) sequences for MDT0000. Each sequence would typically be used by one client per mount, though I recall you had bug LU-1632, and even in LU-3318 it reports that you have 2.1.3 clients that do not have the LU-1632 patch and are consuming FID sequences like crazy. That isn't totally critical at this point, however, since even 9T sequences isn't a huge dent in the 2^64 sequence space if they are being consumed one-at-a-time.
My theory about byte-swabbed FID sequences is out the window I guess. The values look to be all over the place, and I can't see any obvious pattern about what would be causing this.
|
|
Another possibility might be OI-scrub screw up FIDs during update grade, but that is just wild guess. Fan Yong, please comment here.
Chris: If you create a new file on this system right now. Does the file shows reasonable FID with it. You can simply get the FID by "lfs path2fid xxxx", Thanks.
|
|
I have some more information about the PPC situation. We did have PPC clients mount this filesystem in the past. The PPC clients were running a 1.8 flavor while the servers were are 2.1. It is possible that some top-level user directories were created from a PPC login node.
We are not mounting the filesystem from any PPC nodes today.
|
|
Yes, that is working.
|
|
Chris, would you say that the number of broken files is a large fraction of files in the filesystem, or only isolated to specific files/directories?
If you know of a specific directory in the filesystem suffering this problem, could you please check on the MDS with a relatively new version of e2fsprogs:
This will tell us if the FID in the LMA xattr, in the LOV xattr, and in the directory entry data. It will also tell us if the file was recently created, or is old. It would also be useful to use the "stat" command to check the crtime of several of the files with bad FIDs, to see if there is any consistency between new/old creation date.
What Lustre version(s) are the clients on this system?
|
|
Ah, ignore the path2fid request in my previous comment, I see you already did that for Di.
|
|
I just made a temporary patch http://review.whamcloud.com/8213 to make FLDB "working", i.e. all of non-exist sequence will point to MDT0, since you are using single MDT, this can make FLDB work.
But since we do not know the real problem yet, so I am not sure whether this temporary patch will make the FS "working" again.(i.e. I do not know OI table is good or not right now). Though this will help us understand the problem.
|
|
The proplematic top-level directories are easiest to find:
This directory on a lustre client looks like:
Clients are mostly 2.1 flavor still, although there may be some 2.4 out there. We're in the (much longer than hoped) process of upgrading the servers before we do the client upgrades.
|
|
In this case "fid=[0x200000706:0x4f090000:0x0]" has a sequence that looks reasonable "0x200000706", though the OID looks strange "0x4f090000". It should normally be below 128k. In any case, this in itself shouldn't be causing the "can't find sequence" errors, since this particular sequence is valid and should map to MDT0 properly.
The FID in the LMA does not match the one in the LOV EA (which appears to be all zero) since this is a directory and not a regular file. This is not a problem, but I just wanted to see if this data was inconsistent. Could you do this same step with a regular file?
It also looks like your version of debugfs.ldiskfs either does not implement the "-D" option, or the top level directory does not have dirdata that holds the FID.
Since I also have an MDT that has existed since Lustre 1.6 or earlier (currently running 2.1.3 server with 2.4 clients), I'm going to try upgrading this to 2.4.1 to see what happens.
|
|
Hmm, the sequence seems correct, so Andreas's idea of swapped problem might be correct. But OID seems too big here. (The maxim OID of MDT FID should be 0x20000ULL)
Chris, could you please tell me client version(2.1.6?), are there any tag? What is the output of "lfs path2fid dkp"?
Could you please provide -1 debug log of client/MDT when do "stat dkp"? (please clear client cache before doing this "lctl set_param ldlm.*.MDT-mdc*.lru_size=0") Thanks.
|
|
The client I am on happens to be the same lustre version 2.4.0-19chaos. There are probably lustre 2.1.4-[45]chaos clients mounting the filesystem as well.
I'll work on getting logs.
|
|
I spot checked some working top level directories, and large OID numbers look pretty common. Here are some fid EAs from working directories:
Using path2fid, those two are reported on the client, respectively, as:
So it would appear that debugfs is printing the fid info in the wrong byte order.
(Both client and server are x86_64, and running the same 2.4.0-19chaos version of lustre).
|
|
Interesting. Hmm, what is your debugfs version? debugfs suppose to print things as little endian on x86_64.
Hmm, debugfs is wrong here, then the fid we got from the previous comment is wrong.
Then it means the FID([0x200000706:0x4f090000:0x0]) stored on LMA is wrong.
|
|
See attached client_log.txt, and serveR_log.txt.bz2. Note that client nid is 192.168.115.67@o2ib10, and the server nid is 172.16.64.141@tcp.
|
|
There is too much MDS traffic to reasonably catch that one RPC with -1 debugging. I backed it off to our defaults + rpctrace on the server side. Let me know if there is some other conservative debug setting that would work for you.
|
|
debugfs version is 1.42.7.wc1.1chaos
|
|
Unfortunately, the server log here is not very helpful
This is the log I found. almost the same as we got from the console. Could you please also add "+info +inode" "lctl set_param debug "+info +inode"" on the server side? Hope that would not be too much.
|
|
What is your version of e2fsprogs? The code at the 1.42.7.wc1 is:
fid_be_to_cpu(&lma->lma_self_fid, &lma->lma_self_fid);
fprintf(out, " lma: fid="DFID" compat=%x incompat=%x\n",
PFID(&lma->lma_self_fid), ext2fs_le32_to_cpu(lma->lma_compat),
ext2fs_le32_to_cpu(lma->lma_incompat));
That does the opposite swab of the one that you showed:
static inline void fid_be_to_cpu(struct lu_fid *dst, struct lu_fid *src)
{
dst->f_seq = ext2fs_be64_to_cpu(src->f_seq);
dst->f_oid = ext2fs_be32_to_cpu(src->f_oid);
dst->f_ver = ext2fs_be32_to_cpu(src->f_ver);
}
|
|
ah, on 1.42.7 debugfs is somewhat wrong.
It tries to convert the FID from big endian, but actually LMA store the FID as little endian.
|
|
Chris: Do you still have PPC client(2.1) attached to this server? Could you create a directory from PPC client then use debugfs to see whether the FID in lma is correct?
|
|
Oh, my debugfs version is 1.42.3wc3, I create a ticket LU-4228 to track this problem.
|
|
No. We have not have PPC clients attached to this filesystem.
|
|
It seems that the "oid" in the FID is correct, but the "seq" in the FID is in wrong order. It is quite possible that the FID in LMA, and in dir-ent entry, and in the OI mapping are all consistent with one another, but with the wrong "seq" order. The issue should has been there since Lustre-2.1, but because Lustre-2.1 MDS did not verify the FLDB, so it worked before.
On the other hand, it is NOT all the FIDs with wrong "seq" order. The FID [0x2015be3d6:0xb2a3:0x0] and [0x1e8837c:0xc4256171:0x0] are valid. I suspect that only some special client, such as PPC client, ever generated invalid FID and sent it to the Lustre-2.1 MDT when create (such as the "/ROOT/dkp". So if possible, we can downgrade the MDS to Lustre-2.1, and test whether it is the case or not.
|
|
Since during upgrade(2.1 to 2.4), OI-scrub will unlikely touch the FID(both in LMA and name entry). But the FIDs(in LMA) of "dkp" is clearly wrong(Note: FID in name-entry should be wrong too, we can see it from the debug message)
So probably it is not the problem of 2.4 or upgrading. It is more likely a bug already exists on 2.1. Though we suspect that is probably related with PPC client. As Fan Yong said, we do not do fld lookup(check) for 2.1, so it "works" on 2.1.
|
|
On possible solution here might be (probably there are better ones)
1. Applied patch http://review.whamcloud.com/8213 to MDT, and rebuild and remount the system.
2. Find out all files/directories with wrong sequence, and "migrate" them to the new dir/file.
For migrate I mean mkdir(touch), cp, then remove the old ones.
3. umount MDT server and remove the patch for 8213.
But we still need to find out why this "big endian" FIDs was stored in LMA or name-entry. Does it related with PPC client. is it still exists on 2.1? Andreas, Do you recall any bugs related?
|
|
And the MDT just happily wrote the bogus sequence to disk...that sounds like at least two bugs.
|
|
I have logs if you still want them. Where can I upload large files these days? Is ftp.whamcloud.com still the place?
|
|
According to the timestamp of dpk
What is the lustre version was running at Feb 2012?
|
|
Clearly, b2_1 did not do good job here.  It seems to me ftp.whamcloud.com still works. At least I can access it.
|
|
In Dec 2012 is was running 2.1.2-3chaos. So either that or something earlier.
|
|
I just uploaded LU-4226_client_log2.txt and LU-4226_server_log2.txt.bz2 to ftp.whamcloud.com.
|
|
Chris: Thanks for debug log, so it is clear that, the FID(of dkp) in LMA and name entry are wrong,
It seems OID is correct, but sequence 0x607000002000000 is wrong(likely to be wrongly swapped), So it means the error might happened during seq allocation process, otherwise both OID and sequence will be wrong, since they are always be swapped at the same time except seq allocation. I checked 2.1.2 code, and did not find anything wrong there. If you can tell me the exact lustre version you use on Feb 2012, it will be very helpful. And also please tell me are there any other special patches you might add in that version?
I will keep digging the history here. Thanks.
|
|
I upgraded my home system from 2.1.3 to 2.4.1 (RHEL6 with RPMs from the b2_4 "last_successful_build" yum repo on build.whamcloud.com to include Fan Yong's scrub fix from LU-3934). The filesystem was originally formatted with Lustre 1.4 (not sure which version) and has been upgraded through 1.6, 1.8, and 2.1. It performed an automatic initial scrub on the MDS mount. I enabled "dirdata" and performed another manual scrub without problems. I've verified creating new files create entries with dirdata, old files are accessible (both IGIF and normal FIDs), and have not seen any sign of problems. I also verified the LU-4228 fix for debugfs works properly to print FIDs in the LMA and "ls -lD" to print them from dirdata.
It would be useful to know if these files with the crazy FIDs are still being created, or if they have stopped being created when that happened. If there are a limited number of them, then it may be possible to "lfs_migrate" them to new files with Di's patch applied on the MDS. Alternately, it may also be possible to mount the filesystem as ldiskfs and delete the trusted.lma and trusted.link xattrs from the file (essentially turning the file into a 1.8 upgrade object with IGIF) and then re-run LFSCK on it to clean up the dirdata entries and re-add the files into the OI. I haven't tested that yet, so I'm not 100% sure of what effect it will have. Clients would probably need to be remounted, or at a minimum have their cache flushed.
|
|
Chris, have you considered downgrading to 2.1 again to get the system back up and usable?
|
|
I checked the code since 2.1.0, and did not find anything unusual which can trigger the problem. Unfortunately, I do not have any big endian machine to try this on 2.1.0 to see whether the problem is still there. If the system is still working on 2.1, and if you can find some PPC clients attached to it, that might help us understand the issue. Thanks.
|
|
I think I like the idea to remove the trusted.lma and trusted.link xattrs the best. Walking the filesystem through the direct ldiskfs mount won't be too time consuming.
I'll give that a try on some files in a test filesystem.
|
|
I fear the downgrade option. Upgrades don't work well, and those are at least somewhat tested. Downgrades aren't even tested.
|
|
Chris
We do test downgrades, but certainly nothing as complex as this situation
Peter
|
|
Chris, Di's patch in http://review.whamcloud.com/8213 may also be a quick workaround. It weakens the added "FID sanity check" that was introduced in 2.4 so that invalid FIDs are assumed to be on MDT0000. They will still fail if the object doesn't exist, but it might allow the bad files to become accessible again, and would not involve any permanent change to the filesystem, like deleting the LMA xattr does.
It would also make sense to do a "dd" backup of the MDT if possible. That should be relatively fast even if the target is just a single 4TB SATA drive (est. 5.5h for a 100MB/s source/target, extrapolating a 2TB MDT from the ~1B inodes that LFSCK scanned), and could be done with the filesystem live if necessary (a backup that needs an e2fsck is better than none).
|
|
Andreas, we may try your method of deleting trusted.lma and trusted.link xattrs. One thing I'm unsure about is the semantics of the lfsck_start command with regard to the supported repair types. Can we simultaneously repair the OI and namespace (i.e. FID-in-Dirent and LinkEA)? That is, should we run
in immediate succession, or should we wait until the OI scrub completes before starting the namespace repair, or can both types be started with one command?
|
|
"lctl lfsck_start -M <MDT>" will only trigger OI_scrub check/repair at background. So the command will return immediately, but as for how long the check/repair will last depends on how many files on the MDT.
"lctl lfsck_start -M <MDT> -t namespace" will trigger OI_scrub and namespace check/repair simultaneously. The two components run in parallel at background. Similar as above, the command will return immediately, and the scanning time depends on the files count on the MDT.
Something to be clarified:
1) FID-in-LMA is trusted by OI_scrub. If you drop FID-in-LMA, then OI_scrub will regard it as old file, and generate IGIF mode FID, which is different from original dropped FID, and insert IGIF<=>ino# mapping into the OI table.
2) namespace scanning also trust FID-in-LMA. If the FID-in-LMA does not exist, it will append IGIF mode FID after name-entry in the directory block, and also use the IGIF for linkEA.
3) Because on server-side, the files' FIDs are changed, you have to re-mount the clients to purge out all related cached but staled information.
To be safe, it is better to find a test system firstly to verify wether it works as our expectation or not.
|
|
This is a simple script to check a filesystem for strange looking fids.
It will print tons of errors on the filesystem discussed in this bug, but it would be useful to run it on other filesystems (preferably before upgrade to 2.4) to see if they suffer from the same problems.
|
|
Thanks Andreas, that's quite helpful (though I don't think this does what you intended):
[[ -n "$LAST" && "$F" == "$LAST" ]] && LAST="" && echo "found" ||
continue
Incidentally, we carried out the "remove trusted.{lma,link} from bad files + lfsck" recovery procedure, and it worked pretty much as expected.
|
|
Updated version of checkfid.sh program. The "restart" mechanism was added at the last minute and looked like it was working, but wasn't.
|
|
Any chance to run the checkfid.sh script on any of your other filesystems?
|
|
Yes, it was run (maybe still running) on four of our ldiskfs systems on the SCF. Of the four, only one had bad fids, and that filesystem was the one that BG/P used exclusively. That filesystem has in excess of 1 million files/directories with bad fids.
So that would appear to be anther strong correlation pointing to ppc clients and lack of checking on the servers.
|
|
Andreas, in case checkfid.sh is needed again, it needs to handle sequence numbers that compare as negative integers:
|
|
Ned, Chris: Could you please tell me if OI_scrub fix these bad FIDs? Are there anything else I should do for this ticket? Thanks.
|
|
Btw: we will do more FID validation on the server side in https://jira.hpdd.intel.com/browse/LU-4232. Ah, already attached that ticket to the sub-tasks.
|
|
Di, I don't think there was any way for LFSCK to fix the bad FIDs directly. My understanding is that the LMA xattr was removed from the inodes, and then LFSCK treated this as an upgraded 1.8 filesystem with IGIF FIDs and recreated the LMA.
|
|
The problem was handled as Andreas explained. If the servers now have code to prevent this problem in the fist place, then the ticket is complete.
|
|
time to close this out.
|
Generated at Sat Feb 10 01:40:48 UTC 2024 using Jira 9.4.14#940014-sha1:734e6822bbf0d45eff9af51f82432957f73aa32c.
checkfid.sh
client_log.txt