|
http://review.whamcloud.com/#change,4396
|
|
Patch to move range lookups to FLD has been landed.
|
|
I'm reopening this ticket because I think we're hitting an issue due to the changes made in 4396.
We've updated our MDS on Grove to a 2.3.58 based tag, and now we're having issue mounting the FS on our clients, and the clients which survived the upgrade can't mv files around.
I used systemtap to track the issue down to this -EIO returned in fld_server_lookup:
|
|
Prakash: Here is the fix http://review.whamcloud.com/#change,5134 , Could you please try it? Thanks
|
|
Before your patch, a 2.1.2-4chaos client would say this on trying to mount 2.3.58-whatever:
Now, 2.1.2-4chaos trying to mount 2.0.39 plus patch set 2 of your patch 5134 says:
Since your patch talks about changing the root fid and the error is about an invalid root fid, I'm guessing there is a connection.
|
|
"mv" is still broken as well. Client is running 2.3.58-5chaos, server running 2.3.59 + rev2 of your patch. So no visible improvement from the patch.
|
|
Any error console msg from MDS side?
|
|
Ah, I see where is the problem. old client still use the old fid_is_sane to validate the root_fid in client_common_fill_super, which does not include special sequence FID. So the old client would consider the new root fid to be invalid (see fid_is_sane).
(code snippet to fix this for new client)
I need think how to fix this for the old client.
|
|
Hmm, sorry, I was wrong here "((fid_seq(fid) >= FID_SEQ_START && fid_ver(fid) == 0)" should be able to validate the new special root fid. Maybe fid_ver is contaminated somewhere for ZFS, I will check.
|
|
As Andreas pointed out, I think the issue is that fid_oid(fid) returns 0 for the root fid. fid_is_sane in 2.1:
If fid_oid(fid) was non zero, I'd imagine the function would return true.
|
|
Ah, I did not realize fid_oid(fid) != 0 check in 2.1. I will update the patch to not use 0_oid root fid. Thanks.
|
|
I applied this patch to the MDS in my VM filesystem:
And now the client running a 2.3.54 based tag is failing here:
Prior to the patch the client would fail the same way as the 2.1 client:
|
|
Hmm, there is another defect pointed by Andreas, you probably should change that too.
Change FID_SEQ_SPECIAL to FID_SEQ_ROOT. I will update the patch soon. Thanks
|
Change FID_SEQ_SPECIAL to FID_SEQ_ROOT
I tried this as well, and see the same error.
For example:
As far as I can tell, it's coming from the inlined mdt_body_unpack function:
|
|
I just updated the patchz9 http://review.whamcloud.com/#change,5134), and it works for me locally. Could you please try this. If it still failed, could you please provide -1 debug log on the MDS side?
|
|
What versions are you using for the client, MDS, and OSS? I still see the same error with revision 3 of that patch.
|
|
I used 2.3.59 on all of nodes, but I revert those FLDB changes on the client side. And use 2.1 ldiskfs format disk here, (since my ZFS environment has some problem here). Initially I met the same problem as you described, but with the patch, the problem can be fixed here.
|
|
Please test from an unpatched 2.1 client too, and zfs on the server. Otherwise, I don't think you are testing the right things.
|
|
I just checked ZFS code carefully, I think the problem is here.
I missed one case check inside zfs code. Sure I will test this with 2.1 client + ZFS server.
|
|
I just tried this patch http://review.whamcloud.com/#change,5134 with 2.1 client and ZFS server. It works for me now
Client 2.1.4
Server: 2.3.59 + patch 5134
|
|
We will test it.
|
|
Does not appear to work. Both 2.3.59-based ppc64 clients and 2.1.2 x86_64 clients fail to mount. Error on 2.1.2 x86_64 client is:
and 2.3.59 ppc64 clients say:
|
|
I see the same behavior with revision 4 on the VM cluster I set up, as I did with revision 3. I'll post the lustre debug log from the MDS with "-1" set, but I still think the error is coming from mdt_body_unpack as I mentioned in a comment above. I also tested with ldiskfs, and that does work, so the error is specific to the ZFS OSD.
|
|
Thanks for testing and debug log. I just update the patch 5134 again. But I would like Alex check the patch before further testing. Since I am not so confident about ZFS changes.
|
|
we just discussed this topic with Di. to enable DNE we need another sequence for /ROOT and do not mix it with really local objects (like last_rcvd).
I'd suggest to convert this once upon start and do not introduce more cases to OI code.
the conversion should be trivial to do with existing primitives within OSD:
1) remove FID_SEQ_LOCAL_FILE:MDD_ROOT_INDEX_OID from OI
2) lookup inode/dnode of /ROOT
3) insert FID_SEQ_ROOT:1 into OI
in a single transaction.
then fid of /ROOT becomes a normal one and nothing special is required to handle that.
|
|
Hello, Alex made this patch on zfs part, http://review.whamcloud.com/#change,5249, and I will add it to 5134 and test it locally first. I will let you know the result soon.
|
|
I made a new patch http://review.whamcloud.com/#change,5257, but zfs part change is missing. Per discussion with Alex, except 5249, we still need change the root fid in .. and linkea of all inodes in FS, otherwise mv would not work. Alex is working on that.
|
|
Re-assign this ticket to Alex, since he is working on the patch to change the ROOT FID to the new sequence for ZFS.
|
|
http://review.whamcloud.com/#change,5249
|
|
I tried tag 2.3.62-3chaos which has patch set 1 of Change 5249. The test system had an existing filesystem. I get the following errors mounting the MDS:
|
|
Correction: we have patch set 8 of Change 5249.
|
|
hmm, this doesn't seem directly related to the patch.. the sequence 0x200000bd2 was not found in FLDB preventing access the the corresponded object.
Di, any idea ? ohh... I remember we enforced FLDB to return 0 and do not store records as osd-zfs didn't range lookups.
|
|
Hmm, 0x200000bd2 should be in the first group of sequence [0x0000000200000400-0x0000000240000400), and all of the fld lookup should happen inside memory on the server side. Ah, we should insert the current sequence inside the FLDB as well, since we do not really have fldb for ZFS, do we?
No, we have to let zfs work with FLDB right now because of FID on OST and DNE.
|
|
Ned, Could you please post the result
|
|
Note, the MDT will not mount.
|
|
Sigh, proc fldb entry is removed after mount failure. Could you please post debug log(with -1 level) of MDT mount process? Somehow, the fld entries is not inserted into the memory somehow during the fld_index_init.
|
|
my understanding was this is an existing filesystem formatted in Orion era and FLDB was not maintained at that time.
|
|
-1 debug log for failed MDS mount
|
|
Hmm, this debug log does not include failure information I need. Could you please set larger debug_size and collecting the log just after the mount failure.
|
|
Sigh, indeed, we even bypass index insert for FLDB on ZFS
|
|
Di, what is it you aren't finding in the log? debug_mb was already 641 MB and I basically did
lctl dk /dev/null
echo -1 > /proc/sys/lnet/debug
/etc/init.d/lustre start ; lctl dk /tmp/lustre_log
so that should have captured it. Also I have in the log:
which looks like it contains the whole mount attempt.
|
|
Maybe you grabbed the wrong attachment? It is LU-2240-stout-mds1.1363199014.lustre_log.gz
|
|
If you want to reproduce the mount failure in your test environment, the attached mdt image should do it. I created it with a 2.3.58 lustre build, created some files in the lustre namespace, then exported the pool.
zcat lustre-mdt1.gz > /tmp/lustre-mdt1
zpool import -f -d /tmp lustre-mdt1
mkdir /mnt/mds1
mount.lustre lustre-mdt1/mdt1 /mnt/mds1 # should fail
|
|
Thanks, this would be really helpful. I am cooking the patch now, and will try this locally. Thanks.
|
|
I tried this image, but got this, I probably do sth wrong 
But I updated the patch, could you please tried it? Thanks.
|
|
Did you try the -f switch for zpool import? I'll give the patch a try. Thanks
|
|
Attach mdt image from lustre 2.3.58, formatted with ZFS pool version 28. The previously attached image used pool version 5000 which could not be imported on earlier ZFS releases.
|
|
Di, please try lustre-mdt1-pool_version_28.gz . I deleted the first image because the pool version is not supported by the ZFS release you are running.
My initial test of http://review.whamcloud.com/#change,5249 patch set 10 did not succeed. My VM locked up hard when I ran mount.lustre. I haven't gotten any debug data yet.
|
|
Ned, Thanks. the img works this time. I will try it here. Thanks
|
|
Ned, I just updated the patch, I can mount this image with the patch. Could you please try this patch. Thanks.
|
|
I was able to mount the image with patch set 11. Next I'll try it out on a real filesystem and will let you know how it goes.
|
|
Testing patch-set 11 using VMs, I was able to mount the MDT, and mount the 2.3.62 FS with a 2.1.2 client.
|
|
My tests on a real filesystem (LLNL's stout cluster) with patch set 11 did not find any problems.
|
|
That patch has landed to master. Do you feel confident enough that this is fixed for us to consider this ticket resolved?
|
|
We haven't tested it yet on Sequoia's filesystem, which is the most important consideration for us. But we can always reopen the ticket if we find a problem.
|
|
As per LLNL, ok to mark this as resolved. LLNL will open a new ticket if any problems are found with the patch that has been landed
|
|
I'm reopening this issue. I tried upgrading our Grove-Test filesystem to 2.3.62-4chaos but hit the following crash:
|
|
And enabling D_OTHER I gather this message:
|
|
Mounting a snapshot of the MDT through the POSIX layer, I found that objects in the quota_slave directory and the file seq-200000007-lastid are using FID_SEQ_ROOT. Note the matching inode numbers:
|
|
I think we're getting the -EEXISTS (i.e. -17) error back from zap_add when we try inserting the new root fid (0x200000007:0x1:0x0) into the OI since it already exists.
|
|
I'm curious if this is the commit that is biting us:
|
|
Right, I confirmed in a VM that deleting oi.7/0x200000007:0x1:0x0 avoids the crash, and lets it successfully add the new root fid:
I'm not sure what the seq-200000007-lastid is for, or if it's safe to remove its OI entry. The MDT for the production filesystem has a similar file, but in a non-colliding sequence:
So hopefully we won't run into a problem there.
It would be nice if the conversion code handled these collisions. But since there should be very few affected filesystems in the wild, we could probably live with a manual workaround.
|
|
Sigh, this is the difficulty with following the development branch - you are picking up all of the dirty laundry that is normally put away before the release is made. Typically, we don't want anyone to use development releases for long-lived filesystems for exactly this reason.
Hopefully Mike or Alex configure out something to resolve this easily.
|
|
seq-<SEQ>-lastid stores the last used ID in sequence <SEQ>
|
|
could you check whether your filesystem has been using new quota files now? they're supposed to be in the following sequences:
FID_SEQ_QUOTA = 0x200000005ULL,
FID_SEQ_QUOTA_GLB = 0x200000006ULL,
if so, then it should be OK to just remove old quota files in 0x200000007 sequence.
|
|
Here's what I see on the MDS:
I'm somewhat guessing as to what the on disk format is supposed to look like, but it does appear to be using the new quota sequence numbers (0x200000005ULL and 0x200000006ULL).
So, does this mean I can go ahead and remove these files:
?
|
|
yes, I'd suggest to remove them .. and I'd suggest to take a snapshot just before that  unfortunately I'm unable to reproduce the case locally:
I can't generate such an image (can't even find the code in gerrit using 0x200000007 for quota).
|
|
Sigh.. Well it let me remove files oi.7/0x200000007:0x3:0x0, oi.7/0x200000007:0x4:0x0, and oi.7/0x200000007:0x1:0x0 (inode numbers 414211, 414213, and 414209 respectively) but I'm getting ENOENT when removing the others. Using systemtap, I can see it failing in zfs_zget:
I tried removing the files in the order that they were listed in the "find" command in my previous comment. So the first "rm" for each distinct inode number succeeded, but the following calls for files referencing the same inode number failed. Perhaps due to incorrect accounting of the number of links for a given inode?
In case it's useful, the zdb info regarding these objects is below (AFAIK the inode number correspond to its dmu object number):
I'm beginning to think a reformat is our best option moving forward...
|
|
After talking with Brian some more, I definitely think the issue is the improper handling of the "links" field. The first "rm" actually deleted the object from the dataset, and the subsequent removes got ENOENT because the object was already deleted. So I think the only path forward is to either hack the ZPL or Lustre to remove the entries we're interested in from the ZAPs, or reformat the filesystem. Assuming we wont have this problem on our production FS (which I need to verify, still), I'm going to pursue a reformat of our test FS to get around this.
|
|
I can make another patch to remove those objects, but frankly this isn't a nice way to go (we've made amount of changes to on-disk format from the beginning). so if this is possible, it'd be much much better to start from released version.
to some extent we do check on-disk consistency, with ldiskfs though. the good thing is that attributes like nlink are manipulated the same way on zfs.
|
|
I reformatted out Grove-Test file system using our 2.3.62-4chaos tag. Our Grove-Production filesystem doesn't have any entries in oi.7/0x200000007* so we should be OK to simply upgrade that side of things without a reformat (as far as I can tell). So I'll go ahead and resolve this ticket.
|
|
Prakash, do you think we need to keep this conversion code around for a while? my preference is to drop it as soon as possible.
|
|
Well, we still need to upgrade our production side of things which needs the conversion code. But since it landed in a tag already (2.3.63), I'm personally OK with dropping it from master. We can upgrade using a 2.3.63-based tag which will fix the FIDs, and then later upgrade to a newer tag which wouldn't have the conversion code. I'd imagine that would work just fine, and then the conversion code won't be in the actual 2.4 release.
morrone, how does that sound to you?
|
|
I'd say post-2.4.0 would be a bit safer. But yes, we don't need to keep it around too long.
|
Generated at Sat Feb 10 01:23:32 UTC 2024 using Jira 9.4.14#940014-sha1:734e6822bbf0d45eff9af51f82432957f73aa32c.
LU-2240-stout-mds1.1363199014.lustre_log.gz