|
I doubt it's related to new OI index file changed in 2.4
In 2.14, the calling path is mdd_prepare()-> orph_index_init() -> dt_store_open() // to open a dt_object named as "PENDING"
->dt_reg_open() ->dt_locate() ->lu_object_find() ->lu_object_find_at() ->lu_object_find_try ->lu_object_alloc ->loo_object_init() // Initialize "PENDING" dt_object
->osd_object_init() ->osd_fid_lookup() // lookup "PENDING" dt_object
result = osd_oi_lookup(info, oi, fid, id);
if (result == 0) {
......
} else if (result == -ENOENT)
result = 0;
If in OI index file, "PENDING" is not found there (2.4 changed OI index mechanism, could it be the reason?) osd_oi_lookup() returns 0, then osd_object_init() won't call osd_object_init0() and "PENDING" dt_object does not has LOHA_EXISTS in its common attributes.
|
|
"PENDING" should definitely exist, and I'd think it would be in the special "lookup by name" list instead of needing the OI?
|
|
Add some log, which partially fit my description. During orph_index_init(), it find out that PENDING object has been created.
00000004:00000001:0.0:1362551173.779945:0:4818:0:(mdd_orphans.c:470:orph_index_init()) Process entered
00000020:00000001:0.0:1362551173.779945:0:4818:0:(dt_object.c:377:dt_store_open()) Process entered
00000020:00000001:0.0:1362551173.779952:0:4818:0:(dt_object.c:352:dt_reg_open()) Process entered
00000020:00000001:0.0:1362551173.779952:0:4818:0:(dt_object.c:222:dt_lookup()) Process entered
00000004:00000001:0.0:1362551173.779952:0:4818:0:(osd_handler.c:3804:osd_index_ea_lookup()) Process entered
00000004:00000001:0.0:1362551173.779953:0:4818:0:(osd_handler.c:2753:osd_get_fid_from_dentry()) Process leaving (rc=18446744073709551555 : -61 : ffffffffffffffc3)
00000004:00000001:0.0:1362551173.779954:0:4818:0:(osd_handler.c:1903:osd_ea_fid_get()) Process entered
00000004:00000001:0.0:1362551173.779955:0:4818:0:(osd_handler.c:1939:osd_ea_fid_get()) Process leaving (rc=0 : 0 : 0)
00000004:00000001:0.0:1362551173.779955:0:4818:0:(osd_handler.c:3099:osd_ea_lookup_rec()) Process leaving (rc=0 : 0 : 0)
00000004:00000001:0.0:1362551173.779956:0:4818:0:(osd_handler.c:3816:osd_index_ea_lookup()) Process leaving (rc=1 : 1 : 1)
00000020:00000001:0.0:1362551173.779956:0:4818:0:(dt_object.c:232:dt_lookup()) Process leaving (rc=0 : 0 : 0)
00000020:00000002:0.0:1362551173.779957:0:4818:0:(dt_object.c:355:dt_reg_open()) dt_locate [0x200000001:0x7:0x0] ==> PENDING fid
00000020:00000001:0.0:1362551173.779957:0:4818:0:(dt_object.c:245:dt_locate()) Process entered
00000020:00000001:0.0:1362551173.779957:0:4818:0:(lu_object.c:621:lu_object_find_at()) Process entered
00000020:00000001:0.0:1362551173.779958:0:4818:0:(lu_object.c:550:lu_object_find_try()) Process entered
00000020:00000001:0.0:1362551173.779959:0:4818:0:(lu_object.c:573:lu_object_find_try()) Process leaving (rc=18446612133311027672 : -131940398523944 : ffff88003b6a19d8) ==> found it in the hash table
00000020:00000001:0.0:1362551173.779959:0:4818:0:(lu_object.c:625:lu_object_find_at()) Process leaving (rc=18446612133311027672 : -131940398523944 : ffff88003b6a19d8)
00000020:00000001:0.0:1362551173.779960:0:4818:0:(dt_object.c:253:dt_locate()) Process leaving (rc=18446612133330352192 : -131940379199424 : ffff88003c90f840)
00000020:00000001:0.0:1362551173.779961:0:4818:0:(dt_object.c:360:dt_reg_open()) Process leaving (rc=18446612133330352192 : -131940379199424 : ffff88003c90f840)
00000020:00000001:0.0:1362551173.779961:0:4818:0:(lustre_fid.h:375:fid_flatten32()) Process leaving (rc=4261420800 : 4261420800 : fe001f00)
00000020:00000001:0.0:1362551173.779962:0:4818:0:(dt_object.c:387:dt_store_open()) Process leaving (rc=18446612133330352192 : -131940379199424 : ffff88003c90f840)
00000004:00020000:0.0:1362551173.779963:0:4818:0:(osd_handler.c:2345:osd_index_try()) dt does not exists
00000004:00020000:0.0:1362551173.780032:0:4818:0:(mdd_orphans.c:478:orph_index_init()) "PENDING" is not an index! : rc = -20
so I look back to when this object was created, and I find that the PENDING dt_object was created in osd_prepare() => llo_local_objects_setup() => llo_store_create_index() => llo_create_obj()
00000004:00000001:0.0:1362555707.896340:0:4934:0:(mdt_handler.c:4881:mdt_object_init()) Process leaving (rc=0 : 0 : 0)
00000004:00000001:0.0:1362555707.896341:0:4934:0:(cmm_object.c:187:cml_object_init()) Process entered
00000004:00000010:0.0:1362555707.896341:0:4934:0:(mdd_object.c:248:mdd_object_alloc()) kmalloced 'mdd_obj': 136 at ffff88003d75cb40.
00000004:00000001:0.0:1362555707.896341:0:4934:0:(cmm_object.c:209:cml_object_init()) Process leaving (rc=0 : 0 : 0)
00000004:00000001:0.0:1362555707.896342:0:4934:0:(mdd_object.c:271:mdd_object_init()) Process entered
00000004:00000010:0.0:1362555707.896342:0:4934:0:(osd_handler.c:298:osd_object_alloc()) kmalloced 'mo': 192 at ffff88003d75cc00.
00000004:00000001:0.0:1362555707.896343:0:4934:0:(mdd_object.c:282:mdd_object_init()) Process leaving (rc=0 : 0 : 0)
00000004:00000001:0.0:1362555707.896343:0:4934:0:(osd_handler.c:444:osd_object_init()) Process entered
00000004:00000001:0.0:1362555707.896344:0:4934:0:(osd_handler.c:386:osd_fid_lookup()) Process entered
00000004:00000002:0.0:1362555707.896344:0:4934:0:(osd_handler.c:397:osd_fid_lookup()) osd_oi_lookup [0x200000001:0x7:0x0] returns -2 oi lookup cannot find the obj
00000004:00000001:0.0:1362555707.896344:0:4934:0:(osd_handler.c:421:osd_fid_lookup()) Process leaving (rc=0 : 0 : 0)
00000004:00000001:0.0:1362555707.896345:0:4934:0:(osd_handler.c:453:osd_object_init()) Process leaving (rc=0 : 0 : 0)
check out b2_1 version of osd_oi_lookup
if (fid_seq(fid) == FID_SEQ_LOCAL_FILE)
return -ENOENT;
2.1 does not use oi for local file lookup, that's the root cause.
|
|
Try to remove the check "if (fid_seq(fid) == FID_SEQ_LOCAL_FILE)" from "osd_oi_lookup()".
|
|
For the record, my test shows that removing the check works on this scenario, but panics on 2.1.4 formatted system.
|
|
To clarify, in 2.4 the "PENDING" FID is stored in the OI file, but in 2.1 it is not? And 2.1 does not like this?
Is it still possible to get a patch into 2.1.5 to fix this? Is this caused by the initial LFSCK scan added in 2.4 to fix the OI for local objects?
|
|
In 2.4, the "PENDING" FID is added into local file, but it is NOT in 2.1. We can back port OI scrub related patches (including initial OI scrub) to b2_1, part of the back porting are as following:
http://review.whamcloud.com/#change,4620
http://review.whamcloud.com/#change,4621
http://review.whamcloud.com/#change,4623
http://review.whamcloud.com/#change,4624
http://review.whamcloud.com/#change,4626
http://review.whamcloud.com/#change,4627
http://review.whamcloud.com/#change,4628
http://review.whamcloud.com/#change,4629
http://review.whamcloud.com/#change,4630
I do not think the back porting can be ready in a short time. So the better way is to fix b2_1 with removing some check. If there are other bugs after removing, we can try to resolve them one by one. I think it will be faster than the whole back porting.
|
|
Well, we should not really crash unfixed 2.1 servers.
and 2.1.5 should be released any moment now too so there's not tiem for any development to fit there.
We need to think up something in 2.4, and fast I think
|
|
just posted a 2.1 patch to port only necessary ldiskfs based OI implementation at http://review.whamcloud.com/5731
|
|
Will there also be a patch for master?
|
|
Jodi,
No, master does not need a patch, this patch is to make 2.1.4 server OI implementation align to that of master.
|
|
Can yu please rebase this on top of current b2_1?
|
|
ok, and done
|
just posted a 2.1 patch to port only necessary ldiskfs based OI implementation at http://review.whamcloud.com/5731
Here are the test configuration and result:
Lustre b2_1 build: http://build.whamcloud.com/job/lustre-reviews/14375/
Lustre master build: http://build.whamcloud.com/job/lustre-master/1369/
Distro/Arch: RHEL6.3/x86_64
Clean upgrade and downgrade path: b2_1->master->b2_1
After downgrading from master to b2_1, mounting the server targets and clients succeeded. However, on the MDS node, "lctl dl" showed that:
Dmesg on the MDS node showed that:
Test report is still in Maloo import queue.
|
|
Here is the test report: https://maloo.whamcloud.com/test_sessions/f77f5032-9cd5-11e2-802d-52540035b04c
|
|
So is my reaing right that the test actually failed because all OSCs are in inactive state, so it won't be possible to create any new files on such a filesystem? I do not see any attempts for the test to actually create anything post-downgrade so perhaps it's a case we are missing?
|
|
I don't know about the test case, but the latest error has something about CATALOGS file changing in 2.4.
the CATALOGS write by 2.1 is as follows (logid is i_ino+ __u64 0x0+i_generation)
- od -x /mnt/mds1/CATALOGS
0000000 0021 0000 0000 0000 0000 0000 0000 0000
0000020 2a9d 1d4f 0000 0000 0000 0000 0000 0000
0000040 0022 0000 0000 0000 0000 0000 0000 0000
0000060 2a9e 1d4f 0000 0000 0000 0000 0000 0000
0000100 0023 0000 0000 0000 0000 0000 0000 0000
0000120 2a9f 1d4f 0000 0000 0000 0000 0000 0000
0000140
after 2.4 mounted it, the CATALOGS logic arrays changes to
- od -x /mnt/mds1/CATALOGS
0000000 0002 0000 0000 0000 0001 0000 0000 0000
0000020 0000 0000 0000 0000 0000 0000 0000 0000
0000040 0004 0000 0000 0000 0001 0000 0000 0000
0000060 0000 0000 0000 0000 0000 0000 0000 0000
0000100 0006 0000 0000 0000 0001 0000 0000 0000
0000120 0000 0000 0000 0000 0000 0000 0000 0000
0000140
|
|
Without those patches having on-disk change porting backward, the upgrade then downgrade test would be a headache.
|
So is my reaing right that the test actually failed because all OSCs are in inactive state, so it won't be possible to create any new files on such a filesystem?
Right, touching a new file on Lustre client failed as follows:
I do not see any attempts for the test to actually create anything post-downgrade so perhaps it's a case we are missing?
The upgrade/downgrade testing needs to be performed as per wiki page https://wiki.hpdd.intel.com/display/ENG/Upgrade+and+Downgrade+Testing. As we can see, the data creating/verifying steps are included in the post-downgrade phases. However, these test cases are not covered in one test script currently. The script which detected the issue in this ticket was upgrade-downgrade.sh, which covered extra quotas and OST pools testing along the upgrade/downgrade path.
The issue is that while performing this script to do downgrade testing, the extra quotas testing was disabled (due to the new way of setting quotas on master branch), only OST pools testing was covered along the downgrade path, which only verified that the existing files/directories could be accessed and the striping info was correct, but did not create new files.
So, in order to cover all of the test cases in the wiki page, we need improve upgrade-downgrade.sh to make the quotas codes work on master branch, and also need run the other two test scripts {clean,rolling}-upgrade-downgrade.sh before the test cases covered by them are added into upgrade-downgrade.sh.
|
|
This log shows that 2.4 MDT start cannot find the old llog objects, and created new ones in 2.4 way (using llog_osd_ops), which 2.1 code (using llog_lvfs_ops) cannot recognise.
|
|
Actually, I found that when 2.1 formatted disk upgraded 2.4, all files' size becomes 0, their content are lost.
|
|
Bobijam, that sounds like a very critical problem. Does conf-sanity test 32 not detect this problem during 2.1 to 2.4 upgrade? Is that problem repeatable?
Please file a separate bug for that problem and make it a 2.4 blocker until it is better understood and fixed.
|
|
this is rather strange.. I do remember Li Wei improved conf-sanity/32 to verify actual data with md5sum.
|
|
Is it possible that the disk2_1-ldiskfs.tar.bz2 was created with an outdated compatible b2_1 version? I just tested with current b2_1 and master branch, the procedure is
1. create a lustre filesystem with b2_1 version
2. copy /etc/* to this filesystem
3. umount it
4. mount the file system with master version, succeeded
5. 'ls -l' the filesystem, all files are there but all sized 0 with no content.
|
|
iirc, Li Wei did put a function to create such an image in conf-sanity.sh
|
|
strangely, I don't even need master version, with b2_1 alone, format & mount & copy files & umount & mount it again, all files become 0.
btw, I'm using current b2_1 lustre code with linux-2.6.32-279.22.1.el6 kernel code.
|
|
Fired a ticket (LU-3141) for the 'empty file' issue.
|
|
it turns out that the 'empty file' issue is not a bug, I was using llmountcleanup.sh which calls 'umount -f' and it does not flush dirty data.
I'll push another patch which can pass the upgrade & downgrade test on my VM environment.
|
|
Wang Di,
The upgrade test shows these messages on client side
I noticed that it's from your recent patch, what the issue could it be? Is it potentially harmful?
|
|
Just update what I found. My local test shows the error message is from lov_attr_get_raid0()
while in lov_merge_lvb_kms(), stack variable 'fid' has no effect except for displaying debug messages.
WangDi, need your input.
|
|
Di, it appears that this might relate to http://review.whamcloud.com/5820 incorrectly converting struct llog_logid to use struct ost_id. The fields in that structure were already used a bit strangely, and I think that the ostid_to_fid() and fid_to_ostid() macros may be mangling the fields in this structure, since they were never really used as OST id/seq/, but rather MDS inode/0/generation.
On 1.8 the CATALOGS file looks like oid=ino/ogr=0/ogen=gen:
On 2.1 (upgraded from 1.8, if it matters) the CATALOGS file looks like oid=ino/oseq=1/ogen=gen:
On 2.4 (new) the CATALOGS file looks like (f_seq=1/f_oid=oid/ogen=0):
I suspect that the latter change is unintentional, since I think it breaks the on-disk and on-wire llog_logid protocol. I'm just running another test with a pre 6794d7654b4c459519a9e6d85ed439c8c594c2e7 build to see what the CATALOGS file looks like.
Di, could you please work up a patch that reverts the changes to llog_logid and its usage, and we can see if that fixes this latest issue.
|
|
Only the latest issue may be related to LU-2684, not the original one.
|
|
The pre-LU-2684 CATALOGS file looks like:
so indeed this is an unintentional compatibility breakage with the new code.
|
|
Bobi: Could you please try this patch? http://review.whamcloud.com/#change,6034 It seems lmm_oi is being overwritten by some other threads. As Andreas said it might related with ostid_to_fid for llog object. So this patch will use oi_id/oi_seq directly to identify the log object, so to avoid ostid_to_fid conversion. Could you please try a few times, I guess this problem can not be reproduced often, at least I can not reproduce it locally. Thanks.
|
|
http://review.whamcloud.com/6034 does not solve the issue. The error message is not from llog handling, it's from stat/ls the file created by 2.1 system.
attached is the -1 log.
|
|
I have an example of ostid here extracted from lov_merge_lvb_kms()
LustreError: 8277:0:(lustre_idl.h:705:ostid_to_fid()) bad MDT0 id, 0x51:1024 ost_idx:0
LustreError: 8277:0:(lustre_idl.h:706:ostid_to_fid()) 0x51:0x200000400
The second log message format is as follows
CERROR("bad MDT0 id, "DOSTID" ost_idx:%u\n",
POSTID(ostid), ost_idx);
CERROR(LPX64":"LPX64"\n", ostid->oi.oi_id, ostid->oi.oi_seq);
|
|
Hmm, the problem is that in 2.1, we define the lsm/lmm in this
But lmm_object_seq/lmm_object_id is actually normal MDT FIDS, i.e. lmm_object_id/lmm_object_seq will be f_oid/normal_seq, and when unpack lmm to lsm on 2.4,
it will use ostid_le_to_cpu()
And treat the ostid as normal FID, which cause the problem.
Sigh, it seems we do not have better way to convert this special ostid to the real FID.
|
|
http://review.whamcloud.com/#change,6037 Bobi: please check this one. Thanks!
|
|
yes, with http://review.whamcloud.com/5731 on b2_1 and http://review.whamcloud.com/6034 and http://review.whamcloud.com/6037 on master, the downgrade and upgrade test passed with no noise.
|
|
Change/6034 and 6037 have been merged into http://review.whamcloud.com/#change,6044
|
|
Landed for 2.4
|
|
http://review.whamcloud.com/#change,5731 hasn't landed on b2_1 yet, need that to be claimed to be fixed.
|
|
Reducing 2.4 blocker, but keeping open until lands on b2_1.
|
|
Please also make changes to b1_8, hit the same LBUG after downgrade from the latest tag-2.3.64 to 1.8.9
|
|
Do we support 2.x to 1.8 downgrade?
|
|
Sorry, I was downgraded the system to b2_1 by mistake, 1.8.9<->2.4 doesn't have this issue.
|
|
landed on b2_1 for 2.1.6
|
|
Yes, I think that we should support 1.8.x->2.4->1.8.9-wc1 upgrade & downgrade. Of course, w/o dirdata and large_xattr, etc. used.
|
|
I also think that 2.3->2.4->2.3 upgrade & downgrade should be supported. But this bug occurs there as well. Is the appropriate response to reopen this ticket and mark it as affecting 2.3 (and I suppose 2.2 too)? Or should we make a clone?
Also, I think it is a pretty bad story that we'd have to get a fix onto b2_3 to make the upgrade/downgrade scenario work. Because even if we get the existing patch landed to b2_3 you can't revert to 2.3.0.
|
|
I hit this issue when downgrade from 2.5 to 2.3.0, will create a new issue for that.
|
Generated at Sat Feb 10 01:29:04 UTC 2024 using Jira 9.4.14#940014-sha1:734e6822bbf0d45eff9af51f82432957f73aa32c.
2.4-start-mdt.log.tar.bz2