|
Mike
It looks like some work was started but not completed relating to this area under LU-9699
Peter
|
|
I've just updated patch under LU-9699, preparing its port to 2.12
|
|
Thank you!
|
|
Hi Mike... so I have some update!
We have applied the backported patch on MGS and all MDS, on top of 2.12.7.
Then this morning we tried to add an OST. The MGS propagated the config to the clients, but not to the MDTs. The local config llog for oak-MDT* on the MGS seem all corrupt now (but I do have backups and they are not corrupt on MDT). This is new.
This is the log from the MGS when we started new OST index 331 (oak-OST014b):
New OST is not visible from MDTs (no logs) and not shown by lctl dl:
New OST is visible from clients but not filling up (which makes sense):
System is still running fine so there is that, no crash. But now the problem is that the config llogs seem corrupt so I'm worried we will need to restore them from backups:
same for CONFIGS/oak-MDT0001 to 5 on the MGS.
If I run llog_print on MGS, we can see the new OST (which explains why clients can see it):
But it seems to be in memory only as it is not to be found in the MGS oak-client file:
I'm attaching oak-MDT0000 before (ok but without OST 331) as oak-MDT0000.backup-20210923 and after (corrupted) as oak-MDT0000.after-ost331. Do you see what could have caused this (and how to fix it)? Thanks!
|
|
Hi Mike,
My bad. I should have used debugfs to check those config files, as properly documented in man llog_reader! They are not corrupted when I extract them from the MGS via debugfs:
And the OST has been properly added to it, this is a diff of before/after for oak-MDT0000:
So the main question remains, why are MDTs not aware of this new OST? How can I troubleshoot that? Thanks and sorry for the confusion.
|
|
Stephane, such problem can be result of LU-13356 issue, could you check if related patch is in your tree?
|
|
Hello,
This ticket could be related to LU-14802, this is also a configuration issue with the LU-13356 on the background.
|
|
Thank you both! We do have the patch from LU-13356 on the Lustre servers but not on all clients yet. It is being deployed.
|
|
Hi Mike and Etienne,
We still think that in our case, we have another problem when adding new OSTs. With the patch from LU-9699, MDS don't crash anymore on this assert, but we still can't add OSTs without restarting all MDS. I believe that the following messages might give us a hint:
Reminder: these OSTs are not the ones being added.
When a new OST is added, the MGS is updated (and that is OK) revokes the config lock which triggers an update on clients (this is OK too), and also on MDTs, and class_config_llog_handler() where this message is produced is called with cfg->cfg_instance == NULL. In my understanding, cfg_instance should be defined so that new llog can be appended and so the MDTs can then add the new OST at the end without re-processing the full config. My current guess is that this works for a certain length of config llog (until OST0142 / OST0145 depending on the MDT, this is reproducible - do we reach a buffer limit here?), and when the second part of the config buffer is processed, we are losing the instance (and thus also cfg_last_idx?) so logs are processed at the beginning of the buffer which here happens to be OST0142 / OST0145 depending on the MDT as their config llog is not exactly the same. I've spent some time trying to look for a possible defect but I haven't found anything yet.
Another thing I am suspecting is that it could have started when we removed a few OSTs: we previously disabled 12 OSTs on this system, we used the experimental del_ost which behaves like llog_cancel on specific indexes, and I noticed they are marked as EXCLUDE and not SKIP. So even a lctl clear_conf wouldn't help. This is an issue too if we want to definitely get rid of them.
To come back to our 'Skip config outside markers' issue, perhaps there is a llog buffer issue when EXCLUDE markers are previously used. I will try to reproduce to create a large config like this on a test system, but I wanted to share that with you in case you have some insights. Thanks!
|
|
Hello,
it's been a while but it's still a problem when adding new OSTs, I've taken a fresh look and after some efforts I was able to reproduce using the catalogs from the MGS on a dummy test filesystem, this should make it easy to get more infos out of it.
So first for the symptoms we were chasing, Stéphane was correct: the cld_cfg.cfg_last_idx for the oak-MDTxxxx catalogs are wrong on MDS, so old events are being processed when they already had been applied.
My reproducing occurence had the following dmesg:
[ 3486.702628] Lustre: 4657:0:(obd_config.c:1641:class_config_llog_handler()) Skip config outside markers, (inst: 0000000000000000, uuid: , flags: 0x0)
[ 3486.710037] LustreError: 4657:0:(genops.c:556:class_register_device()) oak-OST0147-osc-MDT0002: already exists, won't add
[ 3486.714672] LustreError: 4657:0:(obd_config.c:1835:class_config_llog_handler()) MGC10.42.17.51@tcp: cfg command failed: rc = -17
[ 3486.719065] Lustre: cmd=cf001 0:oak-OST0147-osc-MDT0002 1:osp 2:oak-MDT0002-mdtlov_UUID
[ 3486.722883] LustreError: 2907:0:(mgc_request.c:599:do_requeue()) failed processing log: -17
And with loglevels enabled we can see this in dk
mgc_request.c:2058:mgc_process_log()) Process log oak-MDT0002-0000000000000000 from 3203
With the catalog as follow on mgs:
... last_idx: 3259
...
#3203 (144)setup 0:oak-OST0147-osc-MDT0002 1:oak-OST0147_UUID
...
#3259 (last line)
So: the last_idx here was 3202 when it should have been 3259 to start processing from 3260 and the discreptancy made the MDS reprocess oak-OST0147-osc-MDT0002's setup, leading to the error message, but the real problem is that last_idx doesn't match.
This can also be confirmed with crash for manual inspection (I assume it got +1'd parsing that erroneous line):
crash> mod -s mgc
crash> list -H config_llog_list -s config_llog_data.cld_cfg,cld_logname -l config_llog_data3.cld_list_chain
...
ffff8bdcb6d3fa70
cld_cfg = {
cfg_instance = 0,
cfg_sb = 0xffff8bdc48767000,
cfg_uuid = {
uuid = "\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000"
},
cfg_callback = 0xffffffffc0949980 <class_config_llog_handler>,
cfg_last_idx = 3203,
cfg_flags = 2,
cfg_lwp_idx = 327,
cfg_sub_clds = 31
}
cld_logname = 0xffff8bdcb6d3fae5 "oak-MDT0002"
...
The previous fix was also targetting side effects, but what really needs fixing is the index so we don't reprocess events.
Now, why is that index off?
This is just a supposition for now but upon a closer look, it turns out the CONFIGS/oak-MDT0002 catalog offsets don't keep up between the original on MGT and the copy on the MDT.
Here's what it looks like on MGT:
...
rec #3258 type=10620000 len=88 offset 508600
rec #3259 type=10620000 len=224 offset 508688
Header size : 8192 llh_size : 80
Time : Thu Sep 26 05:36:36 2019
Number of records: 3132 cat_idx: 0 last_idx: 3259
Target uuid : config_uuid
-----------------------
#01 (224)marker 2015 (flags=0x01, v2.10.8.0) oak-MDT0002-mdtlov 'lov setup' Thu Sep 26 05:36:36 2019-
#02 (128)attach 0:oak-MDT0002-mdtlov 1:lov 2:oak-MDT0002-mdtlov_UUID
#03 (176)lov_setup 0:oak-MDT0002-mdtlov 1:(struct lov_desc)
uuid=oak-MDT0002-mdtlov_UUID stripe:cnt=1 size=1048576 offset=18446744073709551615 pattern=0x1
...
#18 (224)marker 2018 (flags=0x01, v2.10.8.0) oak-MDT0002 'add osc(copied)' Thu Sep 26 05:36:36 2019-
#19 (224)EXCLUDE START marker 2019 (flags=0x11, v2.10.8.0) oak-OST0003 'add osc' Thu Sep 26 05:36:36 2019-Fri Mar 19 22:29:33 2021
#20 (088)add_uuid nid=10.0.2.102@o2ib5(0x500050a000266) 0: 1:10.42.17.114@tcp
#23 (088)add_uuid nid=10.0.2.101@o2ib5(0x500050a000265) 0: 1:10.42.17.114@tcp
#26 (224)END EXCLUDE END marker 2019 (flags=0x12, v2.10.8.0) oak-OST0003 'add osc' Thu Sep 26 05:36:36 2019-Fri Mar 19 22:29:33 2021
#27 (224)END marker 2019 (flags=0x02, v2.10.8.0) oak-MDT0002 'add osc(copied)' Thu Sep 26 05:36:36 2019-
#28 (224)marker 2020 (flags=0x01, v2.10.8.0) oak-MDT0002 'add osc(copied)' Thu Sep 26 05:36:36 2019-
#29 (224)EXCLUDE START marker 2021 (flags=0x11, v2.10.8.0) oak-OST0001 'add osc' Thu Sep 26 05:36:36 2019-Fri Mar 19 22:29:22 2021
...
#3172 (112)add_conn 0:oak-OST0142-osc-MDT0002 1:10.42.17.114@tcp
#3173 (136)lov_modify_tgts add 0:oak-MDT0002-mdtlov 1:oak-OST0142_UUID 2:322 3:1
#3174 (224)END marker 6129 (flags=0x02, v2.12.7.0) oak-OST0142 'add osc' Thu Sep 9 15:37:12 2021-
#3175 (224)marker 6137 (flags=0x01, v2.12.7.0) oak-OST0143 'add osc' Thu Sep 9 15:37:38 2021-
#3176 (088)add_uuid nid=10.0.2.104@o2ib5(0x500050a000268) 0: 1:10.42.17.114@tcp
#3177 (128)attach 0:oak-OST0143-osc-MDT0002 1:osc 2:oak-MDT0002-mdtlov_UUID
#3178 (144)setup 0:oak-OST0143-osc-MDT0002 1:oak-OST0143_UUID 2:10.42.17.114@tcp
#3179 (088)add_uuid nid=10.0.2.103@o2ib5(0x500050a000267) 0: 1:10.42.17.114@tcp
#3180 (112)add_conn 0:oak-OST0143-osc-MDT0002 1:10.42.17.114@tcp
#3181 (136)lov_modify_tgts add 0:oak-MDT0002-mdtlov 1:oak-OST0143_UUID 2:323 3:1
#3182 (224)END marker 6137 (flags=0x02, v2.12.7.0) oak-OST0143 'add osc' Thu Sep 9 15:37:38 2021-
#3183 (224)marker 6145 (flags=0x01, v2.12.7.0) oak-OST0145 'add osc' Thu Sep 9 15:38:03 2021-
#3184 (088)add_uuid nid=10.0.2.104@o2ib5(0x500050a000268) 0: 1:10.42.17.114@tcp
#3185 (128)attach 0:oak-OST0145-osc-MDT0002 1:osc 2:oak-MDT0002-mdtlov_UUID
#3186 (144)setup 0:oak-OST0145-osc-MDT0002 1:oak-OST0145_UUID 2:10.42.17.114@tcp
#3187 (088)add_uuid nid=10.0.2.103@o2ib5(0x500050a000267) 0: 1:10.42.17.114@tcp
#3188 (112)add_conn 0:oak-OST0145-osc-MDT0002 1:10.42.17.114@tcp
#3189 (136)lov_modify_tgts add 0:oak-MDT0002-mdtlov 1:oak-OST0145_UUID 2:325 3:1
#3190 (224)END marker 6145 (flags=0x02, v2.12.7.0) oak-OST0145 'add osc' Thu Sep 9 15:38:03 2021-
#3191 (224)marker 6153 (flags=0x01, v2.12.7.0) oak-OST0146 'add osc' Thu Sep 9 15:53:39 2021-
...
#3240 (224)marker 6201 (flags=0x01, v2.12.7.0) oak-OST014d 'add osc' Mon Oct 4 08:10:30 2021-
#3241 (088)add_uuid nid=10.0.2.104@o2ib5(0x500050a000268) 0: 1:10.42.17.114@tcp
#3242 (128)attach 0:oak-OST014d-osc-MDT0002 1:osc 2:oak-MDT0002-mdtlov_UUID
#3243 (144)setup 0:oak-OST014d-osc-MDT0002 1:oak-OST014d_UUID 2:10.42.17.114@tcp
#3244 (088)add_uuid nid=10.0.2.103@o2ib5(0x500050a000267) 0: 1:10.42.17.114@tcp
#3245 (112)add_conn 0:oak-OST014d-osc-MDT0002 1:10.42.17.114@tcp
#3246 (136)lov_modify_tgts add 0:oak-MDT0002-mdtlov 1:oak-OST014d_UUID 2:333 3:1
#3247 (224)END marker 6201 (flags=0x02, v2.12.7.0) oak-OST014d 'add osc' Mon Oct 4 08:10:30 2021-
#3248 (224)marker 6209 (flags=0x01, v2.12.7.0) oak-OST014e 'add osc' Mon Oct 4 08:13:45 2021-
#3249 (088)add_uuid nid=10.0.2.103@o2ib5(0x500050a000267) 0: 1:10.42.17.114@tcp
#3250 (128)attach 0:oak-OST014e-osc-MDT0002 1:osc 2:oak-MDT0002-mdtlov_UUID
#3251 (144)setup 0:oak-OST014e-osc-MDT0002 1:oak-OST014e_UUID 2:10.42.17.114@tcp
#3252 (088)add_uuid nid=10.0.2.104@o2ib5(0x500050a000268) 0: 1:10.42.17.114@tcp
#3254 (112)add_conn 0:oak-OST014e-osc-MDT0002 1:10.42.17.114@tcp
#3255 (136)lov_modify_tgts add 0:oak-MDT0002-mdtlov 1:oak-OST014e_UUID 2:334 3:1
#3256 (224)END marker 6209 (flags=0x02, v2.12.7.0) oak-OST014e 'add osc' Mon Oct 4 08:13:45 2021-
#3257 (224)marker 6362 (flags=0x01, v2.12.7.0) oak 'quota.mdt' Thu Jan 20 21:23:29 2022-
#3258 (088)param 0:oak 1:quota.mdt=gp
#3259 (224)END marker 6362 (flags=0x02, v2.12.7.0) oak 'quota.mdt' Thu Jan 20 21:23:29 2022-
And here's the MDT copy:
rec #3188 type=10620000 len=224 offset 500632
rec #3189 type=10620000 len=224 offset 500856
rec #3190 type=10620000 len=88 offset 501080
rec #3191 type=10620000 len=224 offset 501168
Header size : 8192 llh_size : 80
Time : Fri Jan 21 18:36:56 2022
Number of records: 3132 cat_idx: 0 last_idx: 3191
Target uuid :
-----------------------
#01 (224)marker 2015 (flags=0x01, v2.10.8.0) oak-MDT0002-mdtlov 'lov setup' Thu Sep 26 05:36:36 2019-
#02 (128)attach 0:oak-MDT0002-mdtlov 1:lov 2:oak-MDT0002-mdtlov_UUID
#03 (176)lov_setup 0:oak-MDT0002-mdtlov 1:(struct lov_desc)
uuid=oak-MDT0002-mdtlov_UUID stripe:cnt=1 size=1048576 offset=18446744073709551615 pattern=0x1
...
#18 (224)marker 2018 (flags=0x01, v2.10.8.0) oak-MDT0002 'add osc(copied)' Thu Sep 26 05:36:36 2019-
#19 (224)EXCLUDE START marker 2019 (flags=0x11, v2.10.8.0) oak-OST0003 'add osc' Thu Sep 26 05:36:36 2019-Fri Mar 19 22:29:33 2021
#20 (088)add_uuid nid=10.0.2.102@o2ib5(0x500050a000266) 0: 1:10.42.17.114@tcp
#21 (088)add_uuid nid=10.0.2.101@o2ib5(0x500050a000265) 0: 1:10.42.17.114@tcp
#22 (224)END EXCLUDE END marker 2019 (flags=0x12, v2.10.8.0) oak-OST0003 'add osc' Thu Sep 26 05:36:36 2019-Fri Mar 19 22:29:33 2021
#23 (224)END marker 2019 (flags=0x02, v2.10.8.0) oak-MDT0002 'add osc(copied)' Thu Sep 26 05:36:36 2019-
#24 (224)marker 2020 (flags=0x01, v2.10.8.0) oak-MDT0002 'add osc(copied)' Thu Sep 26 05:36:36 2019-
#25 (224)EXCLUDE START marker 2021 (flags=0x11, v2.10.8.0) oak-OST0001 'add osc' Thu Sep 26 05:36:36 2019-Fri Mar 19 22:29:22 2021
..
#3172 (224)marker 6201 (flags=0x01, v2.12.7.0) oak-OST014d 'add osc' Mon Oct 4 08:10:30 2021-
#3173 (088)add_uuid nid=10.0.2.104@o2ib5(0x500050a000268) 0: 1:10.42.17.114@tcp
#3174 (128)attach 0:oak-OST014d-osc-MDT0002 1:osc 2:oak-MDT0002-mdtlov_UUID
#3175 (144)setup 0:oak-OST014d-osc-MDT0002 1:oak-OST014d_UUID 2:10.42.17.114@tcp
#3176 (088)add_uuid nid=10.0.2.103@o2ib5(0x500050a000267) 0: 1:10.42.17.114@tcp
#3177 (112)add_conn 0:oak-OST014d-osc-MDT0002 1:10.42.17.114@tcp
#3178 (136)lov_modify_tgts add 0:oak-MDT0002-mdtlov 1:oak-OST014d_UUID 2:333 3:1
#3179 (224)END marker 6201 (flags=0x02, v2.12.7.0) oak-OST014d 'add osc' Mon Oct 4 08:10:30 2021-
#3181 (224)marker 6209 (flags=0x01, v2.12.7.0) oak-OST014e 'add osc' Mon Oct 4 08:13:45 2021-
#3182 (088)add_uuid nid=10.0.2.103@o2ib5(0x500050a000267) 0: 1:10.42.17.114@tcp
#3183 (128)attach 0:oak-OST014e-osc-MDT0002 1:osc 2:oak-MDT0002-mdtlov_UUID
#3184 (144)setup 0:oak-OST014e-osc-MDT0002 1:oak-OST014e_UUID 2:10.42.17.114@tcp
#3185 (088)add_uuid nid=10.0.2.104@o2ib5(0x500050a000268) 0: 1:10.42.17.114@tcp
#3186 (112)add_conn 0:oak-OST014e-osc-MDT0002 1:10.42.17.114@tcp
#3187 (136)lov_modify_tgts add 0:oak-MDT0002-mdtlov 1:oak-OST014e_UUID 2:334 3:1
#3188 (224)END marker 6209 (flags=0x02, v2.12.7.0) oak-OST014e 'add osc' Mon Oct 4 08:13:45 2021-
#3189 (224)marker 6362 (flags=0x01, v2.12.7.0) oak 'quota.mdt' Thu Jan 20 21:23:29 2022-
#3190 (088)param 0:oak 1:quota.mdt=gp
#3191 (224)END marker 6362 (flags=0x02, v2.12.7.0) oak 'quota.mdt' Thu Jan 20 21:23:29 2022-
In particular:
- the last_idx doesn't match
- from 20 onwards, the copy llog content is the same but the index drifts off: it goes up for EXCLUDE blocks then oscillate between 48/49 (some holes on mdt that aren't on mgs), then climbs again when there are records pertaining OST[0000-000b] which had been excluded previously showing up again later. I've attached a post-processed catalog with diff/index on mgt/index on mdt: llogdiff
From my understanding, the catalog should be identical on both nodes as the local copy will be used during mds start, so differences there will likely cause problems. It's also safe to say the difference is probably due to the EXCLUDE blocks as suspected from the start...
So we now have two ways forward:
- try to fix the copy so excluded indexes don't affect overall index (e.g. copy holes too)
- remove EXCLUDED rows on catalogs and pretend this all never happened.
since exclude has never been merged I'd be tempted to put a rug on this and try to fix catalogs, but unfortunately I can't think of any way of fixing them nicely without having to shut down the whole filesystem (e.g. nuke the MGS and have targets reregister is most certainly not safe with thousands of clients mounted)
Do you have any suggestion?
Thanks!
|
|
umounting/mounting the MDT target should sync the local configuration with the MGS if the MDT successfully get a lock on llog config ressource.
The job is done by mgc_llog_local_copy().
If the MDT fail to sync the configuration the following message should appear (with debug=+mgc):
You can get more information from : https://review.whamcloud.com/40448/ ("LU-14090 mgs: no local logs flag")
If the MDT local configuration is different from the MGS configuration after the copy, it can mean that there is a bug in llog_backup().
|
|
Hi Etienne! Thanks for the reply.
Yes, in my case the copy was successful, so there definitely is a bug in the copy as I pointed out.
I'm not sure if it's worth fixing this bug for a feature that never was released (there probably isn't anyone else with EXCLUDE catalog entries...) but it might actually be easier to fix the copy than to try to fix the catalogs – I'll admit I haven't looked yet and it's worth a shot.
|
|
Well, it looks like we can have EXCLUDE entries, so it's probably worth a check anyway...
- first, a confirmation that this is a problem:
we use the same cld->cld_cfg (which stores the last_idx processed) whether we look at the remote catalog (in mgc_process_cfg_log, rctxt) or local one (ctxt) , so if the indexes don't match (at least the last index) we'll get bad behaviour as observed: we need copy to preserve the indices.
- looking at llog_backup:
probing on llog_osd_write_rec I can see that the rec->lrh_index is correct (mgs index#), but the writing idx argument is -1 so we're just appending, basically incrementing the record# by 1 everytime and not skipping things like the original record does. So as observed it basically copies the right content but doesn't care about the index.
Unfortunately, the code doesn't let us modify an unset index, so we'd need to keep track of the last written index, compare with the one we want to write and create fake records and cancel these to have the index artificially match?
FWIW I've kludged llog_reader to take a peek at what the "holes" are about and it's cancelled llogs from the exluded block, as well as padding log entries (which are also present on the copy at different offset when they are needed) – unfortunately llog_osd_pad is not exposed so we can't abuse it but we could just write entries with LLOG_PAD_MAGIC type empty or minimal size and immediately cancel it to unset the header bit...? Looking back I don't see any clean way of clearing the bit actually, it might not be so easy :|
Anyway, ETIMEDOUT for today – will keep digging. Please speak up if you have better ideas than what I'm rambling about 
|
|
Hi Dominique,
I think issue is that the mgc_llog_local_copy/llog_backup do not reproduce gap from the original config llog. The copy is done via llog_process that ignore cancel record (from bitmap llog header).
So your original:
is remplaced in local copy by:
The copy don't keep the "holes" in the indexes created by "del_ost", that why you do not have the same cfg_last_idx.
So it mess up the config update mechanism because the MDT rely on the local last index to get new record from the MGS config.
So if you try to remove the EXCLUDE record it will increase the index difference between local and original. This will not help.
I am working on a patch to reproduce index gap on the local copy.
|
|
"Etienne AUJAMES <eaujames@ddn.com>" uploaded a new patch: https://review.whamcloud.com/46545
Subject: LU-15000 llog: check for index gaps in llog_backup()
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: fa9c281196d7683e76096aa20b3f08b7ccaca0be
|
|
The patch above implement what you have described (without LLOG_PAD_MAGIC). But like you said this is not totaly accurate because of the chunk padding.
Maybe the easiest way is to take the local backup contents (without "holes") and correct the MGS original with it.
|
|
Hi Etienne,
Thank you!
Re: your last comment, we were thinking of doing that (do a copy of each MDT's local backup to the MGS), which could fix the immediate issue on the MDTs when adding new OSTs, but wouldn't that mess up mounted clients in that case, especially after we add more OSTs? Are clients' config based on indexes? Our main goal is to avoid re-mounting all clients but still be able to add new OSTs, which is tricky due to this index problem.
|
|
Hi Etienne, thanks for the patch!
That's pretty much what I had in mind, yes. This came back haunting me a bit while I sleep and yes it's not perfect because of padding: we can't control when each side will add their padding (because length differences), so the last index written might be 1 too big and we might have the opposite problem now (last index bigger means we'll skip one record we should have processed)
So I've been thinking it's safer to keep the holes, and just append dummy small logs at the end until we get the right last index to fake it up. It's safer because the dummy logs being small would reduce the need to add padding, but unfortunately it's still not perfect because if it does require padding at the last dummy log appended we'll overrun the last index again 
I can't think of any safe copy mechanism that preserves indices right now as we don't know the side of skipped records northe offset within the changelog of the entry immediately after a hole, and unless we can reproduce that we won't be able to match the automatic padding. I plan to look further this way this weekend.
Stéphane, this is not a problem: clients don't have a copy of the MDT changelogs, so we can just restart the MDT and they'll get a new copy which will fix all our problems. If we want to play safe we can also backport the tunefs --nolocallogs patch, but even without it the servers will try to get a new copy and throw away the old one if copy worked, so it should work as far as I understand. That won't impact clients more than a normal failover, so if we can get this right this is a good solution.
|
|
Hi again, sorry for the spam Talking with Stéphane gave me another idea: if we can't copy the offsets reliably, we can just get rid of them.
In mgc_llog_local_copy the MDS will make a local copy first with llog_backup(env, obd, lctxt, lctxt, logname, temp_log); (note they use lctxt twice which is LLOG_CONFIG_ORIG_CTXT).
This process is identical and will get rid of any cancelled changelog, so if we just do the same on the MGS for any log that has holes when it starts we won't need to worry about these anymore.
It's not perfect either though: we still have the problem while the MGS is running immediately after cancel events, so I guess we need a better time to do that.. asynchronously a while after any cancel event? hmm... Well, just a wild idea if we can't get copy right. I still intend on looking further this weekend.
|
|
@Stephane Thiell the issue with correcting the MGS configs is that the target copies have last indexes lower that the original ones. Clients/targets are notified only for "new" records in mounted state, so they will ignore indexes between original and the copy.
One solution could be to artificially generate rec after correcting MGS (with "lctl config_param <param>" and "lctl config_param -d <param>") until the correct configs index reach the old configs "last indexes". Then clients/targets should take into account new records (e.g adding new OSTs).
|
|
"Etienne AUJAMES <eaujames@ddn.com>" uploaded a new patch: https://review.whamcloud.com/46552
Subject: LU-15000 llog: read canceled records in llog_backup
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 0c5471748b18d40893309aad3943b8aed5e91f1a
|
|
Hi Etienne,
Today, we applied your patch on MGS/MDS (on top of 2.12.7) and we were then able to add 16 OSTs without issue. So your patch worked for us. This will also allow us to consider using llog_cancel/del_ost in the future again.
Thank you!
|
|
Hi Stephane,
Great for you.
I will add some reviewers to land the patch.
|
|
"Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/46552/
Subject: LU-15000 llog: read canceled records in llog_backup
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: d8e2723b4e9409954846939026c599b0b1170e6e
|
|
Landed for 2.16
|
|
"Etienne AUJAMES <eaujames@ddn.com>" uploaded a new patch: https://review.whamcloud.com/47515
Subject: LU-15000 llog: read canceled records in llog_backup
Project: fs/lustre-release
Branch: b2_12
Current Patch Set: 1
Commit: 57b085cb1068c1a7a7a35376ccf1db10f489c6a9
|
|
@Peter Jones could you please consider this patch for inclusion in LTS releases (at least for 2.15)?
|
|
"Jian Yu <yujian@whamcloud.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/48898
Subject: LU-15000 llog: read canceled records in llog_backup
Project: fs/lustre-release
Branch: b2_15
Current Patch Set: 1
Commit: 51bf31c545e2bc67b214de7ae8fd415c1776c3bf
|
|
"Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/48898/
Subject: LU-15000 llog: read canceled records in llog_backup
Project: fs/lustre-release
Branch: b2_15
Current Patch Set:
Commit: 7ba5dc8e895c693f68d49e7ffc46483710d67beb
|
Generated at Sat Feb 10 03:14:35 UTC 2024 using Jira 9.4.14#940014-sha1:734e6822bbf0d45eff9af51f82432957f73aa32c.
llogdiff
oak-MDT0000.after-ost331
oak-MDT0000.llog.txt