https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-4481 Lustre <p>Hi,</p> <p>On a customer cluster, changelogs refuse to start, probably because of an internal data corruption.<br/> Here are the messages we can see when mounting the MDT:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>1373184833 2013 Jul 7 10:13:53 bcluster111 kern err kernel LustreError: 14143:0:(llog_lvfs.c:199:llog_lvfs_read_header()) bad log header magic: 0x10670000 (expected 0x10645539) 1373184833 2013 Jul 7 10:13:53 bcluster111 kern err kernel LustreError: 14143:0:(llog_obd.c:320:cat_cancel_cb()) Cannot find handle for log 0x1490186b: -5 1373184833 2013 Jul 7 10:13:53 bcluster111 kern err kernel LustreError: 14133:0:(llog_obd.c:393:llog_obd_origin_setup()) llog_process() with cat_cancel_cb failed: -5 1373184833 2013 Jul 7 10:13:53 bcluster111 kern err kernel LustreError: 14133:0:(llog_obd.c:220:llog_setup_named()) obd mdd_obd-scratch3-MDT0000 ctxt 14 lop_setup=ffffffffa0501cc0 failed -5 1373184833 2013 Jul 7 10:13:53 bcluster111 kern err kernel LustreError: 14133:0:(mds_log.c:218:mds_changelog_llog_init()) changelog users llog setup failed -5 1373184835 2013 Jul 7 10:13:55 bcluster111 kern err kernel LustreError: 14133:0:(mdd_device.c:216:mdd_changelog_llog_init()) no changelog user context 1373184835 2013 Jul 7 10:13:55 bcluster111 kern err kernel LustreError: 14133:0:(mdd_device.c:254:mdd_changelog_init()) Changelog setup during init failed -22 1373184835 2013 Jul 7 10:13:55 bcluster111 kern warning kernel Lustre: scratch3-MDT0000: used disk, loading </pre> </div></div> <p>So the MDt is started, but without changelogs.</p> <p>And if we try to look at changelog_users with lctl:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># lctl get_param mdd.scratch3-MDT0000.changelog_users error: get_param: read('/proc/fs/lustre/mdd/scratch3-MDT0000/changelog_users') failed: No such device or address </pre> </div></div> <p>The problem is the customer needs Lustre changelogs because they are consumed by Robinhood to monitor activity on the file system.</p> <p>So the first thing we need is a way to restart changelogs as soon as possible. We already tried any administrative lustre command (lfs or lctl) to cleanup things, but it did not work due to the fact that the feature did not start. Manually cleaning OBJETS files is not a thing we tried, for fear of making the situation even worse.</p> <p>After the changelogs will be restarted on site, we will need a fix so that changelogs can deal with corrupted data and start afresh in that case.</p> <p>But again, the really first thing we need is a helping hand to clean things on site and restart changelogs ASAP.</p> <p>TIA,<br/> Sebastien.</p> LU-4481

Impossible to start changelogs after corruption

Bug Critical Resolved Fixed Bruno Faccini Sebastien Buisson Tue, 14 Jan 2014 08:56:14 +0000 Wed, 13 Oct 2021 03:04:39 +0000 Wed, 13 Oct 2021 03:04:39 +0000 Lustre 2.1.6 0 7 <p>Hello Seb,<br/> Can you run debugfs on the concerned MDT and do a "stat changelog_users" and "stat changelog_catalog" to provide their infos?<br/> Then can you mount the MDT as ldiskfs and do a "xxd changelog_users" to provide its corrupted content?<br/> BTW, the "0x10670000" wrong value, vs expected LLOG_HDR_MAGIC, looks like CHANGELOG_USER_REC … <br/> In the same time, I am trying to setup a platform to test a <span class="error">&#91;manual?&#93;</span> way for you to recover from this.</p> <p>Hi Bruno,</p> <p>Please find attached two files:</p> <ul class="alternate" type="square"> <li>debugfs_stat: stat of the 2 files changelog_users and changelog_catalog in debugfs;</li> <li>changelog_users_xxd: xxd of the file changelog_users with the MDT mounted in ldiskfs.</li> </ul> <p>HTH,<br/> Sebastien.</p> <p>Bruno,</p> <p>As there is this message "Cannot find handle for log 0x1490186b: -5" in the logs, people on site have also stated this file and taken an od dump:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@bcluster111 OBJECTS] # pwd /mnt/scratch3/mdt/0_ldiskfs/OBJECTS [root@bcluster111 OBJECTS] # od -tx4 1490186b:dbc122a8 | more 0000000 00000028 00000001 10670000 00000000 0000020 00000001 00000000 b3ef6c29 00000000 0000040 00000028 00000001 00000002 00000058 0000060 00000000 00000005 00000001 00000000 0000100 00000000 00000000 00000000 00000000 * 0000140 00000000 00000000 00000003 00000000 0000160 00000000 00000000 00000000 00000000 * 0020000 00000000 00000000 00002000 00000001 0020020 00000001 00000000 b3ef4519 00000000 0020040 00000028 00000001 0020050 [root@bcluster111 OBJECTS] # od -tx4 149025cb:672131de | head -5 0000000 00002000 00000000 10645539 00000000 0000020 51001a9c 00000000 00000012 00000058 0000040 00000040 00000002 00000001 00000000 0000060 00000000 00000000 00000000 00000000 * [root@bcluster111 OBJECTS] # stat 1490186b:dbc122a8 File: `1490186b:dbc122a8' Size: 8232 Blocks: 24 IO Block: 4096 regular file Device: fd01h/64769d Inode: 344987755 Links: 1 Access: (0666/-rw-rw-rw-) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2013-07-12 11:59:30.727996524 +0200 Modify: 2012-08-21 14:42:09.320548695 +0200 Change: 2012-08-21 14:42:09.320548695 +0200 </pre> </div></div> <p>Maybe you could find this helpful.</p> <p>Sebastien.</p> <p>Yes thanks, it was the next step/need I would have requested since changelog_user is ok and pointing to OBJECTS/149025cb:672131de !!<br/> And this is this last file which is corrupted, as the "Cannot find handle for log 0x1490186b: -5" msg you point indicated.</p> <p>Will try to get back soon with a bypass/reconstruct procedure.</p> <p>Hello Seb,<br/> I made some more work/tests in order for you to be able to restart Change-Logs.<br/> A "normal" OBJECTS file pointed by changelog_users with a single/cl1 id registered looks like :</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>0000000 00002000 00000000 10645539 00000000 0000020 52dd1c70 00000000 00000002 00000058 0000040 00000000 00000005 00000001 00000000 0000060 00000000 00000000 00000000 00000000 * 0000120 00000000 00000000 00000003 00000000 0000140 00000000 00000000 00000000 00000000 * 0017760 00000000 00000000 00002000 00000001 0020000 00000028 00000001 10670000 00000000 0020020 00000001 00000000 00000dd7 00000000 0020040 00000028 00000001 0020050 </pre> </div></div> <p>So your is badly corrupted and lacks its full header record !!… But BTW, there is no need to try reconstruct it, since as I understood you restarted and used the MDT without Change-Logs enabled, so a full RobinHood scan will be required to re-populate its database from scratch.</p> <p>Then, to be able to restart Change-Logs, you will need to umount/stop the MDT, mount it as LDISKFS, move/mv both CONFIGS/changelog_<span class="error">&#91;catalog,users&#93;</span> to new-names, re-start/mount the MDT, and re-register the change-log user/id configured in RobinHood.</p> <p>Moving both CONFIGS/changelog_<span class="error">&#91;catalog,users&#93;</span> to new-names is strongly required since it will allow for later OBJECTS/* related files clean-up.</p> <p>Additionally, when you say "we will need a fix so that changelogs can deal with corrupted data and start afresh in that case", do you mean that the procedure I described before should be automatic during MDT mount/start ?</p> <p>Hi Bruno,</p> <p>Thanks for the procedure, I have forwarded it to onsite support team.</p> <p>Concerning the fix, what I meant was that MDT should be able to cope with a corrupted OBJECTS file and start changelog feature even in that case, for instance by ignoring it. But after reading your last comment, it seems that this "restart in degraded mode" way would lead to an implicit reset of changelogs config (users and catalog). So this may be a little bit too strong...<br/> On the other hand, I think resetting changelogs config (via lctl or lfs command) should be possible even if the feature is not started on the MDT. That would avoid mounting the MDT as ldiskfs and manually moving files.<br/> What do you think?</p> <p>Thanks!<br/> Sebastien.</p> <p>Hi,</p> <p>Concerning the procedure, I have confirmation from the Support team that it worked fine. Changelogs are now functional on site, thanks!</p> <p>Cheers,<br/> Sebastien.</p> Development End date Wed, 29 Jan 2014 08:56:14 +0000 Rank 1|hzwcrr: Rank (Obsolete) 12272 Severity Start date Tue, 14 Jan 2014 08:56:14 +0000