[LU-4477] Changelogs not working anymore after upgrading from 2.1 to 2.4

• Status: Resolved
• Project: Lustre
• Affects Version/s: Lustre 2.4.1
• Type: Bug
• Priority: Major
• Reporter: Sebastien Buisson (Inactive)
• Assignee: Lai Siyao
• Resolution: Duplicate
• Severity: 3

Description

Hi,

On a customer site we have a file system originally formatted with Lustre 2.1. On this file system several changelog users were registered, and changelogs were working fine.

Then this file system was upgraded to 2.4. But it broke changelogs usage, as no changelog user is registered anymore. Moreover, former changelog records are unreachable.

As noted by Aurelien Degremont in his email (https://lists.01.org/pipermail/hpdd-discuss/2014-January/000713.html), it seems to be due to changes in the llog interface in 2.4 (changelog_catalog and changelog_users file are now in root MDT directory, changelog_catalogs do not point to other files in OBJECTS/ directory anymore, etc).

We would need a method to migrate the 'changelogs configuration' from 2.1 to 2.4.

Thanks,
Sebastien.

Comments

[Peter Jones]

Lai

Could you please help with this one

Thanks

Peter

[Lai Siyao]

Firstly I tried to add compatible code in 2.4 and master to keep back-compatible, but I realised that it complicated related code, and will be used only once. So a tool to migrate as you suggested should be better. And since this is seldom used, and to not complicate the utils, IMO we can just mount mdt device as ldiskfs and migrate changelogs directly, so the steps is as follows:
1. mount -t ldiskfs <mdt-device> /mnt/temp
2. cat /mnt/temp/CONFIGS/changelog_catalog > /mnt/temp/changelog_catalog
3. mv /mnt/temp/CONFIGS/changelog_catalog /tmp/changelog_catalog.backup
4. remount mdt device

You can do the same for 'changelog_users', which doesn't look to be necessary, though it will be cleaner to move it out of CONFIGS directory.

[Aurelien Degremont (Inactive)]

I've already tried this trick and it does not work.
It seems the llog structure has been changed too and it does not find the subcatalog. After restarting the MDT, it complains that it could not find some objects for the changelog_catalog.

[Lai Siyao]

I tested 2.1 -> master, which works for me, I'll test 2.4 later.

[Lai Siyao]

I just tested 2.1 -> 2.4, which works too. Could you upload your changelog_catalog file? I can do some test locally.

[Aurelien Degremont (Inactive)]

Unfortunately, I've restarted a new Changelog flow for this filesystem as I really needed it. It is difficult for me to modify it now to check this.

When I tried the same kind of tricks you are proposing, here the error log I got:

192:llog_cat_id2handle()) foo-MDD0000: error opening log id 0x0:123815571:fc5c411af: rc = -2
795:cat_cancel_cb()) foo-MDD0000: cannot find handle for llog 0x0:123815571: -2

But the OBJECT file is there!

If I do

$ mount -t ldiskfs /dev/sdd /mnt/foo
$ cd /mnt/foo
$ printf "%x\n" 123815571
7614693
$ ls -l OBJECTS/7614693\:f5c411af
-rw-rw-rw 1 root root 6929088 Dec 16 16:52 OBJECTS/7614693:f5c411af

So I do not understand why MDD layer is return rc = -2.
That's why I thought the OBJECTS file directory has changed too.

[Lai Siyao]

I'll reproduce it tomorrow and see what happens.

[Lai Siyao]

Latest 2.4 and master branch can support this trick, but 2.4.1 not, I'll see which patch is missing.

[Lai Siyao]

Commit 8d306c0617a89476b24a44eb0267f33e2f971ef2 http://review.whamcloud.com/7625 can fix this issue, which has been landed to 2.4.2, could you try it on your system?

[Peter Jones]

duplicate of lu-3934

[Aurelien Degremont (Inactive)]

Lai, we should be able to try this in 4 weeks. Systems are currently in production and we cannot test this upgrade sooner.