Hongchao
Can you please advise?
Thanks
Peter
So to summarize this differently, changelogs are on but not recorded. The changelog_size doesn't increase, for example, for oak-MDT0001, it stays at mdd.oak-MDT0001.changelog_size=4169832.
[root@oak-md1-s1 ~]# lctl get_param mdd.oak-MDT0001.changelog_* mdd.oak-MDT0001.changelog_deniednext=60 mdd.oak-MDT0001.changelog_gc=0 mdd.oak-MDT0001.changelog_max_idle_indexes=2097446912 mdd.oak-MDT0001.changelog_max_idle_time=2592000 mdd.oak-MDT0001.changelog_min_free_cat_entries=2 mdd.oak-MDT0001.changelog_min_gc_interval=3600 mdd.oak-MDT0001.changelog_size=4169832 mdd.oak-MDT0001.changelog_mask= CREAT MKDIR HLINK SLINK MKNOD UNLNK RMDIR RENME RNMTO CLOSE LYOUT TRUNC SATTR HSM MTIME CTIME MIGRT FLRW RESYNC mdd.oak-MDT0001.changelog_users= current index: 9422682426 ID index (idle seconds) cl2 9422682426 (49316)
But changelogs are enabled:
Oct 21 08:32:23 oak-md1-s1 kernel: Lustre: oak-MDD0001: changelog on Oct 21 08:32:23 oak-md1-s1 kernel: Lustre: 41861:0:(mdd_device.c:545:mdd_changelog_llog_init()) oak-MDD0001 : orphan changelog records found, starting from index 0 to index 9422682426, being cleared now
Hi Hongchao,
Should I just stop each MDT and rename the file changelog_catalog and changelog_users at the ldiskfs level? Any other recommendation to not leak too much space on MDT if I do that?
Thanks!
Stephane
Hi Hongchao,
This morning, we tried the procedure of renaming the changelog files under ldiskfs like found in other tickets, eg:
mv changelog_catalog changelog_catalog.bak mv changelog_users changelog_users.bak
But even after that, the changelogs are still not recorded (size is stuck at 33016):
[root@oak-md1-s2 ~]# lctl get_param mdd.oak-MDT0000.changelog_* mdd.oak-MDT0000.changelog_deniednext=60 mdd.oak-MDT0000.changelog_gc=0 mdd.oak-MDT0000.changelog_max_idle_indexes=2097446912 mdd.oak-MDT0000.changelog_max_idle_time=2592000 mdd.oak-MDT0000.changelog_min_free_cat_entries=2 mdd.oak-MDT0000.changelog_min_gc_interval=3600 mdd.oak-MDT0000.changelog_size=33016 mdd.oak-MDT0000.changelog_mask= CREAT MKDIR HLINK SLINK MKNOD UNLNK RMDIR RENME RNMTO CLOSE LYOUT TRUNC SATTR HSM MTIME CTIME MIGRT FLRW RESYNC mdd.oak-MDT0000.changelog_users= current index: 1 ID index (idle seconds) cl1 0 (254)
[root@oak-gw02 ~]# lfs changelog_clear oak-MDT0000 cl1 0 [root@oak-gw02 ~]# echo $? 0 [root@oak-gw02 ~]# lfs changelog oak-MDT0000 cl1 2 00MARK 13:22:02.346104356 2020.10.28 0x0 t=[0x40001:0x0:0x0] ef=0xf u=0:0 nid=0@<0:0> p=[0:0x50:0x2]
Also old changelog IDs are still working, which seems broken:
[root@oak-gw02 ~]# lfs changelog oak-MDT0000 cl2 2 00MARK 13:22:02.346104356 2020.10.28 0x0 t=[0x40001:0x0:0x0] ef=0xf u=0:0 nid=0@<0:0> p=[0:0x50:0x2] [root@oak-gw02 ~]# lfs changelog_clear oak-MDT0000 cl2 0 lfs changelog_clear: cannot purge records for 'cl2': No such file or directory (2) changelog_clear: no changelog user: cl2
How could we repair this and start with fresh changelogs? This issue happened after our 2.10 to 2.12 upgrade on our 4 MDTs. Thanks!
Hi,
I think there are two separate issues:
- the initial problem before changelogs were manually cleared.
The message "orphan changelog records found, starting from index 0 to index 22264575363, being cleared now" indicates that something very weird went wrong, it means the current reader (cl1) was at index 22264575363 which is ok but it found an entry for index 0 in the changelog catalog itself.
I've tried quite a few things including zero-crossing catalog before upgrading 2.10->2.12 but couldn't reproduce this, honestly no idea :/ That being said I was able to reproduce the exact same message by editing the catalog manually and adding a changelog with index 0 and it worked fine even with holes in the catalog; I don't think anything really bad happened.
- lack of changelog generation is completely unrelated and was due to nodemaps and audit_mode not defaulting to 1 on upgrade, that was the real problem.
commit 942a9853f7b4c6fe22729468f1802ab782087e4e from 2.11 says "By default, audit_mode flag is set to 1 in newly created nodemap entries. And it is also set to 1 on 'default' nodemap." but existing maps are read back from disk and fill the void with 0 ; I'll check quickly if that could be improved.
Unfortunately I don't see how to detect if the stored nodemap comes from an upgrade or not, the record has nothing like a "format version" that would let it check.
With hindsight 20/20 that probably should have been added together with the new field, at this point even if we add one now we won't know if it's someone who explicitely had it disabled or not... I'm the type that never reads the doc but maybe it's already documented to be careful with nodemaps on upgrade for that reason?
Unless someone has a miracle idea on how to properly init the nodemaps with nmf_enable_audit / NM_FL_ENABLE_AUDIT set on upgrade I guess this LU can be closed 