Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-11205

Failure to clear the changelog for user 1 on MDT

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Duplicate
    • Icon: Major Major
    • None
    • Lustre 2.12.0, Lustre 2.10.4, Lustre 2.10.6
    • CentOS 7.4 (3.10.0-693.2.2.el7_lustre.pl1.x86_64)
    • 3
    • 9223372036854775807

      Hello,

      We're seeing the following messages on Oak's MDT in 2.10.4:

      Aug 03 09:21:39 oak-md1-s2 kernel: Lustre: 11137:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22
Aug 03 09:31:38 oak-md1-s2 kernel: Lustre: 11271:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22

      Robinhood (also running 2.10.4) shows this:

      2018/08/03 10:00:47 [13766/22] ChangeLog | ERROR: llapi_changelog_clear("oak-MDT0000", "cl1", 13975842301) returned -22
2018/08/03 10:00:47 [13766/22] EntryProc | Error -22 performing callback at stage STAGE_CHGLOG_CLR.
2018/08/03 10:00:47 [13766/16] llapi | cannot purge records for 'cl1'
2018/08/03 10:00:47 [13766/16] ChangeLog | ERROR: llapi_changelog_clear("oak-MDT0000", "cl1", 13975842303) returned -22
2018/08/03 10:00:47 [13766/16] EntryProc | Error -22 performing callback at stage STAGE_CHGLOG_CLR.
2018/08/03 10:00:47 [13766/4] llapi | cannot purge records for 'cl1'
2018/08/03 10:00:47 [13766/4] ChangeLog | ERROR: llapi_changelog_clear("oak-MDT0000", "cl1", 13975842304) returned -22
2018/08/03 10:00:47 [13766/4] EntryProc | Error -22 performing callback at stage STAGE_CHGLOG_CLR.

      Oak's MDT usage is as follow:

      [root@oak-md1-s2 ~]# df -h -t lustre
Filesystem                  Size  Used Avail Use% Mounted on
/dev/mapper/md1-rbod1-mdt0  1.3T  131G 1022G  12% /mnt/oak/mdt/0
[root@oak-md1-s2 ~]# df -i -t lustre
Filesystem                    Inodes     IUsed     IFree IUse% Mounted on
/dev/mapper/md1-rbod1-mdt0 873332736 266515673 606817063   31% /mnt/oak/mdt/0

      I'm concerned that the MDT might fill up with changelogs. Could you please assist in troubleshooting this issue?
      Thanks!
      Stephane

        1. dk.1547747365.gz
          8.56 MB
        2. dk.1547747668.gz
          8.58 MB
        3. dk.1547828521.gz
          7.96 MB
        4. dk_ornl_20190328.gz
          4.59 MB
        5. f2_llog_reader_20190328.gz
          1.03 MB
        6. dk_ornl_20190328_1216.gz
          4.61 MB
        7. ornl_0x1_0xd8_0x0.gz
          1.19 MB
        8. lu-11205-ssec.log.gz
          647 kB
        9. changelog-reader.tgz
          0.8 kB

            tappro Mikhail Pershin
            sthiell Stephane Thiell
            Votes:
            3 Vote for this issue
            Watchers:
            26 Start watching this issue

              Created:
              Updated:
              Resolved: