Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-11205

Failure to clear the changelog for user 1 on MDT

    XMLWordPrintable

Details

    • Bug
    • Resolution: Duplicate
    • Major
    • None
    • Lustre 2.12.0, Lustre 2.10.4, Lustre 2.10.6
    • CentOS 7.4 (3.10.0-693.2.2.el7_lustre.pl1.x86_64)
    • 3
    • 9223372036854775807

    Description

      Hello,

      We're seeing the following messages on Oak's MDT in 2.10.4:

      Aug 03 09:21:39 oak-md1-s2 kernel: Lustre: 11137:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22
      Aug 03 09:31:38 oak-md1-s2 kernel: Lustre: 11271:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22
      

      Robinhood (also running 2.10.4) shows this:

      2018/08/03 10:00:47 [13766/22] ChangeLog | ERROR: llapi_changelog_clear("oak-MDT0000", "cl1", 13975842301) returned -22
      2018/08/03 10:00:47 [13766/22] EntryProc | Error -22 performing callback at stage STAGE_CHGLOG_CLR.
      2018/08/03 10:00:47 [13766/16] llapi | cannot purge records for 'cl1'
      2018/08/03 10:00:47 [13766/16] ChangeLog | ERROR: llapi_changelog_clear("oak-MDT0000", "cl1", 13975842303) returned -22
      2018/08/03 10:00:47 [13766/16] EntryProc | Error -22 performing callback at stage STAGE_CHGLOG_CLR.
      2018/08/03 10:00:47 [13766/4] llapi | cannot purge records for 'cl1'
      2018/08/03 10:00:47 [13766/4] ChangeLog | ERROR: llapi_changelog_clear("oak-MDT0000", "cl1", 13975842304) returned -22
      2018/08/03 10:00:47 [13766/4] EntryProc | Error -22 performing callback at stage STAGE_CHGLOG_CLR.
      

      Oak's MDT usage is as follow:

      [root@oak-md1-s2 ~]# df -h -t lustre
      Filesystem                  Size  Used Avail Use% Mounted on
      /dev/mapper/md1-rbod1-mdt0  1.3T  131G 1022G  12% /mnt/oak/mdt/0
      [root@oak-md1-s2 ~]# df -i -t lustre
      Filesystem                    Inodes     IUsed     IFree IUse% Mounted on
      /dev/mapper/md1-rbod1-mdt0 873332736 266515673 606817063   31% /mnt/oak/mdt/0
      

      I'm concerned that the MDT might fill up with changelogs. Could you please assist in troubleshooting this issue?
      Thanks!
      Stephane

      Attachments

        1. ornl_0x1_0xd8_0x0.gz
          1.19 MB
        2. lu-11205-ssec.log.gz
          647 kB
        3. f2_llog_reader_20190328.gz
          1.03 MB
        4. dk.1547828521.gz
          7.96 MB
        5. dk.1547747668.gz
          8.58 MB
        6. dk.1547747365.gz
          8.56 MB
        7. dk_ornl_20190328.gz
          4.59 MB
        8. dk_ornl_20190328_1216.gz
          4.61 MB
        9. changelog-reader.tgz
          0.8 kB

        Issue Links

          Activity

            People

              tappro Mikhail Pershin
              sthiell Stephane Thiell
              Votes:
              3 Vote for this issue
              Watchers:
              26 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: