Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-4102

lots of multiply-claimed blocks in e2fsck

Details

    • Bug
    • Resolution: Cannot Reproduce
    • Major
    • None
    • Lustre 1.8.8
    • e2fsprogs 1.41.90.wc2
    • 3
    • 11017

    Description

      After a power loss, an older e2fsck (e2fsprogs 1.41.90.wc2) was run on the OSTs. It found tons of multiply-claimed blocks, including for the /O directory. Here's an example of one of the inodes:

      File ... (inode #17825793, mod time Wed Aug 15 19:02:25 2012)
        has 1 multiply-claimed block(s), shared with 1 file(s):
              /O (inode #84934657, mod time Wed Aug 15 19:02:25 2012)
      Clone multiply-claimed blocks? yes
      
      Inode 17825793 doesn't have an associated directory entry, it eventually gets put into lost+found.
      

      So the questions are:

      • how could this have happened? My slightly-informed-probably-wrong theory is that the journal got corrupted and it replayed some old inodes back into existence. I noticed there were a lot of patches dealing with journal checksums committed after 1.41.90.
      • what's the best way to deal with these? Cloning takes forever when you are talking about TB sized files. I tested the delete extended option, and it looks like it deletes both sides of the file. It would be nice if it just deleted the unlinked side. Right now my plan is to create a debugfs script from the read-only e2fsck output, but if there is a better way, that would be good.

      Thanks.

      Attachments

        1. e2fsck.patch1
          2 kB
          Kit Westneat
        2. lfs2_unusual_syslog_msgs_2013-10-19.txt
          67 kB
          Nathan Dauchy
        3. ost15.log.tgz
          599 kB
          Kit Westneat

        Issue Links

          Activity

            [LU-4102] lots of multiply-claimed blocks in e2fsck
            adilger Andreas Dilger made changes -
            Link New: This issue is related to LU-16171 [ LU-16171 ]
            adilger Andreas Dilger made changes -
            Description Original: After a power loss at NOAA, an older e2fsck (e2fsprogs 1.41.90.wc2) was run on the OSTs. It found tons of multiply-claimed blocks, including for the /O directory. Here's an example of one of the inodes:

            File ... (inode #17825793, mod time Wed Aug 15 19:02:25 2012)
              has 1 multiply-claimed block(s), shared with 1 file(s):
                    /O (inode #84934657, mod time Wed Aug 15 19:02:25 2012)
            Clone multiply-claimed blocks? yes

            Inode 17825793 doesn't have an associated directory entry, it eventually gets put into lost+found.

            So the questions are:
            * how could this have happened? My slightly-informed-probably-wrong theory is that the journal got corrupted and it replayed some old inodes back into existence. I noticed there were a lot of patches dealing with journal checksums committed after 1.41.90.
            * what's the best way to deal with these? Cloning takes forever when you are talking about TB sized files. I tested the delete extended option, and it looks like it deletes both sides of the file. It would be nice if it just deleted the unlinked side. Right now my plan is to create a debugfs script from the read-only e2fsck output, but if there is a better way, that would be good.

            Thanks.
            New: After a power loss, an older e2fsck (e2fsprogs 1.41.90.wc2) was run on the OSTs. It found tons of multiply-claimed blocks, including for the /O directory. Here's an example of one of the inodes:
            {noformat}
            File ... (inode #17825793, mod time Wed Aug 15 19:02:25 2012)
              has 1 multiply-claimed block(s), shared with 1 file(s):
                    /O (inode #84934657, mod time Wed Aug 15 19:02:25 2012)
            Clone multiply-claimed blocks? yes

            Inode 17825793 doesn't have an associated directory entry, it eventually gets put into lost+found.
            {noformat}
            So the questions are:
            * how could this have happened? My slightly-informed-probably-wrong theory is that the journal got corrupted and it replayed some old inodes back into existence. I noticed there were a lot of patches dealing with journal checksums committed after 1.41.90.
            * what's the best way to deal with these? Cloning takes forever when you are talking about TB sized files. I tested the delete extended option, and it looks like it deletes both sides of the file. It would be nice if it just deleted the unlinked side. Right now my plan is to create a debugfs script from the read-only e2fsck output, but if there is a better way, that would be good.

            Thanks.
            pjones Peter Jones made changes -
            Resolution New: Cannot Reproduce [ 5 ]
            Status Original: Open [ 1 ] New: Resolved [ 5 ]
            chunteraa Chris Hunter (Inactive) made changes -
            Link New: This issue is related to DDN-370 [ DDN-370 ]
            pjones Peter Jones made changes -
            End date New: 06/May/15
            Start date New: 14/Oct/13
            adilger Andreas Dilger made changes -
            Link New: This issue is related to LU-4745 [ LU-4745 ]
            pjones Peter Jones made changes -
            Link New: This issue is related to DDN-79 [ DDN-79 ]
            pjones Peter Jones made changes -
            Labels Original: mn1 mq314 New: mn1
            adilger Andreas Dilger made changes -
            Labels Original: mn1 New: mn1 mq314
            pjones Peter Jones made changes -
            Reporter Original: Kit Westneat [ kitwestneat ] New: Oz Rentas [ orentas ]

            People

              niu Niu Yawei (Inactive)
              orentas Oz Rentas (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: