Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-2176

ZFS: running racer grounds everything to a standstill

Details

    • Bug
    • Resolution: Cannot Reproduce
    • Major
    • None
    • Lustre 2.4.0
    • 3
    • 5214

    Description

      There seems to be a very strange problem with racer and lustre over zfs.
      When you run SLOW=yes REFORMAT=yes FSTYPE=zfs DURATIOn=2700 sh racer.sh
      the dd stuff at the start of racer mostly never finish by themselves, even though we see heavy activity of zfs threads and some errors about ENOSPC in the logs.

      the actual creates and other stuff usually gets run after the timeout is already finished and everything is being killed.

      Contrast this with an ldiskfs run whre shortly after start we get a lot of messages about "file_create: SIZE=XXX" going all the way until the time is up.

      Alex is already looking into this one.

      Attachments

        Issue Links

          is related to

          Bug - A problem which impairs or prevents the functions of the product. LU-2887 sanity-quota test_12a: slow due to ZFS VMs sharing single disk

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Activity

            [LU-2176] ZFS: running racer grounds everything to a standstill
            adilger Andreas Dilger made changes -
            Resolution New: Cannot Reproduce [ 5 ]
            Status Original: Open [ 1 ] New: Resolved [ 5 ]
            adilger Andreas Dilger added a comment -

            Close old issue, I think ZFS is running this today w/o problems.

            adilger Andreas Dilger added a comment - Close old issue, I think ZFS is running this today w/o problems.
            bzzz Alex Zhuravlev added a comment -

            Oleg, is this still an actual issue?

            bzzz Alex Zhuravlev added a comment - Oleg, is this still an actual issue?
            utopiabound Nathaniel Clark made changes -
            Link New: This issue is related to LU-2887 [ LU-2887 ]
            utopiabound Nathaniel Clark made changes -
            Labels Original: LB zfs New: LB performance zfs
            utopiabound Nathaniel Clark made changes -
            Labels Original: LB New: LB zfs
            jlevi Jodi Levi (Inactive) made changes -
            Priority Original: Blocker [ 1 ] New: Major [ 3 ]
            jlevi Jodi Levi (Inactive) made changes -
            Labels New: LB
            bzzz Alex Zhuravlev added a comment -

            SLOW=yes DURATION=2700 sh racer.sh:

            ldiskfs

            {OST|MDS}SIZE=default: 4740321 opens, 44659 getattrs, 131489 readdirs, 654MB in 145047 reads, 10312MB in 84271 writes
            zfs {OST|MDS}

            SIZE=default: 1287981 opens, 201684 getattrs, 41838 readdirs, 41MB in 5827 reads, 825MB in 36025 writes
            zfs

            {OST|MDS}SIZE=500000: 1830863 opens, 180514 getattrs, 50876 readdirs, 98MB in 17732 reads, 3640MB in 80152 writes
            zfs,4GB,{OST|MDS}

            SIZE=500000: 2941173 opens, 132408 getattrs, 84917 readdirs, 216MB in 7664 reads, 4726MB in 62449 writes

            given zfs's requirements are significantly higher, I think the numbers are OK at the moment.

            bzzz Alex Zhuravlev added a comment - SLOW=yes DURATION=2700 sh racer.sh: ldiskfs {OST|MDS}SIZE=default: 4740321 opens, 44659 getattrs, 131489 readdirs, 654MB in 145047 reads, 10312MB in 84271 writes zfs {OST|MDS} SIZE=default: 1287981 opens, 201684 getattrs, 41838 readdirs, 41MB in 5827 reads, 825MB in 36025 writes zfs {OST|MDS}SIZE=500000: 1830863 opens, 180514 getattrs, 50876 readdirs, 98MB in 17732 reads, 3640MB in 80152 writes zfs,4GB,{OST|MDS} SIZE=500000: 2941173 opens, 132408 getattrs, 84917 readdirs, 216MB in 7664 reads, 4726MB in 62449 writes given zfs's requirements are significantly higher, I think the numbers are OK at the moment.
            bzzz Alex Zhuravlev added a comment - - edited

            http://review.whamcloud.com/4403 should provide a bit more info including amount of data read/written.

            from what I've seen locally racer does work with ZFS, not that great as with ldiskfs, but quite acceptable, IMHO.

            I'd suggest Oleg to see actual numbers locally and, hopefully, close the ticket

            bzzz Alex Zhuravlev added a comment - - edited http://review.whamcloud.com/4403 should provide a bit more info including amount of data read/written. from what I've seen locally racer does work with ZFS, not that great as with ldiskfs, but quite acceptable, IMHO. I'd suggest Oleg to see actual numbers locally and, hopefully, close the ticket

            People

              bzzz Alex Zhuravlev
              green Oleg Drokin
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: