Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-11721

"lfs df -i" prints used blocks percentage instead of inodes

Details

    • Bug
    • Resolution: Fixed
    • Minor
    • Lustre 2.13.0, Lustre 2.12.3
    • Lustre 2.11.0
    • None
    • 3
    • 9223372036854775807

    Description

      When issuing an "lfs df -i", the "IUse%" field shows the percentage of used blocks, instead of the percentage of used inodes.

      Attachments

        Issue Links

          Activity

            [LU-11721] "lfs df -i" prints used blocks percentage instead of inodes

            I suspect the last issue was fixed with LU-12248.

            adilger Andreas Dilger added a comment - I suspect the last issue was fixed with LU-12248 .

            Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/34453/
            Subject: LU-11721 lod: limit statfs ffree if less than OST ffree
            Project: fs/lustre-release
            Branch: b2_12
            Current Patch Set:
            Commit: 805b98c8a9812a2c2bf77b54d30e1bc40aff299e

            gerrit Gerrit Updater added a comment - Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/34453/ Subject: LU-11721 lod: limit statfs ffree if less than OST ffree Project: fs/lustre-release Branch: b2_12 Current Patch Set: Commit: 805b98c8a9812a2c2bf77b54d30e1bc40aff299e

            The actual problem looks mostly to be related to what the 34453 patch was trying to fix - namely that the proxy statfs data returned by "df -i" is not being limited to the number of OST objects in the filesystem as it should. This makes the total and free inodes wildly larger than what "lfs df -i" reports. That patch landed "Fri Mar 15 23:46:21 2019" so it shouldn't be seen in patches that are testing this week unless they haven't been rebased, and I confirmed at least one failure was running a patch that included this fix.

            There is a second, less common failure mode on ZFS (e.g. maybe 10% of cases) shows the "df -i" output is slightly different than the "lfs df -i". That might be due to e.g. ZFS background activity changing the number of free blocks, which affects the free inode estimate. This is not a problem for ldiskfs. We might have to allow a margin of error in the total and free inodes for ZFS.

            adilger Andreas Dilger added a comment - The actual problem looks mostly to be related to what the 34453 patch was trying to fix - namely that the proxy statfs data returned by " df -i " is not being limited to the number of OST objects in the filesystem as it should. This makes the total and free inodes wildly larger than what " lfs df -i " reports. That patch landed "Fri Mar 15 23:46:21 2019" so it shouldn't be seen in patches that are testing this week unless they haven't been rebased, and I confirmed at least one failure was running a patch that included this fix. There is a second, less common failure mode on ZFS (e.g. maybe 10% of cases) shows the " df -i " output is slightly different than the " lfs df -i ". That might be due to e.g. ZFS background activity changing the number of free blocks, which affects the free inode estimate. This is not a problem for ldiskfs. We might have to allow a margin of error in the total and free inodes for ZFS.

            Looking at the client debug logs for the failure, it doesn't record any statfs RPCs at all during the test, and the exports are all reported IDLE. That should definitely not be the case for "lfs df" and "lfs df -i", since the client needs to contact all targets to get the statfs info. Only the "df" statfs can be aggregated, but it shouldn't be cached more than a second. The MDS logs show some activity, but not any of the actual statfs results, because the debug level is too low.

            It looks like the test is not running with full debugging enabled as it should for all sanity tests, so I pushed patch 34493 to ensure this is the case, and hopefully we will get some more usable debug logs after that patch lands.

            adilger Andreas Dilger added a comment - Looking at the client debug logs for the failure, it doesn't record any statfs RPCs at all during the test, and the exports are all reported IDLE. That should definitely not be the case for " lfs df " and " lfs df -i ", since the client needs to contact all targets to get the statfs info. Only the " df " statfs can be aggregated, but it shouldn't be cached more than a second. The MDS logs show some activity, but not any of the actual statfs results, because the debug level is too low. It looks like the test is not running with full debugging enabled as it should for all sanity tests, so I pushed patch 34493 to ensure this is the case, and hopefully we will get some more usable debug logs after that patch lands.

            Andreas Dilger (adilger@whamcloud.com) uploaded a new patch: https://review.whamcloud.com/34493
            Subject: LU-11721 tests: enable full debugging for sanity
            Project: fs/lustre-release
            Branch: master
            Current Patch Set: 1
            Commit: 5a8e480e3380009f5e9fddb7d38389e2b3473d8e

            gerrit Gerrit Updater added a comment - Andreas Dilger (adilger@whamcloud.com) uploaded a new patch: https://review.whamcloud.com/34493 Subject: LU-11721 tests: enable full debugging for sanity Project: fs/lustre-release Branch: master Current Patch Set: 1 Commit: 5a8e480e3380009f5e9fddb7d38389e2b3473d8e

            We are still seeing sanity test 418 fail with ''df and lfs df inodes output mismatch: ..." on master branch after all these patches have landed.

            Here are a couple of recent failures:
            https://testing.whamcloud.com/test_sets/6e908786-4c5c-11e9-9720-52540065bddc
            https://testing.whamcloud.com/test_sets/c65cfdbc-4c4f-11e9-92fe-52540065bddc

            jamesanunez James Nunez (Inactive) added a comment - We are still seeing sanity test 418 fail with ''df and lfs df inodes output mismatch: ..." on master branch after all these patches have landed. Here are a couple of recent failures: https://testing.whamcloud.com/test_sets/6e908786-4c5c-11e9-9720-52540065bddc https://testing.whamcloud.com/test_sets/c65cfdbc-4c4f-11e9-92fe-52540065bddc

            Minh Diep (mdiep@whamcloud.com) uploaded a new patch: https://review.whamcloud.com/34453
            Subject: LU-11721 lod: limit statfs ffree if less than OST ffree
            Project: fs/lustre-release
            Branch: b2_12
            Current Patch Set: 1
            Commit: 96000f6e5b8dd42dc8095aa4eff19fe56d493887

            gerrit Gerrit Updater added a comment - Minh Diep (mdiep@whamcloud.com) uploaded a new patch: https://review.whamcloud.com/34453 Subject: LU-11721 lod: limit statfs ffree if less than OST ffree Project: fs/lustre-release Branch: b2_12 Current Patch Set: 1 Commit: 96000f6e5b8dd42dc8095aa4eff19fe56d493887

            Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/34321/
            Subject: LU-11721 tests: wait for statfs to update on DNE
            Project: fs/lustre-release
            Branch: b2_12
            Current Patch Set:
            Commit: b319390728b1fa6402233d5cb05a1e7d5d09825b

            gerrit Gerrit Updater added a comment - Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/34321/ Subject: LU-11721 tests: wait for statfs to update on DNE Project: fs/lustre-release Branch: b2_12 Current Patch Set: Commit: b319390728b1fa6402233d5cb05a1e7d5d09825b

            Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/34136/
            Subject: LU-11721 utils: print used inodes ratio when using "lfs df -i"
            Project: fs/lustre-release
            Branch: b2_12
            Current Patch Set:
            Commit: 52a9419f7afaa31f8db922526ca78ef08d909ed4

            gerrit Gerrit Updater added a comment - Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/34136/ Subject: LU-11721 utils: print used inodes ratio when using "lfs df -i" Project: fs/lustre-release Branch: b2_12 Current Patch Set: Commit: 52a9419f7afaa31f8db922526ca78ef08d909ed4
            pjones Peter Jones added a comment -

            All landed for 2.13 now

            pjones Peter Jones added a comment - All landed for 2.13 now

            People

              nangelinas Nikitas Angelinas
              nangelinas Nikitas Angelinas
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: