Details

    • Technical task
    • Resolution: Fixed
    • Minor
    • None
    • None
    • None
    • 2829

    Description

      When running million file runs of createmany with the '-o' (open) option we are seeing a drop in performance from it's starting rate of about 500 creates per second, to about 150 creates per second. This drop in performance seems to hit at the same time as when the ZFS ARC's 'arc_meta_used' hits and then exceeds it's 'arc_meta_limit'. We believe that since Lustre is doing it's own caching and holds a reference to all of it's objects, the ARC is unable to limit it's cache to 'arc_meta_limit'. Thus, the ARC is spending useless effort trying to uphold it's limit and drop objects (but can't because of Lustre's references) which is causing the create rate decrease.

      One method to slightly relive this, is to use the ARC prune callback feature that was recently added to the ZFS on Linux project in commit: https://github.com/zfsonlinux/zfs/commit/ab26409db753bb087842ab6f1af943f3386c764f

      This would allow the ARC to notify Lustre that it needs to release some of the objects it is holding so the ARC can free up part of it's cache and uphold it's 'arc_meta_limit'.

      Attachments

        Activity

          [LU-2477] Poor MDS create performance due to ARC cache growth

          Integrated in lustre-dev » i686,client,el6,inkernel #340
          ORI-481 osd-zfs: Register prune function with ARC (Revision c29aef2356638d93b418f23f2835980c8a944703)

          Result = SUCCESS
          Mikhail Pershin : c29aef2356638d93b418f23f2835980c8a944703
          Files :

          • lustre/osd-zfs/osd_internal.h
          • lustre/osd-zfs/osd_handler.c
          hudson Build Master (Inactive) added a comment - Integrated in lustre-dev » i686,client,el6,inkernel #340 ORI-481 osd-zfs: Register prune function with ARC (Revision c29aef2356638d93b418f23f2835980c8a944703) Result = SUCCESS Mikhail Pershin : c29aef2356638d93b418f23f2835980c8a944703 Files : lustre/osd-zfs/osd_internal.h lustre/osd-zfs/osd_handler.c

          Integrated in lustre-dev » x86_64,client,el5,inkernel #340
          ORI-481 osd-zfs: Register prune function with ARC (Revision c29aef2356638d93b418f23f2835980c8a944703)

          Result = SUCCESS
          Mikhail Pershin : c29aef2356638d93b418f23f2835980c8a944703
          Files :

          • lustre/osd-zfs/osd_internal.h
          • lustre/osd-zfs/osd_handler.c
          hudson Build Master (Inactive) added a comment - Integrated in lustre-dev » x86_64,client,el5,inkernel #340 ORI-481 osd-zfs: Register prune function with ARC (Revision c29aef2356638d93b418f23f2835980c8a944703) Result = SUCCESS Mikhail Pershin : c29aef2356638d93b418f23f2835980c8a944703 Files : lustre/osd-zfs/osd_internal.h lustre/osd-zfs/osd_handler.c

          Is there a specific `env` variable I should pass to the `lu_site_purge` function? It's unclear to me exactly how the environments are used.

          My initial thinking is I can use a call to `lu_site_purge` similar to the one found in `osd_device_free` in the osd-zfs layer.

          prakash Prakash Surya (Inactive) added a comment - Is there a specific `env` variable I should pass to the `lu_site_purge` function? It's unclear to me exactly how the environments are used. My initial thinking is I can use a call to `lu_site_purge` similar to the one found in `osd_device_free` in the osd-zfs layer.

          Yes, it would be best to keep this code in the zfs-osd. It just wasn't clear to us if it was possible to drop entries the the lu site safely from within the zfs-osd code. If we can use lu_site_purge() for this that would be perfect.

          behlendorf Brian Behlendorf added a comment - Yes, it would be best to keep this code in the zfs-osd. It just wasn't clear to us if it was possible to drop entries the the lu site safely from within the zfs-osd code. If we can use lu_site_purge() for this that would be perfect.

          hmm. Lustre does not bypass kernel infrastructure related to memory management: it registers lu_cache_shrink() and it's called by the kernel as expected.

          given this callback is specific to ARC, it should be done within zfs-osd which in turn can call lu_site_purge()
          on own site (in contrast with all registered sites).

          bzzz Alex Zhuravlev added a comment - hmm. Lustre does not bypass kernel infrastructure related to memory management: it registers lu_cache_shrink() and it's called by the kernel as expected. given this callback is specific to ARC, it should be done within zfs-osd which in turn can call lu_site_purge() on own site (in contrast with all registered sites).

          Here's a graph of our create rate performance. This is on a freshly rebooted MDS. We believe the first drop to ~300 corresponds to reaching the 'arc_meta_limit' but having enough non-lustre buffers to drop to stay at the limit. And the second drop to ~150 corresponds to all non-lustre buffers dropped and now exceeding the 'arc_meta_limit'.

          prakash Prakash Surya (Inactive) added a comment - Here's a graph of our create rate performance. This is on a freshly rebooted MDS. We believe the first drop to ~300 corresponds to reaching the 'arc_meta_limit' but having enough non-lustre buffers to drop to stay at the limit. And the second drop to ~150 corresponds to all non-lustre buffers dropped and now exceeding the 'arc_meta_limit'.

          Alex, I believe the ARC is already doing this coordination with the kernel. From my understanding, when the ARC reaches this limit, it will try and use the existing shrinkers to have the kernel/ZPL drop any referneces it can, allowing the ARC to drop it completely. The problem is that Lustre bypasses the ZPL and kernel infrastructure, talking directly with the DMU. Thus, bypassing all the existing shrinker infrastucture.

          Without the callback, Lustre pin's it's data in the ARC by always holding a reference to it and there's no current functionality to tell Lustre to drop this reference. This callback is used to inform Lustre to drop what it can.

          Also, I don't think the kernel can sort this out, because it has no idea of the ARC's meta limit. As far as I understand it, the ARC is providing similar functionality as Linux's pagecache, but without using any of the Linux infrastructure. From the kernel's perspective, the ARC is using X bytes and there is no need to reclaim since the system still has plenty of available memory. But from the ARC's perspective, it want's to limit itself to Y bytes, and when X surpasses Y, it needs to reclaim no matter the available memory on the system as a whole.

          prakash Prakash Surya (Inactive) added a comment - Alex, I believe the ARC is already doing this coordination with the kernel. From my understanding, when the ARC reaches this limit, it will try and use the existing shrinkers to have the kernel/ZPL drop any referneces it can, allowing the ARC to drop it completely. The problem is that Lustre bypasses the ZPL and kernel infrastructure, talking directly with the DMU. Thus, bypassing all the existing shrinker infrastucture. Without the callback, Lustre pin's it's data in the ARC by always holding a reference to it and there's no current functionality to tell Lustre to drop this reference. This callback is used to inform Lustre to drop what it can. Also, I don't think the kernel can sort this out, because it has no idea of the ARC's meta limit. As far as I understand it, the ARC is providing similar functionality as Linux's pagecache, but without using any of the Linux infrastructure. From the kernel's perspective, the ARC is using X bytes and there is no need to reclaim since the system still has plenty of available memory. But from the ARC's perspective, it want's to limit itself to Y bytes, and when X surpasses Y, it needs to reclaim no matter the available memory on the system as a whole.

          Changing this bug summary to reflect issues with ARC cache shrinking. Other aspects of this performance include xattr overhead (addressed in ORI-361 - SA-based xattrs).

          adilger Andreas Dilger added a comment - Changing this bug summary to reflect issues with ARC cache shrinking. Other aspects of this performance include xattr overhead (addressed in ORI-361 - SA-based xattrs).

          I tend to think the better approach would be to ask kernel for more pages and let kernel to sort this problem out (using all shrinkers, pagecache, etc) - it might be that a lot of memory is consumed by somebody else in some cases.

          bzzz Alex Zhuravlev added a comment - I tend to think the better approach would be to ask kernel for more pages and let kernel to sort this problem out (using all shrinkers, pagecache, etc) - it might be that a lot of memory is consumed by somebody else in some cases.

          People

            bzzz Alex Zhuravlev
            prakash Prakash Surya (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: