[LU-2477] Poor MDS create performance due to ARC cache growth

• Type: Technical task
• Resolution: Fixed
• Priority: Minor
• Assignee: Alex Zhuravlev
• Reporter: Prakash Surya (Inactive)

Description

When running million file runs of createmany with the '-o' (open) option we are seeing a drop in performance from it's starting rate of about 500 creates per second, to about 150 creates per second. This drop in performance seems to hit at the same time as when the ZFS ARC's 'arc_meta_used' hits and then exceeds it's 'arc_meta_limit'. We believe that since Lustre is doing it's own caching and holds a reference to all of it's objects, the ARC is unable to limit it's cache to 'arc_meta_limit'. Thus, the ARC is spending useless effort trying to uphold it's limit and drop objects (but can't because of Lustre's references) which is causing the create rate decrease.

One method to slightly relive this, is to use the ARC prune callback feature that was recently added to the ZFS on Linux project in commit: https://github.com/zfsonlinux/zfs/commit/ab26409db753bb087842ab6f1af943f3386c764f

This would allow the ARC to notify Lustre that it needs to release some of the objects it is holding so the ARC can free up part of it's cache and uphold it's 'arc_meta_limit'.

Comments

[Alex Zhuravlev]

any update here? iirc, the prune function was landed and we fixed memory leak in SA-related code.

[Build Master (Inactive)]

Integrated in lustre-dev » x86_64,client,el6,inkernel #340
ORI-481 osd-zfs: Register prune function with ARC (Revision c29aef2356638d93b418f23f2835980c8a944703)

Result = SUCCESS
Mikhail Pershin : c29aef2356638d93b418f23f2835980c8a944703
Files :

• lustre/osd-zfs/osd_internal.h
• lustre/osd-zfs/osd_handler.c

[Build Master (Inactive)]

Integrated in lustre-dev » x86_64,server,el5,inkernel #340
ORI-481 osd-zfs: Register prune function with ARC (Revision c29aef2356638d93b418f23f2835980c8a944703)

Result = SUCCESS
Mikhail Pershin : c29aef2356638d93b418f23f2835980c8a944703
Files :

• lustre/osd-zfs/osd_handler.c
• lustre/osd-zfs/osd_internal.h

[Build Master (Inactive)]

Integrated in lustre-dev » i686,client,el5,inkernel #340
ORI-481 osd-zfs: Register prune function with ARC (Revision c29aef2356638d93b418f23f2835980c8a944703)

Result = SUCCESS
Mikhail Pershin : c29aef2356638d93b418f23f2835980c8a944703
Files :

• lustre/osd-zfs/osd_handler.c
• lustre/osd-zfs/osd_internal.h

[Build Master (Inactive)]

Integrated in lustre-dev » x86_64,server,el6,inkernel #340
ORI-481 osd-zfs: Register prune function with ARC (Revision c29aef2356638d93b418f23f2835980c8a944703)

Result = SUCCESS
Mikhail Pershin : c29aef2356638d93b418f23f2835980c8a944703
Files :

• lustre/osd-zfs/osd_handler.c
• lustre/osd-zfs/osd_internal.h

[Build Master (Inactive)]

Integrated in lustre-dev » i686,server,el5,inkernel #340
ORI-481 osd-zfs: Register prune function with ARC (Revision c29aef2356638d93b418f23f2835980c8a944703)

Result = SUCCESS
Mikhail Pershin : c29aef2356638d93b418f23f2835980c8a944703
Files :

• lustre/osd-zfs/osd_handler.c
• lustre/osd-zfs/osd_internal.h

[Build Master (Inactive)]

Integrated in lustre-dev » i686,client,el6,inkernel #340
ORI-481 osd-zfs: Register prune function with ARC (Revision c29aef2356638d93b418f23f2835980c8a944703)

Result = SUCCESS
Mikhail Pershin : c29aef2356638d93b418f23f2835980c8a944703
Files :

• lustre/osd-zfs/osd_internal.h
• lustre/osd-zfs/osd_handler.c

[Build Master (Inactive)]

Integrated in lustre-dev » x86_64,client,el5,inkernel #340
ORI-481 osd-zfs: Register prune function with ARC (Revision c29aef2356638d93b418f23f2835980c8a944703)

Result = SUCCESS
Mikhail Pershin : c29aef2356638d93b418f23f2835980c8a944703
Files :

• lustre/osd-zfs/osd_internal.h
• lustre/osd-zfs/osd_handler.c

[Prakash Surya (Inactive)]

http://review.whamcloud.com/#change,2032

[Prakash Surya (Inactive)]

Is there a specific `env` variable I should pass to the `lu_site_purge` function? It's unclear to me exactly how the environments are used.

My initial thinking is I can use a call to `lu_site_purge` similar to the one found in `osd_device_free` in the osd-zfs layer.

[Brian Behlendorf]

Yes, it would be best to keep this code in the zfs-osd. It just wasn't clear to us if it was possible to drop entries the the lu site safely from within the zfs-osd code. If we can use lu_site_purge() for this that would be perfect.