Whamcloud Community JIRA https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 [LU-569] make lu_object cache size adjustable https://jira.whamcloud.com/browse/LU-569 Lustre <p>lu_object cache is specified to consume 20% of total memory. This limits 200 clients can be mounted on one node. We should make it adjustable so that customers have a chance to configure it by their needs.</p> LU-569 make lu_object cache size adjustable Improvement Minor Closed Won't Fix Jinshan Xiong Jinshan Xiong Fri, 5 Aug 2011 15:22:08 +0000 Fri, 20 Jan 2012 16:38:00 +0000 Tue, 9 Aug 2011 22:45:20 +0000 0 2 <p>patch is at: <a href="http://review.whamcloud.com/1188" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/1188</a></p> <p>In addition to just allowing Lustre to consume more memory on the client, I think it is also/more important to determine WHY it is consuming so much memory, and try to reduce the actual memory used. Is it because of too-large hash tables, that could be started at a small size and dynamically grown only as needed? Is it because of other large/static arrays per mountpoint?</p> <p>My 1.8 client consumes about 7MB after flushing the LDLM cache (lctl get_param memused). It should be fairly straight forward to run with +malloc debug for a second/third/fourth mount and dump the debug logs, parse with lustre/tests/leakfinder.pl (may need some fixing) and determine where all of the memory is being used.</p> <p>The memory usage is because it allocates a large hash table when it is mounted. With this patch and set lu_cache_percent to be 1, I can run 1K client on one node without any problem.</p> <p>I agree it will be good to have dynamic hash table size, especially on the server size. Personally I don't think we need it on clients because it's not desirable for clients to have incredible # of objects. </p> <p>the reason we don't allow rehash lu_site especially on server side is because if we want to enabled "rehash" (by using flag cfs_hash_create(..CFS_HASH_REHASH)), then there has to be a single rwlock to protect the whole hash-table, which could be overhead for such a high contention hash-table. </p> <p>Liang, is this needed also for a hash table that can only grow? Probably yes, but just to confirm. Is the improved hash table code already landed on master?</p> <p>Unfortunately (I think) there is no way to know when the lu_cache is set up there is no way to know whether there is going to be a server or only a client on that node. I also assume that it is not possible/safe to share the lu_cache on the client between mountpoints. </p> <p>I wonder if we might have some scalable method for hash table resize that does not need a single rwlock for the whole table? One option is to implement rehash as two independent hash tables, and as long as the migration of entries from the old table to the new table is done while locking both the source and target bucket then it should be transparent to the users,band relatively low contention (only two of all the buckets are locked at one time).</p> <p>Andreas, yes it's only for hash table can grow and it's already on master for a while.<br/> I suspect that growing of hash table with one million entries and millions of elements will consume very long time (probably a few seconds on a busy smp server) and too expensive, i.e: if we want to increase the hash entries from 512K to 1M, then we have to:<br/> 1) alloc a few megabytes as hash head<br/> 2) initialize 1 million hash head<br/> 3) move millions of elements from old hash list to new hash list <br/> It will be even more expensive if we don't have the rwlock and just lock different buckets to move elements, and the worst case is that we have to lock/unlock different target buckets for each element moving. Although we do relax CPU while rehashing so other threads still can access the hash table, but I'm still a little nervous if we have such kind of heavy operations on servers.</p> <p>Another thing we need to notice is lu_site is not using high-level cfs_hash APIs like cfs_hash_find/add/del which will hide locks of cfs_hash, lu_site will directly refer to cfs_hash locks and low-level bucket APIs, so it can use those hash locks to protect it's own data, for example, counters and LRU for shrinker, some waitq etc. Which means we need to make some changes to lu_site if we want to enable rehash.</p> <p>I think there is another option to support growing of lu_site, we can have multiple cfs_hash tables for the lu_site, i.e: 64 hash tables, and hash objects to different hash tables, any of these hash tables can grow when necessary and we don't need to worry about "big rehash" with millions of elements, global lock wouldn't be an issue either because we have many of these hash tables.</p> <p>btw: shouldn't caller of lu_site_init() know about which stack (server/client) the lu_site is created for? If so can we just pass in a flag or whatever to indicate client stack to use smaller hash table?</p> <p>Rehash is way too complex for me. Yes, we can add a parameter in lu_site_init() so that client and server can have different size of hash table. However, I'm afraid that we still may need a way to be able to configure it for special needs - for example, I have to mount 1K mountpoints to test the scalability of IR.</p> <p>If the lu_site_init() picks a very small hash table size for clients, say 4096 entries, does that prevent you from mounting 1k clients on a single node? Is the lu_cache hash table the only significant memory user for each client mount? How much memory does the lu_cache hash table use on a server if it uses the default lu_htable_order() value?</p> <p>If the entries can be as small as 4096, I think that is absolutely fine.</p> <p>I don't know how much exactly memory it consumes - it is prorated by memory size, but after I changed lu_cache_percent from 20 to 1, I could mount 1K mountpoints - it used to be 200 at most.</p> <p>it's kind of off-topic, I think we can improve cfs_hash to make it support rehash-in-bucket in the future:</p> <ul> <li>user can provide (not necessary) two levels hash functions <ul> <li>the first is for bucket-hash (each bucket has one lock and N entries (hlist_head))</li> <li>the second is for entry-hash inside bucket (hash element to hlist_head in that bucket)</li> </ul> </li> <li>rehash can only happen in each bucket <ul> <li>better scalability, because we don't do rehash for the whole hash table in one batch</li> <li>no element moving between buckets, so we don't need rwlock or lock dance for bucket locking</li> </ul> </li> </ul> <p>It will help, but if you are using an evenly distributed hash function, I could say the time for each first-level bucket to be rehashed will be really close.<br/> I just don't understand the intention of rehashing feature, BTW.</p> <p>yes, they should be close, but it doesn't matter if they are handled by different threads on different CPUs, instead of "hog" one thread on one CPU for seconds.<br/> we want to rehash(or grow hash-table) just because we don't want to allocate huge amount of <em>big</em> hash tables, for example, obd_class::exp_lock_hash, it is possible that there are tens of thousands of locks in this hash although most cases it shouldn't be that many, so we should allocate a small hash table on initializing of export and grow it only if necessary, as we can have over hundreds of thousands exports on server, it will save a lot of memory. </p> <p>btw: although not fully tested, I remember the new cfs_hash can support "shrink" of hash-table which is non-blocking too, we probably should test and enable it in the future.</p> <p>I'll use this patch for IR test only.</p> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=sles11,ib_stack=inkernel/285/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,sles11,inkernel #285</a><br/> <a href="https://jira.whamcloud.com/browse/LU-569" title="make lu_object cache size adjustable" class="issue-link" data-issue-key="LU-569"><del>LU-569</del></a>: Make lu_object cache size adjustable</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=c8d7c99ec50c81a33eea43ed1c535fa4d65cef23" class="external-link" target="_blank" rel="nofollow noopener">c8d7c99ec50c81a33eea43ed1c535fa4d65cef23</a><br/> Files : </p> <ul> <li>lustre/obdclass/lu_object.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=server,distro=el6,ib_stack=inkernel/285/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,server,el6,inkernel #285</a><br/> <a href="https://jira.whamcloud.com/browse/LU-569" title="make lu_object cache size adjustable" class="issue-link" data-issue-key="LU-569"><del>LU-569</del></a>: Make lu_object cache size adjustable</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=c8d7c99ec50c81a33eea43ed1c535fa4d65cef23" class="external-link" target="_blank" rel="nofollow noopener">c8d7c99ec50c81a33eea43ed1c535fa4d65cef23</a><br/> Files : </p> <ul> <li>lustre/obdclass/lu_object.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=server,distro=el5,ib_stack=ofa/285/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,server,el5,ofa #285</a><br/> <a href="https://jira.whamcloud.com/browse/LU-569" title="make lu_object cache size adjustable" class="issue-link" data-issue-key="LU-569"><del>LU-569</del></a>: Make lu_object cache size adjustable</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=c8d7c99ec50c81a33eea43ed1c535fa4d65cef23" class="external-link" target="_blank" rel="nofollow noopener">c8d7c99ec50c81a33eea43ed1c535fa4d65cef23</a><br/> Files : </p> <ul> <li>lustre/obdclass/lu_object.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=client,distro=el6,ib_stack=inkernel/285/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,client,el6,inkernel #285</a><br/> <a href="https://jira.whamcloud.com/browse/LU-569" title="make lu_object cache size adjustable" class="issue-link" data-issue-key="LU-569"><del>LU-569</del></a>: Make lu_object cache size adjustable</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=c8d7c99ec50c81a33eea43ed1c535fa4d65cef23" class="external-link" target="_blank" rel="nofollow noopener">c8d7c99ec50c81a33eea43ed1c535fa4d65cef23</a><br/> Files : </p> <ul> <li>lustre/obdclass/lu_object.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=ubuntu1004,ib_stack=inkernel/285/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,ubuntu1004,inkernel #285</a><br/> <a href="https://jira.whamcloud.com/browse/LU-569" title="make lu_object cache size adjustable" class="issue-link" data-issue-key="LU-569"><del>LU-569</del></a>: Make lu_object cache size adjustable</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=c8d7c99ec50c81a33eea43ed1c535fa4d65cef23" class="external-link" target="_blank" rel="nofollow noopener">c8d7c99ec50c81a33eea43ed1c535fa4d65cef23</a><br/> Files : </p> <ul> <li>lustre/obdclass/lu_object.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=el6,ib_stack=inkernel/285/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,el6,inkernel #285</a><br/> <a href="https://jira.whamcloud.com/browse/LU-569" title="make lu_object cache size adjustable" class="issue-link" data-issue-key="LU-569"><del>LU-569</del></a>: Make lu_object cache size adjustable</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=c8d7c99ec50c81a33eea43ed1c535fa4d65cef23" class="external-link" target="_blank" rel="nofollow noopener">c8d7c99ec50c81a33eea43ed1c535fa4d65cef23</a><br/> Files : </p> <ul> <li>lustre/obdclass/lu_object.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=server,distro=el5,ib_stack=inkernel/285/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,server,el5,inkernel #285</a><br/> <a href="https://jira.whamcloud.com/browse/LU-569" title="make lu_object cache size adjustable" class="issue-link" data-issue-key="LU-569"><del>LU-569</del></a>: Make lu_object cache size adjustable</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=c8d7c99ec50c81a33eea43ed1c535fa4d65cef23" class="external-link" target="_blank" rel="nofollow noopener">c8d7c99ec50c81a33eea43ed1c535fa4d65cef23</a><br/> Files : </p> <ul> <li>lustre/obdclass/lu_object.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=server,distro=el5,ib_stack=inkernel/285/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,server,el5,inkernel #285</a><br/> <a href="https://jira.whamcloud.com/browse/LU-569" title="make lu_object cache size adjustable" class="issue-link" data-issue-key="LU-569"><del>LU-569</del></a>: Make lu_object cache size adjustable</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=c8d7c99ec50c81a33eea43ed1c535fa4d65cef23" class="external-link" target="_blank" rel="nofollow noopener">c8d7c99ec50c81a33eea43ed1c535fa4d65cef23</a><br/> Files : </p> <ul> <li>lustre/obdclass/lu_object.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=el5,ib_stack=ofa/285/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,el5,ofa #285</a><br/> <a href="https://jira.whamcloud.com/browse/LU-569" title="make lu_object cache size adjustable" class="issue-link" data-issue-key="LU-569"><del>LU-569</del></a>: Make lu_object cache size adjustable</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=c8d7c99ec50c81a33eea43ed1c535fa4d65cef23" class="external-link" target="_blank" rel="nofollow noopener">c8d7c99ec50c81a33eea43ed1c535fa4d65cef23</a><br/> Files : </p> <ul> <li>lustre/obdclass/lu_object.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=el5,ib_stack=inkernel/285/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,el5,inkernel #285</a><br/> <a href="https://jira.whamcloud.com/browse/LU-569" title="make lu_object cache size adjustable" class="issue-link" data-issue-key="LU-569"><del>LU-569</del></a>: Make lu_object cache size adjustable</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=c8d7c99ec50c81a33eea43ed1c535fa4d65cef23" class="external-link" target="_blank" rel="nofollow noopener">c8d7c99ec50c81a33eea43ed1c535fa4d65cef23</a><br/> Files : </p> <ul> <li>lustre/obdclass/lu_object.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=server,distro=el6,ib_stack=inkernel/285/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,server,el6,inkernel #285</a><br/> <a href="https://jira.whamcloud.com/browse/LU-569" title="make lu_object cache size adjustable" class="issue-link" data-issue-key="LU-569"><del>LU-569</del></a>: Make lu_object cache size adjustable</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=c8d7c99ec50c81a33eea43ed1c535fa4d65cef23" class="external-link" target="_blank" rel="nofollow noopener">c8d7c99ec50c81a33eea43ed1c535fa4d65cef23</a><br/> Files : </p> <ul> <li>lustre/obdclass/lu_object.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=client,distro=el5,ib_stack=ofa/285/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,client,el5,ofa #285</a><br/> <a href="https://jira.whamcloud.com/browse/LU-569" title="make lu_object cache size adjustable" class="issue-link" data-issue-key="LU-569"><del>LU-569</del></a>: Make lu_object cache size adjustable</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=c8d7c99ec50c81a33eea43ed1c535fa4d65cef23" class="external-link" target="_blank" rel="nofollow noopener">c8d7c99ec50c81a33eea43ed1c535fa4d65cef23</a><br/> Files : </p> <ul> <li>lustre/obdclass/lu_object.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=client,distro=el5,ib_stack=inkernel/285/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,client,el5,inkernel #285</a><br/> <a href="https://jira.whamcloud.com/browse/LU-569" title="make lu_object cache size adjustable" class="issue-link" data-issue-key="LU-569"><del>LU-569</del></a>: Make lu_object cache size adjustable</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=c8d7c99ec50c81a33eea43ed1c535fa4d65cef23" class="external-link" target="_blank" rel="nofollow noopener">c8d7c99ec50c81a33eea43ed1c535fa4d65cef23</a><br/> Files : </p> <ul> <li>lustre/obdclass/lu_object.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=server,distro=el5,ib_stack=ofa/285/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,server,el5,ofa #285</a><br/> <a href="https://jira.whamcloud.com/browse/LU-569" title="make lu_object cache size adjustable" class="issue-link" data-issue-key="LU-569"><del>LU-569</del></a>: Make lu_object cache size adjustable</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=c8d7c99ec50c81a33eea43ed1c535fa4d65cef23" class="external-link" target="_blank" rel="nofollow noopener">c8d7c99ec50c81a33eea43ed1c535fa4d65cef23</a><br/> Files : </p> <ul> <li>lustre/obdclass/lu_object.c</li> </ul> Related LU-19 Development Rank 1|hzv8fj: Rank (Obsolete) 4900