https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-55 Lustre <p>Because <a href="https://jira.whamcloud.com/browse/LU-56" title="Finish SMP scalability work" class="issue-link" data-issue-key="LU-56"><del>IT-2</del></a> is internal task which is only visible to Whamcloud engineers, so I create this public ticket to tracking efforts and record discussions.</p> <p>all patches:</p> <p><a href="http://git.whamcloud.com/gitweb/?p=fs/lustre-dev.git;a=shortlog;h=refs/heads/liang/b_smp" class="external-link" target="_blank" rel="nofollow noopener">http://git.whamcloud.com/gitweb/?p=fs/lustre-dev.git;a=shortlog;h=refs/heads/liang/b_smp</a></p> LU-55

Finish SMP scalability work (public tracking ticket)

Improvement Major Closed Fixed Liang Zhen Liang Zhen Tue, 25 Jan 2011 00:28:25 +0000 Tue, 25 Jan 2011 09:59:27 +0000 Tue, 25 Jan 2011 09:59:27 +0000 Lustre 2.0.0 Lustre 2.0.0 0 4 <p>On large scale SMP system, SMP scalability of diskfs will be bottleneck of these operations under shared directory, there is no easy solution of this, I'm trying to workout a pdirops patch to improve SMP performance of shared directory, at the same time I'm thinking about adding a metadata changing operations driver to improvement performance:</p> <p>following descriptions are just copied form <a href="https://jira.whamcloud.com/browse/LU-56" title="Finish SMP scalability work" class="issue-link" data-issue-key="LU-56"><del>IT-2</del></a></p> <ul> <li>this driver includes two parts: <ul> <li>MDD dir fid cache</li> <li>Metadata transaction schedulers</li> </ul> </li> <li>each CPU has a local fid cache (it's a 8 buckets hash-table, each bucket is a 8 items LRU, so we can cache 64 fids by default, of course we can increase number of buckets)</li> <li>transaction schedulers are percpu threads pool in MDD layer</li> <li>each time we create/remove file from a directory, we push the fid into the local CPU fid cache(LRU)</li> <li>if the fid is already in the cache, then just increase a refcount(# of hits) on the fid and move the fid to top of LRU</li> <li>when we push a new fid into local cache, the fid at bottom of LRU will be popped out</li> <li>if refcount(# of hits) of fid in local CPU cache &gt;= N (default is 32), we try to lookup the fid in global fid cache, which is a hash-table as well (do nothing if it already has reference to fid in global cache). <ul> <li>if the fid is not in the global cache, we just add it to global cache and set refcount to 1 (1 CPU is using it)</li> <li>if the fid is already in global cache, we increase refcount of fid in global cache</li> </ul> </li> <li>if refcount of fid in global cache &lt; M, we just return -1 and continue to run transaction in context of current thread (mdt service thread)</li> <li>if refcount of fid in global cache is &gt;= M (M is number of CPUs, which can be 2, 4... as many as we can get good performance of shared directory with ldiskfs/ext4, it's tunable), we specify M CPUs for the fid and return one CPU id within those M CPUs, if caller get a CPU id different with current CPU id, it will launch a MDD level "transaction request", and wakeup one MDD transaction thread to handle the "transaction request"</li> </ul> <p>By this way, we only have very low overhead for common code (almost no contention), almost nothing changed for uniq directory operations. At the same time, we decreased contention on pdirops because changes on shared directory are localized on a few CPUs.</p> <p>Also, we probably can sort unlink requests inside MDD schedulers to decrease disk seek somehow, but I'm not sure how much it can help </p> <p>I moved <a href="https://jira.whamcloud.com/browse/LU-56" title="Finish SMP scalability work" class="issue-link" data-issue-key="LU-56"><del>IT-2</del></a> to the Lustre project, so now it is visible to all. </p> Development Rank 1|hzw3lr: Rank (Obsolete) 10671