https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-6226 Lustre <p>after running sanity.sh i see very bad distribution for a fid's between OI files.</p> <p><span class="error">&#91;root@rhel6-64 tmp&#93;</span># debugfs -R 'ls -l' lustre-mdt1 | grep oi.16 | grep -v 8192<br/> debugfs 1.42.7.wc1 (12-Apr-2013)<br/> 22 100644 (1) 0 0 876544 9-Feb-2015 21:11 oi.16.0<br/> 39 100644 (1) 0 0 491520 9-Feb-2015 21:11 oi.16.17<br/> 70 100644 (1) 0 0 28672 9-Feb-2015 21:11 oi.16.48<br/> 71 100644 (1) 0 0 2424832 9-Feb-2015 21:11 oi.16.49<br/> 72 100644 (1) 0 0 483328 9-Feb-2015 21:11 oi.16.50<br/> 73 100644 (1) 0 0 389120 9-Feb-2015 21:11 oi.16.51</p> <p>all other files 8k byte size - so likely unused. </p> <p>I don't see any reason to have 64 OI files if 5 only used.</p> <p>Please fix a hash function to have better distribution.</p> LU-6226

lu-822 have a bad hash function for a switching files

Improvement Critical Open Unresolved WC Triage Alexey Lyashkov Mon, 9 Feb 2015 18:23:58 +0000 Fri, 28 Aug 2015 18:13:39 +0000 0 5 <p>probably you should try to use many more clients?</p> <p>did you really think - it function have a dependence to number a clients?</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">+<span class="code-keyword">static</span> inline struct osd_oi * +osd_fid2oi(struct osd_device *osd, <span class="code-keyword">const</span> struct lu_fid *fid) +{ + <span class="code-keyword">if</span> (!fid_is_norm(fid)) + <span class="code-keyword">return</span> NULL; + + LASSERT(osd-&gt;od_oi_table != NULL &amp;&amp; osd-&gt;od_oi_count &gt;= 1); + /* It can work even od_oi_count equals to 1 although it's unexpected, + * the only reason we set it to 1 is <span class="code-keyword">for</span> performance measurement */ + <span class="code-keyword">return</span> &amp;osd-&gt;od_oi_table[fid-&gt;f_seq &amp; (osd-&gt;od_oi_count - 1)]; +} </pre> </div></div> <p>f_seq &amp; osd-&gt;od_oi_count - will don't depend to number a clients as uses only lower bits.</p> <p>why don't use fid_hash() instead ?</p> <p>f_seq depends on the number of the clients indirectly.</p> <p>and we use only lower bits of it ? <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/> seq may be in range 0 .. 64 not more <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/> </p> <p>you don't answer - why don't use fid_hash with better distribution ? if we have one line change and replace fid-&gt;f_seq with fid_hash(fid) distribution will changed dramatically and all files will be used.</p> <p>seq isn't limited at all. seq is per client, then every client uses upto 128K oids and gets a new sequences. imagine 10K clients mount and start to generate roughly similar workload, they all will be sending roughly same f_oid..</p> <p>Alex, </p> <p>please look carefully to function - seq &amp; (fid_files_num -1) so only lower bits used - so seq limited to 1-64.</p> <p>probably Liang can comment better. I'd suggest to use more clients to see how the function behaves really. a single client is a very special case. and even in this case few OI files were used which is good.</p> <p>yes, my intention is hashing different clients to different OI files, so we don't have all clients contend on all OIs all the time (drawback of fid_hash). Also, because we have 64 OI files, so seq &amp; (osd-&gt;od_oi_count - 1) should be between 0-63, is there any issue at here? </p> <p>Liang,</p> <p>distribution too bad in case few clients, as lower bits is mostly same for these clients (seq always allocated 128 fid's range), so only few files used after test. from disk perspective any parallel access add additional seeks so not have a differences is different clients work on same file or not.</p> <p>I don't think performance bottleneck is going to be OI operations when there are very few clients, also, why do we bother to use all OI files parallelly for handful clients? There was a single OI for whole filesystem in 2.0/2.1, multiple OIs were added to improve aggregated performance for many clients, not for single client.</p> Related LU-822 Development Rank 1|hzx5zj: Rank (Obsolete) 17426 Severity