https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-8130 Lustre <p>The linux kernel has its own resizable hashtable which can be used in place of libcfs hash code. The linux rhashtable was developed for the networking layer and that layer has very high requirements for performance. Some of the benefits are low latency as well as lockless lookups. Migrating Lustre to rhashtable should mean HUGE PERFORMANCE gains!!!!!</p> LU-8130

Migrate from libcfs hash to rhashtable

Improvement Minor Open Unresolved James A Simmons James A Simmons Thu, 12 May 2016 00:07:59 +0000 Tue, 5 Dec 2023 20:02:54 +0000 Lustre 2.9.0 Upstream 0 12 <p>I've copied Liang so he is aware of this activity in the upstream client</p> <p>James Simmons (uja.ornl@yahoo.com) uploaded a new patch: <a href="https://review.whamcloud.com/27967" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/27967</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8130" title="Migrate from libcfs hash to rhashtable" class="issue-link" data-issue-key="LU-8130">LU-8130</a> libcfs: improve hash handling<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: ed9f0eef1b663c7deea0c55397703d76a93c714a</p> <p>James Simmons (uja.ornl@yahoo.com) uploaded a new patch: <a href="https://review.whamcloud.com/32036" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/32036</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8130" title="Migrate from libcfs hash to rhashtable" class="issue-link" data-issue-key="LU-8130">LU-8130</a> ptlrpc: convert conn_hash to rhashtable<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 815f5ee910a43ee7d9742dd7287b62e5acb16f1a</p> <p>James Simmons (uja.ornl@yahoo.com) uploaded a new patch: <a href="https://review.whamcloud.com/32038" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/32038</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8130" title="Migrate from libcfs hash to rhashtable" class="issue-link" data-issue-key="LU-8130">LU-8130</a> osc: convert osc_quota hash to rhashtable<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 178b746066dfa53a2b58e9ccfa5ce9340db9f900</p> <p>James Simmons (uja.ornl@yahoo.com) uploaded a new patch: <a href="https://review.whamcloud.com/32102" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/32102</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8130" title="Migrate from libcfs hash to rhashtable" class="issue-link" data-issue-key="LU-8130">LU-8130</a> libcfs: add rhashtable support<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: e02b4501bc5648ae51efefda7299336b58c7562d</p> <p>James Simmons (uja.ornl@yahoo.com) uploaded a new patch: <a href="https://review.whamcloud.com/32111" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/32111</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8130" title="Migrate from libcfs hash to rhashtable" class="issue-link" data-issue-key="LU-8130">LU-8130</a> lov: convert lov_pool to use rhashtable<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 96d01db1689afa5aa2c7eb11271b5c8c85a62d8b</p> <p>James Simmons (uja.ornl@yahoo.com) uploaded a new patch: <a href="https://review.whamcloud.com/32408" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/32408</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8130" title="Migrate from libcfs hash to rhashtable" class="issue-link" data-issue-key="LU-8130">LU-8130</a> ldlm: store name directly in namespace.<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: c5be2189469cfca37100892e8435444bc275c369</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/32408/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/32408/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8130" title="Migrate from libcfs hash to rhashtable" class="issue-link" data-issue-key="LU-8130">LU-8130</a> ldlm: store name directly in namespace.<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 520c2959244ee366444dbc0b0d82bc05f4f9f5d6</p> <p>James Simmons (uja.ornl@yahoo.com) uploaded a new patch: <a href="https://review.whamcloud.com/32662" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/32662</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8130" title="Migrate from libcfs hash to rhashtable" class="issue-link" data-issue-key="LU-8130">LU-8130</a> lov: convert lov_pool to use rhashtable<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: cf30ce9a8293d80ac1c5e3cab6dbd6bd78c3f4cf</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/32102/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/32102/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8130" title="Migrate from libcfs hash to rhashtable" class="issue-link" data-issue-key="LU-8130">LU-8130</a> libcfs: prepare rhashtable support<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: bef1069506c38e6d8ebdb4393cccb5a33978a61e</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/32036/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/32036/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8130" title="Migrate from libcfs hash to rhashtable" class="issue-link" data-issue-key="LU-8130">LU-8130</a> ptlrpc: convert conn_hash to rhashtable<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 7b3f9e5d6c509fabcec3cbd71e541a84987db2ff</p> <p>James Simmons (uja.ornl@yahoo.com) uploaded a new patch: <a href="https://review.whamcloud.com/33485" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33485</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8130" title="Migrate from libcfs hash to rhashtable" class="issue-link" data-issue-key="LU-8130">LU-8130</a> nrs: convert NRS crr to rhashtable<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 9f98d556c0a75e681517046d1136b89d870f851b</p> <p>James Simmons (uja.ornl@yahoo.com) uploaded a new patch: <a href="https://review.whamcloud.com/33518" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33518</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8130" title="Migrate from libcfs hash to rhashtable" class="issue-link" data-issue-key="LU-8130">LU-8130</a> obd: convert obd_nid_hash to rhashtable<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 7520485fe431f844c1269907b2f1ae8fa9ddc37a</p> <p>Andreas Dilger (adilger@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/33595" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33595</a><br/> Subject: Revert "<a href="https://jira.whamcloud.com/browse/LU-8130" title="Migrate from libcfs hash to rhashtable" class="issue-link" data-issue-key="LU-8130">LU-8130</a> ptlrpc: convert conn_hash to rhashtable"<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 2479fdefff231b9144f07e5b327f8918859a8a47</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/33595/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33595/</a><br/> Subject: Revert "<a href="https://jira.whamcloud.com/browse/LU-8130" title="Migrate from libcfs hash to rhashtable" class="issue-link" data-issue-key="LU-8130">LU-8130</a> ptlrpc: convert conn_hash to rhashtable"<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 8dec4efe47fa8772ded8d6816638703dbbe04379</p> <p>James Simmons (uja.ornl@yahoo.com) uploaded a new patch: <a href="https://review.whamcloud.com/33789" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33789</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8130" title="Migrate from libcfs hash to rhashtable" class="issue-link" data-issue-key="LU-8130">LU-8130</a> libcfs: support latest rhashtable and hash API<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: c831343618d934c48d7f755243a9f03167f2a117</p> <p>Andreas Dilger (adilger@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/33873" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33873</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8130" title="Migrate from libcfs hash to rhashtable" class="issue-link" data-issue-key="LU-8130">LU-8130</a> libcfs: don't include rhashtable if unavailable<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 63e1e630f5f8ce6fa3119795ef109d01ee26bd0d</p> <p>Andreas Dilger (adilger@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/34020" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34020</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8130" title="Migrate from libcfs hash to rhashtable" class="issue-link" data-issue-key="LU-8130">LU-8130</a> libcfs: don't include rhashtable if unavailable<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: e616db3c0b6e78ab1a4fbdee8d8791bec574e7e0</p> <p>James Simmons (uja.ornl@yahoo.com) uploaded a new patch: <a href="https://review.whamcloud.com/34036" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34036</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8130" title="Migrate from libcfs hash to rhashtable" class="issue-link" data-issue-key="LU-8130">LU-8130</a> libcfs: support latest rhashtable API<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 430f9fe1c2de9aeec992746d79bc40d46b8ab577</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/33789/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33789/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8130" title="Migrate from libcfs hash to rhashtable" class="issue-link" data-issue-key="LU-8130">LU-8130</a> libcfs: port working hash from upstream<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 1658ae30a0e97e7f4018d8cba67e459078470d1a</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/34036/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34036/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8130" title="Migrate from libcfs hash to rhashtable" class="issue-link" data-issue-key="LU-8130">LU-8130</a> libcfs: support latest rhashtable API<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 8de7221201c0707245c9ee2ef7cdd1d54207b3ee</p> <p>James Simmons (uja.ornl@yahoo.com) uploaded a new patch: <a href="https://review.whamcloud.com/34429" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34429</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8130" title="Migrate from libcfs hash to rhashtable" class="issue-link" data-issue-key="LU-8130">LU-8130</a> obd: convert obd uuid hash to rhashtable<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 9977b8882a943e64f62200be32dd202491b9a038</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/34020/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34020/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8130" title="Migrate from libcfs hash to rhashtable" class="issue-link" data-issue-key="LU-8130">LU-8130</a> libcfs: don't include rhashtable if unavailable<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: <br/> Commit: 29d627f860bc1963f2103ea441577dbd18d71344</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/33485/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33485/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8130" title="Migrate from libcfs hash to rhashtable" class="issue-link" data-issue-key="LU-8130">LU-8130</a> nrs: convert NRS CRR to rhashtable<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 9df933d9b4a63e06d93d2d07cf5e670fcc10c8fc</p> <p>Minh Diep (mdiep@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/35143" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35143</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8130" title="Migrate from libcfs hash to rhashtable" class="issue-link" data-issue-key="LU-8130">LU-8130</a> libcfs: support latest rhashtable API<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: b45863f7d65a8d0a65742d20cf63928ec7cab74b</p> <p>Minh Diep (mdiep@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/35179" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35179</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8130" title="Migrate from libcfs hash to rhashtable" class="issue-link" data-issue-key="LU-8130">LU-8130</a> libcfs: port working hash from upstream<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: 7b0c6d2f3ddd40f51d663b4e381b64a1cf55dd22</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/35143/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35143/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8130" title="Migrate from libcfs hash to rhashtable" class="issue-link" data-issue-key="LU-8130">LU-8130</a> libcfs: support latest rhashtable API<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: <br/> Commit: ac8d93c9f6f96d4b343a6888f90bc588ea1ab1b0</p> <p>Andreas Dilger (adilger@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/35565" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35565</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8130" title="Migrate from libcfs hash to rhashtable" class="issue-link" data-issue-key="LU-8130">LU-8130</a> libcfs: don't include rhashtable if unavailable<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: 5ed35622502dbfece2b1e956b30ccd9819537ca2</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/35565/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35565/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8130" title="Migrate from libcfs hash to rhashtable" class="issue-link" data-issue-key="LU-8130">LU-8130</a> libcfs: don't include rhashtable if unavailable<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: <br/> Commit: f28d7ed454d6abcd776aed1c84013fbf3055651e</p> <p>James Simmons (jsimmons@infradead.org) uploaded a new patch: <a href="https://review.whamcloud.com/36216" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36216</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8130" title="Migrate from libcfs hash to rhashtable" class="issue-link" data-issue-key="LU-8130">LU-8130</a> lu_object: factor out extra per-bucket data<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 130e6db7b777497a5b4304e9c470c51cad8cde3a</p> <p>James Simmons (jsimmons@infradead.org) uploaded a new patch: <a href="https://review.whamcloud.com/36218" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36218</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8130" title="Migrate from libcfs hash to rhashtable" class="issue-link" data-issue-key="LU-8130">LU-8130</a> ldlm: separate buckets from ldlm hash table<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: d19d12114c0acc0682d2b6b4a03b60bfbdecdcf2</p> <p>James Simmons (jsimmons@infradead.org) uploaded a new patch: <a href="https://review.whamcloud.com/36219" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36219</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8130" title="Migrate from libcfs hash to rhashtable" class="issue-link" data-issue-key="LU-8130">LU-8130</a> ldlm: add a counter to the per-namespace data<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 31ff85c526e70d0b8aa1a5673c08a795eee150df</p> <p>James Simmons (jsimmons@infradead.org) uploaded a new patch: <a href="https://review.whamcloud.com/36220" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36220</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8130" title="Migrate from libcfs hash to rhashtable" class="issue-link" data-issue-key="LU-8130">LU-8130</a> ldlm: simplify ldlm_ns_hash_defs[]<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: dd4d5c5d1be06783898b67025076982fdcf4c7fa</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/35179/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35179/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8130" title="Migrate from libcfs hash to rhashtable" class="issue-link" data-issue-key="LU-8130">LU-8130</a> libcfs: port working hash from upstream<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: <br/> Commit: 15db0a3070ec42228d0ac666b4ff089c645f9a46</p> <p>James Simmons (jsimmons@infradead.org) uploaded a new patch: <a href="https://review.whamcloud.com/36432" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36432</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8130" title="Migrate from libcfs hash to rhashtable" class="issue-link" data-issue-key="LU-8130">LU-8130</a> obd: remove used HASH_CL_ENV_<span class="error">&#91;BKT&#93;</span>_BITS<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: e8e12d73bad4ba9809ef6fd9cf676ac2fdc58fc9</p> <p>James Simmons (jsimmons@infradead.org) uploaded a new patch: <a href="https://review.whamcloud.com/36707" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36707</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8130" title="Migrate from libcfs hash to rhashtable" class="issue-link" data-issue-key="LU-8130">LU-8130</a> lu_object: convert lu_object cache to rhashtable<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 7ec36418fc77b8803c01e1f6ea36a4902b64d4ec</p> <p>I just gather some performance numbers  for the lu_object cache with a single MDS server. Without this work I get:</p> <p>Operation Max Min Mean Std Dev<br/> --------- &#8212; --- ---- -------<br/> Directory creation: 3206199.336 5254.046 5802.473 375.173<br/> Directory stat : 33705.331 19990.084 24055.134 5191.633<br/> Directory removal : 6574.092 4983.478 6028.141 561.790<br/> File creation : 14538.167 14125.797 14398.448 143.081<br/> File stat : 46441.048 40937.383 43133.248 2223.355<br/> File read : 22429.094 21230.384 21899.047 467.894<br/> File removal : 11246.008 10773.246 10999.512 153.015<br/> Tree creation : 140.564 53.583 93.303 37.284<br/> Tree removal : 60.962 42.381 48.522 6.706</p> <p>With the lu_object rhashtable patches I get:</p> <p>Operation Max Min Mean Std Dev<br/> --------- &#8212; --- ---- -------<br/> Directory creation: 7561.995 6424.013 6811.299 397.400<br/> Directory stat : 44986.490 25812.764 33376.039 7310.581<br/> Directory removal : 7702.135 5550.633 6278.318 788.811<br/> File creation : 15871.038 13575.319 15249.266 850.982<br/> File stat : 113588.704 50862.965 70728.833 22659.169<br/> File read : 31272.666 28991.341 30024.974 828.311<br/> File removal : 12205.639 11399.966 11866.223 262.320<br/> Tree creation : 90.461 68.164 73.028 8.736<br/> Tree removal : 61.626 40.385 51.510 7.180</p> <p>This is with a single MDS but I expect similar scaling with a DNE setup. </p> <p>It will be interesting to see the impact of moving the ldlm locks to rhashtables.</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/36432/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36432/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8130" title="Migrate from libcfs hash to rhashtable" class="issue-link" data-issue-key="LU-8130">LU-8130</a> obd: remove used HASH_CL_ENV_<span class="error">&#91;BKT&#93;</span>_BITS<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: d2dc1749dde1a4a471a111de633f4c53a4b22a83</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/36218/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36218/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8130" title="Migrate from libcfs hash to rhashtable" class="issue-link" data-issue-key="LU-8130">LU-8130</a> ldlm: separate buckets from ldlm hash table<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: d234e2cf5f557eadd4b65758ca76f783fb6ed9a1</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/34429/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34429/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8130" title="Migrate from libcfs hash to rhashtable" class="issue-link" data-issue-key="LU-8130">LU-8130</a> obd: convert obd uuid hash to rhashtable<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: e40b008e8877c3345347500d10af07abd8651d56</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/36219/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36219/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8130" title="Migrate from libcfs hash to rhashtable" class="issue-link" data-issue-key="LU-8130">LU-8130</a> ldlm: add a counter to the per-namespace data<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: f9314d6e9259e6c7683ed614931ab2506356ba88</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/36220/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36220/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8130" title="Migrate from libcfs hash to rhashtable" class="issue-link" data-issue-key="LU-8130">LU-8130</a> ldlm: simplify ldlm_ns_hash_defs[]<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 416142145c9dc66f9692907c6dcea96c82c35bdd</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/36216/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36216/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8130" title="Migrate from libcfs hash to rhashtable" class="issue-link" data-issue-key="LU-8130">LU-8130</a> lu_object: factor out extra per-bucket data<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: e6f7f8a7b349d69d49d2cb883798fd5c679c48c3</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/32662/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/32662/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8130" title="Migrate from libcfs hash to rhashtable" class="issue-link" data-issue-key="LU-8130">LU-8130</a> lov: convert lo<span class="error">&#91;v|d&#93;</span>_pool to use rhashtable<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 2124e07ddde7789ebf0e8ec5600f2aff31788123</p> <p>Neil Brown (neilb@suse.de) uploaded a new patch: <a href="https://review.whamcloud.com/37965" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37965</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8130" title="Migrate from libcfs hash to rhashtable" class="issue-link" data-issue-key="LU-8130">LU-8130</a> lu_object: use RCU to free lu_object_header<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: d33fc68cf25ac8be14bc04ce92b733b5f65ec9d4</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/37965/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37965/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8130" title="Migrate from libcfs hash to rhashtable" class="issue-link" data-issue-key="LU-8130">LU-8130</a> lu_object: use RCU to free lu_object_header<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: b25e8a4bc86ee245be19dc05c085a2b0f4fe4a43</p> <p>patch <a href="https://review.whamcloud.com/#/c/36707" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/36707</a> (patchset 27) has a regression and mdtest failed on DoM configuration.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># lfs setstripe -L mdt -E 1M /ai400/dom/ # salloc -p 40n --nodes=40 --ntasks-per-node=16 mpirun --allow-run-as-root /work/tools/bin/mdtest -t -F -P -w 3901 -e 3901 -d /ai400/dom/mdt_hard -n 10000 -a POSIX -N 1 -i 3 -p 10 -- started at 06/03/2020 13:24:08 -- mdtest-3.3.0+dev was launched with 640 total task(s) on 40 node(s) Command line used: /work/tools/bin/mdtest '-t' '-F' '-P' '-w' '3901' '-e' '3901' '-d' '/ai400/dom/mdt_hard' '-n' '10000' '-a' 'POSIX' '-N' '1' '-i' '3' '-p' '10' Path: /ai400/dom FS: 47.4 TiB Used FS: 0.1% Inodes: 371.8 Mi Used Inodes: 0.0% Nodemap: 1111111111111111000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 V-0: Rank 0 Line 2137 Shifting ranks by 16 for each phase. 640 tasks, 6400000 files [ec01:08604] 639 more processes have sent help message help-mpi-btl-openib.txt / error in device init [ec01:08604] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages ior ERROR: open64("/ai400/dom/mdt_hard/test-dir.1-0/mdtest_tree.0/file.mdtest.24.9377", 66, 0664) failed, errno 16, Device or resource busy (aiori-POSIX.c:413) -------------------------------------------------------------------------- MPI_ABORT was invoked on rank 24 in communicator MPI_COMM_WORLD with errorcode -1. NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. You may or may not see output from other processes, depending on exactly when Open MPI kills them. -------------------------------------------------------------------------- -------------------------------------------------------------------------- An MPI communication peer process has unexpectedly disconnected. This usually indicates a failure in the peer process (e.g., a crash or otherwise exiting without calling MPI_FINALIZE first). Although this local MPI process will likely now behave unpredictably (it may even hang or crash), the root cause of this problem is the failure of the peer -- that is what you need to investigate. For example, there may be a core file that you can examine. More generally: such peer hangups are frequently caused by application bugs or other external events. Local host: ec40 Local PID: 4108 Peer host: ec02 -------------------------------------------------------------------------- [ec01:08604] 47 more processes have sent help message help-mpi-btl-tcp.txt / peer hung up salloc: Relinquishing job allocation 4561 </pre> </div></div> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/33518/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33518/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8130" title="Migrate from libcfs hash to rhashtable" class="issue-link" data-issue-key="LU-8130">LU-8130</a> obd: convert obd_nid_hash to rhashtable<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 580ef453d1d6f901c3227a037137a7db9e600009</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/33616/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33616/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8130" title="Migrate from libcfs hash to rhashtable" class="issue-link" data-issue-key="LU-8130">LU-8130</a> ptlrpc: convert conn_hash to rhashtable<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 37b29a8f709aa4bd2e22cbb3549257b28cd85610</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/36707/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36707/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8130" title="Migrate from libcfs hash to rhashtable" class="issue-link" data-issue-key="LU-8130">LU-8130</a> lu_object: convert lu_object cache to rhashtable<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: aff14dbc522e140945f9207d4e9bbd27d0326819</p> <p>Hi James,</p> <blockquote> <p>James Simmons (uja.ornl@yahoo.com) uploaded a new patch: <a href="https://review.whamcloud.com/34036" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34036</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8130" title="Migrate from libcfs hash to rhashtable" class="issue-link" data-issue-key="LU-8130">LU-8130</a> libcfs: support latest rhashtable API</p></blockquote> <p>In old kernel (like kernel-3.10.0-327.36.1.el7), the codes in linux/rhashtable.h (<span class="nobr"><a href="https://jira.whamcloud.com/secure/attachment/35299/35299_rhashtable.h" title="rhashtable.h attached to LU-8130">rhashtable.h^{<img class="rendericon" src="https://jira.whamcloud.com/images/icons/link_attachment_7.gif" height="7" width="7" align="absmiddle" alt="" border="0"/>}</a></span>) are quite different from those in the new kernel. E.g., "struct rhashtable_compare_arg" is not defined. The differences are causing build failures with old kernel as follows:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> CC [M] /root/lustre-release/libcfs/libcfs/linux/linux-hash.o In file included from /root/lustre-release/libcfs/libcfs/linux/linux-hash.c:33:0: /root/lustre-release/libcfs/include/libcfs/linux/linux-hash.h:71:13: error: ‘struct rhashtable_iter’ declared inside parameter list [-Werror] struct rhashtable_iter *iter) ^ /root/lustre-release/libcfs/include/libcfs/linux/linux-hash.h:71:13: error: its scope is only this definition or declaration, which is probably not what you want [-Werror] /root/lustre-release/libcfs/include/libcfs/linux/linux-hash.h: In function ‘rhashtable_walk_enter’: /root/lustre-release/libcfs/include/libcfs/linux/linux-hash.h:76:2: error: implicit declaration of function ‘rhashtable_walk_init’ [-Werror=implicit-function-declaration] return rhashtable_walk_init(ht, iter); ^ /root/lustre-release/libcfs/include/libcfs/linux/linux-hash.h: In function ‘rhltable_init’: /root/lustre-release/libcfs/include/libcfs/linux/linux-hash.h:98:2: error: passing argument 2 of ‘rhashtable_init’ discards ‘const’ qualifier from pointer target type [-Werror] return rhashtable_init(&amp;hlt-&gt;ht, params); &lt;~snip~&gt; </pre> </div></div> <p>Here is a log that contains all of the build failures: <span class="nobr"><a href="https://jira.whamcloud.com/secure/attachment/35300/35300_make.log" title="make.log attached to LU-8130">make.log^{<img class="rendericon" src="https://jira.whamcloud.com/images/icons/link_attachment_7.gif" height="7" width="7" align="absmiddle" alt="" border="0"/>}</a></span> </p> <p>Is there a proper way to fix the codes to make them compatible with old kernel?</p> <blockquote><p>Is there a proper way to fix the codes to make them compatible with old kernel?</p></blockquote> <p>This can be resolved by using HAVE_SERVER_SUPPORT to disable the rhashtable codes while building client. The rhashtable codes are only used by lu_env in lustre/obdclass/lu_object.c on server.</p> <p>That RHEL7.2 correct? Do we even test RHEL7.2 client. I know for 2.13.X you can't even build server rpms for RHEL7.2. Is this a 2.12 issue?</p> <p>Yes, James. That's a 2.12 issue on RHEL 7.2 client. After adding HAVE_SERVER_SUPPORT check for rhashtable codes in libcfs/include/libcfs/linux/linux-hash.h, the issue is resolved.</p> <p>Ah, that makes sense. rhashtable is barely uses in 2.12 so it should be a easy fix.</p> <p>James Simmons (jsimmons@infradead.org) uploaded a new patch: <a href="https://review.whamcloud.com/40113" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/40113</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8130" title="Migrate from libcfs hash to rhashtable" class="issue-link" data-issue-key="LU-8130">LU-8130</a> nrs: convert NRS ORR / TRR to rhashtable<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 4b29db0c73802bb9ab32a39a25ca93cbad9f36be</p> <p>James Simmons (jsimmons@infradead.org) uploaded a new patch: <a href="https://review.whamcloud.com/41092" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/41092</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8130" title="Migrate from libcfs hash to rhashtable" class="issue-link" data-issue-key="LU-8130">LU-8130</a> obdclass: convert nid_stats to rhashtable<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 9aa6d2d5f5f7deac7b0fb596524ff9ea991b6e24</p> <p>"James Simmons &lt;jsimmons@infradead.org&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/45882" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/45882</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8130" title="Migrate from libcfs hash to rhashtable" class="issue-link" data-issue-key="LU-8130">LU-8130</a> ldlm: convert ldlm_resource hash to rhashtable<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: df3f27db84994843a9908ced17e7000410267e11</p> <p>"James Simmons &lt;jsimmons@infradead.org&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/51840" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/51840</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8130" title="Migrate from libcfs hash to rhashtable" class="issue-link" data-issue-key="LU-8130">LU-8130</a> libcfs: free xa_node instead of radix_tree_node<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 7553c5aa770d0e8c61630e9cc1deff4a13e02b4c</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/51840/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/51840/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8130" title="Migrate from libcfs hash to rhashtable" class="issue-link" data-issue-key="LU-8130">LU-8130</a> libcfs: don't use radix tree for xarray<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 778791dd7da107710c2311935a24cfd7e7a5fd85</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/40113/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/40113/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8130" title="Migrate from libcfs hash to rhashtable" class="issue-link" data-issue-key="LU-8130">LU-8130</a> nrs: convert NRS ORR/TRR to rhashtable<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 42bf5f78ba23ed5a7a908c921a5d8ded7544a315</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/32038/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/32038/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8130" title="Migrate from libcfs hash to rhashtable" class="issue-link" data-issue-key="LU-8130">LU-8130</a> osc: convert osc_quota hash to xarray<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: ac8c28f959d87c1ae852d9daccce3c2a3f7a6f39</p> Blocker LU-14535 Related LU-9859 LU-14698 LU-16365 LU-6800 LU-11624 LU-14752 LU-10401 LU-16253 LU-17191 LU-17329 LU-12511 Development End date Thu, 12 May 2016 00:07:59 +0000 Rank 1|hzybbz: Rank (Obsolete) 9223372036854775807 Start date Thu, 12 May 2016 00:07:59 +0000