https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-1201 Lustre <p>Lustre use crypto hashes algo in two ways: PTLRPC (ptlrpc/sec_bulk.c), OST(ost/ost_handler.c)/OSC (osc/osc_request.c). <br/> OST,OSC use crc32, crc32c, adler for checksumming (compute_checksum() function) <br/> PTLRPC uses crc32, adler, md5, sha1-512 ( kernel crypto api for all, excluding crc32 adler) <br/> All subsystems go through bulk pages and update checksum. <br/> To resolve conflicts with different implementation of checksumming, a new crypto hash interface is needed at libcfs. It should use kernel crypto api for hash calculation for kernel modules, and lustre implementation for user mode. Previus checksum calculation should be changed to the new libcfs crypto hash api. And adding new hash algo would be a simple task.</p> LU-1201

Lustre crypto hash cleanup

Improvement Minor Resolved Fixed WC Triage Alexander Boyko Fri, 9 Mar 2012 10:48:19 +0000 Wed, 21 Nov 2012 19:30:00 +0000 Sun, 1 Jul 2012 03:00:53 +0000 Lustre 2.3.0 0 6 <p><a href="http://review.whamcloud.com/2276" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/2276</a></p> <p>In addition to the questions in the patch, I was wondering why all of the different hash functions were being added? </p> <p>I don't quite understand how "ptlrpc" hash functions differ from "ost" hash functions? Are you referring to sptlrpc (ie Kerberos)? To confirm, is there a protocol change associated with this patch or not?</p> <p>If there isn't a good reason to add a hash function, I don't necessarily want to just add them because we can. One reason against this is because there are only a limited number of hash functions we can use in the protocol, and also because I'd like to move to "piece-wise" hashes that could be computed on the page and then aggregated in the RPC without having to recompute the checksum again. </p> <p>I'm really want to see the performance numbers by this changes. (any improvements with kernel crypto APIs, instead of current crc32c codes?) The reason of what I made a patch crc32c/w h/w accelerator checksum patch a couple years ago was that for single client's performance improvements. (there was only single thread ptlrpc, checksum on the client was calculated on that thread without in parallel. So, needed off-load checksum calculation)<br/> The server side was not much performance difference regardless the checksum=on (alder or crc32c) or checksum=off. Because, the servers can calculate the checksum in parallel.</p> <p>That's true that single client's performance was improved by 40-50% (900MB/sec -&gt; 1.5GB/sec) in 1.8.x code with crc32c patch, but still hitting 1.5GB/s performance limitation due to single threaded ptlrpc. The recent 2.2 branch supports multiple ptlrpc, and it's breaking 1.6GB/s limitation right now and really higher than previous 1.8's numbers. </p> <p>the single client performance on 2.2 branch, crc32c algorithm still is faster than alder, but crc32c, adler and no checksum's performance number are getting close on 2.2 now..</p> <p>This is logs from libcfs cfs_crypto_test_hashes()<br/> Crypto hash algorithm null speed = -1 MB/s <br/> Crypto hash algorithm adler32 speed = 2241 MB/s <br/> Crypto hash algorithm crc32 speed = 360 MB/s <br/> Crypto hash algorithm md5 speed = 428 MB/s <br/> Crypto hash algorithm sha1 speed = 178 MB/s <br/> Crypto hash algorithm sha256 speed = 117 MB/s <br/> Crypto hash algorithm sha384 speed = 174 MB/s <br/> Crypto hash algorithm sha512 speed = 175 MB/s <br/> Crypto hash algorithm crc32c speed = 2602 MB/s (for sw table ~330 MB/s)</p> <p>VMware, cpu i5-2415M, for crc32c used crc32c_intel module <br/> Libcfs use 1MB buffer for testing, and calculate hash during 1sec with the same buffer.</p> <p>This patch provide libcfs interface for crypto hashes.<br/> &gt;&gt;In addition to the questions in the patch, I was wondering why all of the different hash functions were being added?<br/> because SPTLRPC compute hash for this num of algo. No protocol changes, hash calculation only.</p> <p>This patch drop lustre crc32c implementation because linux kernel crypto have two implementation of crc32c 1) table sw 2) hw (crc32c_intel same code like lustre have). One more - supporting of crc32 instruction should be detected at run time, not in compile (lustre #ifdef X86_FEATURE_XMM4_2).<br/> <a href="https://jira.whamcloud.com/browse/LU-241" title="support crc32c with hardware accelerated instruction as one of lustre checksums" class="issue-link" data-issue-key="LU-241"><del>LU-241</del></a> add crc32c algo for lustre osc/ost only, no sptlrpc.<br/> Now(this patch), all crypto hashes algorithms are at one place, same interface, and all parts of lustre can use it. </p> <p>In the the current implementation, crc32c (and fastest algorithm) should be detected automatically at the run time. I've tested same rpm packages on the clients with both of Nehalem based CPU and more old CPU(Core2 Duo) which doesn't have sse4_2 feature.<br/> e.g) mounted a lustre filesystem from both clients. Here is the deafult checksum_type after just mounted.</p> <ul class="alternate" type="square"> <li>Nehalem based CPU's client<br/> &#35; cat /proc/fs/lustre/osc/*/checksum_type<br/> crc32 adler <span class="error">&#91;crc32c&#93;</span><br/> crc32 adler <span class="error">&#91;crc32c&#93;</span> </li> </ul> <ul class="alternate" type="square"> <li>Core2 Due CPU's client<br/> &#35; cat /proc/fs/lustre/osc/*/checksum_type<br/> crc32 <span class="error">&#91;adler&#93;</span> <br/> crc32 <span class="error">&#91;adler&#93;</span></li> </ul> <p>yes you are right<br/> #ifdef X86_FEATURE_XMM4_2<br/> if (cpu_has_xmm4_2)<br/> ret |= OBD_CKSUM_CRC32C;<br/> #endif</p> <p>This patch don`t improve any checksum calculation. It present libcfs crypto hash interface for checksum calculation and fix all lustre checksums to it. Base on kernel crypto api we got crc32c sw implementation, so clients wich don`t support sse_4.2 can use crc32c.</p> <p>OK, but, why we need to use crc32c sw implementation, instead of adler when if the client doesn't have Nehalem processor? (btw, Nehalem processor was released out 3 years ago...) Your numbers showed us adler (2241 MB/s) is faster than crc32 sw table (~330 MB/s). Also, my recent checksum performance testing on 2.2 branch, with adler, crc32c's and no-checksum's performance are very similar on the single client.</p> <p>Here is qucik performance results on between single OSS and single client over the QDR Infiniband. I tested this on ramdisk (created 8 x OST on ramdisk) instead of general storage to show real checkum performance. Tested server are 2 x HP DL360 (2.93GHz x 12 CPU cores, 48GB memory) </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>WRITE(2.1.1) #proc nochksum crc32 adler crc32c 1 759 289 550 2 1367 567 1063 4 2663 1146 2023 8 3084 2183 1309 WRITE(2.2 branch) 1 748 747 748 749 2 1427 1401 1378 1387 4 2669 2465 2594 2621 8 3152 2470 3073 3148 READ(2.1.1) nochksum crc32 adler crc32c 1 1031 323 829 2 1785 360 921 4 2456 373 963 8 2145 376 954 READ(2.2 branch) 1 998 925 954 938 2 1826 1703 1716 1691 4 2986 2653 2850 2836 8 2875 2724 2659 2855 </pre> </div></div> <p>Alexander, thank you for the checksum performance results. Based on these numbers, the adler32 algorithm should be used for nodes that do not have hardware crc32c support, but h/w crc32c is slightly faster so it should be preferred if available. Does this match your test results? What is the output of "lctl get_param osc.*.checksum_type" (as Ihara showed in his recent comment)?</p> <p>Is the printing of "Crypto hash algorithm null speed = -1 MB/s" a defect?</p> <p>Ihara, thank you also for the benchmark results. What test is being run? Is it a single-threaded or multi-threaded load? The multi-threaded ptlrpcd change, along with moving the bulk RPC checksums to be in ptlrpcd context is what improved this performance.</p> <p>Alexander, also note that if the OST does not support crc32c in hardware, then it should not reply to the client that crc32c is supported. It is much more efficient for the OSS in this case to use adler32, since it uses much less CPU, and the speed on the client is approximately the same. It is more important that the server CPU usage is as efficient as possible.</p> <p>Andreas, the numbers were tested on IOR with multiple processes </p> {1,2,4 and 8} <p> on the single client.<br/> IOR -a POSIX -k -t 1m -b 1g -F -w -o /mnt/lustre/file<br/> sync; echo 1 &gt; /proc/sys/vm/drop_caches<br/> IOR -a POSIX -t 1m -b 1g -F -r -o /mnt/lustre/file</p> <p>The numbers were on 2.1.1 is pretty same as what we saw 1.8.x. And, yes, the checksum with any algorithms are improved on 2.2.</p> <p>&gt;&gt;Does this match your test results? What is the output of "lctl get_param osc.*.checksum_type" (as Ihara showed in his recent comment)?<br/> Yes, the fastest algo would be used by default, if server support it.<br/> &gt;&gt;Is the printing of "Crypto hash algorithm null speed = -1 MB/s" a defect?<br/> No, upper layer disable checksumming for ost/osc. -1 means no algo, or unsupported.</p> <p>&gt;&gt;Alexander, also note that if the OST does not support crc32c in hardware, then it should not reply to the client that crc32c is supported. It is much more efficient for the OSS in this case to use adler32, since it uses much less CPU, and the speed on the client is approximately the same. It is more important that the server CPU usage is as efficient as possible.</p> <p>In this point of view the server should not reply to the client that crc32 is supported too. Because adler based on simple add instruction and all hardware support it. One reason to support crc32 is very old client. With this patch the code always support crc32c, may be this should be changed for sw implementation in the server reply.</p> <p>I think, that not only speed performance is the main parameter for algo, need to look at cpu load on real hardware.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>WRITE(2.1.1) #proc nochksum crc32 adler crc32c 1 759 289 550 2 1367 567 1063 4 2663 1146 2023 8 3084 2183 1309 </pre> </div></div> <p>Why adler for 8 clients is slower than the crc32 ?</p> <p>The same question for read</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>READ(2.2 branch) 1 998 925 954 938 2 1826 1703 1716 1691 4 2986 2653 2850 2836 8 2875 2724 2659 2855 </pre> </div></div> <p>Base on checksum performance adler32 is 6-7 times faster than crc32 table implementation. But raw IOR lustre result show different value.</p> <p>&gt; Why adler for 8 clients is slower than the crc32 ?<br/> don't know why, still, but it was same results on 2 x OSS, 8 x OST(4 OSTs per OSS) configuration, instead of 1 x OSS, 8 OSTs.</p> <p>This was tested on ramdisk /dev/ram, it should be also tested with real disks, instead of ram disk.</p> <blockquote> <p>In this point of view the server should not reply to the client that crc32 is supported too. Because adler based on simple add instruction and all hardware support it. One reason to support crc32 is very old client. With this patch the code always support crc32c, may be this should be changed for sw implementation in the server reply.</p></blockquote> <p>The need to support old crc32-only clients has passed. According to <a href="https://bugzilla.lustre.org/show_bug.cgi?id=13805" class="external-link" target="_blank" rel="nofollow noopener">https://bugzilla.lustre.org/show_bug.cgi?id=13805</a>, the adler32 support was landed in Lustre 1.6.5, and while 1.8 clients and servers need to maintain interoperability with 1.6 clients and servers (these apparently still exist in the world), the 2.3 servers where this patch will land only need interoperability with 1.8 clients.</p> <p>I would suggest that the OST use the cfs_crypto_test_hashes() results, and eliminate any results which are slower than, say, 1/2 of the adler32 value. It would be worthwhile to include a comment referencing b=13805 that added adler32 in 1.6.5 to make it clear that all 1.8.x clients will be able to support this.</p> <p>Andreas - you marked the patch "I would prefer that you didn't submit this" &#8211; is this only because of the default algorithm selection issue? Would it make more sense here to send an explicit OSS preference back to the clients instead of "faking" the connect flags? <br/> The broader cleanup and unification of the sptlrpc and bulk checksum is acceptable to you?</p> <p>Shuichi - your reported 2.2 numbers are using this patch?</p> <p>Nathan, it is only because of the default algorithm selection. The rest of the patch is generally OK I think. My only remaining </p> <p>As for sending the OSS preference back - that would be better, but more complex. It would need a speed/ratio to be sent back for each checksum so that the client could pick the best combination of local and remote checksums. However, I tend to think that this decision is better for the server to make based on the algorithms that the client understands, and the speed of the algorithms that the server supports.</p> <p>It would also be possible for the client to bias the checksum selection by excluding algorithms like software CRC32/CRC32c if they are extremely slow and then letting the OST chose from the remainder based on performance. In essence, this is what is done in the existing code in a much less flexible manner - the client excludes CRC32c if it is not implemented in hardware, as does the OST, and the remaining choices are sent back to the client (ADLER normally, and CRC32 as a fallback for 1.6 clients). Having actual performance metrics could makes this selection less "binary", which will become increasingly important as new algorithms are added.</p> <p>Essentially, what I'm proposing is an improvement of the existing selection semantics that take both client and server performance into account, rather than ignoring the server performance entirely as the proposed patch does. It could be implemented in two ways, and I think my original proposal of having the client chose the fastest algorithm is not as good as having the server do it. New proposal below:</p> <ul class="alternate" type="square"> <li>client computes checksum speeds</li> <li>client discards algorithms below some performance threshold <ul class="alternate" type="square"> <li>e.g. 3/4 of (geometric mean, average, fastest), ...</li> <li>this will keep all algorithms if they are reasonably close in performance, otherwise using just the mean/median would always exclude <em>some</em> algorithm(s) needlessly</li> <li>and always includes ADLER for interoperability</li> </ul> </li> <li>we do not need to include CRC32 <ul class="alternate" type="square"> <li>this can be very slow for the client, even if it is fast on the server</li> <li>ADLER is faster in software and is universally supported</li> <li>CEC32 may be included if it is fast (e.g. implemented in hardware)</li> </ul> </li> <li>client sends (possibly reduced) list of acceptable checksums (bits) to server in connect</li> <li>server discards client algorithms it doesn't understand, and those below the performance threshold calculated based on remaining algorithms</li> <li>the presence of ADLER would ensure that there is always <em>some</em> algorithm which is common to both the client and server</li> <li>only the fastest server algorithm(s) will be returned to the client, if there is a big difference</li> <li>client can make a final choice if there is one</li> </ul> <p>My only concern is that this removes the ability of the users/admin to make the decision. The other approach would be to just negotiate raw capabilities, always default to adler, and let the admin make the final selection with lctl conf_param.</p> <p>Nathan, note that this is only describing the behaviour of selecting the default checksum algorithm. This isn't really any different than what is done today, with the exception that instead of using the actual checksum performance the old code used hard-coded defaults based on the order that the checksums were listed in obd_cksum_pack() (or similar, can't check right now).</p> <p>If there is a checksum algorithm listed via conf_param it would override the one chosen by default. I guess the caveat is that if the algorithm is excluded because it is much slower than the others, then it may not be available for selection. That is true of the code today also - if crc32c is not in hardware on either the client or server, then it is excluded and couldn't be selected by the administrator either.</p> <p>I've never tested this patches on 2.2. All results were on vanilla-2.2 branch.<br/> Here is single client's performance ratio(tested with 12 IOzone threads). This is tested on real disks, instead of ramdisk. The results overall, it seems to be reasonable.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Performance ratio(Write) v2.2 nochecksum 1.00 adler 0.88 crc32 0.68 crc32c 0.93 Performance ratio(Read) v2.2 nochecksum 1.00 adler 0.88 crc32 0.71 crc32c 0.91 </pre> </div></div> <blockquote><p>I guess the caveat is that if the algorithm is excluded because it is much slower than the others, then it may not be available for selection. That is true of the code today also - if crc32c is not in hardware on either the client or server, then it is excluded and couldn't be selected by the administrator either.</p></blockquote> <p>Exactly my concern. If there are tons of clients and few servers, it may make sense to use the fastest server algorithm regardless of the client algo speed to optimize the overall system rate. That's the reason we wanted to be able to use the SW crc-32c algo (on the clients) in the first place.</p> <p>If the concern is the client eliminating the checksum algorithm choices prematurely, then the client could send all of the known algorithms to the OST. Only the OST would drop the algorithms that are totally underperforming (e.g. crc32 or crc32c in s/w). The final performance/config selection would be in the hands of the client/administrator. That would ensure that the client doesn't pick a locally fast algorithm (e.g. crc32c h/w) that would significantly impact the OST CPU usage (e.g. crc32c s/w), without reducing the (sensible) choices available to the user.</p> <p>That sounds like a good solution:<br/> 1. Client sends all available local algos<br/> 2. server drops non-client-supported from list, then replies with remaining algos that perform at 50% or better of the fastest algo remaining on server<br/> 3. client chooses locally fastest of what's left as the default.</p> <p>A quick question.<br/> Do we keep RHE5 patchless client on lustre-2.3?<br/> If so, RHEL5's client may see some performance drops on 2.3 from 2.2 if they upgrade.<br/> Because, crc32c is implemented in the lustre itself and crc32c-intel is not available on that kernel, so alder will be selected on 2.3.</p> <p>Ihara, while it is true that there may be some minor degradation in the case of RHEL 5 clients, based on the test results you posted this performance loss will be minor (or still an improvement over earlier versions of Lustre) due to the multi-threaded ptlrpcd speedups. </p> <p>Also, the chance of users wanting to stick with RHEL5 for stanility, but moving to a new development version of Lustre is not very likely. </p> <p>I do test on RH5 with kernel version 2.6.18-238.19.1, it support crc32c hw (kernel compile), and have better crc32c performance than 2.6.32-... kernels.</p> <p>Alexander, Ihara,<br/> have you done any performance comparison with checksums enabled on 1.8 for comparison?</p> <p>I had couple of benchmarks on 1.8 and 2.2 for comparison, but not yet with this patches. I'm thinking to do same benchmarking on 1.8 and 2.2 with this patches in a couple of days.<br/> btw, we are seeing single client's performance regressions on 2.x due to <a href="https://jira.whamcloud.com/browse/LU-744" title="Single client&#39;s performance degradation on 2.1" class="issue-link" data-issue-key="LU-744"><del>LU-744</del></a>. So, at this moment, we might need to run with file size &lt; client's memory size for more fair comparison.. However, In the case of file size &gt; client's memory size, we still see less client's performance regardless checksum=on/off.</p> <p>Hit kernel panic with this patches on Sandybridge server. I just filed on <a href="https://jira.whamcloud.com/browse/LU-1379" title="kernel BUG at crypto/api.c:336!" class="issue-link" data-issue-key="LU-1379"><del>LU-1379</del></a>.</p> <p>Can you attach your config.h? Looks like invalid configure, I can`t reproduce issue on the same kernel.</p> <p>ok, the problem is fixed with correct config.h.</p> <p>Andreas, <br/> attached is an checksum comparison on various lustre version. most of 2.2 numbers (weather checksum is enabled or not) were lower than 2.1 and 1.8 except read performance. we might be having some regressions on 2.2 client. need to open new ticket of this?</p> <p>Ihara, yes this spreadsheet should be attached to a new bug, along with details of the test being run. Are these results from testing with 24x ramdisk OSTs, as before? Jinshan is looking at this problem, please assign it to him.</p> <p>The new results are contrary to the previous test results in this bug, which showed 2.2 being faster than 2.1 for 8x OSTs.</p> <p>It looks like there is already a bug <a href="https://jira.whamcloud.com/browse/LU-744" title="Single client&#39;s performance degradation on 2.1" class="issue-link" data-issue-key="LU-744"><del>LU-744</del></a> for tracking the 2.x performance degradation.</p> <p>Ihara, have you checksumming benchmark results without this patch(for the same hw and configuration)?</p> <p>No big performance differences between 2.2 and 2.2/w this patches, but still lower than 2.1.2.</p> <p><a href="https://jira.whamcloud.com/browse/LU-744" title="Single client&#39;s performance degradation on 2.1" class="issue-link" data-issue-key="LU-744"><del>LU-744</del></a>, I've filed before, but I think this is another regression compared to what we saw here.<br/> As far as I tested on 2.2, the performance goes down when write/read size exceed client's memory size. This is <a href="https://jira.whamcloud.com/browse/LU-744" title="Single client&#39;s performance degradation on 2.1" class="issue-link" data-issue-key="LU-744"><del>LU-744</del></a> and same problem happens on 2.1.x.</p> <p>In order to avoid this regression for this checksum comparison among on 1.8.7, 2.1.2 and 2.2, I used file size &lt; client's memory size. </p> <p>Anyway, I don't think this regression is related to <a href="https://jira.whamcloud.com/browse/LU-1201" title="Lustre crypto hash cleanup" class="issue-link" data-issue-key="LU-1201"><del>LU-1201</del></a>, I will open the new ticket.</p> <p>Hi Ihara, in <a href="https://jira.whamcloud.com/browse/LU-744" title="Single client&#39;s performance degradation on 2.1" class="issue-link" data-issue-key="LU-744"><del>LU-744</del></a> you saw 2.2 clients performed better than 2.1, however, here you have seen the opposite. Is this only because the file size is less than memory size?</p> <p>Hi Jay, </p> <p>Yes, I didn't write/read the larger file size than client's memory size to avoid <a href="https://jira.whamcloud.com/browse/LU-744" title="Single client&#39;s performance degradation on 2.1" class="issue-link" data-issue-key="LU-744"><del>LU-744</del></a> issue.</p> <p>I opened new ticket on <a href="https://jira.whamcloud.com/browse/LU-1408" title="single client&#39;s performance regression test" class="issue-link" data-issue-key="LU-1408"><del>LU-1408</del></a> for single client's performance regression on 2.2. Also, during my testing for <a href="https://jira.whamcloud.com/browse/LU-1408" title="single client&#39;s performance regression test" class="issue-link" data-issue-key="LU-1408"><del>LU-1408</del></a>, I saw another big performance difference between the latest b2_1 branch and 2.1.2RC0 tag. This is filed on <a href="https://jira.whamcloud.com/browse/LU-1413" title="difference of single client&#39;s performance between b2_1 and 2.1.2RC0" class="issue-link" data-issue-key="LU-1413"><del>LU-1413</del></a> as another regression. </p> <p>So, please move foward to on <a href="https://jira.whamcloud.com/browse/LU-744" title="Single client&#39;s performance degradation on 2.1" class="issue-link" data-issue-key="LU-744"><del>LU-744</del></a>, <a href="https://jira.whamcloud.com/browse/LU-1408" title="single client&#39;s performance regression test" class="issue-link" data-issue-key="LU-1408"><del>LU-1408</del></a> and <a href="https://jira.whamcloud.com/browse/LU-1413" title="difference of single client&#39;s performance between b2_1 and 2.1.2RC0" class="issue-link" data-issue-key="LU-1413"><del>LU-1413</del></a> for single client performance regression and different behaviors on various lustre version.</p> <p>Landed for 2.3</p> <p>patch reverted as breaks build for external OFED</p> <p>new review req at <a href="http://review.whamcloud.com/#change,3098" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,3098</a><br/> two lines code was changed. Build succcessful.</p> <p>Landed for 2.3</p> <p>Xyratex-bug-id: <a href="http://jira-nss.xy01.xyratex.com:8080/browse/MRP-337" class="external-link" target="_blank" rel="nofollow noopener">MRP-337</a><br/> Xyratex-bug-id: <a href="http://jira-nss.xy01.xyratex.com:8080/browse/MRP-471" class="external-link" target="_blank" rel="nofollow noopener">MRP-471</a></p> Related LU-744 Development Rank 1|hzv6en: Rank (Obsolete) 4572