https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-1346 Lustre <p>Migrate libcfs to emulate Linux kernel APIs, so that when submitting Linux client to upstream kernel, we don't need the abstraction layer which will be rejected by kernel maintainers. The majority of libcfs are cfs_* or ll_* wrappers, and can be cleaned up with scripts doing like "s/cfs_spin_lock/spin_lock". For other functions that cannot be cleaned up this way, we need to look at them in a case-by-case basis.</p> LU-1346

cleanup libcfs

Improvement Minor Resolved Fixed Keith Mannthey Peng Tao emc patch Thu, 26 Apr 2012 22:00:52 +0000 Mon, 30 Sep 2013 10:50:14 +0000 Mon, 16 Sep 2013 20:50:17 +0000 Lustre 2.4.0 Lustre 2.5.0 Lustre 2.4.0 Lustre 2.5.0 0 11 <p>Per the discussion with Andreas etc., we can use a script to do libcfs cleanup at compiling time first. We can implement the script incrementally, when the script is good enough we can apply it to lustre tree to change code directly. The code change will be huge and will break most in-flight patches and break different branch rebase, the script can help to resolve the confliction as well.</p> <p>We ever considered implementing the script based on coccinelle, it is potentially a good tool to resolve collateral evolution issues. But it cannot well process some "#ifdef .... #else ... #endif" cases, or some complicate macro codes. So we implemented a sed script to do libcfs cleanup, together with some direct code changes.</p> <p>3 patches submitted for review: <br/> <a href="http://review.whamcloud.com/2829" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/2829</a> cleanup libcfs lock and bit operations<br/> <a href="http://review.whamcloud.com/2830" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/2830</a> cleanup libcfs file operations<br/> <a href="http://review.whamcloud.com/2831" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/2831</a> cleanup libcfs memory operations</p> <p><a href="http://review.whamcloud.com/4775" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/4775</a> cleanup libcfs primitive (linux-prim.h)<br/> <a href="http://review.whamcloud.com/4776" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/4776</a> cleanup macros in kp30.h<br/> <a href="http://review.whamcloud.com/4777" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/4777</a> tcpip/time/type related cleanup<br/> <a href="http://review.whamcloud.com/4778" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/4778</a> cleanup macros in portals_compat25.h<br/> <a href="http://review.whamcloud.com/4779" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/4779</a> cleanup cfs_curproc_xxx macros<br/> <a href="http://review.whamcloud.com/4780" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/4780</a> cleanup list operations<br/> <a href="http://review.whamcloud.com/4781" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/4781</a> replace CFS_CAP_XXX with kernel definition</p> <p>The above patches are only script plus some other code changes that not-easy to be handled by script. I submitted only for review, we can merge them and run the script to generate the final patch at appropriate time.</p> <p>There are some other cleanup patches have not been submitted for example LWT code and some other obsolete macros cleanup.</p> <p><a href="http://review.whamcloud.com/#change,2829" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,2829</a> (cleanup libcfs lock and bit operations) has landed.<br/> <a href="http://review.whamcloud.com/#change,2830" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,2830</a> (remove cfs_ file wrappers) has landed.</p> <p>I just tried out the latest master to discover that it no longer builds on SLES11 SP1. The api for mem_cache_create has a extra argument in the base line 2.6.32 kernel which was removed later.</p> <p>Sorry it was another problem. Your patch doesn't impact SLES11 SP1.</p> <p>I split <br/> <a href="http://review.whamcloud.com/4775" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/4775</a> cleanup libcfs primitive (linux-prim.h)<br/> into several patches, to facilitate review and merging</p> <p><a href="http://review.whamcloud.com/#/c/6954/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/6954/</a><br/> <a href="http://review.whamcloud.com/#/c/6955/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/6955/</a><br/> <a href="http://review.whamcloud.com/#/c/6956/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/6956/</a><br/> <a href="http://review.whamcloud.com/#/c/6957/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/6957/</a><br/> <a href="http://review.whamcloud.com/#/c/6959/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/6959/</a></p> <p>The procfs primitive cleanup is dropped because it will be soon changed with the procfs change in 3.10 kernel support.</p> <p>The last one patch (6959) is partial and converts only atomic primitives in libcfs directory. I'll update incremental patches to clean other modules.</p> <p>There seems to be some build problem related to this patch set. A rebuild failed as well so it likely has to do with the base change itself somehow. </p> <p>I don't have SLES11 environment but I tried to build the patchset against SLES11SP2 kernel on my CentOS box, and no error was found. I couldn't build rpms though because they require different spec file rules. And the error seems to be from rpmbuild.</p> <p>Could you please help to retrieve below file?<br/> /var/lib/jenkins/workspace/lustre-reviews/arch/x86_64/build_type/client/distro/sles11sp2/ib_stack/inkernel/BUILD/find-requires</p> <p>The error message says:<br/> error: Failed to find Requires:<br/> Finding Requires: /var/lib/jenkins/workspace/lustre-reviews/arch/x86_64/build_type/client/distro/sles11sp2/ib_stack/inkernel/BUILD/find-requires<br/> error: line 674: Dependency tokens must begin with alpha-numeric, '_' or '/': rm -rf $RPM_BUILD_ROOT</p> <p>It seems that find-requires scanned some script and found undefined variable $RPM_BUILD_ROOT. But the patchset doesn't change any scripts except for libcfscleanup.sed which clearly doesn't reference $RPM_BUILD_ROOT.</p> <p>Can you try patch <a href="http://review.whamcloud.com/#/c/5491" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/5491</a>. That might fix it up for you.</p> <p>I am working to find a way to get the malformed file so we can better see what went wrong. We have a separate internal ticket tracking this build issue. </p> <p>The present thought is the first patch in this series is wrecking havoc is this odd way. I will see about a debug rebase ontop of the SuSE build change. </p> <p>Thanks James. It seems that <a href="http://review.whamcloud.com/#/c/5491" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/5491</a> is related. I'll update to include it and see if it fixes the build issue.</p> <p>Turned out to be a bug in the base patch. The error message was misleading. I've fixed it and now build succeeded.</p> <p>I am glad this got sorted it. I agree the error message was very misleading. </p> <p>Following up the cfs_atomic primitive change, below seven patches convert the rest pieces of code incrementally. They can each be merged independently.<br/> <a href="http://review.whamcloud.com/7070" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/7070</a><br/> <a href="http://review.whamcloud.com/7072" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/7072</a><br/> <a href="http://review.whamcloud.com/7073" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/7073</a><br/> <a href="http://review.whamcloud.com/7074" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/7074</a><br/> <a href="http://review.whamcloud.com/7075" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/7075</a><br/> <a href="http://review.whamcloud.com/7076" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/7076</a><br/> <a href="http://review.whamcloud.com/7077" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/7077</a></p> <p>These 7 patches are each dependant on about 10 other patches. We will need to land these changes in order and a lowish level patch needing a rebase is going to cause the whole stack to be rebuilt and tested at least once more. </p> <p><a href="http://review.whamcloud.com/4779" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/4779</a> is low in the stack and out of date. </p> <p>The lowest patch is <a href="http://review.whamcloud.com/4776" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/4776</a> and is being testing. 4777 and 4778 look to to land once 4776 finishes testing. </p> <p>If there is any way to break the dependencies up the speed at which this code will land could improve.</p> <p>shall I rebase 4779 and all the patches that rely on 4770 now, or wait until 4776/4777/4778 on are landed?</p> <p>They are dependent mostly because they all modify the libcfs-cleanup.sed file.</p> <p>Can you hold off rebasing 4779. I'm working on getting Lustre working with a 3.10.2 kernel and that version of the kernel impacts process name space handling. First I need to handle the proc changes in 3.10+ </p> <p>James, do you mean you are working on procfs change in 3.10? It is tracked on <a href="https://jira.whamcloud.com/browse/LU-3319" title="Adapt to 3.10 upstream kernel proc_dir_entry change" class="issue-link" data-issue-key="LU-3319"><del>LU-3319</del></a>. I thought that liuying is working on it. You may want to coordinate with each other so don't waste the efforts.</p> <p>James posted <a href="http://review.whamcloud.com/#/c/7454" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/7454</a> for gnilnd</p> <p>Follow on work for this ticket is being tracked in <a href="https://jira.whamcloud.com/browse/LU-3963" title="cleanup libcfs wrappers" class="issue-link" data-issue-key="LU-3963"><del>LU-3963</del></a></p> <p>please revert<br/> commit b22fb817507ff52c02de38435fe90d758e852105<br/> Author: James Simmons &lt;uja.ornl@gmail.com&gt;<br/> Date: Fri Sep 13 09:44:00 2013 -0400</p> <p> <a href="https://jira.whamcloud.com/browse/LU-1346" title="cleanup libcfs" class="issue-link" data-issue-key="LU-1346"><del>LU-1346</del></a> libcfs: replace CFS_CAP_XXX with kernel definition</p> <p>that defines needed because Linux capability now have size more then lustre wire protocol so we don't able to pack all capability into wire. That is main reason to introduce these macroses, please look to commit and ticket where it's adds to lustre tree.</p> <p>I see you are drop compatibility with different OS'es and kernels, but that is very very bad idea.</p> <p>and PLEASE REMEMBER - any on wire constants MUST be platform independent to avoid situation when it's different on different sides of lustre cluster. </p> <p>as additional objection about that work - 'fls' cleanup landed some time ago. that is broke lustre to build on system already have these functions but with different arguments as you expected. <br/> and forget to make cfs_gettimeofday to gettimeofday changes in some places.</p> <p>The patch just did a bunch of renaming of variables. cfs_cap_t is still a __u32. Can you point to exactly the break is where you believe it to be.</p> <p>Please push fixes for the fls and gettimeofday issues. Patches always welcomed <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>OS defined flags should be never applied to the custom type, so CAP_* flags may be applied to the kernel_cap type, other is incorrect by definition, but kernel_cap_t can't be put in _u32 in wire.<br/> reason to have CFS_CAP* to be similar CAP_* defines - capability with older (before patch) clients, but we define only few bits from kernel bitmask, so if we will be need some additional on server side we will be able to pack data from kernel bits to unused bits in wire data.<br/> so avoid a wire protocol changes as long as possible. It's my bad - i forget to add CFS_CAP checks into wiretest script to notify - it's wire specific data and should be don't changed without real reasons.</p> <p>as about breakage.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">In file included from /Users/shadow/work/lustre/work/WC-review/mac-os/libcfs/include/libcfs/posix/libcfs.h:108, from /Users/shadow/work/lustre/work/WC-review/mac-os/libcfs/include/libcfs/libcfs.h:45, from nidstrings.c:43: /Users/shadow/work/lustre/work/WC-review/mac-os/libcfs/include/libcfs/user-bitops.h:80: error: conflicting types <span class="code-keyword">for</span> ‘fls’ /usr/include/strings.h:90: error: previous declaration of ‘fls’ was here </pre> </div></div> <p>so part for cfs_fls need to be reverted.</p> <p>one more additional bug</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">gcc -DHAVE_CONFIG_H -I. -I../.. -D__arch_lib__ -D_LARGEFILE64_SOURCE=1 -include /Users/shadow/work/lustre/work/WC-review/mac-os/config.h -I/Users/shadow/work/lustre/work/WC-review/mac-os/libcfs/include -I/Users/shadow/work/lustre/work/WC-review/mac-os/lnet/include -I/Users/shadow/work/lustre/work/WC-review/mac-os/lustre/include -g -Wall -fPIC -D_GNU_SOURCE -g -O2 -Werror -MT libcfs_a-posix-debug.o -MD -MP -MF .deps/libcfs_a-posix-debug.Tpo -c -o libcfs_a-posix-debug.o `test -f <span class="code-quote">'posix/posix-debug.c'</span> || echo <span class="code-quote">'./'</span>`posix/posix-debug.c cc1: warnings being treated as errors posix/posix-debug.c: In function ‘libcfs_debug_vmsg2’: posix/posix-debug.c:210: warning: implicit declaration of function ‘cfs_gettimeofday’ </pre> </div></div> <p>Okay we can revert the last patch. The caps patch had less of a impact on the code than the other patches for <a href="https://jira.whamcloud.com/browse/LU-1346" title="cleanup libcfs" class="issue-link" data-issue-key="LU-1346"><del>LU-1346</del></a>. Proper capabilities support will need to be explored later.</p> <p>Thanks! I will send maces/freebsd support patches with fuse client son (i hope in next two weeks).</p> <p>I created patch with some cleanups need to compile lustre with clang. most of them is c89 style fixes with correct includes.<br/> please inspect it.</p> <p><a href="http://review.whamcloud.com/7803" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/7803</a></p> Duplicate LU-3963 Development Epic client Rank 1|hzvodb: Rank (Obsolete) 7767