https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-11200 Lustre <p>I am trying to bring up arm64 support for centos7 and it turns out centos 7 / rhel 7 for arm64 uses kernel 4.14 that we do not have any ldiskfs patches for.</p> <p>I checked and the closest match is ubuntu18 4.15 kernel - the ldiskfs series has only a single trivial reject in ext4-data-in-dirent.patch</p> <p>We also need a way to select it in configure that's a bit more complicated since I don't know how to do it easily yet.</p> <p>Then there are compile errors:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lockd.c:135:74: error: macro "DEFINE_TIMER" requires 4 arguments, but only 2 given static CFS_DEFINE_TIMER(waiting_locks_timer, waiting_locks_callback, 0, 0); ^ </pre> </div></div> LU-11200

Centos 8 arm64 server support

New Feature Minor Resolved Fixed James A Simmons Oleg Drokin Fri, 3 Aug 2018 02:04:04 +0000 Thu, 11 Jun 2020 08:51:20 +0000 Thu, 11 Jun 2020 08:51:20 +0000 Lustre 2.12.0 Lustre 2.14.0 0 10 <p>It probably makes sense to add a small patch to the 4.14 ldiskfs to make it match Ubuntu to resolve the conflict rather than creating a whole new set of patches. </p> <p>I just forked that one patch and reuse the ubuntu patches otherwise, that's what we typically did in the past.</p> <p>Actually the latest RHEL alt kernels for ARM/Power8 are 4.15 kernels. I was looking at doing this work since I have been assigned ARM server support. I have been running RHEL ARM clients for some time <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>Never mind I was wrong. We are at 4.11. Need to look update our clients kernels.</p> <p>I'm going to take over this ticket since I need to go to a ARM conference in November to present ARM lustre server support.</p> <p>I am fine with it. Hopefully it does not take too long to have the patches to compile current master on rhel7 arm64. I sent you the ldiskfs series update, but it still needs the configure magic of course. - please make it a separate patch from all the other possible compile fixes (the DEFINE_TIMER and whatever else might crop up) so we can enable arm64 builder permanently and make sure it no longer breaks.</p> <p>James Simmons (uja.ornl@yahoo.com) uploaded a new patch: <a href="https://review.whamcloud.com/32939" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/32939</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11200" title="Centos 8 arm64 server support" class="issue-link" data-issue-key="LU-11200"><del>LU-11200</del></a> libcfs: handle DECLARE_TIMER reduced to two arguments<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: fffd922836c7cae6bafd650a41232d845edaf8cf</p> <p>James Simmons (uja.ornl@yahoo.com) uploaded a new patch: <a href="https://review.whamcloud.com/32940" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/32940</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11200" title="Centos 8 arm64 server support" class="issue-link" data-issue-key="LU-11200"><del>LU-11200</del></a> ldiskfs: add support for RHEL7.5 alt kernel<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: f0cf010cadf6a16848d7ec3e02099df6087a0317</p> <p>I have attached to this ticket the lustre version of the e2fsprogs which is the latest in the lustre-master-test test branch.</p> <p>I assume from the e2fsprogs attachments that you didn't have any problems building the <tt>master-lustre-test</tt> branch on ARM?</p> <p>No problem. Same for Power8. Only have trouble with Ubuntu systems. I will track that down.</p> <p>James</p> <p>Is anything needed for clients to work - either on master or on b2_10?</p> <p>Peter</p> <p>Technically I'm testing on a Power8 which is at RHE7.5 alt using the 4.14 kernel. The ARM system I have is in the progress of moving up to RHE7.5 alt. Both platforms use the same kernel. For my testing on Power8 I need the two patches from this ticket as well as the patch from <a href="https://jira.whamcloud.com/browse/LU-11224" title="T10PI assume several kernel features are enabled." class="issue-link" data-issue-key="LU-11224"><del>LU-11224</del></a>. I also have <a href="https://jira.whamcloud.com/browse/LU-11096" title="Lustre doesn&#39;t build if kernel has no blk integrity" class="issue-link" data-issue-key="LU-11096"><del>LU-11096</del></a> which might be needed for ARM. For Power8 its not required. That is for master. For b2_10 support it would be a lot of work. Lots of bits are missing from LNet to make 64K page systems to work.</p> <p>You probably mean something different than <a href="https://jira.whamcloud.com/browse/LU-11124" title="Add &quot;lfs getstripe -N&quot; option to print mirror count" class="issue-link" data-issue-key="LU-11124"><del>LU-11124</del></a> "<tt>Add 'lfs getstripe -N' option to print mirror count</tt>"...</p> <p>Yes a typo. I updated the comment to reflect the proper LU</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/32939/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/32939/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11200" title="Centos 8 arm64 server support" class="issue-link" data-issue-key="LU-11200"><del>LU-11200</del></a> libcfs: handle DECLARE_TIMER reduced to two arguments<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: ac40000d4bde21f807a68cb2add326ea5d77385c</p> <p>Client support for ARM and Power8 restored. Now for server support.</p> <p>James</p> <p>So what (if anything) would be needed on b2_10 to offer ARM/Power8 client support?</p> <p>Peter</p> <p>First we need to port a bunch of patches to support newer kernels. Then figure out what to do with the ko2iblnd stack with 64K and map_on_demand work.</p> <p>Found an issue with RHEL7 alt kernel for my Power8 system. Now I can attempt to mount ldiskfs but I'm encountering:</p> <p><span class="error">&#91;156572.269381&#93;</span> LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodela</p> <p>lloc</p> <p><span class="error">&#91;156794.172421&#93;</span> INFO: task mount.lustre:37664 blocked for more than 120 seconds.</p> <p><span class="error">&#91;156794.172515&#93;</span>       Tainted: P        W  OE  ------------   4.14.0-49.6.1.el7a.ppc64le #1</p> <p><span class="error">&#91;156794.172524&#93;</span> "echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs" disables this message.</p> <p><span class="error">&#91;156794.172556&#93;</span> mount.lustre    D    0 37664  37663 0x00042080</p> <p><span class="error">&#91;156794.172605&#93;</span> Call Trace:</p> <p><span class="error">&#91;156794.172636&#93;</span> <span class="error">&#91;c000001dd7f16ba0&#93;</span> <span class="error">&#91;c00000000001cde0&#93;</span> __switch_to+0x330/0x660</p> <p><span class="error">&#91;156794.172699&#93;</span> <span class="error">&#91;c000001dd7f16c00&#93;</span> <span class="error">&#91;c000000000c5ff64&#93;</span> __schedule+0x354/0xaf0</p> <p><span class="error">&#91;156794.172758&#93;</span> <span class="error">&#91;c000001dd7f16cd0&#93;</span> <span class="error">&#91;c000000000c60748&#93;</span> schedule+0x48/0xc0</p> <p><span class="error">&#91;156794.172817&#93;</span> <span class="error">&#91;c000001dd7f16d00&#93;</span> <span class="error">&#91;c000000000c65e88&#93;</span> rwsem_down_read_failed+0x148/0x1f0</p> <p><span class="error">&#91;156794.172888&#93;</span> <span class="error">&#91;c000001dd7f16d80&#93;</span> <span class="error">&#91;c000000000c65038&#93;</span> down_read+0x78/0x80</p> <p><span class="error">&#91;156794.172964&#93;</span> <span class="error">&#91;c000001dd7f16db0&#93;</span> <span class="error">&#91;d00000000dc31dc4&#93;</span> ldiskfs_readdir+0x704/0xa40 <span class="error">&#91;ldiskfs&#93;</span></p> <p><span class="error">&#91;156794.173046&#93;</span> <span class="error">&#91;c000001dd7f16ed0&#93;</span> <span class="error">&#91;d00000000e94f718&#93;</span> osd_ios_general_scan+0x148/0x350 <span class="error">&#91;osd_ldiskfs&#93;</span></p> <p><span class="error">&#91;156794.173136&#93;</span> <span class="error">&#91;c000001dd7f16fb0&#93;</span> <span class="error">&#91;d00000000e95ab28&#93;</span> osd_initial_OI_scrub+0x178/0x1770 <span class="error">&#91;osd_ldiskfs&#93;</span></p> <p><span class="error">&#91;156794.173226&#93;</span> <span class="error">&#91;c000001dd7f17150&#93;</span> <span class="error">&#91;d00000000e95cdfc&#93;</span> osd_scrub_setup+0x8bc/0x1300 <span class="error">&#91;osd_ldiskfs&#93;</span></p> <p><span class="error">&#91;156794.173314&#93;</span> <span class="error">&#91;c000001dd7f172e0&#93;</span> <span class="error">&#91;d00000000e923b18&#93;</span> osd_device_alloc+0x6e8/0xa90 <span class="error">&#91;osd_ldiskfs&#93;</span></p> <p><span class="error">&#91;156794.173424&#93;</span> <span class="error">&#91;c000001dd7f173c0&#93;</span> <span class="error">&#91;d00000002821e478&#93;</span> class_setup+0xaf8/0x10d0 <span class="error">&#91;obdclass&#93;</span></p> <p><span class="error">&#91;156794.173516&#93;</span> <span class="error">&#91;c000001dd7f17510&#93;</span> <span class="error">&#91;d000000028229924&#93;</span> class_process_config+0x1d64/0x3980 <span class="error">&#91;obdclass&#93;</span></p> <p><span class="error">&#91;156794.173620&#93;</span> <span class="error">&#91;c000001dd7f17640&#93;</span> <span class="error">&#91;d000000028232dc8&#93;</span> do_lcfg+0x358/0x890 <span class="error">&#91;obdclass&#93;</span></p> <p><span class="error">&#91;156794.173713&#93;</span> <span class="error">&#91;c000001dd7f17770&#93;</span> <span class="error">&#91;d000000028237af4&#93;</span> lustre_start_simple+0x1d4/0x460 <span class="error">&#91;obdclass&#93;</span></p> <p><span class="error">&#91;156794.173817&#93;</span> <span class="error">&#91;c000001dd7f17840&#93;</span> <span class="error">&#91;d000000028279404&#93;</span> osd_start+0x714/0xd40 <span class="error">&#91;obdclass&#93;</span></p> <p><span class="error">&#91;156794.173911&#93;</span> <span class="error">&#91;c000001dd7f17960&#93;</span> <span class="error">&#91;d000000028287508&#93;</span> server_fill_super+0x268/0x1cc0 <span class="error">&#91;obdclass&#93;</span></p> <p><span class="error">&#91;156794.174015&#93;</span> <span class="error">&#91;c000001dd7f17a60&#93;</span> <span class="error">&#91;d00000002823cb40&#93;</span> lustre_fill_super+0x7c0/0x1050 <span class="error">&#91;obdclass&#93;</span></p> <p><span class="error">&#91;156794.174098&#93;</span> <span class="error">&#91;c000001dd7f17b20&#93;</span> <span class="error">&#91;c0000000004494d0&#93;</span> mount_nodev+0x160/0x390</p> <p><span class="error">&#91;156794.174178&#93;</span> <span class="error">&#91;c000001dd7f17b90&#93;</span> <span class="error">&#91;d000000028234ad4&#93;</span> lustre_mount+0x54/0x70 <span class="error">&#91;obdclass&#93;</span></p> <p><span class="error">&#91;156794.174249&#93;</span> <span class="error">&#91;c000001dd7f17be0&#93;</span> <span class="error">&#91;c00000000044b97c&#93;</span> mount_fs+0x8c/0x230</p> <p><span class="error">&#91;156794.174309&#93;</span> <span class="error">&#91;c000001dd7f17c80&#93;</span> <span class="error">&#91;c000000000487978&#93;</span> vfs_kern_mount+0x78/0x1b0</p> <p> </p> <p>Hi James, <br/> I've tested your latest patch series (patch set 8), got crash. Please see attached vmcore-dmesg.txt (<a href="https://jira.whamcloud.com/secure/attachment/32378/vmcore-dmesg.txt" class="external-link" rel="nofollow">https://jira.whamcloud.com/secure/attachment/32378/vmcore-dmesg.txt</a>) it was kernel-alt on x86_64 system though.<br/> I will try patch on arm system too.</p> <p>I think I resolved the locking issues for ldiskfs. The problem was try_lookup_one_len() can return NULL. In the original ldiskfs scrub code the dentry was then allocated in that case. So if try_lookup_one_len() doesn't find a dentry we have to create one.</p> <p>Shuichi Ihara try both:</p> <p><a href="https://review.whamcloud.com/#/c/34714" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/34714</a></p> <p><a href="https://review.whamcloud.com/#/c/32940" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/32940</a></p> <p>Hi,</p> <p>Thank you for your patches and efforts !</p> <p>I've tested the latest series on an ARM64 machine (VM hosted on a ThunderX2 with Infiniband to be precise).<br/> First off, when using the 'vanilla' CentOS distributed 4.14.0-49.10.1.el7a.aarch64 kernel, it seems ldiskfs's kmod complains about missing 'fscrypt' symbols.<br/> Thus I recompiled the kernel enabling/forcing EXT4_ENCRYPTION, with that I can load ldiskfs's kmods fine.<br/> However, when running llmount.sh (naively, no modifications to any config file and using LDISKFS_MKFS_OPTS="^metadata_csum" because of "LDISKFS-fs (loop0): Couldn't mount because of unsupported optional features (400)") I get :</p> <p><span class="error">&#91;bgerdeb@lustretest ~&#93;</span>$ sudo FSTYPE=ldiskfs LDISKFS_MKFS_OPTS="^metadata_csum" /usr/lib64/lustre/tests/llmount.sh <br/> Stopping clients: lustretest /mnt/lustre (opts:-f)<br/> Stopping clients: lustretest /mnt/lustre2 (opts:-f)<br/> lustretest: executing set_hostid<br/> Loading modules from /usr/lib64/lustre/tests/..<br/> detected 56 online CPUs by sysfs<br/> libcfs will create CPU partition based on online CPUs<br/> Formatting mgs, mds, osts<br/> Format mds1: /tmp/lustre-mdt1<br/> Failed to initialize ZFS library: 256</p> <p>mkfs.lustre FATAL: Unable to build fs /dev/loop0 (256)</p> <p>mkfs.lustre FATAL: mkfs failed 256</p> <p>Furthermore, dmesg shows :</p> <p>[ 164.229457] libcfs: loading out-of-tree module taints kernel.<br/> [ 164.231426] libcfs: module verification failed: signature and/or required key missing - tainting kernel<br/> [ 164.239068] LNet: HW NUMA nodes: 1, HW CPU cores: 56, npartitions: 28<br/> [ 164.244595] alg: No test for adler32 (adler32-zlib)<br/> [ 165.001287] Lustre: DEBUG MARKER: lustretest: executing set_hostid<br/> [ 165.140092] Lustre: Lustre: Build Version: 2.12.54<br/> [ 165.210256] LNet: Added LNI 10.40.16.14@tcp <span class="error">&#91;8/1792/0/180&#93;</span><br/> [ 165.211486] LNet: Accept secure, port 988<br/> [ 166.796493] Lustre: 6332:0:(gss_svc_upcall.c:1119:gss_init_svc_upcall()) Init channel is not opened by lsvcgssd, following request might be dropped until lsvcgssd is active<br/> [ 166.799447] Key type lgssc registered<br/> [ 166.949177] Lustre: Echo OBD driver; <a href="http://www.lustre.org/" class="external-link" target="_blank" rel="nofollow noopener">http://www.lustre.org/</a><br/> [ 177.165649] loop: module loaded<br/> <span class="error">&#91;...&#93;</span><br/> [ 4728.518978] Lustre: DEBUG MARKER: lustretest: executing set_hostid<br/> [ 4738.611605] print_req_error: I/O error, dev loop0, sector 0<br/> [ 4738.617781] print_req_error: I/O error, dev loop0, sector 0<br/> [ 4738.623807] print_req_error: I/O error, dev loop0, sector 0<br/> [ 4738.629806] print_req_error: I/O error, dev loop0, sector 0<br/> [ 4738.636358] print_req_error: I/O error, dev loop0, sector 0<br/> [ 4738.642873] print_req_error: I/O error, dev loop0, sector 0<br/> [ 4759.617658] print_req_error: I/O error, dev loop0, sector 0<br/> [ 4759.650171] print_req_error: I/O error, dev loop0, sector 0<br/> [ 4759.655604] print_req_error: I/O error, dev loop0, sector 0<br/> [ 4759.661274] print_req_error: I/O error, dev loop0, sector 0<br/> [ 4759.666946] print_req_error: I/O error, dev loop0, sector 0<br/> [ 4759.672655] print_req_error: I/O error, dev loop0, sector 0</p> <p>Those last 'print_req_error' seem to be inherent to this precise version of the kernel/EXT4 drivers (Ubuntu's bug tracker is full of those errors).<br/> I also encountered them when running llmount.sh with FSTYPE=zfs on this kernel (4.14.0-49.10.1.el7a.aarch64, both vanilla and my builds) crashing llmount.sh</p> <p>When compiling Lustre (master + both patches cherry picked) against the latest CentOS 7.6 aarch64 kernel (yesterday's 4.14.0-115.8.1.el7a.aarch64, as well as 4.14.0-115.7.1.el7a.aarch64) with <b>only</b> ZFS enabled, I can llmount.sh just fine.<br/> Note that I haven't run much of auster on this (I need to migrate the VM, not enough space left...), but I'm not getting <b>any</b> print_req_error.</p> <p>metadata_csum is new. As for the loop device you might need the new e2fsprogs being worked on for RHEL8. In fact give me some time and I will build you updated e2fsprogs for you. I do plan on updating the <a href="https://jira.whamcloud.com/browse/LU-11200" title="Centos 8 arm64 server support" class="issue-link" data-issue-key="LU-11200"><del>LU-11200</del></a> patch due to RHEL8 changes as well. </p> <p>Thanks a lot ! Looking forward to trying out those patches !</p> <p>Updated the e2fsprogs. I hope to have the RHEL7.6alt patches ready over the next few days.</p> <p>Thanks ! Downloaded and reinstalled your e2fsprogs over mine, tested and got the same result as above.<br/> I might redo a clean build and install later with your e2fsprogs just to make sure.<br/> Looking forward to the 7.6alt patches !</p> <p>Just updated the ldiskfs patch. Currently we are running 4.14.43-200 and that kernel is nearly identical to the Ubuntu 18 LTS kernel so its a really easy update. That patch plus the <a href="https://jira.whamcloud.com/browse/LU-12137" title="update client to use iterate_shared" class="issue-link" data-issue-key="LU-12137"><del>LU-12137</del></a> patch should get you mounting a file system. The only issue is that <a href="https://jira.whamcloud.com/browse/LU-12137" title="update client to use iterate_shared" class="issue-link" data-issue-key="LU-12137"><del>LU-12137</del></a> isn't complete on some recovery points but you should be able to mount ldiskfs on ARM.</p> <p>I can't seem to find a 4.14.43-200 kernel (for aarch64). The closest I have is <a href="https://koji.fedoraproject.org/koji/buildinfo?buildID=1015632" class="external-link" target="_blank" rel="nofollow noopener">Fedora's 4.14.13-200</a>. Is it that one ?</p> <p>I saw 4.14.43-200 kernels for Fedora x86 but I can't seem to find one for ARM. The kernel I'm using is directly from the vendor using RedHat. Can you try to see if you have build issues with this version? I do see the latest fedora is 30 for ARM and it is a 5.0 kernel which is being worked on <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>So I tried with the aforementioned kernel : the good news is that I only had to do minor changes to your rhel7.5alt patch (see : <a href="https://jira.whamcloud.com/secure/attachment/32821/patch_to_patches_ldiskfsset20.txt" class="external-link" rel="nofollow">this file</a>), which I already had to do before but failed to mention it seems...) for it to apply to this kernel's EXT4 tree.</p> <p>The bad news is that it fails to compile with GCC 5.4.0 where it succeeded before; here are the errors :</p> <p>In file included from include/linux/kernel.h:10:0,<br/> from include/linux/list.h:9,<br/> from include/linux/wait.h:7,<br/> from include/linux/wait_bit.h:8,<br/> from include/linux/fs.h:6,<br/> from /home/bgerdeb/lustre_build/lustre/lustrearm/ldiskfs/dir.c:25:<br/> /home/bgerdeb/lustre_build/lustre/lustrearm/ldiskfs/dir.c: In function ‘__ldiskfs_check_dir_entry’:<br/> */home/bgerdeb/lustre_build/lustre/lustrearm/ldiskfs/dir.c:71:22: error: implicit declaration of function ‘__LDISKFS_DIR_REC_LEN’ <span class="error">&#91;-Werror=implicit-function-declaration&#93;</span><br/> if (unlikely(rlen &lt; __LDISKFS_DIR_REC_LEN(1)))*<br/> ^<br/> include/linux/compiler.h:77:42: note: in definition of macro ‘unlikely’</p> <ol> <li>define unlikely<img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/error.png" height="16" width="16" align="absmiddle" alt="" border="0"/> __builtin_expect(!!<img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/error.png" height="16" width="16" align="absmiddle" alt="" border="0"/>, 0)<br/> ^<br/> */home/bgerdeb/lustre_build/lustre/lustrearm/ldiskfs/ldiskfs.h:2031:77: error: invalid operands to binary &amp; (have ‘struct ldiskfs_dir_entry_2 *’ and ‘int’)<br/> #define LDISKFS_DIR_REC_LEN(name_len) (((name_len) + 8 + LDISKFS_DIR_ROUND) &amp; \<br/> ^*<br/> include/linux/compiler.h:77:42: note: in definition of macro ‘unlikely’</li> <li>define unlikely<img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/error.png" height="16" width="16" align="absmiddle" alt="" border="0"/> __builtin_expect(!!<img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/error.png" height="16" width="16" align="absmiddle" alt="" border="0"/>, 0)<br/> ^<br/> /home/bgerdeb/lustre_build/lustre/lustrearm/ldiskfs/dir.c:75:27: note: in expansion of macro ‘LDISKFS_DIR_REC_LEN’<br/> else if (unlikely(rlen &lt; LDISKFS_DIR_REC_LEN(de)))<br/> ^<br/> */home/bgerdeb/lustre_build/lustre/lustrearm/ldiskfs/dir.c: In function ‘ldiskfs_htree_store_dirent’:<br/> /home/bgerdeb/lustre_build/lustre/lustrearm/ldiskfs/dir.c:451:26: error: ‘LDISKFS_DIRENT_LUFID’ undeclared (first use in this function)<br/> if (dirent-&gt;file_type &amp; LDISKFS_DIRENT_LUFID)<br/> ^*<br/> /home/bgerdeb/lustre_build/lustre/lustrearm/ldiskfs/dir.c:451:26: note: each undeclared identifier is reported only once for each function it appears in<br/> */home/bgerdeb/lustre_build/lustre/lustrearm/ldiskfs/dir.c:452:16: error: implicit declaration of function ‘ldiskfs_get_dirent_data_len’ <span class="error">&#91;-Werror=implicit-function-declaration&#93;</span><br/> extra_data = ldiskfs_get_dirent_data_len(dirent);<br/> ^*<br/> /home/bgerdeb/lustre_build/lustre/lustrearm/ldiskfs/dir.c: At top level:<br/> cc1: warning: unrecognized command line option ‘-Wno-stringop-overflow’<br/> cc1: warning: unrecognized command line option ‘-Wno-stringop-truncation’<br/> cc1: warning: unrecognized command line option ‘-Wno-format-truncation’</li> </ol> <p><span class="error">&#91;... make -j12 so it goes on ..&#93;</span></p> <p>/home/bgerdeb/lustre_build/lustre/lustrearm/lustre/osd-ldiskfs/osd_handler.c:3695:20: warning: ‘struct ldiskfs_dentry_param’ declared inside parameter list<br/> const struct lu_fid *fid)<br/> ^<br/> /home/bgerdeb/lustre_build/lustre/lustrearm/lustre/osd-ldiskfs/osd_handler.c:3695:20: warning: its scope is only this definition or declaration, which is probably not what you want<br/> */home/bgerdeb/lustre_build/lustre/lustrearm/lustre/osd-ldiskfs/osd_handler.c: In function ‘osd_get_ldiskfs_dirent_param’:<br/> /home/bgerdeb/lustre_build/lustre/lustrearm/lustre/osd-ldiskfs/osd_handler.c:3699:8: error: dereferencing pointer to incomplete type ‘struct ldiskfs_dentry_param’<br/> param-&gt;edp_magic = 0;<br/> ^*<br/> */home/bgerdeb/lustre_build/lustre/lustrearm/lustre/osd-ldiskfs/osd_handler.c:3703:21: error: ‘LDISKFS_LUFID_MAGIC’ undeclared (first use in this function)<br/> param-&gt;edp_magic = LDISKFS_LUFID_MAGIC;<br/> ^*<br/> /home/bgerdeb/lustre_build/lustre/lustrearm/lustre/osd-ldiskfs/osd_handler.c:3703:21: note: each undeclared identifier is reported only once for each function it appears in<br/> */home/bgerdeb/lustre_build/lustre/lustrearm/lustre/osd-ldiskfs/osd_handler.c: In function ‘osd_add_dot_dotdot_internal’:<br/> /home/bgerdeb/lustre_build/lustre/lustrearm/lustre/osd-ldiskfs/osd_handler.c:3746:31: error: passing argument 1 of ‘osd_get_ldiskfs_dirent_param’ from incompatible pointer type <span class="error">&#91;-Werror=incompatible-pointer-types&#93;</span><br/> osd_get_ldiskfs_dirent_param(dot_dot_ldp, dot_dot_fid);<br/> ^*<br/> /home/bgerdeb/lustre_build/lustre/lustrearm/lustre/osd-ldiskfs/osd_handler.c:3694:13: note: expected ‘struct ldiskfs_dentry_param *’ but argument is of type ‘struct ldiskfs_dentry_param *’<br/> static void osd_get_ldiskfs_dirent_param(struct ldiskfs_dentry_param *param,<br/> ^<br/> */home/bgerdeb/lustre_build/lustre/lustrearm/lustre/osd-ldiskfs/osd_handler.c:3749:9: error: dereferencing pointer to incomplete type ‘struct ldiskfs_dentry_param’<br/> dot_ldp-&gt;edp_magic = 0;<br/> ^<br/> /home/bgerdeb/lustre_build/lustre/lustrearm/lustre/osd-ldiskfs/osd_handler.c:3751:7: error: implicit declaration of function ‘ldiskfs_add_dot_dotdot’ <span class="error">&#91;-Werror=implicit-function-declaration&#93;</span><br/> rc = ldiskfs_add_dot_dotdot(oth-&gt;ot_handle, parent_dir,<br/> ^*<br/> */home/bgerdeb/lustre_build/lustre/lustrearm/lustre/osd-ldiskfs/osd_handler.c: In function ‘osd_get_fid_from_dentry’:<br/> /home/bgerdeb/lustre_build/lustre/lustrearm/lustre/osd-ldiskfs/osd_handler.c:5074:22: error: ‘LDISKFS_DIRENT_LUFID’ undeclared (first use in this function)<br/> if (de-&gt;file_type &amp; LDISKFS_DIRENT_LUFID) </p> { ^* */home/bgerdeb/lustre_build/lustre/lustrearm/lustre/osd-ldiskfs/osd_handler.c: In function ‘__osd_ea_add_rec’: /home/bgerdeb/lustre_build/lustre/lustrearm/lustre/osd-ldiskfs/osd_handler.c:5407:6: error: dereferencing pointer to incomplete type ‘struct ldiskfs_dentry_param’ ldp-&gt;edp_magic = 0; ^* */home/bgerdeb/lustre_build/lustre/lustrearm/lustre/osd-ldiskfs/osd_handler.c:5409:32: error: passing argument 1 of ‘osd_get_ldiskfs_dirent_param’ from incompatible pointer type [-Werror=incompatible-pointer-types] osd_get_ldiskfs_dirent_param(ldp, fid); ^* /home/bgerdeb/lustre_build/lustre/lustrearm/lustre/osd-ldiskfs/osd_handler.c:3694:13: note: expected ‘struct ldiskfs_dentry_param *’ but argument is of type ‘struct ldiskfs_dentry_param *’ static void osd_get_ldiskfs_dirent_param(struct ldiskfs_dentry_param *param, ^ */home/bgerdeb/lustre_build/lustre/lustrearm/lustre/osd-ldiskfs/osd_handler.c:5427:22: error: ‘LDISKFS_DIRENT_LUFID’ undeclared (first use in this function) de-&gt;file_type = LDISKFS_DIRENT_LUFID | ^* */home/bgerdeb/lustre_build/lustre/lustrearm/lustre/osd-ldiskfs/osd_handler.c: In function ‘osd_ldiskfs_filldir’: /home/bgerdeb/lustre_build/lustre/lustrearm/lustre/osd-ldiskfs/osd_handler.c:6550:22: error: ‘LDISKFS_DIRENT_LUFID’ undeclared (first use in this function) } <p> else if (d_type &amp; LDISKFS_DIRENT_LUFID) {<br/> ^*<br/> *In file included from /home/bgerdeb/lustre_build/lustre/lustrearm/lustre/osd-ldiskfs/osd_handler.c:56:0:<br/> /home/bgerdeb/lustre_build/lustre/lustrearm/lustre/osd-ldiskfs/osd_handler.c: In function ‘osd_dotdot_has_space’:<br/> /home/bgerdeb/lustre_build/lustre/lustrearm/ldiskfs/ldiskfs.h:2031:77: error: invalid operands to binary &amp; (have ‘struct ldiskfs_dir_entry_2 *’ and ‘int’)<br/> #define LDISKFS_DIR_REC_LEN(name_len) (((name_len) + 8 + LDISKFS_DIR_ROUND) &amp; \<br/> ^*<br/> /home/bgerdeb/lustre_build/lustre/lustrearm/lustre/osd-ldiskfs/osd_handler.c:6711:6: note: in expansion of macro ‘LDISKFS_DIR_REC_LEN’<br/> if (LDISKFS_DIR_REC_LEN(de) &gt;=<br/> ^<br/> */home/bgerdeb/lustre_build/lustre/lustrearm/lustre/osd-ldiskfs/osd_handler.c:6712:6: error: implicit declaration of function ‘__LDISKFS_DIR_REC_LEN’ <span class="error">&#91;-Werror=implicit-function-declaration&#93;</span><br/> __LDISKFS_DIR_REC_LEN(2 + 1 + sizeof(struct osd_fid_pack)))<br/> ^*<br/> */home/bgerdeb/lustre_build/lustre/lustrearm/lustre/osd-ldiskfs/osd_handler.c: In function ‘osd_dirent_reinsert’:<br/> /home/bgerdeb/lustre_build/lustre/lustrearm/lustre/osd-ldiskfs/osd_handler.c:6762:20: error: ‘LDISKFS_DIRENT_LUFID’ undeclared (first use in this function)<br/> de-&gt;file_type |= LDISKFS_DIRENT_LUFID;<br/> ^*<br/> */home/bgerdeb/lustre_build/lustre/lustrearm/lustre/osd-ldiskfs/osd_handler.c:6775:31: error: passing argument 1 of ‘osd_get_ldiskfs_dirent_param’ from incompatible pointer type <span class="error">&#91;-Werror=incompatible-pointer-types&#93;</span><br/> osd_get_ldiskfs_dirent_param(ldp, fid);<br/> ^*<br/> /home/bgerdeb/lustre_build/lustre/lustrearm/lustre/osd-ldiskfs/osd_handler.c:3694:13: note: expected ‘struct ldiskfs_dentry_param *’ but argument is of type ‘struct ldiskfs_dentry_param *’<br/> static void osd_get_ldiskfs_dirent_param(struct ldiskfs_dentry_param *param,<br/> ^<br/> CC <span class="error">&#91;M&#93;</span> /home/bgerdeb/lustre_build/lustre/lustrearm/lustre/obdclass/class_obd.o<br/> /home/bgerdeb/lustre_build/lustre/lustrearm/lustre/osd-ldiskfs/osd_handler.c: At top level:<br/> cc1: warning: unrecognized command line option ‘-Wno-stringop-overflow’<br/> cc1: warning: unrecognized command line option ‘-Wno-stringop-truncation’<br/> cc1: warning: unrecognized command line option ‘-Wno-format-truncation’</p> <p><b>Note, I always cherry pick your patches on top of master. Will rebase and retry.</b></p> <p>EDIT : Tried to put in bold the errors, of course thanks to Atlassian's wonderful text editor, it failed.</p> <p>It seems Fedora is a different beast then RHEL7.6alt. Would you be willing to try Fedora30 since its a 5.0 kernel and that support is being worked on. I'm hoping we can just use the same ldiskfs patch set as the 5.0 series.</p> <p>Talking to Marvell at ISC this week and they seemed to think that focusing on RHEL8 should be the first priority in this arena.</p> <p>Are you suggesting we drop RHEL7 alt support?  Why that seems like a good idea people might be stuck with what their vendor supports <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/sad.png" height="16" width="16" align="absmiddle" alt="" border="0"/> I think their might be a delay for some time before RHEL8 is picked up ARM vendors. For ORNL we will be getting new ARM servers in soon and one set will be RHEL7 alt and the second set will be RHEL8. Thankfully other vendors haven't made the mistake of rolling their own special ARM only kernels.</p> <p>I am working to the assumption that it will take a little while to get Arm server support working so focusing on something with a longer shelf life would be best</p> <p>We are much closer than you think. The challenge has been getting people to test the needed changes. Now that I have worked out most of the issues for <a href="https://jira.whamcloud.com/browse/LU-11893" title="doesn&#39;t handle logical network interface properly." class="issue-link" data-issue-key="LU-11893"><del>LU-11893</del></a> I can focus on finishing this. I talked to Shaun from Cray and he is willing to work with me to finishing this work up.</p> <p>I have resolved the ldiskfs locking issues that ARM testers have reported. Should be in a good position to finish this work off.</p> <p>I have tested your patches against Fedora 30's 5.1.1 kernel, and the patches do not apply to the ext4.<br/> I have also tested CentOS's <a href="https://mirrors.tripadvisor.com/centos-vault/altarch/7.5.1804/kernel/Source/SPackages/kernel-4.14.43-201.el7.src.rpm" class="external-link" target="_blank" rel="nofollow noopener">4.14.43-201</a> and the ialloc.c differs non trivially from what "rhel7.6alt/ext4-corrupted-inode-block-bitmaps-handling-patches.patch" expects.</p> <p>I'd advise to use the upstream CentOS 7.6alt kernel : <a href="http://vault.centos.org/centos/7/os/Source/SPackages/kernel-alt-4.14.0-115.el7a.0.1.src.rpm" class="external-link" target="_blank" rel="nofollow noopener">here</a> which is the one you get doing a netinstall by default (if you don't run yum update afterwards).<br/> The next to latest one is this one : <a href="http://vault.centos.org/centos/7/updates/Source/SPackages/kernel-alt-4.14.0-115.7.1.el7a.src.rpm" class="external-link" target="_blank" rel="nofollow noopener">here</a><br/> For the latest one, kernel-alt-4.14.0-115.8.1.el7a, I can't seem to find the src rpm but <a href="https://www.redhat.com/archives/rhsa-announce/2019-June/msg00007.html" class="external-link" target="_blank" rel="nofollow noopener">here is the changelog</a> : it doesn't seem to include any changes to EXT4 (only XFS).<br/> I'll try to get a diff of 7.1 and 8.1's ext4 to make sure.</p> <p>Concerning 7.6 vs RHEL8, I'm happy to test both.<br/> Since the timeline to CentOS 8 is not clear, 7.6alt is still (the most) relevant (and since I haven't tested CentOS 8, there is no assurance it is as stable as 7.6alt yet). </p> <p>I have tested out the latest lustre-release master on RHEL8 ARM64 VM and can confirm that it builds installs, runs and passes (most of) the sanity test suite with LDISKFS (in all-on-one-node configuration at the moment) !</p> <p>Here are the results : <span class="nobr"><a href="https://jira.whamcloud.com/secure/attachment/33271/33271_results-ldiskfs-rhel8-2507.yml" title="results-ldiskfs-rhel8-2507.yml attached to LU-11200">results-ldiskfs-rhel8-2507.yml^{<img class="rendericon" src="https://jira.whamcloud.com/images/icons/link_attachment_7.gif" height="7" width="7" align="absmiddle" alt="" border="0"/>}</a></span> !<br/> Will investigate further tomorrow, try to isolate the FAILS.</p> <p>Baptiste I updated <a href="https://review.whamcloud.com/#/c/34714/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/34714</a> This should make RHLE8 ARM fully functional.</p> <p>Thanks a lot !<br/> Will take a look at it this week and report back (here I guess ? Let me know if I should open an issue/if there is a more "à propos" issue elsewhere)</p> <p>The last patch for this ticket was landed in 2.14, and RHEL8 clients are working for all arches.</p> Blocker LU-12323 Related LU-11527 LU-6387 LU-6766 LU-11224 LU-11832 LU-9793 LU-11878 LU-11246 LU-11440 LU-12137 LU-10300 Development Rank 1|i0007j: Rank (Obsolete) 9223372036854775807