https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-4199 Lustre <p>We are interested in Lustre on ARM based processor platform. It runs on standard Linux kernel + specific patches today. This encourages Lustre runs on ARM based platform. So, we have ported few Lustre codes to adapt ARM processor and Lustre client runs on it.</p> ARM compatibility processor LU-4199

Lustre client support for ARM platform

New Feature Minor Resolved Fixed James A Simmons Shuichi Ihara patch Fri, 1 Nov 2013 23:06:36 +0000 Sat, 11 Mar 2017 05:10:24 +0000 Sat, 11 Mar 2017 05:10:24 +0000 Lustre 2.6.0 0 15 <p>Sounds interesting. A couple of people were asking about ARM support in Lustre at SC last year. Do you have anything ready to contribute yet?</p> <p>Peter, Yes, we have patch and tested on tiny ARM board. We will push patch here very soon.</p> <p>Here is the patch.<br/> <a href="http://review.whamcloud.com/8144" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/8144</a><br/> But there is problem. Folloing newly merged patch removes LIBCFS_FUNC_DUMP_TRACE test since dump_trace() function is always defined on X86 platform. However, there is no dump_trace() function for ARM platform (e.g. Linux-3.6). We are trying to figure out what is the best way to fix this problem.<br/> <a href="http://git.whamcloud.com/?p=fs/lustre-release.git;a=commit;h=8250f49e3d0431db6a6363f959d2cce65684c74e" class="external-link" target="_blank" rel="nofollow noopener">http://git.whamcloud.com/?p=fs/lustre-release.git;a=commit;h=8250f49e3d0431db6a6363f959d2cce65684c74e</a></p> <p>We have built Lustre codes (both kernel modules and user space utilities) with <a href="http://review.whamcloud.com/8144" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/8144</a> using Lustre cross build environment.<br/> And built Lustre kernel modules and utilizes have been tested on Raspberry PI <a href="http://www.raspberrypi.org/" class="external-link" target="_blank" rel="nofollow noopener">http://www.raspberrypi.org/</a> which is ARMv6 compatibility processor based board. We are still investigating pcakging (.e.g. RPM, deb) for ARM based Linux distribution. The following steps are still a little bit complicated, but it works well.</p> <p>Host system (x86_64, CentOS6.4)</p> <p>The kernel re-compile is NOT required for the lustre and it's NOT part of Lustre. Howerver, the latest kenrel source tree has some updates from original pre-installed kernel. I did re-compile and installed before Lustre build.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Creating "dummy root directory" to transfer modules and binaries for Rasberry Pi from host system # mkdir -p /usr/src/dummyroot/boot Setup Cross build environment and re-compile kernel with them # cd /usr/src/ # git clone --depth 5 https://github.com/raspberrypi/tools.git # export PATH=/usr/src/tools/arm-bcm2708/arm-bcm2708-linux-gnueabi/bin:$PATH # git clone -b rpi-3.6.y --depth 5 git://github.com/raspberrypi/linux.git rpi-3.6.y # cd rpi-3.6.y # cp arch/arm/configs/bcmrpi_defconfig .config # yes "" | make ARCH=arm CROSS_COMPILE=arm-bcm2708-linux-gnueabi- oldconfig # make ARCH=arm CROSS_COMPILE=arm-bcm2708-linux-gnueabi- Install kernel images and modules to "dummy root" # make ARCH=arm CROSS_COMPILE=arm-bcm2708-linux-gnueabi- \ INSTALL_PATH=/usr/src/dummyroot/boot install # make ARCH=arm CROSS_COMPILE=arm-bcm2708-linux-gnueabi- INSTALL_MOD_PATH=/usr/src/dummyroot </pre> </div></div> <p>Lustre build against rpi-3.6.y linux source tree</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Checkout LU-4199 patches and build Lustre with Cross compile option (compiled modules and binaries are installed to dummy root directory as well) # sh ./autogen.sh # ./configure --with-linux=/usr/src/rpi-3.6.y/ --without-o2ib \ --host=arm-bcm2708-linux-gnueabi --prefix=/usr/src/dummyroot # make ARCH=arm CROSS_COMPILE=arm-bcm2708-linux-gnueabi- CROSS_PATH=/usr/src/dummyroot # make ARCH=arm CROSS_COMPILE=arm-bcm2708-linux-gnueabi- CROSS_PATH=/usr/src/dummyroot install Export "dummy root" directory for Raspberry Pi # exportfs *:/usr/src/dummyroot </pre> </div></div> <p>Target system(ARM, Raspberry Pi, Raspbian)</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Boot Raspberry Pi and login the system, then mount host system's dummy root directory via NFS. # mount &lt;host system's IP address&gt;:/usr/src/dummyroot /mnt Install kernel image, modules (include Lustre) and Lustre user utility # cp /mnt/boot/vmlinuz-3.6.11+ /boot/kernel.img # rsync -av --exclude=/boot /mnt/ / # depmod -a 3.6.11+ # sync; reboot </pre> </div></div> <p>Now, it can mounts the lustre from Rasberry PI board</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>root@raspberrypi:~# mount -t lustre 192.168.1.27@tcp:/lustre /lustre root@raspberrypi:~# df -t lustre Filesystem 1K-blocks Used Available Use% Mounted on 192.168.1.27@tcp:/lustre 984248 38024 896224 5% /lustre </pre> </div></div> <p>We can add back in testing for dump_trace again. The test was broken before since it only tested on x86 platforms anyways. Looking at the latest kernel code only x86 and parisc has dump_trace.</p> <p>Looked at the code more closely and it appears dump_trace is x86 specific. The parisc platform only uses it internally.</p> <p>thanks for checking, so "#ifdef CONFIG_X86" would be fine. we added this in linux-debug.c.</p> <p>I seen in your patches you have implemented UMP versions of cfs_cpu_ht_nsiblings and cfs_cpt_table_print. What does cat /proc/sys/lnet/cpu_partition_table look like for you with the default function?</p> <p>Hi James,</p> <p>Thanks for your review! The default cfs_cpu_ht_nsiblings and cfs_cpt_table_print only work for SMP. Since the uniprocessor version of "struct cfs_cpt_table" does not have necccessary fields, I am afraid it is impossible to use default cfs_cpt_table_print() for uniprocessor systems. We had to add new versions of those functions, otherwise the compiling will fail.</p> <p>Thanks!</p> <p>Oh I missed that struct cfs_cpt_table is different for the UMP case. I also see the dependency of HAVE_LIBCFS_CPT on CONFIG_SMP. I will changed my review of your patch.</p> <p>James, Thanks for checking.</p> <p>No problem. I also included Alexey on the review to make sure we don't break his Darwin port.</p> <p>Patch <a href="http://review.whamcloud.com/#/c/8144" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/8144</a> is ready for inspect and possible landing.</p> <p>I've splitted the patch into multiple patches.<br/> 1. <a href="https://jira.whamcloud.com/browse/LU-4199" title="Lustre client support for ARM platform" class="issue-link" data-issue-key="LU-4199"><del>LU-4199</del></a> libcfs: remove assertion of spin_is_locked()<br/> <a href="http://review.whamcloud.com/#/c/8144" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/8144</a><br/> The newly merged nrs_tbf.c is updated.<br/> 2. <a href="https://jira.whamcloud.com/browse/LU-4199" title="Lustre client support for ARM platform" class="issue-link" data-issue-key="LU-4199"><del>LU-4199</del></a> libcfs: add wrapper of dump_trace()<br/> <a href="http://review.whamcloud.com/#/c/8872" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/8872</a><br/> 3. <a href="https://jira.whamcloud.com/browse/LU-4199" title="Lustre client support for ARM platform" class="issue-link" data-issue-key="LU-4199"><del>LU-4199</del></a> libcfs: add CPU table functions for uniprocessor<br/> <a href="http://review.whamcloud.com/#/c/8873" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/8873</a><br/> 4. <a href="https://jira.whamcloud.com/browse/LU-4199" title="Lustre client support for ARM platform" class="issue-link" data-issue-key="LU-4199"><del>LU-4199</del></a> libcfs: add wrapper of PAGE_SIZE<br/> <a href="http://review.whamcloud.com/#/c/8877" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/8877</a><br/> This is a new problem when compiling Lustre on Raspberry Pi. Compilation of userspace utils fails becuasee PAGE_SIZE is missing. It seems we should not use PAGE_SIZE from userspace because page.h will not be exported to usespace in the future.<br/> 5. <a href="https://jira.whamcloud.com/browse/LU-4199" title="Lustre client support for ARM platform" class="issue-link" data-issue-key="LU-4199"><del>LU-4199</del></a> build: add ARM support in build system<br/> <a href="http://review.whamcloud.com/8878" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/8878</a><br/> Oleg has given some advices on simpling the script of lustre-build-linux.m4. However, I don't have MPSS or k1om test environment, so it seems easy for me to break the script when building on those architectures. Any idea about how to improve it?</p> <p> We used to hit following BUGs when compiling, but it is gone now.<br/> /home/lixi/lustre/lustre-release.git/lustre/lov/lov_pack.c: In function 'lov_getstripe':<br/> /home/lixi/lustre/lustre-release.git/lustre/lov/lov_pack.c:632:9: error: duplicate case value<br/> /home/lixi/lustre/lustre-release.git/lustre/lov/lov_pack.c:632:9: error: previously used here</p> <p>The kernel crashed when running "ls" on Lustre client of Raspberry Pi. lmv_read_entry() hit "Unable to handle kernel NULL pointer dereference at virtual address 00000003". We did not see that problem when the version is 2.5.50. We are trying to figure out since which version the problem starts.</p> <p>It is confirmed that 4e0c8aeb94 (<a href="https://jira.whamcloud.com/browse/LU-3531" title="DNE2: striped directory" class="issue-link" data-issue-key="LU-3531"><del>LU-3531</del></a> llite: move dir cache to MDC layer) is the first bad commit that has crash problem of lmv_read_entry().</p> <p>In the last seperate patches, I did not include the change which adds packed attribute to some structures. Those packed attribute fixed the problem when compiling Lustre codes:<br/> /home/lixi/lustre/lustre-release.git/lustre/lov/lov_pack.c: In function 'lov_getstripe':<br/> /home/lixi/lustre/lustre-release.git/lustre/lov/lov_pack.c:632:9: error: duplicate case value<br/> /home/lixi/lustre/lustre-release.git/lustre/lov/lov_pack.c:632:9: error: previously used here</p> <p>Please note this problem happens when the Lustre version is 2.5.0. For some reason, that problem did not show up when I rebased the patches to the later versions (from 2.5.1 to 2.5.54+). I don't know why and so I did not commit that part of change. I will investigate more on this.</p> <p>lmv_read_entry() crashes because lli_lmv_md field of struct ll_inode_info is not inited in ll_lli_init().</p> <p>Following is the patch which fixes this problem.<br/> <a href="http://review.whamcloud.com/9042" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/9042</a></p> <p>While testing these patch I found a problem with struct cfs_cpt_table in libcfs/include/libcfs/libcfs_cpu.h. It has a nodemask_t that is not a pointer but the struct cfs_cpt_table in libcfs/include/libcfs/linux/libcfs_cpu.h is a pointer. So this can be fixed in one of two ways. We could change it to a pointer and then update the table allocation function. The other is just remove the nodemask and cpumask in struct cfs_cpt_table since they are not used. The reason this was caught is that ost_handler.c function ost_setup accesses the ctb_nodemask directly. It really should use<br/> cfs_cpt_nodemask instead. Which would be the preferred fix?</p> <p>Only two patches left.</p> <p><a href="http://review.whamcloud.com/#/c/8144" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/8144</a><br/> <a href="http://review.whamcloud.com/#/c/8878" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/8878</a></p> <p>All that is left for this ticket is to see if lustre can cross compile.</p> <p>Currently Lustre master branch can be build natively on ARM/PPC. IMO cross compiling can be handled by a separate ticket since that functionality touch many types of platforms.</p> <p>This ticket can be closed now.</p> <p>close per request</p> Related LU-4668 LU-6766 Development End date Thu, 25 Jun 2015 23:06:36 +0000 Rank 1|hzw7sf: Rank (Obsolete) 11408 Start date Fri, 1 Nov 2013 23:06:36 +0000