https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-17095 Lustre <p>Occasionally when building Lustre packages (both for RHEL and Ubuntu) the build will fail with an error about a missing <tt>modules.order</tt> file and then the build will fail:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00:05:29.164 Type 'make' to build Lustre. 00:05:29.232 + make -j10 -s 00:05:29.244 Making all in . 00:06:22.277 cat: /tmp/rpmbuild-lustre-jenkins-fmslEJ8U/BUILD/lustre-2.14.0_ddn101//modules.order: No such file or directory 00:06:22.390 Making all in lustre-iokit 00:06:22.398 Making all in obdfilter-survey 00:06:22.401 Making all in sgpdd-survey 00:06:22.404 Making all in ost-survey 00:06:22.406 Making all in ior-survey 00:06:22.409 Making all in mds-survey 00:06:22.412 Making all in stats-collect 00:06:22.419 Making all in libcfs 00:06:22.426 Making all in libcfs 00:06:22.434 Making all in linux 00:06:22.438 Making all in util 00:06:22.441 Making all in crypto 00:06:22.446 make[5]: *** No rule to make target 'libcfs.ko', needed by 'all-am'. Stop. </pre> </div></div> <p><a href="https://build.whamcloud.com/job/lustre-reviews/97659/arch=x86_64,build_type=client,distro=el9.2,ib_stack=inkernel/console" class="external-link" target="_blank" rel="nofollow noopener">https://build.whamcloud.com/job/lustre-reviews/97659/arch=x86_64,build_type=client,distro=el9.2,ib_stack=inkernel/console</a><br/> <a href="https://build.whamcloud.com/job/lustre-b_es6_0/514/arch=x86_64,build_type=client,distro=rocky9.2,ib_stack=inkernel/console" class="external-link" target="_blank" rel="nofollow noopener">https://build.whamcloud.com/job/lustre-b_es6_0/514/arch=x86_64,build_type=client,distro=rocky9.2,ib_stack=inkernel/console</a><br/> <a href="https://build.whamcloud.com/job/lustre-b_es6_0/520/arch=x86_64,build_type=client,distro=sles15sp4,ib_stack=mlx/console" class="external-link" target="_blank" rel="nofollow noopener">https://build.whamcloud.com/job/lustre-b_es6_0/520/arch=x86_64,build_type=client,distro=sles15sp4,ib_stack=mlx/console</a><br/> <a href="https://build.whamcloud.com/job/lustre-b_es6_0/522/arch=x86_64,build_type=client,distro=rocky9.2,ib_stack=inkernel/console" class="external-link" target="_blank" rel="nofollow noopener">https://build.whamcloud.com/job/lustre-b_es6_0/522/arch=x86_64,build_type=client,distro=rocky9.2,ib_stack=inkernel/console</a><br/> <a href="https://build.whamcloud.com/job/lustre-b_es-reviews/12517/arch=x86_64,build_type=client,distro=el9.0,ib_stack=mlx/console" class="external-link" target="_blank" rel="nofollow noopener">https://build.whamcloud.com/job/lustre-b_es-reviews/12517/arch=x86_64,build_type=client,distro=el9.0,ib_stack=mlx/console</a><br/> <a href="https://build.whamcloud.com/job/lustre-b_es-reviews/12531/arch=x86_64,build_type=client,distro=el9.1,ib_stack=inkernel/console" class="external-link" target="_blank" rel="nofollow noopener">https://build.whamcloud.com/job/lustre-b_es-reviews/12531/arch=x86_64,build_type=client,distro=el9.1,ib_stack=inkernel/console</a><br/> <a href="https://build.whamcloud.com/job/lustre-b_es-reviews/12541/arch=x86_64,build_type=client,distro=el9.1,ib_stack=inkernel/console" class="external-link" target="_blank" rel="nofollow noopener">https://build.whamcloud.com/job/lustre-b_es-reviews/12541/arch=x86_64,build_type=client,distro=el9.1,ib_stack=inkernel/console</a><br/> <a href="https://build.whamcloud.com/job/lustre-b_es-reviews/12574/arch=x86_64,build_type=client,distro=sles15sp4,ib_stack=inkernel/console" class="external-link" target="_blank" rel="nofollow noopener">https://build.whamcloud.com/job/lustre-b_es-reviews/12574/arch=x86_64,build_type=client,distro=sles15sp4,ib_stack=inkernel/console</a></p> LU-17095

modules.order: No such file or directory, No rule to make target 'libcfs.ko' needed by 'all-am'

Bug Minor Resolved Fixed Jian Yu Andreas Dilger build Thu, 7 Sep 2023 03:01:55 +0000 Thu, 23 Nov 2023 13:26:34 +0000 Thu, 28 Sep 2023 15:11:37 +0000 Lustre 2.16.0 Lustre 2.15.4 0 5 <p>It isn't clear what is causing the problem, since it is intermittently failing. It appears to happen more often with newer kernels, and possibly with more concurrency (see "<tt>make -j 10</tt>" in the description).</p> <p>The <tt>modules.order</tt> file itself is not generated directly from the Lustre build, but appears to be part of the kernel build infrastructure:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[lustre-head]$ git grep modules.order .gitignore:modules.order build/.gitignore:/modules.order ldiskfs/.gitignore:/modules.order ldiskfs/autoMakefile.am: rm -rf linux linux-stage ldiskfs*.h trace modules.order lustre-iokit/.gitignore:/modules.order [linux-git]$ git grep modules.order .gitignore:modules.order Documentation/dontdiff:modules.order Documentation/kbuild/kbuild.rst:modules.order Documentation/target/tcm_mod_builder.rst: -rw-r--r-- 1 root root 49 2010-10-05 03:23 modules.order Makefile:export MODORDER := $(extmod_prefix)modules.order Makefile: @sed 's:^\(.*\)\.o$$:kernel/\1.ko:' modules.order &gt; $(MODLIB)/modules.order Makefile: -o -name '*.symtypes' -o -name 'modules.order' \ scripts/Makefile.build:subdir-modorder := $(sort $(filter %/modules.order, $(obj-m))) scripts/Makefile.build:targets-for-modules += $(obj)/modules.order scripts/Makefile.build:$(subdir-modorder): $(obj)/%/modules.order: $(obj)/% ; scripts/Makefile.build:# Rule to create modules.order file : </pre> </div></div> <p>It seems possible that the <tt>modules.order</tt> generation is racy with higher concurrency, and one <tt>make</tt> thread is trying to access it without having a proper make dependency on it, while another <tt>make</tt> thread hasn't created it yet?</p> <p>This is likely a bug in the kernel build system. It might be possible to work around this by executing something <em>like</em> "<tt>make modules.order</tt>" to build this file serially instead of using "<tt>make -j10 ...</tt>", but this command doesn't actually work.</p> <p>Just digging a bit through the kernel build code, I found a few things that might be related.</p> <p>I had removed <tt>modules.order</tt> at the start of a local build, and it appears that it was recreated during the later module linking stage:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># ls -l modules.order ls: cannot access modules.order: No such file or directory # fg make -j10 CC [M] /usr/src/lustre-exa/lustre/obdclass/upcall_cache.o CC [M] /usr/src/lustre-exa/lustre/ptlrpc/llog_server.o LD [M] /usr/src/lustre-exa/lustre/quota/lquota.o : CC [M] /usr/src/lustre-exa/lustre/ptlrpc/gss/gss_keyring.o LD [M] /usr/src/lustre-exa/lustre/ptlrpc/ptlrpc.o LD [M] /usr/src/lustre-exa/lustre/ptlrpc/gss/ptlrpc_gss.o Building modules, stage 2. MODPOST 34 modules CC /usr/src/lustre-exa/ldiskfs/ldiskfs.mod.o CC /usr/src/lustre-exa/libcfs/libcfs/libcfs.mod.o CC /usr/src/lustre-exa/lnet/klnds/o2iblnd/ko2iblnd.mod.o : ^Z [1]+ Stopped make -j10 [root@centos7 lustre-exa]# ls -l modules.order 4 -rwxrwx---. 1 root 1686 Sep 6 22:08 modules.order* </pre> </div></div> <p>In <tt>Documentation/kbuild/kbuild.rst</tt> I was looking for <tt>MODPOST</tt> and found a few things of interest:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>modules.order ------------- This file records the order in which modules appear in Makefiles. This is used by modprobe to deterministically resolve aliases that match multiple modules. KBUILD_MODPOST_NOFINAL ---------------------- KBUILD_MODPOST_NOFINAL can be set to skip the final link of modules. This is solely useful to speed up test compiles. KBUILD_VERBOSE -------------- Set the kbuild verbosity. Can be assigned same values as "V=...". </pre> </div></div> <p>Removing <tt>modules.order</tt> and running with "<tt>make V=1</tt>" produces output for all of the modules like:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>make -f scripts/Makefile.build obj=/usr/src/lustre-exa make -f scripts/Makefile.build obj=/usr/src/lustre-exa/ldiskfs (cat /dev/null; echo kernel//usr/src/lustre-exa/ldiskfs/ldiskfs.ko;) &gt; /usr/src/lustre-exa/ldiskfs/modules.order make -f scripts/Makefile.build obj=/usr/src/lustre-exa/libcfs make -f scripts/Makefile.build obj=/usr/src/lustre-exa/libcfs/libcfs (cat /dev/null; echo kernel//usr/src/lustre-exa/libcfs/libcfs/libcfs.ko;) &gt; /usr/src/lustre-exa/libcfs/libcfs/modules.order (cat /dev/null; cat /usr/src/lustre-exa/libcfs/libcfs/modules.order;) &gt; /usr/src/lustre-exa/libcfs/modules.order : : (cat /dev/null; cat /usr/src/lustre-exa/lustre/fid/modules.order; cat /usr/src/lustre-exa/lustre/obdclass/modules.order; cat /usr/src/lustre-exa/lustre/ptlrpc/modules.order; cat /usr/src/lustre-exa/lustre/obdecho/modules.order; cat /usr/src/lustre-exa/lustre/mgc/modules.order; cat /usr/src/lustre-exa/lustre/tests/kernel/modules.order; cat /usr/src/lustre-exa/lustre/ost/modules.order; cat /usr/src/lustre-exa/lustre/mgs/modules.order; cat /usr/src/lustre-exa/lustre/mdt/modules.order; cat /usr/src/lustre-exa/lustre/mdd/modules.order; cat /usr/src/lustre-exa/lustre/ofd/modules.order; cat /usr/src/lustre-exa/lustre/quota/modules.order; cat /usr/src/lustre-exa/lustre/osp/modules.order; cat /usr/src/lustre-exa/lustre/lod/modules.order; cat /usr/src/lustre-exa/lustre/lfsck/modules.order; cat /usr/src/lustre-exa/lustre/lov/modules.order; cat /usr/src/lustre-exa/lustre/osc/modules.order; cat /usr/src/lustre-exa/lustre/mdc/modules.order; cat /usr/src/lustre-exa/lustre/lmv/modules.order; cat /usr/src/lustre-exa/lustre/llite/modules.order; cat /usr/src/lustre-exa/lustre/fld/modules.order; cat /usr/src/lustre-exa/lustre/lz4/modules.order; cat /usr/src/lustre-exa/lustre/gzip/modules.order; cat /usr/src/lustre-exa/lustre/osd-ldiskfs/modules.order;) &gt; /usr/src/lustre-exa/lustre/modules.order (cat /dev/null; cat /usr/src/lustre-exa/ldiskfs/modules.order; cat /usr/src/lustre-exa/libcfs/modules.order; cat /usr/src/lustre-exa/lnet/modules.order; cat /usr/src/lustre-exa/lustre/modules.order;) &gt; /usr/src/lustre-exa/modules.order make -f ./scripts/Makefile.modpost find /usr/src/lustre-exa/.tmp_versions -name '*.mod' | xargs -r grep -h '\.ko$' | sort -u | sed 's/\.ko$/.o/' | scripts/mod/modpost -m -a -i ./Module.symvers -I /usr/src/lustre-exa/Module.symvers -o /usr/src/lustre-exa/Module.symvers -w -s -T - </pre> </div></div> <p>so this is definitely the part of the build that is causing problems, but I couldn't see what was going wrong. </p> <p>It seems likely that this behavior depends on the kernel which the modules were built against (I'm testing with el8.7), </p> <p>"Jian Yu &lt;yujian@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/52323" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/52323</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-17095" title="modules.order: No such file or directory, No rule to make target &#39;libcfs.ko&#39; needed by &#39;all-am&#39;" class="issue-link" data-issue-key="LU-17095"><del>LU-17095</del></a> build: avoid modules.order nonexistence failure<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 61e9a196313f788a1d86f4672be1fdad8e5298c2</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/52323/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/52323/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-17095" title="modules.order: No such file or directory, No rule to make target &#39;libcfs.ko&#39; needed by &#39;all-am&#39;" class="issue-link" data-issue-key="LU-17095"><del>LU-17095</del></a> build: avoid modules.order nonexistence failure<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: dbe4f860977455a9abe50165645a025bb6c46350</p> <p>Landed for 2.16</p> <p>"Jian Yu &lt;yujian@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/52545" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/52545</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-17095" title="modules.order: No such file or directory, No rule to make target &#39;libcfs.ko&#39; needed by &#39;all-am&#39;" class="issue-link" data-issue-key="LU-17095"><del>LU-17095</del></a> build: avoid modules.order nonexistence failure<br/> Project: fs/lustre-release<br/> Branch: b2_15<br/> Current Patch Set: 1<br/> Commit: 3d581b2f5a2516c55b6c93033b6724f941fb5079</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/52545/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/52545/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-17095" title="modules.order: No such file or directory, No rule to make target &#39;libcfs.ko&#39; needed by &#39;all-am&#39;" class="issue-link" data-issue-key="LU-17095"><del>LU-17095</del></a> build: avoid modules.order nonexistence failure<br/> Project: fs/lustre-release<br/> Branch: b2_15<br/> Current Patch Set: <br/> Commit: 25b536a8c26c5209b1bd54e0cf9cf3aa0b829bad</p> Related Development Rank 1|i03ux3: Rank (Obsolete) 9223372036854775807 Severity