https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-12755 Lustre <p>After successfully creating packages for Red Hat 7.7</p> <p>(e.g. lustre-2.12.57_35_g55a7e2d-1.el7.x86_64.rpm)</p> <p>I get CPU soft lockups when trying to create an MGS with LDISKFS backend.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [mkfs.lustre:31220]</pre> </div></div> <p>More details from log:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Sep 6 10:41:00 mgs1 kernel: Call Trace:Sep 6 10:41:00 mgs1 kernel: [&lt;ffffffff9bd73365&gt;] queued_spin_lock_slowpath+0xb/0xf Sep 6 10:41:00 mgs1 kernel: [&lt;ffffffff9bd81ad0&gt;] _raw_spin_lock+0x20/0x30 Sep 6 10:41:00 mgs1 kernel: [&lt;ffffffff9b865e2e&gt;] igrab+0x1e/0x60 Sep 6 10:41:00 mgs1 kernel: [&lt;ffffffffc06bd88b&gt;] ldiskfs_quota_off+0x3b/0x130 [ldiskfs] Sep 6 10:41:00 mgs1 kernel: [&lt;ffffffffc06c091d&gt;] ldiskfs_put_super+0x4d/0x400 [ldiskfs] Sep 6 10:41:00 mgs1 kernel: [&lt;ffffffff9b84b13d&gt;] generic_shutdown_super+0x6d/0x100 Sep 6 10:41:00 mgs1 kernel: [&lt;ffffffff9b84b5b7&gt;] kill_block_super+0x27/0x70 Sep 6 10:41:00 mgs1 kernel: [&lt;ffffffff9b84b91e&gt;] deactivate_locked_super+0x4e/0x70 Sep 6 10:41:00 mgs1 kernel: [&lt;ffffffff9b84c0a6&gt;] deactivate_super+0x46/0x60 Sep 6 10:41:00 mgs1 kernel: [&lt;ffffffff9b86abff&gt;] cleanup_mnt+0x3f/0x80 Sep 6 10:41:00 mgs1 kernel: [&lt;ffffffff9b86ac92&gt;] __cleanup_mnt+0x12/0x20 Sep 6 10:41:00 mgs1 kernel: [&lt;ffffffff9b6c1c0b&gt;] task_work_run+0xbb/0xe0 Sep 6 10:41:00 mgs1 kernel: [&lt;ffffffff9b62cc65&gt;] do_notify_resume+0xa5/0xc0 Sep 6 10:41:00 mgs1 kernel: [&lt;ffffffff9bd8c23b&gt;] int_signal+0x12/0x17 Sep 6 10:41:00 mgs1 kernel: Code: 47 fe ff ff 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 66 90 b9 01 00 00 00 8b 17 85 d2 74 0d 83 fa 03 74 08 f3 90 &lt;8b&gt; 17 85 d2 75 f3 89 d0 f0 0f b1 0f 39 c2 75 e3 5d 66 90 c3 0f</pre> </div></div> <p>I also tried to go for an MDS/MGS pair on the DL380 but mkfs.lustre got stuck the same way </p> <p>as seen on VMware.</p> Red Hat 7.7 on VMware <br/> Red Hat 7.7 on HPE ProLiant DL380 Gen10 <br/> Red Hat 7.7 on HPE Synergy 480 Gen10 LU-12755

CPU soft lockup on mkfs.lustre

Bug Blocker Resolved Fixed Jian Yu Tamas Kazinczy hang mkfs.lustre Thu, 12 Sep 2019 14:23:15 +0000 Mon, 14 Oct 2019 12:35:07 +0000 Sat, 28 Sep 2019 04:01:04 +0000 Upstream Lustre 2.13.0 Lustre 2.12.3 0 10 <p>Tamas,</p> <p>Are you able to provide dmesg from your node?  (You can attach a file here)</p> <p>Also, what version of e2fsprogs are you using?  (output of rpm -qa | grep e2fsprogs would work for this)</p> <p>If possible, it would also be nice to get this:</p> <p>After gathering dmesg, if you can also (while the mkfs is hung) do:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>echo t &gt; /proc/sysrq-trigger </pre> </div></div> <p>Then gather dmesg again and attach that as well. (Gathering dmesg twice is necessary because this can produce so much output that it will overwrite your existing dmesg.)</p> <p>Furthermore, I would suggest that you focus on the tip of b2_12 rather than using master in your tests. The intention is to add support for RHEL 7.7 with the upcoming 2.12.3 release and the current b2_12 is close to the release version.</p> <p>We have:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>e2fsprogs.x86_64 1.45.2.wc1-0.el7 @e2fsprogs-wc</pre> </div></div> <p> </p> <p>I've just attached dmesg outputs.</p> <p>Just to confirm I also hit the same issue using the usual Lustre development environment.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>$ git reflog 219c519 HEAD@{0}: clone: from https://review.whamcloud.com/fs/lustre-release $ rpm -q e2fsprogs e2fsprogs-1.45.2.wc1-0.el7.x86_64 $ uname -r 3.10.0-1062.el7.x86_64</pre> </div></div> <p> I'm just using RHEL 7.7</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> git cloned lustre master branch sh ./autogen.sh ./configure make REFORMAT=: ./lustre/tests/llmount.sh </pre> </div></div> <p>Hi Tamas and Aurelien,</p> <p>I didn't hit the above issue after building Lustre 2.12.57 from source codes on a VirtualBox RHEL 7.7 vm against kernel 3.10.0-1062.el7.</p> <p>Before compiling Lustre codes, did you patch the kernel? </p> <p>FYI, here are the steps I performed on a RHEL 7.7 vm:<br/> Build kernel:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>git clone git://git.whamcloud.com/fs/lustre-release.git cd lustre-release/ sh ./autogen.sh cd mkdir -p kernel/rpmbuild/{BUILD,RPMS,SOURCES,SPECS,SRPMS} cd kernel echo '%_topdir %(echo $HOME)/kernel/rpmbuild' &gt; ~/.rpmmacros rpm -ivh /root/kernelrpm/kernel-3.10.0-1062.el7.src.rpm cd /root/kernel/rpmbuild/ rpmbuild -bp --target=`uname -m` ./SPECS/kernel.spec cd /root/lustre-release/lustre/kernel_patches/series/ for patch in $(&lt;"3.10-rhel7.7.series"); do patch_file="$HOME/lustre-release/lustre/kernel_patches/patches/${patch}"; cat "${patch_file}" &gt;&gt; "$HOME/lustre-kernel-x86_64-lustre.patch"; done cp ~/lustre-kernel-x86_64-lustre.patch ~/kernel/rpmbuild/SOURCES/patch-3.10.0-lustre.patch sed -i.inst -e '/find $RPM_BUILD_ROOT\/lib\/modules\/$KernelVer/a\ cp -a fs/ext3/* $RPM_BUILD_ROOT/lib/modules/$KernelVer/build/fs/ext3 \ cp -a fs/ext4/* $RPM_BUILD_ROOT/lib/modules/$KernelVer/build/fs/ext4' \ -e '/^# empty final patch to facilitate testing of kernel patches/i\ Patch99995: patch-%{version}-lustre.patch' \ -e '/^ApplyOptionalPatch linux-kernel-test.patch/i\ ApplyOptionalPatch patch-%{version}-lustre.patch' \ -e '/^%define listnewconfig_fail 1/s/1/0/' \ ~/kernel/rpmbuild/SPECS/kernel.spec cd echo '# x86_64' &gt; ~/kernel/rpmbuild/SOURCES/kernel-3.10.0-x86_64.config cat ~/lustre-release/lustre/kernel_patches/kernel_configs/kernel-3.10.0-3.10-rhel7.7-x86_64.config &gt;&gt; ~/kernel/rpmbuild/SOURCES/kernel-3.10.0-x86_64.config cd ~/kernel/rpmbuild buildid="_lustre" rpmbuild -ba --with firmware --target x86_64 --with baseonly --without kabichk --define "buildid ${buildid}" ~/kernel/rpmbuild/SPECS/kernel.spec cd /root/kernel/rpmbuild/RPMS/x86_64/ rpm -ivh kernel-3.10.0-1062.el7_lustre.x86_64.rpm kernel-devel-3.10.0-1062.el7_lustre.x86_64.rpm kernel-headers-3.10.0-1062.el7_lustre.x86_64.rpm kernel-tools-3.10.0-1062.el7_lustre.x86_64.rpm kernel-tools-libs-3.10.0-1062.el7_lustre.x86_64.rpm --force sync shutdown -r 0 </pre> </div></div> <p>Build Lustre:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>cd lustre-release/ ./configure --with-linux=/root/kernel/rpmbuild/BUILD/kernel-3.10.0-1062.el7/linux-3.10.0-1062.el7_lustre.x86_64/ make </pre> </div></div> <p>Here is the wiki page for compiling Lustre: <a href="http://wiki.lustre.org/Compiling_Lustre" class="external-link" target="_blank" rel="nofollow noopener">http://wiki.lustre.org/Compiling_Lustre</a>.</p> <p>Hi Jian,</p> <p>No, I did not patched the kernel but I though this is needed only when using project quota feature?</p> <p>I just realized I downgraded my kernel to 3.10.0-957.10.1.el7_lustre, which is the patched version from Whamcloud. So it seems really related to the kernel patches.</p> <p>Hey Jian,</p> <p>I forgot to tell that I'd like to have patchless servers.</p> <p>Thank you Aurelien and Tamas for the info.<br/> I can reproduce the issue with patchless server build. With vfs-project-quotas-rhel7.patch applied to kernel, the issue disappeared. I'm looking into the related ldiskfs patches.</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=degremoa" class="user-hover" rel="degremoa">degremoa</a>, are you also seeing the hang with the same stack trace (stuck on a spinlock in the quota inode during unmount)? While it is a bit confusing, <tt>mkfs.lustre</tt> is mounting the filesystem internally to create Lustre configuration files. </p> <p>I'd guess that there is something wrong with the quota inodes stored in the superblock, like they are not initialized, or the spinlock is not initialized, which only shows up because we are formatting then mounting and unmounting the filesystem so quickly.</p> <p>Hi Andreas,<br/> While running mkfs.lustre, ldiskfs_write_ldd() mounted the device temporarily to write CONFIGS/mountdata. The hang occurred while unmounting the device.<br/> I compared ldiskfs patch ext4-projid-ignore-maxquotas.patch and kernels for RHEL 7.7 and 7.6, and found the following <tt>ext4_quota_off_umount()</tt> was added into <tt>ext4_put_super()</tt> for RHEL 7.7 kernel 3.10.0-1062.el7:</p> <div class="code panel" style="border-width: 1px;"><div class="codeHeader panelHeader" style="border-bottom-width: 1px;"><b>ext4_quota_off_umount() and ext4_put_super()</b></div><div class="codeContent panelContent"> <pre class="code-java"> #ifdef CONFIG_QUOTA <span class="code-keyword">static</span> <span class="code-object">int</span> ext4_quota_off(struct super_block *sb, <span class="code-object">int</span> type); <span class="code-keyword">static</span> inline void ext4_quota_off_umount(struct super_block *sb) { <span class="code-object">int</span> type; <span class="code-comment">/* Use our quota_off function to clear inode flags etc. */</span> <span class="code-keyword">for</span> (type = 0; type &lt; MAXQUOTAS; type++) ext4_quota_off(sb, type); } #<span class="code-keyword">else</span> <span class="code-keyword">static</span> inline void ext4_quota_off_umount(struct super_block *sb) { } #endif <span class="code-keyword">static</span> void ext4_put_super(struct super_block *sb) { struct ext4_sb_info *sbi = EXT4_SB(sb); struct ext4_super_block *es = sbi-&gt;s_es; <span class="code-object">int</span> aborted = 0; <span class="code-object">int</span> i, err; ext4_unregister_li_request(sb); ext4_quota_off_umount(sb); &lt;~snip~&gt; } </pre> </div></div> <p>inode_lock(inode) and inode_unlock(inode) were also added into ext4_quota_off() for RHEL 7.7 kernel.</p> <p>Can you please check the upstream kernel to see if there is some additional fix to this code? If yes, then it should be backported tinour ldiskfs series, and also reported to RH. </p> <p>Sure, Andreas. Let me do this.</p> <p>Hi Andreas,<br/> The ext4_quota_off_umount() function and ext4_quota_on/off() changes were added by the following Linux kernel commit:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>commit 957153fce8d226612de05938ca18101b7d608328 Author: Jan Kara &lt;jack@suse.cz&gt; Date: Thu Apr 6 15:40:06 2017 +0200 ext4: Set flags on quota files directly Currently immutable and noatime flags on quota files are set by quota code which requires us to copy inode-&gt;i_flags to our on disk version of quota flags in GETFLAGS ioctl and ext4_do_update_inode(). Move to setting / clearing these on-disk flags directly to save that copying. Signed-off-by: Jan Kara &lt;jack@suse.cz&gt; </pre> </div></div> <p>However after that, there is no additional fix related to the issue in this ticket.</p> <p>I'm wondering why after applying vfs-project-quotas-rhel7.patch, the issue disappeared.</p> <p>Andreas, for some unknown reason my system keeps crashing when I try to reproduce this issue. Kernel crashes instead of hitting this during MDT formatting.</p> <p>But I think you got everything you need for this problem anyway, no?</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=yujian" class="user-hover" rel="yujian">yujian</a>, it looks like this may indeed be related to the project quota patch. </p> <p>With <tt>vfs-project-quotas-rhel7.patch</tt> applied the kernel properly handles the project quota type (index 2, i.e. the 3rd quota type), and without it, the RHEL7 kernel doesn't handle this quota type. In ldiskfs, we need to make sure that the loop in <tt>ext4_quota_off_umount()</tt> is limiting the <tt>LDISKFS_MAXQUOTAS</tt> loop to the kernel <tt>MAXQUOTAS</tt>. Otherwise, it is trying to dereference <tt>sb-&gt;s_inode<span class="error">&#91;2&#93;</span></tt> which is not an inode at all, just random memory.</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=degremoa" class="user-hover" rel="degremoa">degremoa</a>, I think we are good. It shouldn't be too hard to make a patch now that we (likely) know what the problem is.</p> <p>It looks like in <tt>ldiskfs/kernel_patches/patches/rhel7/ext4-projid-feature-support.patch</tt> it should check how many quotas that the kernel supports:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-comment">/* <span class="code-object">Number</span> of quota types we support */</span> +#<span class="code-keyword">if</span> MAXQUOTAS &gt; 2 +#define EXT4_MAXQUOTAS 3 +#<span class="code-keyword">else</span> #define EXT4_MAXQUOTAS 2 +#endif </pre> </div></div> <p>or something similar, so that it doesn't try to use project quotas on an unpatched RHEL7 kernel.</p> <p>Thank you very much, Andreas. Let me push a patch and verify it.</p> <p>Looking into this a bit more (c.f. <a href="https://jira.whamcloud.com/browse/LU-20" title="patchless server kernel" class="issue-link" data-issue-key="LU-20"><del>LU-20</del></a>), it seems that we may want the same code to module with both a patched and unpatched kernel, so it is probably better to keep <tt>EXT4_MAXQUOTAS=3</tt> and decide whether to disable the project quota at unmount during runtime rather than at compile time. Something similar to how quota is enabled for the various types:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> @@ -819,7 +819,7 @@ <span class="code-keyword">static</span> inline void ext4_quota_off_umount <span class="code-object">int</span> type; <span class="code-comment">/* Use our quota_off function to clear inode flags etc. */</span> - <span class="code-keyword">for</span> (type = 0; type &lt; MAXQUOTAS; type++) - ext4_quota_off(sb, type); + <span class="code-keyword">for</span> (type = 0; type &lt; EXT4_MAXQUOTAS; type++) + <span class="code-keyword">if</span> (EXT4_SB(sb)-&gt;s_qf_names[i]) + ext4_quota_off(sb, type); } </pre> </div></div> <p>The existing check "<tt>if (!inode0</tt>" inside <tt>ext4&#95;quota&#95;off()</tt> is incorrect in this case because <tt>sb_dqopt(sb)&#45;&gt;files<span class="error">&#91;type&#93;</span></tt> is located in the kernel superblock, which was not compiled to have enough space for the project quota. It may be that <tt>ext4_orphan_cleanup()</tt> needs a similar change to check <tt>EXT4&#95;SB(sb)&#45;&gt;s&#95;qf&#95;names<span class="error">&#91;i&#93;</span></tt> instead, since that is located inside the ext4/ldiskfs module and will be self-consistent.</p> <p>Jian Yu (yujian@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/36203" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36203</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12755" title="CPU soft lockup on mkfs.lustre" class="issue-link" data-issue-key="LU-12755"><del>LU-12755</del></a> ldiskfs: check s_qf_names in ext4_quota_off_umount()<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: fff55938f6266cdd9d201dae950079371a6fde3b</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=kazinczy" class="user-hover" rel="kazinczy">kazinczy</a> and/or <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=degremoa" class="user-hover" rel="degremoa">degremoa</a> can you verify whether this fix works for you?</p> <p>I just got the test result that the patch fixed the spinlock issue but hit another issue:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Kernel error detected: [ 64.991808] VFS: Busy inodes after unmount of dm-0. Self-destruct in 5 seconds. Have a nice day... </pre> </div></div> <p>Apologies, I missed this important ticket.</p> <p>Jian Yu,</p> <p>Would you collect dumpe2fs -h /dev/dm-0 output?</p> <p>I guess we might load project inode but the logic is we did not iput it caused the problem, would you try following parts to see if it could fix above buys inode problem:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@server ext4]# git diff super.c diff --git a/fs/ext4/super.c b/fs/ext4/super.c index ca8b50c8..e3fccb79 100644 --- a/fs/ext4/super.c +++ b/fs/ext4/super.c @@ -5624,7 +5624,7 @@ static int ext4_enable_quotas(struct super_block *sb) sb_dqopt(sb)-&gt;flags |= DQUOT_QUOTA_SYS_FILE; for (type = 0; type &lt; EXT4_MAXQUOTAS; type++) { - if (qf_inums[type]) { + if (qf_inums[type] &amp;&amp; type &lt; MAXQUOTAS) { err = ext4_quota_enable(sb, type, QFMT_VFS_V1, DQUOT_USAGE_ENABLED); if (err) { </pre> </div></div> <p>Andreas, any reasons that why we did not use first suggested way:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> /* Number of quota types we support */ +#if MAXQUOTAS &gt; 2 +#define EXT4_MAXQUOTAS 3 +#else #define EXT4_MAXQUOTAS 2 +#endif </pre> </div></div> <p>This looks much safe fix.</p> <p>Thank you, Shilong. Let me try the above changes.</p> <p>Shilong, the reason we don't use the static MAXQUOTAS limit at compile time is to allow using the same ldiskfs and osd-ldiskfs module on both patched and unpatched kernels. </p> <p>With the "qf_inums<span class="error">&#91;type&#93;</span> &amp;&amp; type &lt; MAXQUOTAS" change applied, the following failure still occurred:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Kernel error detected: [ 67.386709] VFS: Busy inodes after unmount of dm-0. Self-destruct in 5 seconds. Have a nice day... </pre> </div></div> <p>Let me setup an environment and try locally.</p> <p>Jian, can you please push a separate patch with the "simple" version of the patch, which limits <tt>EXT4_MAX_QUOTAS</tt> to the kernel <tt>MAXQUOTAS</tt> value, if smaller. That will at least tell us if this is the source of the problem. </p> <p>Sure, Andreas.</p> <p>Jian Yu (yujian@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/36241" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36241</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12755" title="CPU soft lockup on mkfs.lustre" class="issue-link" data-issue-key="LU-12755"><del>LU-12755</del></a> ldiskfs: limit EXT4_MAXQUOTAS to MAXQUOTAS<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: b0965235b20ff03c55eaeae6d917d6e3f11968f7</p> <p>OK, the problem is that we leaked USR and Group inode, comments on first patch, I agreed we'd better use first patch.</p> <p>Also it will be interesting that with project quota enabled but unpatched kernel used, I just tested it will fail:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> 642.115121] [&lt;ffffffff93379262&gt;] dump_stack+0x19/0x1b [ 642.115888] [&lt;ffffffffc091606a&gt;] ldiskfs_quota_enable+0x10a/0x120 [ldiskfs] [ 642.116934] [&lt;ffffffffc091807b&gt;] ldiskfs_enable_quotas+0x9b/0x110 [ldiskfs] [ 642.117962] [&lt;ffffffffc091c490&gt;] ldiskfs_fill_super+0x2e60/0x2e80 [ldiskfs] [ 642.118993] [&lt;ffffffff92e4c893&gt;] mount_bdev+0x1b3/0x1f0 [ 642.119790] [&lt;ffffffffc0919630&gt;] ? ldiskfs_calculate_overhead+0x430/0x430 [ldiskfs] [ 642.120951] [&lt;ffffffffc0913605&gt;] ldiskfs_mount+0x15/0x20 [ldiskfs] [ 642.121874] [&lt;ffffffff92e4d1fe&gt;] mount_fs+0x3e/0x1b0 [ 642.122624] [&lt;ffffffff92de1035&gt;] ? __alloc_percpu+0x15/0x20 [ 642.123409] [&lt;ffffffff92e6b377&gt;] vfs_kern_mount+0x67/0x110 [ 642.124225] [&lt;ffffffff92e6dacf&gt;] do_mount+0x1ef/0xce0 [ 642.124983] [&lt;ffffffff92e4521a&gt;] ? __check_object_size+0x1ca/0x250 [ 642.125919] [&lt;ffffffff92ddbe7f&gt;] ? memdup_user+0x4f/0x80 [ 642.126723] [&lt;ffffffff92e6e903&gt;] SyS_mount+0x83/0xd0 [ 642.127521] [&lt;ffffffff9338bede&gt;] system_call_fastpath+0x25/0x2a </pre> </div></div> <p>I think we also need this extra fix for that:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@server ext4]# git diff super.c diff --git a/fs/ext4/super.c b/fs/ext4/super.c index ca8b50c8..e3fccb79 100644 --- a/fs/ext4/super.c +++ b/fs/ext4/super.c @@ -5624,7 +5624,7 @@ static int ext4_enable_quotas(struct super_block *sb) sb_dqopt(sb)-&gt;flags |= DQUOT_QUOTA_SYS_FILE; for (type = 0; type &lt; EXT4_MAXQUOTAS; type++) { - if (qf_inums[type]) { + if (qf_inums[type] &amp;&amp; type &lt; MAXQUOTAS) { err = ext4_quota_enable(sb, type, QFMT_VFS_V1, DQUOT_USAGE_ENABLED); if (err) { </pre> </div></div> <blockquote> <p>I think we also need this extra fix for that:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> [root@server ext4]# git diff <span class="code-keyword">super</span>.c @@ -5624,7 +5624,7 @@ <span class="code-keyword">static</span> <span class="code-object">int</span> ext4_enable_quotas(struct super_block *sb) sb_dqopt(sb)-&gt;flags |= DQUOT_QUOTA_SYS_FILE; <span class="code-keyword">for</span> (type = 0; type &lt; EXT4_MAXQUOTAS; type++) { - <span class="code-keyword">if</span> (qf_inums[type]) { + <span class="code-keyword">if</span> (qf_inums[type] &amp;&amp; type &lt; MAXQUOTAS) { err = ext4_quota_enable(sb, type, QFMT_VFS_V1, DQUOT_USAGE_ENABLED); <span class="code-keyword">if</span> (err) { </pre> </div></div></blockquote> <p>It isn't clear if this will fix all of the problems, since it is taking the MAXQUOTAS value from the kernel at build time, not at runtime. Since we always build against a patched kernel with MAXQUOTAS=3, it wouldn't restrict the code. It probably would help for the case of building against a patched kernel.</p> <p>Yup, I was thinking that to make us ldiskfs.ko module works for both patched and unpatched kernel:</p> <p>using following way to check:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>bool is_proj_supported_by_kernel(struct super_block *sb) { return (sizeof(sb_dqopt(sb)-&gt;info) / sizeof(struct mem_dqinfo)) &gt; PRJQUOTA; } </pre> </div></div> <p>This will still return the size of the structures matching the kernel that ldiskfs was built against, not necessarily the kernel that the module is loaded with. </p> <p>We would need that function to be exported by the running kernel, which would mean to patch the kernel (obviously not possible for the unpatched kernel, and very unlikely to get upstream).</p> <p>Instead, we need to find something that is exported from the <b>running</b> kernel that gives us information about whether it supports project quotas.</p> <p>Hmm, you are right..</p> <p>But haven't found a way to do that since Kernel mostly use static value MAXQUOTAS..</p> <p>I am wondering whether the following way would help use to get the MAXQUOTAS of the running kernel (using a tricky way):</p> <p>1) We call dquot_initialize(inode) in OSD/ldiskfs.<br/> 2) __dquot_initialize() calls inode-&gt;i_sb-&gt;dq_op-&gt;get_projid()<br/> 3) We patch ldiskfs to change ext4_get_projid(). In ext4_get_projid(), it set a static flag to show that the kernel actually supports PRJQUOTA.</p> <p>However, dquot_initialize() depends on ext4_enable_quotas? So might not be workable...</p> <p>In the mount phase, ROOT inode will call dquot_initialize() first time but at that time ext4_enable_quotas() haven't been called, and unfortunately we also need check whether kernel support PRJQUOTA inside ext4_enable_quotas().....</p> <p>So this way won't work..</p> <p>unfortunately it looks hard to make it work ideally, we coud not be blocked at this, at least we need fix common case, how about<br/> we add some warning messages during mount time?</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>diff --git a/fs/ext4/super.c b/fs/ext4/super.c index ca8b50c8..cddf6595 100644 --- a/fs/ext4/super.c +++ b/fs/ext4/super.c @@ -4489,6 +4489,13 @@ no_journal: ratelimit_state_init(&amp;sbi-&gt;s_msg_ratelimit_state, 5 * HZ, 10); kfree(orig_data); +#ifdef HAVE_PROJECT_QUOTA + ext4_msg(sb, KERN_WARNING, + "ext4 module compiled with patched kernel won't work on unpatched kernel"); +#else + ext4_msg(sb, KERN_WARNING, + "ext4 module compiled with unpatched kernel won't work on patched kernel"); +#endif return 0; </pre> </div></div> <p>Mark it as a known issue, at least Administrator knows this issue.</p> <p>The good new is this issue won't exist for RHEL8 and later kernels though..</p> <p>Instead of printing an error, returning error seems better. It will prevent any future problem.</p> <blockquote><p>In the mount phase, ROOT inode will call dquot_initialize() first time but at that time ext4_enable_quotas() haven't been called, and unfortunately we also need check whether kernel support PRJQUOTA inside ext4_enable_quotas().....</p></blockquote> <p>I think we can ext4_enable_quotas() first on group quota and user quota. And then set the flag of sb_has_quota_active(PROJQUOTA) temporarily. And then call dquot_initialize() to check whether MAXQUOTAS == 3.</p> <p>The tricky way seems work with limited testing, Yu Jian could you continue some testing on refreshed patch:</p> <p>1) Built and installed. ldiskfs on patched kernel and then switch it to unpatched kernel.<br/> 2) Built and installed ldiskfs on unpatched kernel and then switch it to patched kernel.</p> <p>To make sure mount and umount Lustre works.</p> <p>Sure, Shilong.</p> <p>Shilong, I assume you mean the updated patch <a href="https://review.whamcloud.com/#/c/36203/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/36203/</a></p> <p>Any reason why ext4_get_projid() is called before ext4_enable_quotas()? I thought ext4_enable_quotas() is always called before any one calls ext4_get_projid().</p> <p>Li Xi,</p> <p>I walkrounded that issue by set PROJ active flag temporariy, and then call dquot_initialize() to call ext4_get_projid().<br/> I just tested fixed version, it seems work:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>+ sb_dqopt(sb)-&gt;flags |= dquot_state_flag(DQUOT_USAGE_ENABLED, PRJQUOTA); + dquot_initialize(root); + sb_dqopt(sb)-&gt;flags &amp;= ~dquot_state_flag(DQUOT_USAGE_ENABLED, PRJQUOTA); </pre> </div></div> <p>By adding some printk debug, it seems detection works.</p> <p>Jian Yu (yujian@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/36270" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36270</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12755" title="CPU soft lockup on mkfs.lustre" class="issue-link" data-issue-key="LU-12755"><del>LU-12755</del></a> ldiskfs: fix project quota unpon unpatched kernel<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: e56b4bc0970275ce6413883794bd52d2dfa7b164</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/36203/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36203/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12755" title="CPU soft lockup on mkfs.lustre" class="issue-link" data-issue-key="LU-12755"><del>LU-12755</del></a> ldiskfs: fix project quota unpon unpatched kernel<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: d780f15a2d63c8bde5ae6345aed85b4b44904fb5</p> <p>Landed for 2.13</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/36270/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36270/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12755" title="CPU soft lockup on mkfs.lustre" class="issue-link" data-issue-key="LU-12755"><del>LU-12755</del></a> ldiskfs: fix project quota unpon unpatched kernel<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: <br/> Commit: 820e374624a584ec0c0a326ec96bf0abeb50cf40</p> <p>I can confirm that <tt>mkfs.lustre</tt> for patchless LDISKFS from b2_12 works fine for me now.</p> <p>Great. Thanks for confirming <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=kazinczy" class="user-hover" rel="kazinczy">kazinczy</a></p> Development Epic hang server Epic/Theme Lustre-2.12.57 Rank 1|i00mov: Rank (Obsolete) 9223372036854775807 Severity