[LU-12755] CPU soft lockup on mkfs.lustre

• Type: Bug
• Resolution: Fixed
• Priority: Blocker
• Fix Version/s: Lustre 2.13.0, Lustre 2.12.3
• Affects Version/s: Upstream
• Labels: hang mkfs.lustre
• Environment: Red Hat 7.7 on VMware ; Red Hat 7.7 on HPE ProLiant DL380 Gen10 ; Red Hat 7.7 on HPE Synergy 480 Gen10

Epic/Theme

Lustre-2.12.57

• Severity: 3

Epic

hang server

• Assignee: Jian Yu
• Reporter: Tamas Kazinczy (Inactive)

Description

After successfully creating packages for Red Hat 7.7

(e.g. lustre-2.12.57_35_g55a7e2d-1.el7.x86_64.rpm)

I get CPU soft lockups when trying to create an MGS with LDISKFS backend.

NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [mkfs.lustre:31220]

More details from log:

Sep  6 10:41:00 mgs1 kernel: Call Trace:Sep  6 10:41:00 mgs1 kernel: [<ffffffff9bd73365>] queued_spin_lock_slowpath+0xb/0xf
Sep  6 10:41:00 mgs1 kernel: [<ffffffff9bd81ad0>] _raw_spin_lock+0x20/0x30
Sep  6 10:41:00 mgs1 kernel: [<ffffffff9b865e2e>] igrab+0x1e/0x60
Sep  6 10:41:00 mgs1 kernel: [<ffffffffc06bd88b>] ldiskfs_quota_off+0x3b/0x130 [ldiskfs]
Sep  6 10:41:00 mgs1 kernel: [<ffffffffc06c091d>] ldiskfs_put_super+0x4d/0x400 [ldiskfs]
Sep  6 10:41:00 mgs1 kernel: [<ffffffff9b84b13d>] generic_shutdown_super+0x6d/0x100
Sep  6 10:41:00 mgs1 kernel: [<ffffffff9b84b5b7>] kill_block_super+0x27/0x70
Sep  6 10:41:00 mgs1 kernel: [<ffffffff9b84b91e>] deactivate_locked_super+0x4e/0x70
Sep  6 10:41:00 mgs1 kernel: [<ffffffff9b84c0a6>] deactivate_super+0x46/0x60
Sep  6 10:41:00 mgs1 kernel: [<ffffffff9b86abff>] cleanup_mnt+0x3f/0x80
Sep  6 10:41:00 mgs1 kernel: [<ffffffff9b86ac92>] __cleanup_mnt+0x12/0x20
Sep  6 10:41:00 mgs1 kernel: [<ffffffff9b6c1c0b>] task_work_run+0xbb/0xe0
Sep  6 10:41:00 mgs1 kernel: [<ffffffff9b62cc65>] do_notify_resume+0xa5/0xc0
Sep  6 10:41:00 mgs1 kernel: [<ffffffff9bd8c23b>] int_signal+0x12/0x17
Sep  6 10:41:00 mgs1 kernel: Code: 47 fe ff ff 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 66 90 b9 01 00 00 00 8b 17 85 d2 74 0d 83 fa 03 74 08 f3 90 <8b> 17 85 d2 75 f3 89 d0 f0 0f b1 0f 39 c2 75 e3 5d 66 90 c3 0f

I also tried to go for an MDS/MGS pair on the DL380 but mkfs.lustre got stuck the same way 

as seen on VMware.

Comments

[Li Xi]

In the mount phase, ROOT inode will call dquot_initialize() first time but at that time ext4_enable_quotas() haven't been called, and unfortunately we also need check whether kernel support PRJQUOTA inside ext4_enable_quotas().....

I think we can ext4_enable_quotas() first on group quota and user quota. And then set the flag of sb_has_quota_active(PROJQUOTA) temporarily. And then call dquot_initialize() to check whether MAXQUOTAS == 3.

[Li Xi]

Instead of printing an error, returning error seems better. It will prevent any future problem.

[Wang Shilong (Inactive)]

The good new is this issue won't exist for RHEL8 and later kernels though..

[Wang Shilong (Inactive)]

unfortunately it looks hard to make it work ideally, we coud not be blocked at this, at least we need fix common case, how about
we add some warning messages during mount time?

diff --git a/fs/ext4/super.c b/fs/ext4/super.c
index ca8b50c8..cddf6595 100644
--- a/fs/ext4/super.c
+++ b/fs/ext4/super.c
@@ -4489,6 +4489,13 @@ no_journal:
        ratelimit_state_init(&sbi->s_msg_ratelimit_state, 5 * HZ, 10);
 
        kfree(orig_data);
+#ifdef  HAVE_PROJECT_QUOTA
+       ext4_msg(sb, KERN_WARNING, 
+               "ext4 module compiled with patched kernel won't work on unpatched kernel");
+#else
+       ext4_msg(sb, KERN_WARNING, 
+               "ext4 module compiled with unpatched kernel won't work on patched kernel");
+#endif
        return 0;

Mark it as a known issue, at least Administrator knows this issue.

[Wang Shilong (Inactive)]

In the mount phase, ROOT inode will call dquot_initialize() first time but at that time ext4_enable_quotas() haven't been called, and unfortunately we also need check whether kernel support PRJQUOTA inside ext4_enable_quotas().....

So this way won't work..

[Li Xi]

I am wondering whether the following way would help use to get the MAXQUOTAS of the running kernel (using a tricky way):

1) We call dquot_initialize(inode) in OSD/ldiskfs.
2) __dquot_initialize() calls inode->i_sb->dq_op->get_projid()
3) We patch ldiskfs to change ext4_get_projid(). In ext4_get_projid(), it set a static flag to show that the kernel actually supports PRJQUOTA.

However, dquot_initialize() depends on ext4_enable_quotas? So might not be workable...

[Wang Shilong (Inactive)]

Hmm, you are right..

But haven't found a way to do that since Kernel mostly use static value MAXQUOTAS..

[Andreas Dilger]

This will still return the size of the structures matching the kernel that ldiskfs was built against, not necessarily the kernel that the module is loaded with.

We would need that function to be exported by the running kernel, which would mean to patch the kernel (obviously not possible for the unpatched kernel, and very unlikely to get upstream).

Instead, we need to find something that is exported from the running kernel that gives us information about whether it supports project quotas.

[Wang Shilong (Inactive)]

Yup, I was thinking that to make us ldiskfs.ko module works for both patched and unpatched kernel:

using following way to check:

bool is_proj_supported_by_kernel(struct super_block *sb)
{
    return (sizeof(sb_dqopt(sb)->info) / sizeof(struct mem_dqinfo)) > PRJQUOTA;
}

[Andreas Dilger]

I think we also need this extra fix for that:

[root@server ext4]# git diff super.c

@@ -5624,7 +5624,7 @@ static int ext4_enable_quotas(struct super_block *sb)
 
        sb_dqopt(sb)->flags |= DQUOT_QUOTA_SYS_FILE;
        for (type = 0; type < EXT4_MAXQUOTAS; type++) {
-               if (qf_inums[type]) {
+               if (qf_inums[type] && type < MAXQUOTAS) {
                        err = ext4_quota_enable(sb, type, QFMT_VFS_V1,
                                                DQUOT_USAGE_ENABLED);
                        if (err) {

It isn't clear if this will fix all of the problems, since it is taking the MAXQUOTAS value from the kernel at build time, not at runtime. Since we always build against a patched kernel with MAXQUOTAS=3, it wouldn't restrict the code. It probably would help for the case of building against a patched kernel.

[Wang Shilong (Inactive)]

Also it will be interesting that with project quota enabled but unpatched kernel used, I just tested it will fail:

  642.115121]  [<ffffffff93379262>] dump_stack+0x19/0x1b
[  642.115888]  [<ffffffffc091606a>] ldiskfs_quota_enable+0x10a/0x120 [ldiskfs]
[  642.116934]  [<ffffffffc091807b>] ldiskfs_enable_quotas+0x9b/0x110 [ldiskfs]
[  642.117962]  [<ffffffffc091c490>] ldiskfs_fill_super+0x2e60/0x2e80 [ldiskfs]
[  642.118993]  [<ffffffff92e4c893>] mount_bdev+0x1b3/0x1f0
[  642.119790]  [<ffffffffc0919630>] ? ldiskfs_calculate_overhead+0x430/0x430 [ldiskfs]
[  642.120951]  [<ffffffffc0913605>] ldiskfs_mount+0x15/0x20 [ldiskfs]
[  642.121874]  [<ffffffff92e4d1fe>] mount_fs+0x3e/0x1b0
[  642.122624]  [<ffffffff92de1035>] ? __alloc_percpu+0x15/0x20
[  642.123409]  [<ffffffff92e6b377>] vfs_kern_mount+0x67/0x110
[  642.124225]  [<ffffffff92e6dacf>] do_mount+0x1ef/0xce0
[  642.124983]  [<ffffffff92e4521a>] ? __check_object_size+0x1ca/0x250
[  642.125919]  [<ffffffff92ddbe7f>] ? memdup_user+0x4f/0x80
[  642.126723]  [<ffffffff92e6e903>] SyS_mount+0x83/0xd0
[  642.127521]  [<ffffffff9338bede>] system_call_fastpath+0x25/0x2a

I think we also need this extra fix for that:

[root@server ext4]# git diff super.c
diff --git a/fs/ext4/super.c b/fs/ext4/super.c
index ca8b50c8..e3fccb79 100644
--- a/fs/ext4/super.c
+++ b/fs/ext4/super.c
@@ -5624,7 +5624,7 @@ static int ext4_enable_quotas(struct super_block *sb)
 
        sb_dqopt(sb)->flags |= DQUOT_QUOTA_SYS_FILE;
        for (type = 0; type < EXT4_MAXQUOTAS; type++) {
-               if (qf_inums[type]) {
+               if (qf_inums[type] && type < MAXQUOTAS) {
                        err = ext4_quota_enable(sb, type, QFMT_VFS_V1,
                                                DQUOT_USAGE_ENABLED);
                        if (err) {