[LU-15171] corrupted in-inode xattr

• Type: Bug
• Resolution: Fixed
• Priority: Major
• Fix Version/s: Lustre 2.15.0
• Labels: rhel8.3
• Severity: 3
• Assignee: Andrew Perepechko
• Reporter: Shaun Tancheff

Description

LDISKFS-fs error (device md6): ldiskfs_get_inode_usage:818: inode #310: comm ll_ost_io00_570: corrupted in-inode xattr
Aborting journal on device md6-8.
Kernel panic - not syncing: LDISKFS-fs (device md6): panic forced after error

LDISKFS-fs error (device md6): ldiskfs_journal_check_start:61: Detected aborted journal
LDISKFS-fs error (device md6): ldiskfs_journal_check_start:61: Detected aborted journal
LDISKFS-fs (md6): Remounting filesystem read-only
CPU: 9 PID: 55386 Comm: ll_ost_io00_570 4.18.0-305.10.2.x6.0.29.x86_64 #1
Hardware name: Seagate Laguna Seca/Laguna Seca, BIOS v02.0043 09/24/2019
Call Trace:
  dump_stack+0x5c/0x80
  panic+0xe7/0x2a9
  ldiskfs_handle_error.cold.139+0x13/0x13 [ldiskfs]
  __ldiskfs_error_inode+0xaf/0x130 [ldiskfs]
  __xattr_check_inode+0x4a/0x70 [ldiskfs]
  ldiskfs_get_inode_usage+0x195/0x290 [ldiskfs]
  __dquot_transfer+0x8a/0x5d0
  dquot_transfer+0x8e/0x130
  osd_quota_transfer+0x188/0x310 [osd_ldiskfs]
  osd_attr_set+0xd4/0x740 [osd_ldiskfs]
  ofd_write_attr_set+0x7bf/0x1070 [ofd]
  ofd_commitrw_write+0x222/0x1a50 [ofd]
  ofd_commitrw+0x458/0xa60 [ofd]
  tgt_brw_write+0x18de/0x2390 [ptlrpc]
  tgt_request_handle+0xc93/0x1a00 [ptlrpc]
  ptlrpc_server_handle_request+0x323/0xbd0 [ptlrpc]
  ptlrpc_main+0xc06/0x1550 [ptlrpc]

Issue Links

• is related to: LU-16082 old-style Lustre EA inodes support is broken in newer kernel

Comments

[Peter Jones]

Landed for 2.15

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/45424/
Subject: LU-15171 osd-ldiskfs: xattr_sem locking is missing for dquot_transfer
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: e6c7fcdaf40b130c39af2e3ee8b108c6e31a8ca8

[Gerrit Updater]

"Andrew Perepechko <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/45424
Subject: LU-15171 osd-ldiskfs: xattr_sem locking is missing for dquot_transfer
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: fb9f97b3481214017be8de89555076262ecaa6e1

[Andrew Perepechko]

It seems that this issue is not really a corruption. At least I haven't found any corruption in the buffer heads from the crash dump

Commit 7a9ca53ae (~v4.13) added the requirement for xattr_sem locking when calling *dquot_transfer:

commit 7a9ca53aea10ad4677a0f347ad7639c304b80194
Author: Tahsin Erdogan <[e-mail]>
Date:   Thu Jun 22 11:46:48 2017 -0400

    quota: add get_inode_usage callback to transfer multi-inode charges

...

diff --git a/fs/ext4/inode.c b/fs/ext4/inode.c
index 962f28a0e176..d9733aa955e9 100644
--- a/fs/ext4/inode.c
+++ b/fs/ext4/inode.c
@@ -5295,7 +5295,14 @@ int ext4_setattr(struct dentry *dentry, struct iattr *attr)
                        error = PTR_ERR(handle);
                        goto err_out;
                }
+
+               /* dquot_transfer() calls back ext4_get_inode_usage() which
+                * counts xattr inode references.
+                */
+               down_read(&EXT4_I(inode)->xattr_sem);
                error = dquot_transfer(inode, attr);
+               up_read(&EXT4_I(inode)->xattr_sem);
+
                if (error) {
                        ext4_journal_stop(handle);
                        return error;
diff --git a/fs/ext4/ioctl.c b/fs/ext4/ioctl.c
index dde8deb11e59..42b3a73143cf 100644
--- a/fs/ext4/ioctl.c
+++ b/fs/ext4/ioctl.c
@@ -373,7 +373,13 @@ static int ext4_ioctl_setproject(struct file *filp, __u32 projid)
 
        transfer_to[PRJQUOTA] = dqget(sb, make_kqid_projid(kprojid));
        if (!IS_ERR(transfer_to[PRJQUOTA])) {
+
+               /* __dquot_transfer() calls back ext4_get_inode_usage() which
+                * counts xattr inode references.
+                */
+               down_read(&EXT4_I(inode)->xattr_sem);
                err = __dquot_transfer(inode, transfer_to);
+               up_read(&EXT4_I(inode)->xattr_sem);
                dqput(transfer_to[PRJQUOTA]);
                if (err)
                        goto out_dirty;
...

In Lustre, we do not take this lock. It seems that a first write race is possible when one thread attempts to modify inode xattrs and another thread performs dquot_transfer which analyzes xattr consistency (and eventually fails).

It seems that we can simply wrap *dquot_transfer() calls with xattr locking for the newer kernels. It should be ok performance-wise for the OST side and, hopefully, for the MDT side either. A proof of concept path will be uploaded for review.