Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-9723

test large xattr (ea_inode) patch interoperability

Details

    • Task
    • Resolution: Fixed
    • Minor
    • None
    • None
    • None
    • 9223372036854775807

    Description

      The large xattr (ea_inode) feature patches are being merged into the upstream 4.13 kernel, but additional changes are being built on top of the EA inode functionality that we use in the ldiskfs patch series. In particular, in the upstream patch series the EA inodes can be shared among multiple parent inodes, and instead of having a backpointer to the parent inode/generation the shared inodes have a refcount and a hash of the xattr value to verify that the correct xattr is referenced. Once we update to a newer vendor kernel that includes these changes, we can remove the patch from ldiskfs to reduce ongoing maintenance efforts, but we can't wait until that time to verify that the feature works properly with existing Lustre filesystems.

      There is supposed to be interoperability functionality in the upstream feature to allow access to existing Lustre EA inodes. We need to test that the ext4 patches being landed to the upstream kernel are able to work with xattrs created by Lustre, and that the upstream e2fsprogs will not consider Lustre EA inodes to be corrupt. This testing needs to be done in the next week or two, to ensure that we can feed back any issues to the upstream ext4 maintainers before that feature is released in an upstream kernel.

      For testing, something like the following process should be sufficient:

      • create an MDT filesystem with an existing RHEL7 kernel with --mkfsoptions="-O ea_inode"
      • mount filesystem as type lustre
      • create large user.test xattrs via setfattr (up to 64KiB) with verifiable data (e.g. filename repeated many times)
      • dump all xattrs via getfattr -d -m user.test testfiles > xattrs.lustre
      • upgrade the kernel to upstream kernel
      • disable the dirdata feature via debugfs -w -R "feature ^dirdata" /dev/MDT and mount it as type ext4
      • dump all xattrs via getfattr -d -m user.test testfiles > xattrs.ext4 and compare to xattrs.lustre to verify xattr consistency and ensure that no errors are generated by the kernel
      • run new e2fsck on updated filesystem to verify that it does not consider the EA inodes as corrupted, at worst it should offer to fix the refcount and hash values of those inodes

      The upstream kernel patches are included on the dev branch of https://git.kernel.org/pub/scm/linux/kernel/git/tytso/ext4.git (all of the patches from author "Tahsin Erdogan") but could potentially also be pushed as a series to the fs/linux-staging branch (with Test-Parameters: forbuildonly since there is no benefit to Lustre testing on them) in order to have Jenkins build the patches for testing via loadjenkinsbuild.

      Attachments

        1. e2fsck_new.log
          2 kB
        2. e2fsck.log
          2 kB
        3. xattrs_after_new_e2fs.lustre
          64 kB
        4. xattrs.ext4
          64 kB
        5. xattrs.lustre
          64 kB

        Issue Links

          Activity

            [LU-9723] test large xattr (ea_inode) patch interoperability

            So it looks like all that is needed is to set err = 0 in the case that the parent inode and generation match ("Do not add ea_inode to the cache") . That must be true or the error would have been printed.

            Could you please give that a try? If that fixes the problem then please attach the patch here for review and then we can send it upstream.

            adilger Andreas Dilger added a comment - So it looks like all that is needed is to set err = 0 in the case that the parent inode and generation match ("Do not add ea_inode to the cache") . That must be true or the error would have been printed. Could you please give that a try? If that fixes the problem then please attach the patch here for review and then we can send it upstream.
            emoly.liu Emoly Liu added a comment - - edited

            adilger and bzzz, the debugging information shows the error " -EFSCORRUPTED" is returned from the following code in fs/ext4/xattr.c: ext4_xattr_inode_verify_hashes()

            static int
            ext4_xattr_inode_verify_hashes(struct inode *ea_inode,
                                           struct ext4_xattr_entry *entry, void *buffer,
                                           size_t size)
            {
                    u32 hash;
            
                    /* Verify stored hash matches calculated hash. */
                    hash = ext4_xattr_inode_hash(EXT4_SB(ea_inode->i_sb), buffer, size);
                    if (hash != ext4_xattr_inode_get_hash(ea_inode))
                            return -EFSCORRUPTED;   //return error in this line
            
                    if (entry) {
                            __le32 e_hash, tmp_data;
            
                            /* Verify entry hash. */
                            tmp_data = cpu_to_le32(hash);
                            e_hash = ext4_xattr_hash_entry(entry->e_name, entry->e_name_len,
                                                           &tmp_data, 1);
                            if (e_hash != entry->e_hash)
                                    return -EFSCORRUPTED;
                    }
                    return 0;
            }
            
            
            

            and then called by ext4_xattr_inode_get():

                    err = ext4_xattr_inode_verify_hashes(ea_inode, entry, buffer, size);
                    /*
                     * Compatibility check for old Lustre ea_inode implementation. Old
                     * version does not have hash validation, but it has a backpointer
                     * from ea_inode to the parent inode.
                     */
                    if (err == -EFSCORRUPTED) {
                            if (EXT4_XATTR_INODE_GET_PARENT(ea_inode) != inode->i_ino ||
                                ea_inode->i_generation != inode->i_generation) {
                                    ext4_warning_inode(ea_inode,
                                                       "EA inode hash validation failed");
                                    goto out;
                            }
                            /* Do not add ea_inode to the cache. */
                            ea_inode_cache = NULL;
                    } else if (err)
                            goto out;
            
            
            
            emoly.liu Emoly Liu added a comment - - edited adilger and bzzz , the debugging information shows the error " -EFSCORRUPTED" is returned from the following code in fs/ext4/xattr.c: ext4_xattr_inode_verify_hashes() static int ext4_xattr_inode_verify_hashes(struct inode *ea_inode, struct ext4_xattr_entry *entry, void *buffer, size_t size) { u32 hash; /* Verify stored hash matches calculated hash. */ hash = ext4_xattr_inode_hash(EXT4_SB(ea_inode->i_sb), buffer, size); if (hash != ext4_xattr_inode_get_hash(ea_inode)) return -EFSCORRUPTED; //return error in this line if (entry) { __le32 e_hash, tmp_data; /* Verify entry hash. */ tmp_data = cpu_to_le32(hash); e_hash = ext4_xattr_hash_entry(entry->e_name, entry->e_name_len, &tmp_data, 1); if (e_hash != entry->e_hash) return -EFSCORRUPTED; } return 0; } and then called by ext4_xattr_inode_get(): err = ext4_xattr_inode_verify_hashes(ea_inode, entry, buffer, size); /* * Compatibility check for old Lustre ea_inode implementation. Old * version does not have hash validation, but it has a backpointer * from ea_inode to the parent inode. */ if (err == -EFSCORRUPTED) { if (EXT4_XATTR_INODE_GET_PARENT(ea_inode) != inode->i_ino || ea_inode->i_generation != inode->i_generation) { ext4_warning_inode(ea_inode, "EA inode hash validation failed"); goto out; } /* Do not add ea_inode to the cache. */ ea_inode_cache = NULL; } else if (err) goto out;
            emoly.liu Emoly Liu added a comment -

            Could you please also run the upstream e2fsck from https://git.kernel.org/pub/scm/fs/ext2/e2fsprogs.git"next" branch against the MDT filesystem after the Lustre e2fsck 1.42.13.wc5 is run to clean up the dirdata errors. That should be possible on the current filesystem if it is still available, but it may prevent debugging the -EUCLEAN error further.

            Here is the output of running the new e2fsck on the MDT device after the Lustre e2fsck 1.42.13.wc5 is run to clean up the dirdata errors.

            [root@centos7-4 e2fsck]# ./e2fsck -d -v -t -t -f -y /tmp/lustre-mdt1 2>&1 | tee e2fsck_new.log
            e2fsck 1.43.5-WIP (17-Feb-2017)
            Pass 1: Checking inodes, blocks, and sizes
            Pass 1: Memory used: 272k/0k (94k/179k), time:  0.00/ 0.00/ 0.00
            Pass 1: I/O read: 1MB, write: 1MB, rate: 521.10MB/s
            Pass 2: Checking directory structure
            Entry '..' in /ROOT/.lustre/fid (53378) has an incorrect filetype (was 18, should be 2).
            Fix? yes
            
            Entry '..' in /ROOT/.lustre/lost+found (53379) has an incorrect filetype (was 18, should be 2).
            Fix? yes
            
            Pass 2: Memory used: 272k/0k (102k/171k), time:  0.00/ 0.00/ 0.00
            Pass 2: I/O read: 1MB, write: 1MB, rate: 660.07MB/s
            Pass 3: Checking directory connectivity
            Peak memory: Memory used: 272k/0k (103k/170k), time:  0.01/ 0.00/ 0.00
            Pass 3A: Memory used: 272k/0k (103k/170k), time:  0.00/ 0.00/ 0.00
            Pass 3A: I/O read: 0MB, write: 0MB, rate: 0.00MB/s
            Pass 3: Memory used: 272k/0k (101k/172k), time:  0.00/ 0.00/ 0.00
            Pass 3: I/O read: 1MB, write: 0MB, rate: 6211.18MB/s
            Pass 4: Checking reference counts
            Pass 4: Memory used: 272k/0k (65k/208k), time:  0.00/ 0.00/ 0.00
            Pass 4: I/O read: 1MB, write: 0MB, rate: 554.32MB/s
            Pass 5: Checking group summary information
            Pass 5: Memory used: 272k/0k (64k/209k), time:  0.00/ 0.00/ 0.00
            Pass 5: I/O read: 1MB, write: 0MB, rate: 547.65MB/s
            
            lustre-MDT0000: ***** FILE SYSTEM WAS MODIFIED *****
            
                     265 inodes used (0.33%, out of 79992)
                       3 non-contiguous files (1.1%)
                       0 non-contiguous directories (0.0%)
                         # of inodes with ind/dind/tind blocks: 1/0/0
                   24543 blocks used (49.09%, out of 50000)
                       0 bad blocks
                       1 large file
            
                     140 regular files
                     116 directories
                       0 character device files
                       0 block device files
                       0 fifos
                       0 links
                       0 symbolic links (0 fast symbolic links)
                       0 sockets
            ------------
                     255 files
            Memory used: 272k/0k (63k/210k), time:  0.03/ 0.01/ 0.00
            I/O read: 2MB, write: 1MB, rate: 74.95MB/s
            
            

            I will upload the result in file e2fsck_new.log.

            It would also be good to check if it is possible to mount the filesystem back as type Lustre on the old kernel after the new e2fsck is run, since this should keep the existing large xattrs as-is so that we can still mount use the old ldiskfs ea_inode code.

            The answer is yes. I can mount the filesystem back as type Lustre on the old kernel after the new e2fsck is run. Here is the output:

            [root@centos7-4 tests]# mount -t lustre   -o loop /tmp/lustre-mdt1 /mnt/lustre-mds1
            [root@centos7-4 tests]# mount -t lustre   -o loop /tmp/lustre-ost1 /mnt/lustre-ost1
            [root@centos7-4 tests]# mount -t lustre -o user_xattr,flock centos7-4@tcp:/lustre /mnt/lustre
            [root@centos7-4 tests]# ls -al /mnt/lustre/testfile 
            -rw-r--r--. 1 root root 0 Jul  7 12:28 /mnt/lustre/testfile
            [root@centos7-4 tests]# getfattr -d -m user.test /mnt/lustre/testfile | tee xattrs_after_new_e2fs.lustre
            getfattr: Removing leading '/' from absolute path names
            # file: mnt/lustre/testfile
            user.test="abcdefghijklmnopqrstuvwxyz123456abcdefghijk...
            
            

            I will upload file xattrs_after_new_e2fs.lustre later too.

            Next, as we discussed, I will add some printk() lines to the places where -EFSCORRUPTED is returned so that we can see why it thinks the xattr is bad.

            emoly.liu Emoly Liu added a comment - Could you please also run the upstream e2fsck from  https://git.kernel.org/pub/scm/fs/ext2/e2fsprogs.git "next" branch against the MDT filesystem  after  the Lustre e2fsck 1.42.13.wc5 is run to clean up the  dirdata  errors. That should be possible on the current filesystem if it is still available, but it may prevent debugging the -EUCLEAN error further. Here is the output of running the new e2fsck on the MDT device  after  the Lustre e2fsck 1.42.13.wc5 is run to clean up the  dirdata  errors. [root@centos7-4 e2fsck]# ./e2fsck -d -v -t -t -f -y /tmp/lustre-mdt1 2>&1 | tee e2fsck_new.log e2fsck 1.43.5-WIP (17-Feb-2017) Pass 1: Checking inodes, blocks, and sizes Pass 1: Memory used: 272k/0k (94k/179k), time: 0.00/ 0.00/ 0.00 Pass 1: I/O read: 1MB, write: 1MB, rate: 521.10MB/s Pass 2: Checking directory structure Entry '..' in /ROOT/.lustre/fid (53378) has an incorrect filetype (was 18, should be 2). Fix? yes Entry '..' in /ROOT/.lustre/lost+found (53379) has an incorrect filetype (was 18, should be 2). Fix? yes Pass 2: Memory used: 272k/0k (102k/171k), time: 0.00/ 0.00/ 0.00 Pass 2: I/O read: 1MB, write: 1MB, rate: 660.07MB/s Pass 3: Checking directory connectivity Peak memory: Memory used: 272k/0k (103k/170k), time: 0.01/ 0.00/ 0.00 Pass 3A: Memory used: 272k/0k (103k/170k), time: 0.00/ 0.00/ 0.00 Pass 3A: I/O read: 0MB, write: 0MB, rate: 0.00MB/s Pass 3: Memory used: 272k/0k (101k/172k), time: 0.00/ 0.00/ 0.00 Pass 3: I/O read: 1MB, write: 0MB, rate: 6211.18MB/s Pass 4: Checking reference counts Pass 4: Memory used: 272k/0k (65k/208k), time: 0.00/ 0.00/ 0.00 Pass 4: I/O read: 1MB, write: 0MB, rate: 554.32MB/s Pass 5: Checking group summary information Pass 5: Memory used: 272k/0k (64k/209k), time: 0.00/ 0.00/ 0.00 Pass 5: I/O read: 1MB, write: 0MB, rate: 547.65MB/s lustre-MDT0000: ***** FILE SYSTEM WAS MODIFIED ***** 265 inodes used (0.33%, out of 79992) 3 non-contiguous files (1.1%) 0 non-contiguous directories (0.0%) # of inodes with ind/dind/tind blocks: 1/0/0 24543 blocks used (49.09%, out of 50000) 0 bad blocks 1 large file 140 regular files 116 directories 0 character device files 0 block device files 0 fifos 0 links 0 symbolic links (0 fast symbolic links) 0 sockets ------------ 255 files Memory used: 272k/0k (63k/210k), time: 0.03/ 0.01/ 0.00 I/O read: 2MB, write: 1MB, rate: 74.95MB/s I will upload the result in file e2fsck_new.log. It would also be good to check if it is possible to mount the filesystem back as type Lustre on the old kernel after the new e2fsck is run, since this should keep the existing large xattrs as-is so that we can still mount use the old ldiskfs  ea_inode  code. The answer is yes. I can mount the filesystem back as type Lustre on the old kernel after the new e2fsck is run. Here is the output: [root@centos7-4 tests]# mount -t lustre -o loop /tmp/lustre-mdt1 /mnt/lustre-mds1 [root@centos7-4 tests]# mount -t lustre -o loop /tmp/lustre-ost1 /mnt/lustre-ost1 [root@centos7-4 tests]# mount -t lustre -o user_xattr,flock centos7-4@tcp:/lustre /mnt/lustre [root@centos7-4 tests]# ls -al /mnt/lustre/testfile -rw-r--r--. 1 root root 0 Jul 7 12:28 /mnt/lustre/testfile [root@centos7-4 tests]# getfattr -d -m user.test /mnt/lustre/testfile | tee xattrs_after_new_e2fs.lustre getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/testfile user.test="abcdefghijklmnopqrstuvwxyz123456abcdefghijk... I will upload file xattrs_after_new_e2fs.lustre later too. Next, as we discussed, I will add some printk() lines to the places where -EFSCORRUPTED is returned so that we can see why it thinks the xattr is bad.
            emoly.liu Emoly Liu added a comment -

            Please check if there are any console messages from ext4 when trying to access the large xattr on the new kernel. This will help debug where the error is coming from.

            I didn't see more messages on the new kernel console and didn't see other useful messages in the system log.

            Could you please also run the upstream e2fsck from https://git.kernel.org/pub/scm/fs/ext2/e2fsprogs.git"next" branch against the MDT filesystem after the Lustre e2fsck 1.42.13.wc5 is run to clean up the dirdata errors. That should be possible on the current filesystem if it is still available, but it may prevent debugging the -EUCLEAN error further.

            It would also be good to check if it is possible to mount the filesystem back as type Lustre on the old kernel after the new e2fsck is run, since this should keep the existing large xattrs as-is so that we can still mount use the old ldiskfs ea_inode code.

            OK, I will download that upstream e2fsck and do these tests.

            emoly.liu Emoly Liu added a comment - Please check if there are any console messages from ext4 when trying to access the large xattr on the new kernel. This will help debug where the error is coming from. I didn't see more messages on the new kernel console and didn't see other useful messages in the system log. Could you please also run the upstream e2fsck from  https://git.kernel.org/pub/scm/fs/ext2/e2fsprogs.git "next" branch against the MDT filesystem  after  the Lustre e2fsck 1.42.13.wc5 is run to clean up the  dirdata  errors. That should be possible on the current filesystem if it is still available, but it may prevent debugging the -EUCLEAN error further. It would also be good to check if it is possible to mount the filesystem back as type Lustre on the old kernel after the new e2fsck is run, since this should keep the existing large xattrs as-is so that we can still mount use the old ldiskfs  ea_inode  code. OK, I will download that upstream e2fsck and do these tests.

            Please check if there are any console messages from ext4 when trying to access the large xattr on the new kernel. This will help debug where the error is coming from.

            Could you please also run the upstream e2fsck from https://git.kernel.org/pub/scm/fs/ext2/e2fsprogs.git "next" branch against the MDT filesystem after the Lustre e2fsck 1.42.13.wc5 is run to clean up the dirdata errors. That should be possible on the current filesystem if it is still available, but it may prevent debugging the -EUCLEAN error further.

            It would also be good to check if it is possible to mount the filesystem back as type Lustre on the old kernel after the new e2fsck is run, since this should keep the existing large xattrs as-is so that we can still mount use the old ldiskfs ea_inode code.

            adilger Andreas Dilger added a comment - Please check if there are any console messages from ext4 when trying to access the large xattr on the new kernel. This will help debug where the error is coming from. Could you please also run the upstream e2fsck from https://git.kernel.org/pub/scm/fs/ext2/e2fsprogs.git "next" branch against the MDT filesystem after the Lustre e2fsck 1.42.13.wc5 is run to clean up the dirdata errors. That should be possible on the current filesystem if it is still available, but it may prevent debugging the -EUCLEAN error further. It would also be good to check if it is possible to mount the filesystem back as type Lustre on the old kernel after the new e2fsck is run, since this should keep the existing large xattrs as-is so that we can still mount use the old ldiskfs ea_inode code.
            emoly.liu Emoly Liu added a comment - - edited

            Andreas, sorry, the test I did yesterday was not correct. Here are the new steps and results:

            • env:
              • lustre: top commit "f7df236 LU-9183"
              • lustre kernel: 3.10.0-514.16.1.el7_lustre.x86_64
              • upstream kernel: top commit "0c5f031 Add linux-next specific files for 20170705" applied with ext4 patch "6f82dfb ext4: change fast symlink test to not rely on i_blocks"; compiled version "4.12.0-lu9723-next-20170705+"
              • e2fsprogs: 1.42.13.wc6-7.el7.x86_64
            • steps:
              # mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/root/lustre-release/lustre/tests/../utils/l_getidentity --backfstype=ldiskfs --device-size=200000 --mkfsoptions="-O ea_inode" --reformat /tmp/lustre-mdt1
              # mkfs.lustre --mgsnode=centos7-4@tcp --fsname=lustre --ost --index=0 --param=sys.timeout=20 --backfstype=ldiskfs --device-size=400000 --reformat /tmp/lustre-ost1
              # mount -t lustre   -o loop /tmp/lustre-mdt1 /mnt/lustre-mds1
              # mount -t lustre   -o loop /tmp/lustre-ost1 /mnt/lustre-ost1
              # mount -t lustre -o user_xattr,flock centos7-4@tcp:/lustre /mnt/lustre
              # touch /mnt/lustre/testfile
              # setfattr -n user.test -v $(for i in `seq 1 2048`; do echo -n "abcdefghijklmnopqrstuvwxyz123456";done) /mnt/lustre/testfile
              # getfattr -d -m user.test /mnt/lustre/testfile > xattrs.lustre
              
              

              Then upgrade to the upstream kernel

              # debugfs -w -R 'feature ^dirdata' /tmp/lustre-mdt1
              debugfs 1.42.13.wc6 (05-Feb-2017)
              Filesystem features: has_journal ext_attr resize_inode dir_index filetype flex_bg ea_inode sparse_super large_file huge_file uninit_bg dir_nlink quota
              # mount -t ext4 /tmp/lustre-mdt1 /mnt/lustre-mds1
              # getfattr -d -m user.test /mnt/lustre-mds1/ROOT/testfile 2>&1 | tee xattrs.ext4
              /mnt/lustre-mds1/ROOT/testfile: user.test: Structure needs cleaning
              # umount /mnt/lustre-mds1
              # e2fsck -d -v -t -t -f -y /tmp/lustre-mdt1 2>&1 | tee e2fsck.log
              e2fsck 1.42.13.wc6 (05-Feb-2017)
              Pass 1: Checking inodes, blocks, and sizes
              Pass 1: Memory used: 264k/0k (83k/182k), time:  0.00/ 0.00/ 0.00
              Pass 1: I/O read: 1MB, write: 1MB, rate: 492.85MB/s
              Pass 2: Checking directory structure
              Entry '[0x200000400:0x1:0x0]' in /update_log_dir (53375) dirdata length set incorrectly.
              Clear? yes
              
              Entry '.lustre' in /ROOT (53376) dirdata length set incorrectly.
              Clear? yes
              
              Entry 'testfile' in /ROOT (53376) dirdata length set incorrectly.
              Clear? yes
              
              Entry '..' in /ROOT/.lustre (53377) dirdata length set incorrectly.
              Clear? yes
              
              Entry 'fid' in /ROOT/.lustre (53377) dirdata length set incorrectly.
              Clear? yes
              
              Entry 'lost+found' in /ROOT/.lustre (53377) dirdata length set incorrectly.
              Clear? yes
              
              Entry '..' in /ROOT/.lustre/fid (53378) dirdata length set incorrectly.
              Clear? yes
              
              Entry '..' in /ROOT/.lustre/lost+found (53379) dirdata length set incorrectly.
              Clear? yes
              
              Pass 2: Memory used: 264k/0k (91k/174k), time:  0.00/ 0.00/ 0.00
              Pass 2: I/O read: 1MB, write: 1MB, rate: 229.78MB/s
              Pass 3: Checking directory connectivity
              Peak memory: Memory used: 264k/0k (91k/174k), time:  0.01/ 0.00/ 0.00
              Pass 3A: Memory used: 264k/0k (91k/174k), time:  0.00/ 0.00/ 0.00
              Pass 3A: I/O read: 0MB, write: 0MB, rate: 0.00MB/s
              Pass 3: Memory used: 264k/0k (89k/176k), time:  0.00/ 0.00/ 0.00
              Pass 3: I/O read: 1MB, write: 0MB, rate: 9803.92MB/s
              Pass 4: Checking reference counts
              Pass 4: Memory used: 264k/0k (61k/204k), time:  0.00/ 0.00/ 0.00
              Pass 4: I/O read: 1MB, write: 0MB, rate: 512.56MB/s
              Pass 5: Checking group summary information
              Pass 5: Memory used: 264k/0k (60k/205k), time:  0.00/ 0.00/ 0.00
              Pass 5: I/O read: 1MB, write: 0MB, rate: 396.04MB/s
              
              lustre-MDT0000: ***** FILE SYSTEM WAS MODIFIED *****
              
                       265 inodes used (0.33%, out of 79992)
                         3 non-contiguous files (1.1%)
                         0 non-contiguous directories (0.0%)
                           # of inodes with ind/dind/tind blocks: 1/0/0
                     24543 blocks used (49.09%, out of 50000)
                         0 bad blocks
                         1 large file
              
                       140 regular files
                       116 directories
                         0 character device files
                         0 block device files
                         0 fifos
                         0 links
                         0 symbolic links (0 fast symbolic links)
                         0 sockets
              ------------
                       255 files
              Memory used: 264k/0k (60k/205k), time:  0.02/ 0.01/ 0.00
              I/O read: 1MB, write: 1MB, rate: 60.79MB/s
              
              

            We can see getfattr report error "Structure needs cleaning" on upstream kernel. When I ran "strace getfattr xxx", it showed the following logs:

            getxattr("/mnt/lustre-mds1/ROOT/testfile", "user.test", 0x0, 0) = 65536
            getxattr("/mnt/lustre-mds1/ROOT/testfile", "user.test", 0x1685b20, 65536) = -1 EUCLEAN (Structure needs cleaning)
            write(2, "/mnt/lustre-mds1/ROOT/testfile: ", 32/mnt/lustre-mds1/ROOT/testfile: ) = 32
            
            

            I will upload xattr.*, e2fsck.log and strace.log later.

            emoly.liu Emoly Liu added a comment - - edited Andreas, sorry, the test I did yesterday was not correct. Here are the new steps and results: env: lustre: top commit "f7df236 LU-9183 " lustre kernel: 3.10.0-514.16.1.el7_lustre.x86_64 upstream kernel: top commit "0c5f031 Add linux-next specific files for 20170705" applied with ext4 patch "6f82dfb ext4: change fast symlink test to not rely on i_blocks"; compiled version "4.12.0-lu9723-next-20170705+" e2fsprogs: 1.42.13.wc6-7.el7.x86_64 steps: # mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/root/lustre-release/lustre/tests/../utils/l_getidentity --backfstype=ldiskfs --device-size=200000 --mkfsoptions="-O ea_inode" --reformat /tmp/lustre-mdt1 # mkfs.lustre --mgsnode=centos7-4@tcp --fsname=lustre --ost --index=0 --param=sys.timeout=20 --backfstype=ldiskfs --device-size=400000 --reformat /tmp/lustre-ost1 # mount -t lustre -o loop /tmp/lustre-mdt1 /mnt/lustre-mds1 # mount -t lustre -o loop /tmp/lustre-ost1 /mnt/lustre-ost1 # mount -t lustre -o user_xattr,flock centos7-4@tcp:/lustre /mnt/lustre # touch /mnt/lustre/testfile # setfattr -n user.test -v $(for i in `seq 1 2048`; do echo -n "abcdefghijklmnopqrstuvwxyz123456";done) /mnt/lustre/testfile # getfattr -d -m user.test /mnt/lustre/testfile > xattrs.lustre Then upgrade to the upstream kernel # debugfs -w -R 'feature ^dirdata' /tmp/lustre-mdt1 debugfs 1.42.13.wc6 (05-Feb-2017) Filesystem features: has_journal ext_attr resize_inode dir_index filetype flex_bg ea_inode sparse_super large_file huge_file uninit_bg dir_nlink quota # mount -t ext4 /tmp/lustre-mdt1 /mnt/lustre-mds1 # getfattr -d -m user.test /mnt/lustre-mds1/ROOT/testfile 2>&1 | tee xattrs.ext4 /mnt/lustre-mds1/ROOT/testfile: user.test: Structure needs cleaning # umount /mnt/lustre-mds1 # e2fsck -d -v -t -t -f -y /tmp/lustre-mdt1 2>&1 | tee e2fsck.log e2fsck 1.42.13.wc6 (05-Feb-2017) Pass 1: Checking inodes, blocks, and sizes Pass 1: Memory used: 264k/0k (83k/182k), time: 0.00/ 0.00/ 0.00 Pass 1: I/O read: 1MB, write: 1MB, rate: 492.85MB/s Pass 2: Checking directory structure Entry '[0x200000400:0x1:0x0]' in /update_log_dir (53375) dirdata length set incorrectly. Clear? yes Entry '.lustre' in /ROOT (53376) dirdata length set incorrectly. Clear? yes Entry 'testfile' in /ROOT (53376) dirdata length set incorrectly. Clear? yes Entry '..' in /ROOT/.lustre (53377) dirdata length set incorrectly. Clear? yes Entry 'fid' in /ROOT/.lustre (53377) dirdata length set incorrectly. Clear? yes Entry 'lost+found' in /ROOT/.lustre (53377) dirdata length set incorrectly. Clear? yes Entry '..' in /ROOT/.lustre/fid (53378) dirdata length set incorrectly. Clear? yes Entry '..' in /ROOT/.lustre/lost+found (53379) dirdata length set incorrectly. Clear? yes Pass 2: Memory used: 264k/0k (91k/174k), time: 0.00/ 0.00/ 0.00 Pass 2: I/O read: 1MB, write: 1MB, rate: 229.78MB/s Pass 3: Checking directory connectivity Peak memory: Memory used: 264k/0k (91k/174k), time: 0.01/ 0.00/ 0.00 Pass 3A: Memory used: 264k/0k (91k/174k), time: 0.00/ 0.00/ 0.00 Pass 3A: I/O read: 0MB, write: 0MB, rate: 0.00MB/s Pass 3: Memory used: 264k/0k (89k/176k), time: 0.00/ 0.00/ 0.00 Pass 3: I/O read: 1MB, write: 0MB, rate: 9803.92MB/s Pass 4: Checking reference counts Pass 4: Memory used: 264k/0k (61k/204k), time: 0.00/ 0.00/ 0.00 Pass 4: I/O read: 1MB, write: 0MB, rate: 512.56MB/s Pass 5: Checking group summary information Pass 5: Memory used: 264k/0k (60k/205k), time: 0.00/ 0.00/ 0.00 Pass 5: I/O read: 1MB, write: 0MB, rate: 396.04MB/s lustre-MDT0000: ***** FILE SYSTEM WAS MODIFIED ***** 265 inodes used (0.33%, out of 79992) 3 non-contiguous files (1.1%) 0 non-contiguous directories (0.0%) # of inodes with ind/dind/tind blocks: 1/0/0 24543 blocks used (49.09%, out of 50000) 0 bad blocks 1 large file 140 regular files 116 directories 0 character device files 0 block device files 0 fifos 0 links 0 symbolic links (0 fast symbolic links) 0 sockets ------------ 255 files Memory used: 264k/0k (60k/205k), time: 0.02/ 0.01/ 0.00 I/O read: 1MB, write: 1MB, rate: 60.79MB/s We can see getfattr report error " Structure needs cleaning " on upstream kernel. When I ran "strace getfattr xxx", it showed the following logs: getxattr("/mnt/lustre-mds1/ROOT/testfile", "user.test", 0x0, 0) = 65536 getxattr("/mnt/lustre-mds1/ROOT/testfile", "user.test", 0x1685b20, 65536) = -1 EUCLEAN (Structure needs cleaning) write(2, "/mnt/lustre-mds1/ROOT/testfile: ", 32/mnt/lustre-mds1/ROOT/testfile: ) = 32 I will upload xattr.*, e2fsck.log and strace.log later.

            Hi Emoly, thank you for your testing. It looks like the upstream kernel ea_inode feature works properly to access the Lustre xattrs, which is great.

            Could you please check if there are any error messages on the console when accessing the large xattr on the upstream kernel. Could you also run the upstream e2fsprogs with the ea_inode feature against the MDT filesystem (possibly after the Lustre e2fsprogs runs to clean up dirdata errors) to see that the new e2fsprogs works properly with existing large xattrs.

            adilger Andreas Dilger added a comment - Hi Emoly, thank you for your testing. It looks like the upstream kernel ea_inode feature works properly to access the Lustre xattrs, which is great. Could you please check if there are any error messages on the console when accessing the large xattr on the upstream kernel. Could you also run the upstream e2fsprogs with the ea_inode feature against the MDT filesystem (possibly after the Lustre e2fsprogs runs to clean up dirdata errors) to see that the new e2fsprogs works properly with existing large xattrs.
            emoly.liu Emoly Liu added a comment -

            I hit some issues when upgrading to the upstream kernel. I'm looking into them.

            emoly.liu Emoly Liu added a comment - I hit some issues when upgrading to the upstream kernel. I'm looking into them.
            emoly.liu Emoly Liu added a comment -

            Just had a check at the latest upstream kernel cloned from git://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git, it included all the ext4 patches I posted above, except the top one "6f82dfb ext4: change fast symlink test to not rely on i_blocks".

            Next, I will compile the upstream kernel with that patch and perform the test. I will update the status later.

            emoly.liu Emoly Liu added a comment - Just had a check at the latest upstream kernel cloned from git://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git, it included all the ext4 patches I posted above, except the top one "6f82dfb ext4: change fast symlink test to not rely on i_blocks". Next, I will compile the upstream kernel with that patch and perform the test. I will update the status later.
            emoly.liu Emoly Liu added a comment - - edited

            Andreas,
            Cloning https://git.kernel.org/pub/scm/linux/kernel/git/tytso/ext4.git is extremely slow for me locally, so I did it on trevis instead. The following commits signed-off by "Tahsin Erdogan", related to ext4, are all needed to apply to upstream kernel?

            -sh-4.1$ git log --author="Tahsin Erdogan" --oneline|grep ext4
            6f82dfb ext4: change fast symlink test to not rely on i_blocks
            ad39acd ext4: add nombcache mount option
            1205d54 ext4: strong binding of xattr inode references
            17bb880 ext4: eliminate xattr entry e_hash recalculation for removes
            de5b47d ext4: reserve space for xattr entries/names
            82fd1c9 ext4: xattr inode deduplication
            30a7eb9 ext4: cleanup transaction restarts during inode deletion
            02749a4 ext4: add ext4_is_quota_file()
            4738740 ext2, ext4: make mb block cache names more explicit
            b6d9029 ext4: move struct ext4_xattr_inode_array to xattr.h
            0421a18 ext4: modify ext4_xattr_ino_array to hold struct inode *
            c1a5d5f ext4: improve journal credit handling in set xattr paths
            65d3000 ext4: ext4_xattr_delete_inode() should return accurate errors
            b347e2b ext4: retry storing value in external inode with xattr block too
            b315529 ext4: fix credits calculation for xattr inode
            7cec191 ext4: fix ext4_xattr_cmp()
            f610910 ext4: fix ext4_xattr_move_to_block()
            9bb21ce ext4: fix ext4_xattr_make_inode_space() value size calculation
            0bd454c ext4: ext4_xattr_value_same() should return false for external data
            990461d ext4: add missing le32_to_cpu(e_value_inum) conversions
            90966693 ext4: clean up ext4_xattr_inode_get()
            bab79b0 ext4: change ext4_xattr_inode_iget() signature
            0eefb10 ext4: extended attribute value size limit is enforced by vfs
            1e7d359 ext4: fix ref counting for ea_inode
            ddfa17e ext4: call journal revoke when freeing ea_inode blocks
            9e1ba00 ext4: ea_inode owner should be the same as the inode owner
            bd3b963 ext4: attach jinode after creation of xattr inode
            1b917ed ext4: do not set posix acls on xattr inodes
            0de5983 ext4: lock inode before calling ext4_orphan_add()
            33d201e ext4: fix lockdep warning about recursive inode locking
            b8cb5a5 ext4: fix quota charging for shared xattr blocks
            
            emoly.liu Emoly Liu added a comment - - edited Andreas, Cloning https://git.kernel.org/pub/scm/linux/kernel/git/tytso/ext4.git is extremely slow for me locally, so I did it on trevis instead. The following commits signed-off by "Tahsin Erdogan", related to ext4, are all needed to apply to upstream kernel? -sh-4.1$ git log --author="Tahsin Erdogan" --oneline|grep ext4 6f82dfb ext4: change fast symlink test to not rely on i_blocks ad39acd ext4: add nombcache mount option 1205d54 ext4: strong binding of xattr inode references 17bb880 ext4: eliminate xattr entry e_hash recalculation for removes de5b47d ext4: reserve space for xattr entries/names 82fd1c9 ext4: xattr inode deduplication 30a7eb9 ext4: cleanup transaction restarts during inode deletion 02749a4 ext4: add ext4_is_quota_file() 4738740 ext2, ext4: make mb block cache names more explicit b6d9029 ext4: move struct ext4_xattr_inode_array to xattr.h 0421a18 ext4: modify ext4_xattr_ino_array to hold struct inode * c1a5d5f ext4: improve journal credit handling in set xattr paths 65d3000 ext4: ext4_xattr_delete_inode() should return accurate errors b347e2b ext4: retry storing value in external inode with xattr block too b315529 ext4: fix credits calculation for xattr inode 7cec191 ext4: fix ext4_xattr_cmp() f610910 ext4: fix ext4_xattr_move_to_block() 9bb21ce ext4: fix ext4_xattr_make_inode_space() value size calculation 0bd454c ext4: ext4_xattr_value_same() should return false for external data 990461d ext4: add missing le32_to_cpu(e_value_inum) conversions 90966693 ext4: clean up ext4_xattr_inode_get() bab79b0 ext4: change ext4_xattr_inode_iget() signature 0eefb10 ext4: extended attribute value size limit is enforced by vfs 1e7d359 ext4: fix ref counting for ea_inode ddfa17e ext4: call journal revoke when freeing ea_inode blocks 9e1ba00 ext4: ea_inode owner should be the same as the inode owner bd3b963 ext4: attach jinode after creation of xattr inode 1b917ed ext4: do not set posix acls on xattr inodes 0de5983 ext4: lock inode before calling ext4_orphan_add() 33d201e ext4: fix lockdep warning about recursive inode locking b8cb5a5 ext4: fix quota charging for shared xattr blocks

            One node is enough. Unfortunately, the test configuration is complicated by the fact we don't have dirdata support for the newer upstream kernel, nor can we mount the MDT with upstream kernel since there is no ldiskfs patch series for it.

            Feel free to ask questions here or via Skype if something isn't clear. I'd like to get feedback to the upstream ext4 maintainer and ea_xattr developer within the next week or so.

            adilger Andreas Dilger added a comment - One node is enough. Unfortunately, the test configuration is complicated by the fact we don't have dirdata support for the newer upstream kernel, nor can we mount the MDT with upstream kernel since there is no ldiskfs patch series for it. Feel free to ask questions here or via Skype if something isn't clear. I'd like to get feedback to the upstream ext4 maintainer and ea_xattr developer within the next week or so.

            People

              emoly.liu Emoly Liu
              adilger Andreas Dilger
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: