[LU-9723] test large xattr (ea_inode) patch interoperability Created: 29/Jun/17 Updated: 10/Aug/18 Due: 07/Jul/17 Resolved: 08/Aug/18 |
|
| Status: | Resolved |
| Project: | Lustre |
| Component/s: | None |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Task | Priority: | Minor |
| Reporter: | Andreas Dilger | Assignee: | Emoly Liu |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | None | ||
| Attachments: |
|
||||||||
| Issue Links: |
|
||||||||
| Rank (Obsolete): | 9223372036854775807 | ||||||||
| Description |
|
The large xattr (ea_inode) feature patches are being merged into the upstream 4.13 kernel, but additional changes are being built on top of the EA inode functionality that we use in the ldiskfs patch series. In particular, in the upstream patch series the EA inodes can be shared among multiple parent inodes, and instead of having a backpointer to the parent inode/generation the shared inodes have a refcount and a hash of the xattr value to verify that the correct xattr is referenced. Once we update to a newer vendor kernel that includes these changes, we can remove the patch from ldiskfs to reduce ongoing maintenance efforts, but we can't wait until that time to verify that the feature works properly with existing Lustre filesystems. There is supposed to be interoperability functionality in the upstream feature to allow access to existing Lustre EA inodes. We need to test that the ext4 patches being landed to the upstream kernel are able to work with xattrs created by Lustre, and that the upstream e2fsprogs will not consider Lustre EA inodes to be corrupt. This testing needs to be done in the next week or two, to ensure that we can feed back any issues to the upstream ext4 maintainers before that feature is released in an upstream kernel. For testing, something like the following process should be sufficient:
The upstream kernel patches are included on the dev branch of https://git.kernel.org/pub/scm/linux/kernel/git/tytso/ext4.git (all of the patches from author "Tahsin Erdogan") but could potentially also be pushed as a series to the fs/linux-staging branch (with Test-Parameters: forbuildonly since there is no benefit to Lustre testing on them) in order to have Jenkins build the patches for testing via loadjenkinsbuild. |
| Comments |
| Comment by Emoly Liu [ 30/Jun/17 ] |
|
Is there any else requirement about the testing environment, e.g. one single node is enough? |
| Comment by Andreas Dilger [ 04/Jul/17 ] |
|
One node is enough. Unfortunately, the test configuration is complicated by the fact we don't have dirdata support for the newer upstream kernel, nor can we mount the MDT with upstream kernel since there is no ldiskfs patch series for it. Feel free to ask questions here or via Skype if something isn't clear. I'd like to get feedback to the upstream ext4 maintainer and ea_xattr developer within the next week or so. |
| Comment by Emoly Liu [ 05/Jul/17 ] |
|
Andreas, -sh-4.1$ git log --author="Tahsin Erdogan" --oneline|grep ext4 6f82dfb ext4: change fast symlink test to not rely on i_blocks ad39acd ext4: add nombcache mount option 1205d54 ext4: strong binding of xattr inode references 17bb880 ext4: eliminate xattr entry e_hash recalculation for removes de5b47d ext4: reserve space for xattr entries/names 82fd1c9 ext4: xattr inode deduplication 30a7eb9 ext4: cleanup transaction restarts during inode deletion 02749a4 ext4: add ext4_is_quota_file() 4738740 ext2, ext4: make mb block cache names more explicit b6d9029 ext4: move struct ext4_xattr_inode_array to xattr.h 0421a18 ext4: modify ext4_xattr_ino_array to hold struct inode * c1a5d5f ext4: improve journal credit handling in set xattr paths 65d3000 ext4: ext4_xattr_delete_inode() should return accurate errors b347e2b ext4: retry storing value in external inode with xattr block too b315529 ext4: fix credits calculation for xattr inode 7cec191 ext4: fix ext4_xattr_cmp() f610910 ext4: fix ext4_xattr_move_to_block() 9bb21ce ext4: fix ext4_xattr_make_inode_space() value size calculation 0bd454c ext4: ext4_xattr_value_same() should return false for external data 990461d ext4: add missing le32_to_cpu(e_value_inum) conversions 90966693 ext4: clean up ext4_xattr_inode_get() bab79b0 ext4: change ext4_xattr_inode_iget() signature 0eefb10 ext4: extended attribute value size limit is enforced by vfs 1e7d359 ext4: fix ref counting for ea_inode ddfa17e ext4: call journal revoke when freeing ea_inode blocks 9e1ba00 ext4: ea_inode owner should be the same as the inode owner bd3b963 ext4: attach jinode after creation of xattr inode 1b917ed ext4: do not set posix acls on xattr inodes 0de5983 ext4: lock inode before calling ext4_orphan_add() 33d201e ext4: fix lockdep warning about recursive inode locking b8cb5a5 ext4: fix quota charging for shared xattr blocks |
| Comment by Emoly Liu [ 05/Jul/17 ] |
|
Just had a check at the latest upstream kernel cloned from git://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git, it included all the ext4 patches I posted above, except the top one "6f82dfb ext4: change fast symlink test to not rely on i_blocks". Next, I will compile the upstream kernel with that patch and perform the test. I will update the status later. |
| Comment by Emoly Liu [ 06/Jul/17 ] |
|
I hit some issues when upgrading to the upstream kernel. I'm looking into them. |
| Comment by Andreas Dilger [ 06/Jul/17 ] |
|
Hi Emoly, thank you for your testing. It looks like the upstream kernel ea_inode feature works properly to access the Lustre xattrs, which is great. Could you please check if there are any error messages on the console when accessing the large xattr on the upstream kernel. Could you also run the upstream e2fsprogs with the ea_inode feature against the MDT filesystem (possibly after the Lustre e2fsprogs runs to clean up dirdata errors) to see that the new e2fsprogs works properly with existing large xattrs. |
| Comment by Emoly Liu [ 07/Jul/17 ] |
|
Andreas, sorry, the test I did yesterday was not correct. Here are the new steps and results:
We can see getfattr report error "Structure needs cleaning" on upstream kernel. When I ran "strace getfattr xxx", it showed the following logs: getxattr("/mnt/lustre-mds1/ROOT/testfile", "user.test", 0x0, 0) = 65536
getxattr("/mnt/lustre-mds1/ROOT/testfile", "user.test", 0x1685b20, 65536) = -1 EUCLEAN (Structure needs cleaning)
write(2, "/mnt/lustre-mds1/ROOT/testfile: ", 32/mnt/lustre-mds1/ROOT/testfile: ) = 32
I will upload xattr.*, e2fsck.log and strace.log later. |
| Comment by Andreas Dilger [ 07/Jul/17 ] |
|
Please check if there are any console messages from ext4 when trying to access the large xattr on the new kernel. This will help debug where the error is coming from. Could you please also run the upstream e2fsck from https://git.kernel.org/pub/scm/fs/ext2/e2fsprogs.git "next" branch against the MDT filesystem after the Lustre e2fsck 1.42.13.wc5 is run to clean up the dirdata errors. That should be possible on the current filesystem if it is still available, but it may prevent debugging the -EUCLEAN error further. It would also be good to check if it is possible to mount the filesystem back as type Lustre on the old kernel after the new e2fsck is run, since this should keep the existing large xattrs as-is so that we can still mount use the old ldiskfs ea_inode code. |
| Comment by Emoly Liu [ 07/Jul/17 ] |
I didn't see more messages on the new kernel console and didn't see other useful messages in the system log.
OK, I will download that upstream e2fsck and do these tests. |
| Comment by Emoly Liu [ 07/Jul/17 ] |
Here is the output of running the new e2fsck on the MDT device after the Lustre e2fsck 1.42.13.wc5 is run to clean up the dirdata errors. [root@centos7-4 e2fsck]# ./e2fsck -d -v -t -t -f -y /tmp/lustre-mdt1 2>&1 | tee e2fsck_new.log
e2fsck 1.43.5-WIP (17-Feb-2017)
Pass 1: Checking inodes, blocks, and sizes
Pass 1: Memory used: 272k/0k (94k/179k), time: 0.00/ 0.00/ 0.00
Pass 1: I/O read: 1MB, write: 1MB, rate: 521.10MB/s
Pass 2: Checking directory structure
Entry '..' in /ROOT/.lustre/fid (53378) has an incorrect filetype (was 18, should be 2).
Fix? yes
Entry '..' in /ROOT/.lustre/lost+found (53379) has an incorrect filetype (was 18, should be 2).
Fix? yes
Pass 2: Memory used: 272k/0k (102k/171k), time: 0.00/ 0.00/ 0.00
Pass 2: I/O read: 1MB, write: 1MB, rate: 660.07MB/s
Pass 3: Checking directory connectivity
Peak memory: Memory used: 272k/0k (103k/170k), time: 0.01/ 0.00/ 0.00
Pass 3A: Memory used: 272k/0k (103k/170k), time: 0.00/ 0.00/ 0.00
Pass 3A: I/O read: 0MB, write: 0MB, rate: 0.00MB/s
Pass 3: Memory used: 272k/0k (101k/172k), time: 0.00/ 0.00/ 0.00
Pass 3: I/O read: 1MB, write: 0MB, rate: 6211.18MB/s
Pass 4: Checking reference counts
Pass 4: Memory used: 272k/0k (65k/208k), time: 0.00/ 0.00/ 0.00
Pass 4: I/O read: 1MB, write: 0MB, rate: 554.32MB/s
Pass 5: Checking group summary information
Pass 5: Memory used: 272k/0k (64k/209k), time: 0.00/ 0.00/ 0.00
Pass 5: I/O read: 1MB, write: 0MB, rate: 547.65MB/s
lustre-MDT0000: ***** FILE SYSTEM WAS MODIFIED *****
265 inodes used (0.33%, out of 79992)
3 non-contiguous files (1.1%)
0 non-contiguous directories (0.0%)
# of inodes with ind/dind/tind blocks: 1/0/0
24543 blocks used (49.09%, out of 50000)
0 bad blocks
1 large file
140 regular files
116 directories
0 character device files
0 block device files
0 fifos
0 links
0 symbolic links (0 fast symbolic links)
0 sockets
------------
255 files
Memory used: 272k/0k (63k/210k), time: 0.03/ 0.01/ 0.00
I/O read: 2MB, write: 1MB, rate: 74.95MB/s
I will upload the result in file e2fsck_new.log.
The answer is yes. I can mount the filesystem back as type Lustre on the old kernel after the new e2fsck is run. Here is the output: [root@centos7-4 tests]# mount -t lustre -o loop /tmp/lustre-mdt1 /mnt/lustre-mds1 [root@centos7-4 tests]# mount -t lustre -o loop /tmp/lustre-ost1 /mnt/lustre-ost1 [root@centos7-4 tests]# mount -t lustre -o user_xattr,flock centos7-4@tcp:/lustre /mnt/lustre [root@centos7-4 tests]# ls -al /mnt/lustre/testfile -rw-r--r--. 1 root root 0 Jul 7 12:28 /mnt/lustre/testfile [root@centos7-4 tests]# getfattr -d -m user.test /mnt/lustre/testfile | tee xattrs_after_new_e2fs.lustre getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/testfile user.test="abcdefghijklmnopqrstuvwxyz123456abcdefghijk... I will upload file xattrs_after_new_e2fs.lustre later too. Next, as we discussed, I will add some printk() lines to the places where -EFSCORRUPTED is returned so that we can see why it thinks the xattr is bad. |
| Comment by Emoly Liu [ 10/Jul/17 ] |
|
adilger and bzzz, the debugging information shows the error " -EFSCORRUPTED" is returned from the following code in fs/ext4/xattr.c: ext4_xattr_inode_verify_hashes() static int
ext4_xattr_inode_verify_hashes(struct inode *ea_inode,
struct ext4_xattr_entry *entry, void *buffer,
size_t size)
{
u32 hash;
/* Verify stored hash matches calculated hash. */
hash = ext4_xattr_inode_hash(EXT4_SB(ea_inode->i_sb), buffer, size);
if (hash != ext4_xattr_inode_get_hash(ea_inode))
return -EFSCORRUPTED; //return error in this line
if (entry) {
__le32 e_hash, tmp_data;
/* Verify entry hash. */
tmp_data = cpu_to_le32(hash);
e_hash = ext4_xattr_hash_entry(entry->e_name, entry->e_name_len,
&tmp_data, 1);
if (e_hash != entry->e_hash)
return -EFSCORRUPTED;
}
return 0;
}
and then called by ext4_xattr_inode_get(): err = ext4_xattr_inode_verify_hashes(ea_inode, entry, buffer, size);
/*
* Compatibility check for old Lustre ea_inode implementation. Old
* version does not have hash validation, but it has a backpointer
* from ea_inode to the parent inode.
*/
if (err == -EFSCORRUPTED) {
if (EXT4_XATTR_INODE_GET_PARENT(ea_inode) != inode->i_ino ||
ea_inode->i_generation != inode->i_generation) {
ext4_warning_inode(ea_inode,
"EA inode hash validation failed");
goto out;
}
/* Do not add ea_inode to the cache. */
ea_inode_cache = NULL;
} else if (err)
goto out;
|
| Comment by Andreas Dilger [ 10/Jul/17 ] |
|
So it looks like all that is needed is to set err = 0 in the case that the parent inode and generation match ("Do not add ea_inode to the cache") . That must be true or the error would have been printed. Could you please give that a try? If that fixes the problem then please attach the patch here for review and then we can send it upstream. |
| Comment by Emoly Liu [ 10/Jul/17 ] |
|
adilger, with the following fix, all the tests can pass. diff --git a/fs/ext4/xattr.c b/fs/ext4/xattr.c
index ce12c3f..c7876c2 100644
--- a/fs/ext4/xattr.c
+++ b/fs/ext4/xattr.c
@@ -451,6 +451,7 @@ static int ext4_xattr_inode_iget(struct inode *parent, unsigned long ea_ino,
}
/* Do not add ea_inode to the cache. */
ea_inode_cache = NULL;
+ err = 0;
} else if (err)
goto out;
|
| Comment by Emoly Liu [ 10/Jul/17 ] |
|
I just uploaded the latest testing results here. |
| Comment by Andreas Dilger [ 11/Jul/17 ] |
|
Emoly, can you please make a patch (git commit) for this on the ext4 "next" branch, with a proper commit comment with Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-9723 and Signed-off-by: line, and then send it to the list with "git send-email --to tytso@mit.edu --cc linux-ext4@vger.kernel.org". This should be done as soon as possible so that it will be included with the patch series going upstream. |
| Comment by Emoly Liu [ 12/Jul/17 ] |
|
adilger, do you mean ext4 "dev" branch? There is no "next" branch at https://git.kernel.org/pub/scm/linux/kernel/git/tytso/ext4.git -sh-4.1$ git branch -a dev * master remotes/origin/3.2-punch-fix remotes/origin/HEAD -> origin/master remotes/origin/backport-to-3.10 remotes/origin/crypto remotes/origin/crypto-3.14 remotes/origin/dev remotes/origin/ext4-tools remotes/origin/for-stable remotes/origin/fscrypt remotes/origin/lazy_journal remotes/origin/master remotes/origin/origin remotes/origin/test remotes/origin/test-mb_generate_buddy-failure remotes/origin/unstable |
| Comment by Emoly Liu [ 12/Jul/17 ] |
|
Done. |
| Comment by Andreas Dilger [ 20/Jul/17 ] |
|
I just talked with Ted, and the patch will be included into the next release. |
| Comment by Emoly Liu [ 07/Aug/17 ] |
|
This patch has been landed to ext4 upstream: commit 191eac33009e6a6d31e87cfa425a20d0e79704b4
Author: Emoly Liu <emoly.liu@intel.com>
Date: Mon Jul 31 00:40:22 2017 -0400
ext4: error should be cleared if ea_inode isn't added to the cache
For Lustre, if ea_inode fails in hash validation but passes parent
inode and generation checks, it won't be added to the cache as well
as the error "-EFSCORRUPTED" should be cleared, otherwise it will
cause "Structure needs cleaning" when running getfattr command.
Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-9723
Cc: stable@vger.kernel.org
Fixes: dec214d00e0d78a08b947d7dccdfdb84407a9f4d
Signed-off-by: Emoly Liu <emoly.liu@intel.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
Reviewed-by: Andreas Dilger <adilger@dilger.ca>
Reviewed-by: tahsin@google.com
|
| Comment by Andreas Dilger [ 08/Aug/18 ] |
|
We are updating to e2fsprogs-1.44 for the 2.12 release, so we will now be using the upstream ea_inode code in e2fsprogs at least. At some point, maybe Ubuntu 18, we will also get the ea_inode feature in ldiskfs, so that will be tested as part of those releases. |
| Comment by James A Simmons [ 09/Aug/18 ] |
|
Actually Ubuntu e2fsprogs is at version 1.44 so it has support for ea_inode already. Its Ubuntu16 that is lacking. BTW how much a difference is their between lustre e2fsprogs version and the normal e2fsprog 1.44 version? |
| Comment by Andreas Dilger [ 10/Aug/18 ] |
|
Many of the features in e2fsprogs-1.42.13 have been merged into e2fsprogs-1.44. The main exception is the dirdata patch. The remaining patches are mostly .spec files for different distros and functional improvements that do not affect the on-disk format. It may be that we could use an unpatched e2fsprogs-1.44 on the OST, but not yet on the MDT (this has not been tested). |