|
More failure instances:
replay-single test 0a: https://testing.hpdd.intel.com/test_sets/5fadea7c-8a19-11e6-91aa-5254006e85c2
recovery-mds-scale: https://testing.hpdd.intel.com/test_sets/ad25283e-8bb2-11e6-91aa-5254006e85c2
|
|
Jian Yu (jian.yu@intel.com) uploaded a new patch: http://review.whamcloud.com/23251
Subject: LU-8729 utils: improve lctl to add freeze and unfreeze operations
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: cb65ea32816a1df5ca23a11a681ff1db43d6cff9
|
|
Hi Hongchao,
I tested the dm-flakey patch on the latest master branch in https://review.whamcloud.com/24733 and found the following issues:
On CentOS 6.8 distro, replay-single test 0c consistently hit the following errors on MDS:
On CentOS 7.3 distro, conf-sanity test 84, replay-single tests 0a, 4b, 5, 18, 19 and replay-ost-single test 9 consistently hit the following errors on MDS or OSS:
The "Buffer I/O error" didn't occur on CentOS 6.8 distro. It also didn't occur on CentOS 7.3 distro before patch set 21 for https://review.whamcloud.com/7200. The CentOS 7.3 kernel version for patch set 20 is 3.10.0-327.36.3.el7, and the kernel version for patch set 21 is 3.10.0-514.el7.
Is there sth wrong with dm-flakey in different kernels?
|
|
On my VM, I have setup dm-flakey manually on 7.3 (3.10.0-514.2.2.el7) and run some basic tests (say, creation, deletion,
copy etc) on it, and it show no error. I aslo looked at the code line of dm-flakey.c and found no clue of the problem so far.
Will update the status of the ticket once there is any progress.
|
|
This whole 'dmsetup load /dev/mapper/mds1_flakey --table "0 4194304 flakey 252:0 0 0 1800 1 drop_writes"' this looks a bit suspicious to me.
Looking into the kenrel documentation, we can see:
According to this, "1" at the end of the table above is spurious? And if a target is unknown then it's just makes io errors to be returned from the io - which seems to be matching what is seen on rhel 7.3?
There's a huge diff in devicemapper code between rhell 7.2 and 7.3, though I don't think API ever had this extra number there.
Anyway I submitted the same patch we have no with just this table changed in https://review.whamcloud.com/#/c/25387/ to see how well would that work and might be that's our fix in the end for newer distros.
|
|
oh, I guess 1 is the "num_features" that I missed
|
|
Jian, when is the last time this was tested? LU-7481 had a patch that was landed 2017-01-27 that fixed a related issue.
|
|
Hi Andreas,
The last time was 2017-03-12 and the fix for LU-7481 was included.
The consistent failures now are the errors in comment:
https://jira.hpdd.intel.com/browse/LU-8729?focusedCommentId=180716&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-180716
|
|
Hi Andreas and Oleg,
The "Buffer I/O error" on device-mapper flakey device is still blocking the testing on patch https://review.whamcloud.com/7200. Hongchao looked at the codes of dm-flakey.c in CentOS 7.3 (kernel version 3.10.0-514.2.2.el7) but didn't find any clue of the problem.
Could you please advise what we need to do to move this forward? Thank you.
|
|
While testing I started to ask myself why are using dm_flakey for ever test. It appears this is only needed for recovery testing to replace lct --device readonly. Should we look to reduce the scope of dm-flakey to only the test that it is needed for?
|
|
After working on this patch today I believe I got a handle on what is going on. So the "Buffer I/O error" are correct. So in the test you have:
mount_client $MOUNT1 || error "mount $MOUNT1 failed"
mount_client $MOUNT2 || error "mount $MOUNT2 failed"
- make sure new superblock labels are sync'd before disabling writes
sync_all_data
sleep 5
replay_barrier $SINGLEMDS
createmany o $DIR1/$tfile%d 1000
This mounts and syncs all the data. Then replay_barrier using the power of the flakey driver sets the disk /dev/mapper/mds1_flakey to read only mode. The creation attempts by createmany on a read only file system is what is generated the "Buffer I/O errors". This is what is supposed to happen. Next is:
do_facet $SINGLEMDS "lctl set_param fail_loc=0x20000709 fail_val=5"
facet_failover --fsck $SINGLEMDS || error "failover: $?"
The facet_failover part is what is important. Here we set a special flag to make sure e2fsck is run on the MDT. Looking at the Maloo logs:
https://testing.hpdd.intel.com/test_logs/2cb1e62c-1f0c-11e7-8920-5254006e85c2/show_text
I noticed the error:
[26120.388883] Lustre: DEBUG MARKER: /usr/sbin/lctl mark conf-sanity test_84: \
@@@@@@ FAIL: e2fsck -d -v -t -t -f -n \/dev\/mapper\/mds1_flakey returned 8, \
should be \<= 4
[26120.544691] Lustre: DEBUG MARKER: conf-sanity test_84: @@@@@@ FAIL: e2fsck \
-d -v -t -t -f -n /dev/mapper/mds1_flakey returned 8, should be <= 4
Running e2fsck shouldn't work on a read only device, mds1_flakey. I fixed up
shutdown_facet which is called right before run_e2fsck in the function
failover_facet(). Now it should not only stop the device but remove the flakey
device and then e2fsck will be run on the native device which is not read only.
Now I see a different failure. It looks like MDT0000 is not suppose to recover
but it does for. Will need to look into it.
|
|
Thank you very much James for helping debug this failure.
While testing I started to ask myself why are using dm_flakey for ever test. It appears this is only needed for recovery testing to replace lct --device readonly. Should we look to reduce the scope of dm-flakey to only the test that it is needed for?
Besides recovery test suites (replay-single, replay-dual, etc), recovery testing also exists in other test suites (sanity, insanity, conf-sanity, etc). The changes will be a lot. What's more, we would like to make mount and unmount transparent, so that developers do not need to know they have to use a special option to create flakey devices while creating a recovery testing related sub-test.
|
|
After doing some experiments, I found out the commits that caused the "Buffer I/O error". They are related to RHEL 7.3 support:
In LU-684 patch https://review.whamcloud.com/7200, replay-single test passed on patch set 20:
https://testing.hpdd.intel.com/test_sessions/c01176f8-a5ef-11e6-b605-5254006e85c2
but failed on patch set 21 because of the above commits:
https://testing.hpdd.intel.com/test_sessions/41fe0cb2-beeb-11e6-92c6-5254006e85c2
The error message on MDS console was:
Lustre: DEBUG MARKER: dmsetup suspend --nolockfs --noflush /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: dmsetup load /dev/mapper/mds1_flakey --table "0 4194304 flakey 252:0 0 0 1800 1 drop_writes"
Lustre: DEBUG MARKER: dmsetup resume /dev/mapper/mds1_flakey
Buffer I/O error on dev dm-6, logical block 524272, async page read
In RHEL 7.3 kernel 3.10.0-514.el7, the above error message came from fs/buffer.c:
static void end_buffer_async_read(struct buffer_head *bh, int uptodate)
{
page = bh->b_page;
if (uptodate) {
set_buffer_uptodate(bh);
} else {
clear_buffer_uptodate(bh);
buffer_io_error(bh, ", async page read");
SetPageError(page);
}
}
|
|
One option is to add a WARN_ON() in end_buffer_async_read(), so that this will dump the full stack. Note that the callpath for end_buffer_async_read() does not come from mark_buffer_async_read(), that is just setting the bh->b_end_io pointer so that the end_buffer_async_read() callback is called when the read from disk is actually finished.
It is probably coming through:
I do see some comments in dm-flakey.c that might indicate problems if the DROP_WRITES state is being changed:
} else if (!test_bit(DROP_WRITES, &fc->flags) &&
!test_bit(ERROR_WRITES, &fc->flags)) {
/*
* Error read during the down_interval if drop_writes
* and error_writes were not configured.
*/
return -EIO;
}
so that may be a result of changing the drop_writes state in the middle of a read. I see just before configuring the dm-flakey device:
Does "--noflush" means there may still be partially-completed reads?
At a higher level, have we considered using the dm-flakey device only for cases when lctl readonly is actually used? It appears that this is only called from replay_barrier(), replay_barrier_nosync(), and replay_barrier_nodf(). I see one call to replay_barrier in conf-sanity.sh::test_84(), but we might consider to move this test over to replay-single.sh. It is also worthwhile to note that dm-flakey is only needed for ldiskfs, not for ZFS filesystems.
|
One option is to add a WARN_ON() in end_buffer_async_read(), so that this will dump the full stack.
The full stack is:
I'm running the test again with "--noflush" removed.
|
|
With "--noflush" removed, the full stack is:
|
|
Hi Andreas,
I also removed "--nolockfs" and got the same failure and full stack. So, the failure is not related to "--nolockfs" and/or "--noflush" options.
|
|
Hi Hongchao,
Could you please proceed with this blocker? Thank you.
|
|
Status Update:
I have tested it in Maloo by disabling different patches contained in http://review.whamcloud.com/22113 and
http://review.whamcloud.com/23560 (the debug patch is tracked at https://review.whamcloud.com/#/c/26788/, the patch set is 15 ~ 21).
the error "Buffer I/O error" still occur.
I looked at the test result in the original patch for LU-684 (https://review.whamcloud.com/#/c/7200/), there is no such problem prior to
patch set 20 and start to occur from patch set 21, the difference of Lustre itself between the two patch set is the two patches
http://review.whamcloud.com/22113 and http://review.whamcloud.com/23560, but the more big different is the change of kernel,
patch set 20 still used RHEL7.2 to test, but patch set 21 began to use RHEL7.3.
Patch Set 20: https://testing.hpdd.intel.com/test_sessions/c01176f8-a5ef-11e6-b605-5254006e85c2
(3.10.0-327.36.3.el7.x86_64)
Patch Set 21: https://testing.hpdd.intel.com/test_sessions/41fe0cb2-beeb-11e6-92c6-5254006e85c2 (3.10.0-514.el7.x86_64)
Then the issue could be caused by the kernel RHEL7.3 (3.10.0-514.el7.x86_64)
|
|
I have setup the RHEL7.3 (3.10.0-514.2.2.el7.x86_64) and reproduce the problem by using ext4 and dm-flakey module
the reproducer is attached.
|
|
Can you try linux-commit: 299f6230bc6d0ccd5f95bb0fb865d80a9c7d5ccc
dm flakey: fix reads to be issued if drop_writes configured
v4.8-rc3 commit 99f3c90d0d ("dm flakey: error READ bios during the
down_interval") overlooked the 'drop_writes' feature, which is meant to
allow reads to be issued rather than errored, during the down_interval.
Fixes: 99f3c90d0d ("dm flakey: error READ bios during the down_interval")
Reported-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: Mike Snitzer <snitzer@redhat.com>
Cc: stable@vger.kernel.org
|
|
Hi James,
Can you try linux-commit: 299f6230bc6d0ccd5f95bb0fb865d80a9c7d5ccc
I tried this in https://review.whamcloud.com/26788 (patch set 25). The same error still occurred:
|
|
I have tested the commit locally, the commit can fix the problem on 3.10.0-514.2.2.el7.x86_64,
but can't on 3.10.0-514.21.1.el7.x86_64, which is being used in our Auotest system.
|
|
The autotest system uses the build with the following kernel version supported by the latest master branch:
So, the commit doesn't work and the failure still exists.
|
|
So the good news is this is fixed in RHEL7.4. If some one really wants to work on RHEL7.3 can compare the sources in RHEL7.4 to RHLE7.3 to see which change fixed this problem so we can include the patch fpr RHEL7.3 if so desired. I updated the patch for LU-684 in the mean time to move this work forward.
|
|
No - let's just focus on RHEL 7.4
|
Generated at Sat Feb 10 02:20:03 UTC 2024 using Jira 9.4.14#940014-sha1:734e6822bbf0d45eff9af51f82432957f73aa32c.
reproducer.sh