|
i've been this in the local testing as well.
|
|
Compairing results with ldiskfs via maloo:
Some zfs runs complete in ~4K seconds (possibly SLOW=yes).
ldisksfs runs complete in ~300 to 900 seconds.
Possible fix is to mark this as SLOW only or SLOW for zfs?
|
|
Will you work on this, Alex? Seems it's a generic zfs sync problem?
|
|
sorry, not anytime soon
|
|
Tag should be "sometimes_except" since it's now a SLOW test for zfs
|
|
Alex, do you think we can learn something with some profiling ??
|
|
Whole logs analysis does not indicate threads (at least ll_ost_io00_0 with pid 18170, which could be traced in Lustre debug-log) were hung. So we may only face a very slow test execution due to some external cause, the perfs were horribly slow :
BTW, my own runs of test_12a on zfs show much better perfs/stats.
Nathaniel, did you hit this problem/time-out multiple times ?? Also, you have already set test_12a as ZFS_SLOW in change 5553 for LU-2874 !
I will try to re-run the same test suite for your original LU-2829/change 5530/patch #2/build 13602 where these https://maloo.whamcloud.com/test_sets/c3a0b364-812d-11e2-b609-52540035b04c tests failed.
|
|
Humm too bad/late, Build 13602 has been removed ...
So back in trying to reproduce the very slow runs of test_12a.
|
|
Most recent failure https://maloo.whamcloud.com/test_sets/bb4b4890-8895-11e2-b643-52540035b04c
Failure Rate: 7.00% of last 100 executions [all branches]
|
|
it might make sense to add some stats to osd-zfs/ so we know for sure how long txg_wait_synced() takes ?
|
|
Lowering priority per discussion with Oleg.
|
|
Lustre Build: http://build.whamcloud.com/job/lustre-b2_4/33/
FSTYPE=zfs
sanity-quota test_7a hung as follows:
Syslog on OSS:
Maloo report: https://maloo.whamcloud.com/test_sets/760750e4-095f-11e3-ad8a-52540035b04c
|
|
Lustre Build: http://build.whamcloud.com/job/lustre-b2_4/33/
FSTYPE=zfs
replay-ost-single test_5 hung, and syslog on OSS showed that:
Maloo report: https://maloo.whamcloud.com/test_sets/15fe8b0a-095e-11e3-ad8a-52540035b04c
The following sub-tests also hung at "txg_sync":
large-scale test_3a: https://maloo.whamcloud.com/test_sets/9ed19422-0963-11e3-ad8a-52540035b04c
obdfilter-survey test_1a: https://maloo.whamcloud.com/test_sets/653914a0-0964-11e3-ad8a-52540035b04c
parallel-scale test_compilebench: https://maloo.whamcloud.com/test_sets/18f7c150-0963-11e3-ad8a-52540035b04c
parallel-scale-nfsv3 test_compilebench: https://maloo.whamcloud.com/test_sets/bd745a9e-0964-11e3-ad8a-52540035b04c
parallel-scale-nfsv4 test_compilebench: https://maloo.whamcloud.com/test_sets/17b573b2-0965-11e3-ad8a-52540035b04c
posix test_1: https://maloo.whamcloud.com/test_sets/73bee652-0965-11e3-ad8a-52540035b04c
The issue in this ticket is blocking the zfs test session on Lustre b2_4 branch:
https://maloo.whamcloud.com/test_sessions/0343b17e-095a-11e3-ad8a-52540035b04c
|
|
Lustre build: http://build.whamcloud.com/job/lustre-b2_4/44/ (2.4.1 RC1)
Distro/Arch: RHEL6.4/x86_64
FSTYPE=zfs
sanity-quota test 12a hit the same failure:
https://maloo.whamcloud.com/test_sets/15cc22b2-1559-11e3-8938-52540035b04c
parallel-scale test simul: https://maloo.whamcloud.com/test_sets/4e4fc084-155a-11e3-8938-52540035b04c
|
|
is there a way to generate crash dump for the case? I'm still scratching my head around the issue. talked to Brian B., but he doesn't see the root cause as well.
|
|
Lustre build: http://build.whamcloud.com/job/lustre-b2_4/45/ (2.4.1 RC2)
Distro/Arch: RHEL6.4/x86_64
FSTYPE=zfs
MDSCOUNT=1
OSTCOUNT=7
MDSSIZE=2097152
OSTSIZE=2097152
In autotest run, parallel-scale test simul timed out in 3600s: https://maloo.whamcloud.com/test_sets/7a8c3714-18ab-11e3-aa54-52540035b04c
Dmesg on OSS node showed that:
In manual test run, parallel-scale test simul passed in 8727s: https://maloo.whamcloud.com/test_sets/aacb3504-1a8d-11e3-8fec-52540035b04c
In the above manual test report, we can see that although all of the sub-tests passed, many of them took very long time to be finished:
I'll set up kdump on the test node and run the test manually again to try to get the vmcore dump file.
|
|
Is this ticket related to LU-2476/LU-2600 which is about the performance issue on ZFS?
|
|
Lustre build: http://build.whamcloud.com/job/lustre-b2_4/45/ (2.4.1 RC2)
Distro/Arch: RHEL6.4/x86_64
FSTYPE=zfs
MDSCOUNT=1
OSTCOUNT=7
MDSSIZE=2097152
OSTSIZE=2097152
In manual test runs, sanity-quota did not hang, it finally passed in 27153s: https://maloo.whamcloud.com/test_sets/529eda72-21ac-11e3-8292-52540035b04c
Among the sub-tests, test 2 and 12a took very long time to be finished:
|
|
could some one run some single test (like simul) with ofd.*.sync_on_lock_cancel set to never please?
|
|
I did submit http://review.whamcloud.com/#/c/7711/ few days ago, but Jira hasn't started even ..
|
could some one run some single test (like simul) with ofd.*.sync_on_lock_cancel set to never please?
Patch for Lustre b2_4 branch is in http://review.whamcloud.com/7725.
A manual test run is ongoing on wtm-[82-85], which are the test nodes where the parallel-scale test was performed against Lustre 2.4.1 RC2 previously. We'll compare the test results.
|
|
Lustre build: http://build.whamcloud.com/job/lustre-b2_4/45/ (2.4.1 RC2)
Distro/Arch: RHEL6.4/x86_64
FSTYPE=zfs
MDSCOUNT=1
OSTCOUNT=7
MDSSIZE=2097152
OSTSIZE=2097152
Without the patch, parallel-scale test simul passed in 8727s: https://maloo.whamcloud.com/test_sets/aacb3504-1a8d-11e3-8fec-52540035b04c
With the patch, parallel-scale test simul passed in 189s: https://maloo.whamcloud.com/test_sets/a0eb119e-25af-11e3-aeff-52540035b04c
|
|
ahaha
|
|
Jian, can you tell which devices were used on OST in that 189s run? thanks.
|
|
yet another question: if we're using raw devices for ZFS pools (not /dev/loop*), then what I/O scheduler is used?
|
|
Hi Alex,
The devices are SCSI disk partitions.
For I/O scheduler:
|
|
thanks. no luck then..
|
|
Jian, would you mind to try with noop please? (on the clean master branch)
|
|
Lustre build: http://build.whamcloud.com/job/lustre-master/1689/
Distro/Arch: RHEL6.4/x86_64
FSTYPE=zfs
Test nodes: wtm-[82-85]
With the default "deadline" I/O scheduler:
parallel-scale test simul passed in 13457s: https://maloo.whamcloud.com/test_sets/a154ee76-2680-11e3-b741-52540035b04c
With the "noop" I/O scheduler:
parallel-scale test simul passed in 13589s: https://maloo.whamcloud.com/test_sets/dae61290-26a4-11e3-94b1-52540035b04c
|
|
given no quick reply with the results, it's going to take another zillion seconds 
would you mind to run manually simul and grab /proc/spl/kstat/zfs/txgs-* few times over the run?
|
|
I guess it makes sense to specify zfs_txg_history=500 (or 1000) when you load zfs module.
|
would you mind to run manually simul and grab /proc/spl/kstat/zfs/txgs-* few times over the run?
I guess it makes sense to specify zfs_txg_history=500 (or 1000) when you load zfs module.
OK, let me do more experiments.
|
|
Hi Alex,
Since the simul test is still running with noop scheduler, I did not stop it and gathered the /proc/spl/kstat/zfs/txgs-* data from both MDS (wtm-84) and OSS (wtm-85) for three times. Please refer to the attachments.
|
|
thanks! learning ...
|
|
do I understand correctly that all OSTs are using the same physical device?
|
do I understand correctly that all OSTs are using the same physical device?
Yes.
|
|
would it be possible to run simul using single OST on a non-shared phys.device?
|
would it be possible to run simul using single OST on a non-shared phys.device?
Sure. I can start a new test run now because the previous simul test run was just finished. I'm uploading the Maloo report.
So, the new test run will be performed with single OST, deadline I/O scheduler, and zfs_txg_history=1000 while loading zfs module.
|
|
please, make sure that single OST uses a physical device nobody else uses (including MDT). so the idea is have one ZFS pool in use on that device and zero concurrency between different ZFS pools. thanks in advance.
|
|
Lustre build: http://build.whamcloud.com/job/lustre-master/1689/
Distro/Arch: RHEL6.4/x86_64
FSTYPE=zfs
Test configuration:
parallel-scale test simul passed in 1950s: https://maloo.whamcloud.com/test_sets/55b02a2c-26b1-11e3-9d3a-52540035b04c
|
|
hmm, sounds interesting.. any idea whether simul's workset depends on the space available? or probably it makes sense to return with OSTSIZE=$((6*2097152)) ?
|
|
I think the performance of simul test does not depend on the available space on OST because no parameter is calculated from the available space on OST, and the simul test mainly does the open/close/unlink/rename/creat/truncate, etc. operations.
In the above test run, I used OSTSIZE=15000000 (about 14G). Just re-ran the test with OSTSIZE=$((6*2097152))=12582912 (about 12G) and it passed in 2095s: https://maloo.whamcloud.com/test_sets/e089858a-26c0-11e3-83d1-52540035b04c
|
|
thanks a lot for help. it seems the issue is due to two factors: 1) expensive sync (to be learned yet) 2) high cost of competing of few pools on a same phys.device. I don't think anybody will be running few pools on a same device in wild..
|
|
You're welcome, Alex.
I just checked the autotest configuration and found the OST devices used by it were:
It was likely that all of the LVs were created from the same PV, especially on virtual machines. Since most of the test nodes used by autotest system are VMs, and ZFS full test runs are performed by autotest regularly, it's hard to bypass the above factor 2).
|
|
frankly I don't think we should be trying to improve ZFS for such an environment - it's not realistic to have many pools on a same device. solution to this isn't obvious either.
|
|
http://review.whamcloud.com/7778
Skip simul test for ZFS, since it takes too long.
|
|
http://review.whamcloud.com/7780 - FOR TEST ONLY patch to run parallel-scale with only 2 OSTs
|
|
Hi Alex and Nathaniel,
FYI, all of the following sub-tests run slowly on ZFS under the current test environment (7 OSTs share the same physical device):
We'd better not to skip those tests for ZFS. We can improve the test scripts to reduce file sizes/counts, iteration times, etc. to make the tests still run for ZFS within a reasonable time (less than 1800s with SLOW=yes).
|
|
can we reconfigure with OSTCOUNT=1 for parallel-scale specifically ?
|
can we reconfigure with OSTCOUNT=1 for parallel-scale specifically ?
This needs changes on autotest system. We can get the result of parallel-scale test with SLOW=no first to see whether those sub-tests can pass within a reasonable time or not.
|
|
In manual test run with SLOW=no and OSTCOUNT=7, parallel-scale test simul passed in 112s: https://maloo.whamcloud.com/test_sets/4b220e1e-28c7-11e3-8951-52540035b04c
However, the following sub-tests still took very long time:
Let's wait for the autotest test result in http://review.whamcloud.com/7778 to do a comparison.
|
Let's wait for the autotest test result in http://review.whamcloud.com/7778 to do a comparison.
In autotest test run with SLOW=no and OSTCOUNT=7, parallel-scale timed out: https://maloo.whamcloud.com/test_sets/487e3fe6-29c3-11e3-b5ea-52540035b04c
We still have to reduce the number of cbench_IDIRS, cbench_RUNS, mbench_NFILES, etc. for ZFS.
|
|
From 7778 https://maloo.whamcloud.com/test_sets/487e3fe6-29c3-11e3-b5ea-52540035b04c
the metabench result seems very strange.
metabench normally runs fairly quickly (~200-500) judging by the results in maloo for ZFS runs.
It looks like the whole system (client-30) just went out to lunch for 4 hours.
|
|
Reduce performance expectation for ZFS in sanity-quota/0, lowest observed over last 4 weeks is ~150.
http://review.whamcloud.com/7848
|
|
I would prefer the approach of setting OSTCOUNT=2 for ZFS-backed test filesystems. This is more in line with real systems, since we will typically only have a single OST per OSS with ZFS instead of 4 or more OSTs per OSS with ldiskfs. I think a lot of tests depend on having at least two OSTs, so OSTCOUNT=1 will probably cause some tests to be skipped.
Another complimentary approach would be to format a single ZFS pool across a few LVs and then have the different OST/MDT targets in their own datasets in the shared pool. That would avoid the extra commits caused by having separate pools. The drawback is that all of the datasets would store their files into the same space, so some of the Lustre tests would be broken if we don't add ZFS reservations for the minimum size of the datasets (e.g. fill one OST and allocate objects to another OST would break).
|
I would prefer the approach of setting OSTCOUNT=2 for ZFS-backed test filesystems.
I just created TEI-790 to ask TEI team for help on this change.
Another complimentary approach would be to format a single ZFS pool across a few LVs and then have the different OST/MDT targets in their own datasets in the shared pool.
This needs change mdsdevname(), ostdevname() and the failover testing support codes for ZFS (http://review.whamcloud.com/6429) in test-framework.sh.
|
|
Since TEI-790 was fixed, I've triggered a full group test session on ZFS against Lustre b2_4 build #47. I'll vet the test results to see whether the timed-out tests can pass or not with OSTCOUNT=2 and SLOW=yes.
|
|
Here is the test result on Lustre b2_4 build #47 with FSTYPE=zfs, OSTCOUNT=2 and SLOW=yes:
https://maloo.whamcloud.com/test_sessions/05b82736-444d-11e3-8472-52540035b04c
Timeout failures with OSTCOUNT=7 on the following sub-tests disappeared with OSTCOUNT=2:
However, timeout failure on sanityn test 33a (LU-2829) still occurred.
And because of LU-3906 (out of space issue), parallel-scale, parallel-scale-nfsv3 and parallel-scale-nfsv4 tests were not really run, then we do not know whether the timeout failures on them disappear or not with OSTCOUNT=2.
|
http://review.whamcloud.com/7778
The above patch and http://review.whamcloud.com/8234 have landed on master branch. They are also needed on Lustre b2_4 and b2_5 branches.
The two patches are combined and back-ported to Lustre b2_4 branch: http://review.whamcloud.com/8284
|
|
Patch http://review.whamcloud.com/8284 landed on Lustre b2_4 branch. I'll trigger ZFS full group test session on Lustre b2_4 build #57 to check the test results.
|
|
Lustre Build: http://build.whamcloud.com/job/lustre-b2_4/58/
Distro/Arch: RHEL6.4/x86_64
FSTYPE=zfs
MDSCOUNT=1
MDSSIZE=2097152
OSTCOUNT=2
OSTSIZE=2097152
replay-ost-single test 5 still timed out: https://maloo.whamcloud.com/test_sets/a76a6b78-5606-11e3-8e94-52540035b04c
|
|
Lustre Build: http://build.whamcloud.com/job/lustre-b2_4/67/
Distro/Arch: RHEL6.4/x86_64
FSTYPE=zfs
MDSCOUNT=1
MDSSIZE=2097152
OSTCOUNT=2
OSTSIZE=8388608
performance-sanity test 4 timed out in 28800s:
https://maloo.whamcloud.com/test_sets/68896e68-6579-11e3-8518-52540035b04c
parallel-scale test metabench timed out in 14400s:
https://maloo.whamcloud.com/test_sets/bdff6c44-6579-11e3-8518-52540035b04c
|
|
Landed for 2.4.2 and 2.6. Will land for 2.5.1 shortly.
|
|
sanity-quota.sh test_12a is still being skipped on ZFS due to this bug, reopen until problem is actually fixed. At a minimum, the skip should only be done in the case of VM nodes, or ZFS-on-LVM or something.
|
|
ZIL will improve this
|
Generated at Sat Feb 10 01:29:04 UTC 2024 using Jira 9.4.14#940014-sha1:734e6822bbf0d45eff9af51f82432957f73aa32c.
txgs.1380193340.log