[LU-797] Test failure on test suite ost-pools, subtest test_14, test_18, test_23 - Whamcloud Community JIRA

Details

Type: Bug
Resolution: Fixed
Priority: Blocker
Fix Version/s: Lustre 2.4.0, Lustre 2.1.4, Lustre 1.8.9
Affects Version/s: Lustre 2.2.0, Lustre 2.1.2, Lustre 2.1.3, Lustre 2.1.4, Lustre 1.8.8
Labels:
None

Severity:
3
Rank (Obsolete):
4734

Description

This issue was created by maloo for sarah <sarah@whamcloud.com>

This issue relates to the following test suite run: https://maloo.whamcloud.com/test_sets/f6c809ee-00ec-11e1-bb4f-52540025f9af.

The sub-test test_14 failed with the following error:

test failed to respond and timed out

Info required for matching: ost-pools 14

Attachments

Issue Links

is duplicated by

LU-1001 Test failure on test suite ost-pools, subtest test_18

Resolved

LU-876 Improvement to ost-pools.sh

Resolved

Trackbacks

Lustre 1.8.7-wc1 release testing tracker Lustre 1.8.7wc1 RC2 Tag: v187WC1RC2 Build:

Lustre 1.8.8-wc1 release testing tracker Lustre 1.8.8wc1 RC2 Tag: v188WC1RC2 Build:

Lustre 2.1.1 release testing tracker Lustre 2.1.1 RC4 Tag: v2110RC4 Build:

Lustre 2.1.2 release testing tracker Lustre 2.1.2 RC2 Tag: v212RC2 Build:

Lustre 2.1.3 release testing tracker Lustre 2.1.3 RC1 Tag: v213RC1 Build:

Changelog 2.1 Changes from version 2.1.0 to version 2.1.1 Server support for kernels: 2.6.18274.12.1.el5 (RHEL5) 2.6.32220.el6 (RHEL6) Client support for unpatched kernels: 2.6.18274.12.1.el5 (RHEL5) 2.6.32220.el6 (RHEL6) 2.6.32.360....

Changelog 2.2 version 2.2.0 Support for networks: o2iblnd OFED 1.5.4 Server support for kernels: 2.6.32220.4.2.el6 (RHEL6) Client support for unpatched kernels: 2.6.18274.18.1.el5 (RHEL5) 2.6.32220.4.2.el6 (RHEL6) 2.6.32.360....

(4 Trackbacks)

Activity

[LU-797] Test failure on test suite ost-pools, subtest test_14, test_18, test_23

Jian Yu added a comment - 30/Dec/12 10:24 PM

Lustre Branch: b1_8
Lustre Build: http://build.whamcloud.com/job/lustre-b1_8/236/
Distro/Arch: RHEL5.8/x86_64
Network: TCP (1GigE)
ENABLE_QUOTA=yes
OSTSIZE=149723171

The ost-pools test 14 timed out: https://maloo.whamcloud.com/test_sets/f8ded3ce-5187-11e2-bbc3-52540035b04c

Jian Yu added a comment - 30/Dec/12 10:24 PM Lustre Branch: b1_8 Lustre Build: http://build.whamcloud.com/job/lustre-b1_8/236/ Distro/Arch: RHEL5.8/x86_64 Network: TCP (1GigE) ENABLE_QUOTA=yes OSTSIZE=149723171 The ost-pools test 14 timed out: https://maloo.whamcloud.com/test_sets/f8ded3ce-5187-11e2-bbc3-52540035b04c

Emoly Liu added a comment - 28/Dec/12 3:41 AM

The related patches for b1_8 are merged into http://review.whamcloud.com/4898

Emoly Liu added a comment - 28/Dec/12 3:41 AM The related patches for b1_8 are merged into http://review.whamcloud.com/4898

Peter Jones added a comment - 19/Dec/12 11:39 AM

Fix landed for 2.1.4 and 2.4

Peter Jones added a comment - 19/Dec/12 11:39 AM Fix landed for 2.1.4 and 2.4

Emoly Liu added a comment - 14/Dec/12 2:20 AM

http://review.whamcloud.com/#change,4474

b2_1 port is at http://review.whamcloud.com/4831

Emoly Liu added a comment - 14/Dec/12 2:20 AM http://review.whamcloud.com/#change,4474 b2_1 port is at http://review.whamcloud.com/4831

Jian Yu added a comment - 10/Dec/12 10:25 AM

Lustre Branch: b2_1
Lustre Build: http://build.whamcloud.com/job/lustre-b2_1/148
Distro/Arch: RHEL6.3/x86_64 (kernel version: 2.6.32-279.14.1.el6)
Network: TCP (1GigE)

The same issue occurred:
https://maloo.whamcloud.com/test_sets/9b37618e-41db-11e2-adcf-52540035b04c

Jian Yu added a comment - 10/Dec/12 10:25 AM Lustre Branch: b2_1 Lustre Build: http://build.whamcloud.com/job/lustre-b2_1/148 Distro/Arch: RHEL6.3/x86_64 (kernel version: 2.6.32-279.14.1.el6) Network: TCP (1GigE) The same issue occurred: https://maloo.whamcloud.com/test_sets/9b37618e-41db-11e2-adcf-52540035b04c

Emoly Liu added a comment - 10/Dec/12 8:50 AM

remove the space of "lfs df" output interllay: http://review.whamcloud.com/4705

b2_1 port is at http://review.whamcloud.com/#change,4784

Emoly Liu added a comment - 10/Dec/12 8:50 AM remove the space of "lfs df" output interllay: http://review.whamcloud.com/4705 b2_1 port is at http://review.whamcloud.com/#change,4784

Niu Yawei (Inactive) added a comment - 29/Nov/12 5:07 AM

remove the space of "lfs df" output interllay: http://review.whamcloud.com/4705

Niu Yawei (Inactive) added a comment - 29/Nov/12 5:07 AM remove the space of "lfs df" output interllay: http://review.whamcloud.com/4705

Niu Yawei (Inactive) added a comment - 05/Nov/12 11:12 PM

http://review.whamcloud.com/#change,4474

Niu Yawei (Inactive) added a comment - 05/Nov/12 11:12 PM http://review.whamcloud.com/#change,4474

Niu Yawei (Inactive) added a comment - 05/Nov/12 10:58 PM

Look at the recent failure:

== ost-pools test 23b: OST pools and OOS ============================================================= 21:49:32 (1345438172)
running as uid/gid/euid/egid 500/500/500/500, groups:
 [touch] [/mnt/lustre/d0_runas_test/f13763]
CMD: client-30vm3 lctl pool_new lustre.testpool
client-30vm3: Pool lustre.testpool created
CMD: client-30vm6.lab.whamcloud.com lctl get_param -n lov.lustre-*.pools.testpool         2>/dev/null || echo foo
CMD: client-30vm3 lctl pool_add lustre.testpool lustre-OST[0000-0006/3]
client-30vm3: OST lustre-OST0000_UUID added to pool lustre.testpool
client-30vm3: OST lustre-OST0003_UUID added to pool lustre.testpool
client-30vm3: OST lustre-OST0006_UUID added to pool lustre.testpool
CMD: client-30vm6.lab.whamcloud.com lctl get_param -n lov.lustre-*.pools.testpool |
                           sort -u | tr '\n' ' ' 
1: 32768+0 records in
32768+0 records out
34359738368 bytes (34 GB) copied, 1656.82 seconds, 20.7 MB/s
2: 32768+0 records in
32768+0 records out
34359738368 bytes (34 GB) copied, 1761.27 seconds, 19.5 MB/s
3: 32768+0 records in
32768+0 records out
34359738368 bytes (34 GB) copied, 1921.64 seconds, 17.9 MB/s
4: 32768+0 records in
32768+0 records out
34359738368 bytes (34 GB) copied, 1941.35 seconds, 17.7 MB/s
5: 32768+0 records in
32768+0 records out
34359738368 bytes (34 GB) copied, 2098.63 seconds, 16.4 MB/s
6: 32768+0 records in
32768+0 records out
34359738368 bytes (34 GB) copied, 2030.07 seconds, 16.9 MB/s

The MAXFREE should be (2000000(KB) * 7(osts)) / (1024 * 1024) = 13GB, which means the available space should be less than 13GB in this pool, however, tests writes much more than the it and didn't fail with ENOSPC. I think we'd add more debug information in the tests, and also break the writting whenever written bytes is greater than available space.

Niu Yawei (Inactive) added a comment - 05/Nov/12 10:58 PM Look at the recent failure: == ost-pools test 23b: OST pools and OOS ============================================================= 21:49:32 (1345438172) running as uid/gid/euid/egid 500/500/500/500, groups: [touch] [/mnt/lustre/d0_runas_test/f13763] CMD: client-30vm3 lctl pool_new lustre.testpool client-30vm3: Pool lustre.testpool created CMD: client-30vm6.lab.whamcloud.com lctl get_param -n lov.lustre-*.pools.testpool 2>/dev/ null || echo foo CMD: client-30vm3 lctl pool_add lustre.testpool lustre-OST[0000-0006/3] client-30vm3: OST lustre-OST0000_UUID added to pool lustre.testpool client-30vm3: OST lustre-OST0003_UUID added to pool lustre.testpool client-30vm3: OST lustre-OST0006_UUID added to pool lustre.testpool CMD: client-30vm6.lab.whamcloud.com lctl get_param -n lov.lustre-*.pools.testpool | sort -u | tr '\n' ' ' 1: 32768+0 records in 32768+0 records out 34359738368 bytes (34 GB) copied, 1656.82 seconds, 20.7 MB/s 2: 32768+0 records in 32768+0 records out 34359738368 bytes (34 GB) copied, 1761.27 seconds, 19.5 MB/s 3: 32768+0 records in 32768+0 records out 34359738368 bytes (34 GB) copied, 1921.64 seconds, 17.9 MB/s 4: 32768+0 records in 32768+0 records out 34359738368 bytes (34 GB) copied, 1941.35 seconds, 17.7 MB/s 5: 32768+0 records in 32768+0 records out 34359738368 bytes (34 GB) copied, 2098.63 seconds, 16.4 MB/s 6: 32768+0 records in 32768+0 records out 34359738368 bytes (34 GB) copied, 2030.07 seconds, 16.9 MB/s The MAXFREE should be (2000000(KB) * 7(osts)) / (1024 * 1024) = 13GB, which means the available space should be less than 13GB in this pool, however, tests writes much more than the it and didn't fail with ENOSPC. I think we'd add more debug information in the tests, and also break the writting whenever written bytes is greater than available space.

Jian Yu added a comment - 21/Aug/12 6:58 AM

More instances occurred during Lustre 2.1.3 RC2 testing:
https://maloo.whamcloud.com/test_sets/9c6295ce-eb32-11e1-ba73-52540035b04c
https://maloo.whamcloud.com/test_sets/68c13076-eb3d-11e1-ba73-52540035b04c

Jian Yu added a comment - 21/Aug/12 6:58 AM More instances occurred during Lustre 2.1.3 RC2 testing: https://maloo.whamcloud.com/test_sets/9c6295ce-eb32-11e1-ba73-52540035b04c https://maloo.whamcloud.com/test_sets/68c13076-eb3d-11e1-ba73-52540035b04c

People

Assignee:: Niu Yawei (Inactive)

Reporter:: Maloo

Votes:: 0 Vote for this issue

Watchers:: 8 Start watching this issue

Dates

Created:: 27/Oct/11 7:33 PM

Updated:: 22/Feb/13 1:24 PM

Resolved:: 19/Dec/12 11:39 AM