[LU-1357] Test failure on test suite performance-sanity, subtest test_4

• Status: Resolved
• Project: Lustre
• Affects Version/s: Lustre 2.1.2, Lustre 1.8.8
• Type: Bug
• Priority: Minor
• Reporter: Maloo
• Assignee: Bob Glossman (Inactive)
• Resolution: Cannot Reproduce
• Severity: 3

Description

This issue was created by maloo for sarah <[e-mail]>

This issue relates to the following test suite run: https://maloo.whamcloud.com/test_sets/95cf2ada-8f19-11e1-8a98-525400d2bfa6.

The sub-test test_4 failed with the following error:

test failed to respond and timed out

Info required for matching: performance-sanity 4

test_4() {
echo "Large file creation performance"
bash mdsrate-create-large.sh
}
run_test 4 "large file create/open/delete ======"

This test seems very slow, which takes more than 3600 seconds in this test. Usually it only take about 30 mins.

It seems both client are doing this

mdsrate S 0000000000000000 0 4989 4988 0x00000080
ffff880074ff99d8 0000000000000082 0000000000000000 ffff880078b9fc00
ffff880074ff99e8 ffffffffa03acc37 ffff880074ff9978 ffffffffa0731a13
ffff88007b83b038 ffff880074ff9fd8 000000000000f4e8 ffff88007b83b038
Call Trace:
[<ffffffffa03acc37>] ? ldlm_prep_elc_req+0x267/0x600 [ptlrpc]
[<ffffffffa0731a13>] ? cfs_alloc+0x63/0x90 [libcfs]
[<ffffffffa07316fe>] cfs_waitq_wait+0xe/0x10 [libcfs]
[<ffffffffa049eb35>] osc_destroy+0x655/0x8f0 [osc]
[<ffffffffa055a9cd>] ? lov_set_add_req+0x2d/0x50 [lov]
[<ffffffffa055f902>] ? lov_prep_destroy_set+0x372/0xa40 [lov]
[<ffffffff8105e7f0>] ? default_wake_function+0x0/0x20
[<ffffffffa054ac07>] lov_destroy+0x357/0xcd0 [lov]
[<ffffffffa03d5045>] ? lustre_msg_buf+0x85/0x90 [ptlrpc]
[<ffffffffa056fff3>] ? lsm_unpackmd_v1+0xf3/0x200 [lov]
[<ffffffffa0555424>] ? lov_alloc_memmd+0x54/0x360 [lov]
[<ffffffffa0400e21>] ? req_capsule_set_size+0x71/0x1b0 [ptlrpc]
[<ffffffffa05fae41>] ll_objects_destroy+0x571/0x1a90 [lustre]
[<ffffffffa03d5045>] ? lustre_msg_buf+0x85/0x90 [ptlrpc]
[<ffffffffa040145b>] ? __req_capsule_get+0x14b/0x6b0 [ptlrpc]
[<ffffffffa05f5446>] ? ll_update_times+0x26/0x110 [lustre]
[<ffffffffa05fce6f>] ll_unlink+0x35f/0x780 [lustre]
[<ffffffff81183fef>] vfs_unlink+0x9f/0xe0
[<ffffffff81182d3a>] ? lookup_hash+0x3a/0x50
[<ffffffff81186573>] do_unlinkat+0x183/0x1c0
[<ffffffff810d4692>] ? audit_syscall_entry+0x272/0x2a0
[<ffffffff8100c5b5>] ? math_state_restore+0x45/0x60
[<ffffffff811865c6>] sys_unlink+0x16/0x20
[<ffffffff8100b0f2>] system_call_fastpath+0x16/0x1b

Issue Links

• is duplicated by: LU-4108 Failure on test suite performance-san...

Comments

[Oleg Drokin]

Dd this ever work before in VMs? I think performance was used to run on real hardware only?

[Peter Jones]

Assign to Bob

[Sarah Liu]

I reran this test on real hardware and cannot reproduce it, well it took 39176s to finish.
https://maloo.whamcloud.com/test_sets/83e2c5be-99ca-11e1-9853-52540035b04c

[Jian Yu]

Lustre Tag: v1_8_8_WC1_RC2
Lustre Build: http://build.whamcloud.com/job/lustre-b1_8/197/
Distro/Arch: RHEL5.8/x86_64(server), RHEL6.2/x86_64(client)
Network: TCP (1GigE)
ENABLE_QUOTA=yes

performance-sanity test 4 also timed out: https://maloo.whamcloud.com/test_sets/216c93fc-a216-11e1-abdc-52540035b04c

[Andreas Dilger]

Close old ticket.