[LU-1724] Test failure on test suite performance-sanity, subtest test_3

• Status: Resolved
• Project: Lustre
• Affects Version/s: Lustre 2.3.0
• Type: Bug
• Priority: Blocker
• Reporter: Maloo
• Assignee: Keith Mannthey (Inactive)
• Resolution: Duplicate
• Severity: 3

Description

This issue was created by maloo for sarah <[e-mail]>

This issue relates to the following test suite run: https://maloo.whamcloud.com/test_sets/0596b3e0-dcd3-11e1-8744-52540035b04c.

The sub-test test_3 failed with the following error:

test_3 returned 1

01:24:41:Lustre: DEBUG MARKER: performance-sanity test_3: @@@@@@ FAIL: test_3 failed with 10

Comments

[Sarah Liu]

This error may caused by the previous failure of test mds-survey

[Keith Mannthey (Inactive)]

What happened to client-27vm3 of this test run? It was the mds and it paniced or ????

[Sarah Liu]

There is a previous failure of mds-survey, it may cause the MDS abnormal.

https://maloo.whamcloud.com/test_sets/7fa0e0bc-dcd2-11e1-8744-52540035b04c

[Keith Mannthey (Inactive)]

from the MDS of the 2nd run

00:54:47:Lustre: DEBUG MARKER: == mds-survey test 2: Metadata survey with stripe_count = 1 == 00:54:45 (1343894085)
00:54:50:Lustre: DEBUG MARKER: lctl dl
00:54:52:LustreError: 17365:0:(echo_client.c:1607:echo_md_lookup()) lookup tests: rc = -2
00:54:52:LustreError: 17365:0:(echo_client.c:1607:echo_md_lookup()) Skipped 2 previous similar messages
00:54:52:LustreError: 17365:0:(echo_client.c:1806:echo_md_destroy_internal()) Can't find child tests: rc = -2
00:54:52:LustreError: 17365:0:(echo_client.c:1806:echo_md_destroy_internal()) Skipped 2 previous similar messages
00:54:54:LustreError: 17387:0:(echo_client.c:1607:echo_md_lookup()) lookup tests1: rc = -2
00:54:54:LustreError: 17387:0:(echo_client.c:1806:echo_md_destroy_internal()) Can't find child tests1: rc = -2
01:04:40:lctl invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0

The OOM killer was out running around. Once OOM is running the system behaviour becomes non-deterministic as it may choose different processes to kill.

Why is the MDS running of of memory?

[Peter Jones]

I think that the theory is due to LU-1548. Do you think that it is reasonable to consider this a follow-on issue, close as a duplicate and reopen if it reoccurs with the LU-1548 fix applied?

[Keith Mannthey (Inactive)]

After reviewing LU-1548 I feel these are very likely the same issue: MDS OOM under the right conditions. I think duping it is the right way to go. There are no MDS logs from the first failure but OOM can cause the network to drop and the 2nd failure was clearly OOM.

[Peter Jones]

Closing as a duplicate of LU-1548. Please reopen if this reoccurs with that issue fixed