[LU-5773] obdfilter-survey test 1c: oom occurred on OSS

• Type: Bug
• Resolution: Fixed
• Priority: Critical
• Fix Version/s: Lustre 2.7.0
• Affects Version/s: Lustre 2.7.0, Lustre 2.5.3
• Labels: 22pl MB mq115
• Environment: Lustre build: https://build.hpdd.intel.com/job/lustre-master/2684; Distro/Arch: RHEL6.5/x86_64
• Severity: 3
• Assignee: Niu Yawei (Inactive)
• Reporter: Jian Yu

Description

While running obdfilter-survey test 1c, oom failure occurred on OSS:

21:17:56:Lustre: DEBUG MARKER: == obdfilter-survey test 1c: Object Storage Targets survey, big batch == 02:50:56 (1412823056)
21:17:56:Lustre: DEBUG MARKER: lctl dl | grep obdfilter
21:17:56:Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@'
21:17:56:Lustre: Echo OBD driver; http://www.lustre.org/
21:17:56:hrtimer: interrupt took 7516 ns
21:17:56:lctl invoked oom-killer: gfp_mask=0x200da, order=0, oom_adj=0, oom_score_adj=0
21:17:56:lctl cpuset=/ mems_allowed=0
21:17:56:Pid: 19467, comm: lctl Not tainted 2.6.32-431.29.2.el6_lustre.x86_64 #1
21:17:56:Call Trace:
21:17:56: [<ffffffff810d07b1>] ? cpuset_print_task_mems_allowed+0x91/0xb0
21:17:56: [<ffffffff81122b80>] ? dump_header+0x90/0x1b0
21:17:56: [<ffffffff81122cee>] ? check_panic_on_oom+0x4e/0x80
21:17:56: [<ffffffff811233db>] ? out_of_memory+0x1bb/0x3c0
21:17:56: [<ffffffff8112fd5f>] ? __alloc_pages_nodemask+0x89f/0x8d0
21:17:56: [<ffffffff81167dea>] ? alloc_pages_vma+0x9a/0x150
21:17:56: [<ffffffff811499dd>] ? do_wp_page+0xfd/0x920
21:17:56: [<ffffffff8133e4f5>] ? misc_open+0x1d5/0x330
21:17:56: [<ffffffff8114a9fd>] ? handle_pte_fault+0x2cd/0xb00
21:17:56: [<ffffffff8118d495>] ? chrdev_open+0x125/0x230
21:17:56: [<ffffffff811ab840>] ? mntput_no_expire+0x30/0x110
21:17:56: [<ffffffff8118d370>] ? chrdev_open+0x0/0x230
21:17:56: [<ffffffff811863bf>] ? __dentry_open+0x23f/0x360
21:17:56: [<ffffffff812284ef>] ? security_inode_permission+0x1f/0x30
21:17:56: [<ffffffff8114b45a>] ? handle_mm_fault+0x22a/0x300
21:17:56: [<ffffffff8104a8d8>] ? __do_page_fault+0x138/0x480
21:17:56: [<ffffffff8152f25e>] ? do_page_fault+0x3e/0xa0
21:17:56: [<ffffffff8152f25e>] ? do_page_fault+0x3e/0xa0
21:17:56: [<ffffffff8152c615>] ? page_fault+0x25/0x30

Maloo report: https://testing.hpdd.intel.com/test_sets/973e0216-4fcd-11e4-8e65-5254006e85c2

Issue Links

• duplicates: LU-5079 conf-sanity test_47 timeout
• duplicates: LU-3366 Test failure obdfilter-survey, subtest test_1c: oom-killer
• is duplicated by: LU-6004 obdfilter-survey test_2a: obdfilter is missing when running "lctl dl"
• is duplicated by: LU-5920 obdfilter-survey test_1c: OST OOM
• is related to: LU-6064 obdfilter-survey test_1c: test failed to respond and timed out
• is related to: LU-4768 ost-survey hangs on client 2.4

Comments

[Andreas Dilger]

Niu or Yu Jian, could you please look into a patch to change obdfilter-survey to reduce the threads count when running in a low-memory VM, so it doesn't hit this OOM? We still want to run this test during autotest to make sure that the test script doesn't break, but it just needs to run basic functionality/stress tests since the performance numbers from a VM are useless.

[Niu Yawei (Inactive)]

Do we still have real machine which has larger memory in our auto-test system? I presume such failure wouldn't occur on that system. Probably we should reduce the OST/thread count for the test_1c to make it runable on the 2G mem VMs?

[Niu Yawei (Inactive)]

Jodi, it seems to me that patch isn't related to this failure.

[Jodi Levi (Inactive)]

Do we need to back port http://review.whamcloud.com/#/c/11971/ to other branches?

[Jian Yu]

More instance on Lustre b2_5 branch:
https://testing.hpdd.intel.com/test_sets/dc5dabda-8074-11e4-a434-5254006e85c2

[Niu Yawei (Inactive)]

Looks dup of LU-3366. Perhaps the vm (OSS) can't afford such test (7 OSTs, 128 brw threads for each OST)?

[James Nunez (Inactive)]

Patch http://review.whamcloud.com/#/c/11971/ modified the ost-survey script, not obdfilter. Is there still a connection with this ticket and the 11971 patch?

[Andreas Dilger]

It appears that http://review.whamcloud.com/11971 was changing the obdfilter-survey script, which landed on Oct 5th, and this bug was filed on Oct 20th.

[Jodi Levi (Inactive)]

Niu,
Could you please have a look at this one?
Thank you!

[Andreas Dilger]

I found something strange in the OST logs - hundreds of lctl processes are running on the node, like it is a fork bomb:

11:04:49:[28448]     0 28448     3820      230   0       0             0 lctl
11:04:49:[28449]     0 28449     3820      229   0       0             0 lctl
11:04:49:[28450]     0 28450     3820      228   1       0             0 lctl
11:04:49:[28451]     0 28451     3820      230   0       0             0 lctl
11:04:49:[28452]     0 28452     3820      229   1       0             0 lctl
[repeats]

[Jian Yu]

More instance on master branch:
https://testing.hpdd.intel.com/test_sets/19e98088-7e99-11e4-ab67-5254006e85c2