https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-5773 Lustre <p>While running obdfilter-survey test 1c, oom failure occurred on OSS:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>21:17:56:Lustre: DEBUG MARKER: == obdfilter-survey test 1c: Object Storage Targets survey, big batch == 02:50:56 (1412823056) 21:17:56:Lustre: DEBUG MARKER: lctl dl | grep obdfilter 21:17:56:Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' 21:17:56:Lustre: Echo OBD driver; http://www.lustre.org/ 21:17:56:hrtimer: interrupt took 7516 ns 21:17:56:lctl invoked oom-killer: gfp_mask=0x200da, order=0, oom_adj=0, oom_score_adj=0 21:17:56:lctl cpuset=/ mems_allowed=0 21:17:56:Pid: 19467, comm: lctl Not tainted 2.6.32-431.29.2.el6_lustre.x86_64 #1 21:17:56:Call Trace: 21:17:56: [&lt;ffffffff810d07b1&gt;] ? cpuset_print_task_mems_allowed+0x91/0xb0 21:17:56: [&lt;ffffffff81122b80&gt;] ? dump_header+0x90/0x1b0 21:17:56: [&lt;ffffffff81122cee&gt;] ? check_panic_on_oom+0x4e/0x80 21:17:56: [&lt;ffffffff811233db&gt;] ? out_of_memory+0x1bb/0x3c0 21:17:56: [&lt;ffffffff8112fd5f&gt;] ? __alloc_pages_nodemask+0x89f/0x8d0 21:17:56: [&lt;ffffffff81167dea&gt;] ? alloc_pages_vma+0x9a/0x150 21:17:56: [&lt;ffffffff811499dd&gt;] ? do_wp_page+0xfd/0x920 21:17:56: [&lt;ffffffff8133e4f5&gt;] ? misc_open+0x1d5/0x330 21:17:56: [&lt;ffffffff8114a9fd&gt;] ? handle_pte_fault+0x2cd/0xb00 21:17:56: [&lt;ffffffff8118d495&gt;] ? chrdev_open+0x125/0x230 21:17:56: [&lt;ffffffff811ab840&gt;] ? mntput_no_expire+0x30/0x110 21:17:56: [&lt;ffffffff8118d370&gt;] ? chrdev_open+0x0/0x230 21:17:56: [&lt;ffffffff811863bf&gt;] ? __dentry_open+0x23f/0x360 21:17:56: [&lt;ffffffff812284ef&gt;] ? security_inode_permission+0x1f/0x30 21:17:56: [&lt;ffffffff8114b45a&gt;] ? handle_mm_fault+0x22a/0x300 21:17:56: [&lt;ffffffff8104a8d8&gt;] ? __do_page_fault+0x138/0x480 21:17:56: [&lt;ffffffff8152f25e&gt;] ? do_page_fault+0x3e/0xa0 21:17:56: [&lt;ffffffff8152f25e&gt;] ? do_page_fault+0x3e/0xa0 21:17:56: [&lt;ffffffff8152c615&gt;] ? page_fault+0x25/0x30 </pre> </div></div> <p>Maloo report: <a href="https://testing.hpdd.intel.com/test_sets/973e0216-4fcd-11e4-8e65-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/973e0216-4fcd-11e4-8e65-5254006e85c2</a></p> Lustre build: <a href="https://build.hpdd.intel.com/job/lustre-master/2684">https://build.hpdd.intel.com/job/lustre-master/2684</a><br/> Distro/Arch: RHEL6.5/x86_64 LU-5773

obdfilter-survey test 1c: oom occurred on OSS

Bug Critical Resolved Fixed Niu Yawei Jian Yu 22pl MB mq115 Mon, 20 Oct 2014 23:01:15 +0000 Thu, 12 May 2016 16:44:18 +0000 Thu, 16 Jul 2015 20:39:17 +0000 Lustre 2.7.0 Lustre 2.5.3 Lustre 2.7.0 0 9 <p>More instance on master branch: <a href="https://testing.hpdd.intel.com/test_sets/ce9211ca-4bb9-11e4-b821-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/ce9211ca-4bb9-11e4-b821-5254006e85c2</a></p> <p>The patch to fix this was landed under <a href="https://jira.whamcloud.com/browse/LU-5079" title="conf-sanity test_47 timeout" class="issue-link" data-issue-key="LU-5079"><del>LU-5079</del></a>.</p> <p>Lustre Build: <a href="https://build.hpdd.intel.com/job/lustre-b2_5/100/" class="external-link" target="_blank" rel="nofollow noopener">https://build.hpdd.intel.com/job/lustre-b2_5/100/</a><br/> Distro/Arch: RHEL6.5/x86_64 + SLES11SP3/x86_64 (Server + Client)</p> <p>The same oom failure occurred on OSS while running obdfilter-survey test 1c:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>automount invoked oom-killer: gfp_mask=0x280da, order=0, oom_adj=0, oom_score_adj=0 automount cpuset=/ mems_allowed=0 Pid: 10749, comm: automount Not tainted 2.6.32-431.29.2.el6_lustre.gf7f3864.x86_64 #1 Call Trace: [&lt;ffffffff810d07b1&gt;] ? cpuset_print_task_mems_allowed+0x91/0xb0 [&lt;ffffffff81122b80&gt;] ? dump_header+0x90/0x1b0 [&lt;ffffffff81122cee&gt;] ? check_panic_on_oom+0x4e/0x80 [&lt;ffffffff811233db&gt;] ? out_of_memory+0x1bb/0x3c0 [&lt;ffffffff8112fd5f&gt;] ? __alloc_pages_nodemask+0x89f/0x8d0 [&lt;ffffffff81167dea&gt;] ? alloc_pages_vma+0x9a/0x150 [&lt;ffffffff8114ae6d&gt;] ? handle_pte_fault+0x73d/0xb00 [&lt;ffffffff811f4570&gt;] ? proc_delete_inode+0x0/0x80 [&lt;ffffffff8128b96e&gt;] ? number+0x2ee/0x320 [&lt;ffffffff8114b45a&gt;] ? handle_mm_fault+0x22a/0x300 [&lt;ffffffff8128b96e&gt;] ? number+0x2ee/0x320 [&lt;ffffffff8104a8d8&gt;] ? __do_page_fault+0x138/0x480 [&lt;ffffffff8128c210&gt;] ? string+0x40/0x100 [&lt;ffffffff8128d776&gt;] ? vsnprintf+0x336/0x5e0 [&lt;ffffffff8152f25e&gt;] ? do_page_fault+0x3e/0xa0 [&lt;ffffffff8152c615&gt;] ? page_fault+0x25/0x30 [&lt;ffffffff8128e71e&gt;] ? copy_user_generic_unrolled+0x3e/0xb0 [&lt;ffffffff811aedb2&gt;] ? seq_read+0x2d2/0x400 [&lt;ffffffff81189a95&gt;] ? vfs_read+0xb5/0x1a0 [&lt;ffffffff81189bd1&gt;] ? sys_read+0x51/0x90 [&lt;ffffffff810e204e&gt;] ? __audit_syscall_exit+0x25e/0x290 [&lt;ffffffff8100b072&gt;] ? system_call_fastpath+0x16/0x1b </pre> </div></div> <p>Maloo report: <a href="https://testing.hpdd.intel.com/test_sets/4bf3d7bc-6874-11e4-acbe-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/4bf3d7bc-6874-11e4-acbe-5254006e85c2</a></p> <p>The patch for <a href="https://jira.whamcloud.com/browse/LU-5079" title="conf-sanity test_47 timeout" class="issue-link" data-issue-key="LU-5079"><del>LU-5079</del></a> was included in this build, so, this is not a duplicate issue.</p> <p>More instance on master branch:<br/> <a href="https://testing.hpdd.intel.com/test_sets/e6ec8538-6b45-11e4-88ff-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/e6ec8538-6b45-11e4-88ff-5254006e85c2</a></p> <p>More instances on Lustre b2_5 branch:<br/> <a href="https://testing.hpdd.intel.com/test_sets/12401f30-7a4e-11e4-b9fd-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/12401f30-7a4e-11e4-b9fd-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/test_sets/393c5232-7a55-11e4-807e-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/393c5232-7a55-11e4-807e-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/test_sets/ee460ed2-7980-11e4-aa22-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/ee460ed2-7980-11e4-aa22-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/test_sets/f55941c6-6a58-11e4-b203-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/f55941c6-6a58-11e4-b203-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/test_sets/87b3b698-5cdd-11e4-8561-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/87b3b698-5cdd-11e4-8561-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/test_sets/bce60eac-4eef-11e4-872e-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/bce60eac-4eef-11e4-872e-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/test_sets/6702eeee-7d55-11e4-943c-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/6702eeee-7d55-11e4-943c-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/test_sets/6502c6d6-7d33-11e4-943c-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/6502c6d6-7d33-11e4-943c-5254006e85c2</a></p> <p>More instance on master branch:<br/> <a href="https://testing.hpdd.intel.com/test_sets/19e98088-7e99-11e4-ab67-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/19e98088-7e99-11e4-ab67-5254006e85c2</a></p> <p>I found something strange in the OST logs - hundreds of lctl processes are running on the node, like it is a fork bomb:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>11:04:49:[28448] 0 28448 3820 230 0 0 0 lctl 11:04:49:[28449] 0 28449 3820 229 0 0 0 lctl 11:04:49:[28450] 0 28450 3820 228 1 0 0 lctl 11:04:49:[28451] 0 28451 3820 230 0 0 0 lctl 11:04:49:[28452] 0 28452 3820 229 1 0 0 lctl [repeats] </pre> </div></div> <p>Niu,<br/> Could you please have a look at this one?<br/> Thank you!</p> <p>It appears that <a href="http://review.whamcloud.com/11971" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/11971</a> was changing the obdfilter-survey script, which landed on Oct 5th, and this bug was filed on Oct 20th.</p> <p>Patch <a href="http://review.whamcloud.com/#/c/11971/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/11971/</a> modified the ost-survey script, not obdfilter. Is there still a connection with this ticket and the 11971 patch?</p> <p>Looks dup of <a href="https://jira.whamcloud.com/browse/LU-3366" title="Test failure obdfilter-survey, subtest test_1c: oom-killer" class="issue-link" data-issue-key="LU-3366"><del>LU-3366</del></a>. Perhaps the vm (OSS) can't afford such test (7 OSTs, 128 brw threads for each OST)?</p> <p>More instance on Lustre b2_5 branch:<br/> <a href="https://testing.hpdd.intel.com/test_sets/dc5dabda-8074-11e4-a434-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/dc5dabda-8074-11e4-a434-5254006e85c2</a></p> <p>Do we need to back port <a href="http://review.whamcloud.com/#/c/11971/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/11971/</a> to other branches?</p> <p>Jodi, it seems to me that patch isn't related to this failure.</p> <p>Do we still have real machine which has larger memory in our auto-test system? I presume such failure wouldn't occur on that system. Probably we should reduce the OST/thread count for the test_1c to make it runable on the 2G mem VMs?</p> <p>Niu or Yu Jian, could you please look into a patch to change obdfilter-survey to reduce the threads count when running in a low-memory VM, so it doesn't hit this OOM? We still want to run this test during autotest to make sure that the test script doesn't break, but it just needs to run basic functionality/stress tests since the performance numbers from a VM are useless.</p> <p>Ok, I'm going to cook a patch soon.</p> <p><a href="http://review.whamcloud.com/13078" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/13078</a></p> <p>Niu Yawei (yawei.niu@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/13101" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/13101</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-5773" title="obdfilter-survey test 1c: oom occurred on OSS" class="issue-link" data-issue-key="LU-5773"><del>LU-5773</del></a> test: reduce thread count<br/> Project: fs/lustre-release<br/> Branch: b2_5<br/> Current Patch Set: 1<br/> Commit: 113e635c1b68afa7698ec3b894647526ce2fef79</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/13078/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/13078/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-5773" title="obdfilter-survey test 1c: oom occurred on OSS" class="issue-link" data-issue-key="LU-5773"><del>LU-5773</del></a> test: reduce thread count<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: d1a717ed189a1245af1f96ecb701cd869956ef75</p> Duplicate LU-5079 LU-3366 LU-6004 LU-5920 Related LU-4768 LU-6064 Development Rank 1|hzwz1r: Rank (Obsolete) 16206 Severity