https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-9966 Lustre <p>This issue was created by maloo for Bob Glossman &lt;bob.glossman@intel.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://testing.hpdd.intel.com/test_sets/32b0aa4c-9502-11e7-ba84-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/32b0aa4c-9502-11e7-ba84-5254006e85c2</a>.</p> <p>The sub-test test_411 failed with the following error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>fail to trigger a memory allocation error </pre> </div></div> <p>test_411 is very new. has been failing since 9/1<br/> some (all?) of the instances of FAIL have been seen on sles12sp2/sles12sp3</p> <p>more:<br/> <a href="https://testing.hpdd.intel.com/test_sets/8c6725ca-8f6c-11e7-b5c2-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/8c6725ca-8f6c-11e7-b5c2-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/test_sets/0a6acf7c-8f8f-11e7-b67f-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/0a6acf7c-8f8f-11e7-b67f-5254006e85c2</a></p> <p>Info required for matching: sanity 411</p> LU-9966

sanity test_411: fail to trigger a memory allocation error

Bug Minor Resolved Fixed Yang Sheng Maloo Sat, 9 Sep 2017 05:53:24 +0000 Sat, 6 Oct 2018 13:39:14 +0000 Sat, 6 Oct 2018 13:39:14 +0000 Lustre 2.12.0 Lustre 2.12.0 0 12 <p>this may be a 100% fail on any sles12. It may never have worked except on RHEL 7.<br/> another:<br/> <a href="https://testing.hpdd.intel.com/test_sets/85081fe4-9536-11e7-b75f-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/85081fe4-9536-11e7-b75f-5254006e85c2</a></p> <p>Bob, it would be helpful if you linked this (and other regressions) to the Jira ticket and patch that added this new test. </p> <p>The patch that added test 411 was <a href="https://review.whamcloud.com/21745" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/21745</a>, "<a href="https://jira.whamcloud.com/browse/LU-8435" title="LBUG (osc_cache.c:1290:osc_completion()) ASSERTION( equi(page-&gt;cp_state == CPS_PAGEIN, cmd == OBD_BRW_READ) )" class="issue-link" data-issue-key="LU-8435"><del>LU-8435</del></a> tests: slab alloc error does not LBUG"</p> <p>In sles12 there is no /sys/fs/cgroup/memory/memory.kmem.limit_in_bytes<br/> Since test_411 uses this it's no surprise the test doesn't work.</p> <p>Is there an equivalent function that could be used instead or should we just skip the test for sles12 (and presumably any other newer kernels)?</p> <blockquote> <p>Is there an equivalent function that could be used instead or should we just skip the test for sles12 (and presumably any other newer kernels)?</p></blockquote> <p>Needs the Author of the test to answer that question.<br/> As far as I can tell there is nothing equivalent in sles12.</p> <p>If the solution is to skip the test when the needed /sys entry isn't there I can push a patch for that. There is already some skip logic there, I would just need to extend it a bit.</p> <p><a href="https://testing.hpdd.intel.com/test_sets/57f429a2-97f8-11e7-b9c6-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/57f429a2-97f8-11e7-b9c6-5254006e85c2</a></p> <p>Looks like this test is failed by permission issue.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>== sanity test 411: Slab allocation error with cgroup does not LBUG ================================== 01:03:59 (1505203439) 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 1.78497 s, 58.7 MB/s /usr/lib64/lustre/tests/sanity.sh: line 16400: /sys/fs/cgroup/memory/osc_slab_alloc/memory.kmem.limit_in_bytes: Permission denied 204800+0 records in 204800+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 23.5257 s, 4.5 MB/s sanity test_411: @@@@@@ FAIL: fail to trigger a memory allocation error Trace dump: </pre> </div></div> <p>The 'osc_slab_alloc/memory.kmem.limit_in_bytes' cannot be changed so trigger action is failed. I'll try to find the cause.</p> <p>Thanks,<br/> YangSheng</p> <p>it reports as "permission denied" but pretty sure it's due to the entry not existing. Easy fix to check for the entry &amp; skip if it doesn't exist, but not sure that's the right approach.</p> <p>I can push a mod that does that for inspection.</p> <p>Bob Glossman (bob.glossman@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/28974" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/28974</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9966" title="sanity test_411: fail to trigger a memory allocation error" class="issue-link" data-issue-key="LU-9966"><del>LU-9966</del></a> test: add a skip test to test_411<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 6c5dcd0caca8f08eb9874b533629f09af3cc1b7f</p> <p>The 'CONFIG_MEMCG_KMEM' is disabled in sles12 default. So kmem.limit_in_bytes is absent. Then skipping is right solution.</p> <p>I see that 'CONFIG_MEMCG_KMEM' is enabled in rhel7 by default. Totally explains why test 411 works on rhel7 and doesn't work on sles12.</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/28974/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/28974/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9966" title="sanity test_411: fail to trigger a memory allocation error" class="issue-link" data-issue-key="LU-9966"><del>LU-9966</del></a> test: add a skip test to test_411<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: f6b0e358f304b006dd24524503bb16d649c5499d</p> <p>Landed for 2.11</p> <p>Seeing this failure on the flr branch:<br/> <a href="https://testing.hpdd.intel.com/test_sets/93f8d4c2-ce65-11e7-9c63-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/93f8d4c2-ce65-11e7-9c63-52540065bddc</a></p> <p>Two failure instances occurred on master branch yesterday:<br/> <a href="https://testing.hpdd.intel.com/test_sets/76978322-06b6-11e8-a7cd-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/76978322-06b6-11e8-a7cd-52540065bddc</a><br/> <a href="https://testing.hpdd.intel.com/test_sets/455d1326-06d9-11e8-bd00-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/455d1326-06d9-11e8-bd00-52540065bddc</a></p> <p>Looks like some allocation errors did occur anyway during these failed test sessions :</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[ 6308.394164] Lustre: DEBUG MARKER: == sanity test 411: Slab allocation error with cgroup does not LBUG ================================== 22:34:17 (1517438057) [ 6311.637464] SLUB: Unable to allocate memory on node -1 (gfp=0x8050) [ 6311.638238] cache: kmalloc-512(0:osc_slab_alloc), object size: 512, buffer size: 512, default order: 1, min order: 0 [ 6311.638238] node 0: slabs: 13, objs: 208, free: 0 [ 6311.670203] SLUB: Unable to allocate memory on node -1 (gfp=0x0) [ 6311.670957] cache: kmalloc-192(0:osc_slab_alloc), object size: 192, buffer size: 192, default order: 0, min order: 0 [ 6311.670957] node 0: slabs: 1, objs: 21, free: 0 [ 6360.020975] Lustre: DEBUG MARKER: /usr/sbin/lctl mark sanity test_411: @@@@@@ FAIL: fail to trigger a memory allocation error [ 6360.203970] Lustre: DEBUG MARKER: sanity test_411: @@@@@@ FAIL: fail to trigger a memory allocation error </pre> </div></div> <p>But not causing "dd" command to fail as it is expected in sanity/test_411:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>== sanity test 411: Slab allocation error with cgroup does not LBUG ================================== 22:34:17 (1517438057) 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 3.13092 s, 33.5 MB/s 204800+0 records in 204800+0 records out 104857600 bytes (105 MB) copied, 48.1542 s, 2.2 MB/s sanity test_411: @@@@@@ FAIL: fail to trigger a memory allocation error Trace dump: = /usr/lib64/lustre/tests/test-framework.sh:5718:error() = /usr/lib64/lustre/tests/sanity.sh:17667:test_411() = /usr/lib64/lustre/tests/test-framework.sh:5994:run_one() = /usr/lib64/lustre/tests/test-framework.sh:6033:run_one_logged() = /usr/lib64/lustre/tests/test-framework.sh:5880:run_test() = /usr/lib64/lustre/tests/sanity.sh:17673:main() Dumping lctl log to /home/autotest/autotest/logs/test_logs/2018-01-31/lustre-reviews-el7-x86_64--review-ldiskfs--1_8_1__54107___60bc072b-48d1-4e5e-bb15-747752d7c9b7/sanity.test_411.*.1517438110.log CMD: trevis-10vm10,trevis-10vm11,trevis-10vm12,trevis-10vm9.trevis.hpdd.intel.com /usr/sbin/lctl dk &gt; /home/autotest/autotest/logs/test_logs/2018-01-31/lustre-reviews-el7-x86_64--review-ldiskfs--1_8_1__54107___60bc072b-48d1-4e5e-bb15-747752d7c9b7/sanity.test_411.debug_log.\$(hostname -s).1517438110.log; dmesg &gt; /home/autotest/autotest/logs/test_logs/2018-01-31/lustre-reviews-el7-x86_64--review-ldiskfs--1_8_1__54107___60bc072b-48d1-4e5e-bb15-747752d7c9b7/sanity.test_411.dmesg.\$(hostname -s).1517438110.log Resetting fail_loc on all nodes...CMD: trevis-10vm10,trevis-10vm11,trevis-10vm12,trevis-10vm9.trevis.hpdd.intel.com lctl set_param -n fail_loc=0 fail_val=0 2&gt;/dev/null done. </pre> </div></div> <p>Since as per my <a href="https://jira.whamcloud.com/browse/LU-8435" title="LBUG (osc_cache.c:1290:osc_completion()) ASSERTION( equi(page-&gt;cp_state == CPS_PAGEIN, cmd == OBD_BRW_READ) )" class="issue-link" data-issue-key="LU-8435"><del>LU-8435</del></a> analysis, kmem/memory cgroup features is known to be buggy with 3.x kernels (even if CONFIG_MEMCG_KMEM is configured by default in 3.x kernel shipped in CentOS/RH distros) and only safe to be used starting with 4.x kernels, why don't we simply skip sanity/test_411 for now, or at least add an other skip-test checking for Kernel 4.x?</p> <p>+1 on master review for <a href="https://jira.whamcloud.com/browse/LU-10680" title="MDT becoming unresponsive in 2.10.3" class="issue-link" data-issue-key="LU-10680"><del>LU-10680</del></a> at <a href="https://testing.hpdd.intel.com/test_sets/9b8d6f46-150e-11e8-a10a-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/9b8d6f46-150e-11e8-a10a-52540065bddc</a></p> <p>Hi, Bruno,</p> <p>I think test_411 just intends to verify not hit on LBUG. So we can avoid to check dd whether success or not. Does it necessary?</p> <p>Thanks,<br/> YangSheng</p> <p>another on master:<br/> <a href="https://testing.hpdd.intel.com/test_sets/a0416dd0-2f8a-11e8-b6a0-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/a0416dd0-2f8a-11e8-b6a0-52540065bddc</a></p> <p>&gt; I think test_411 just intends to verify not hit on LBUG. So we can avoid to check dd whether success or not.<br/> &gt; Does it necessary?</p> <p>YangSheng, yes we may do that, but we could also try to find a way to set the conditions that may cause dd to fail.</p> <p>Yang Sheng (yang.sheng@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/32293" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/32293</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9966" title="sanity test_411: fail to trigger a memory allocation error" class="issue-link" data-issue-key="LU-9966"><del>LU-9966</del></a> tests: sanity-411 check LBUG direct<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: eb5551388e81b16704c6655f6a2d0c469b9d5262</p> <p>Just a note: This issue is also seen with Lustre 2.11 on SLES15 RC4</p> <p>We see a different but related bug with Ubuntu18.</p> <p>Logs show that some allocations are failing but dd is succeeding. Perhaps we should weaken the test to just check that we don't crash.</p> <p>Yes, since the allocation is really depend on situation. So we should avoid to verify dd whether success or not. Anyway, I'll update the patch as this way.</p> <p>+1 on master branch:<br/> <a href="https://testing.whamcloud.com/test_sets/8e14dad8-c18f-11e8-a9d9-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/8e14dad8-c18f-11e8-a9d9-52540065bddc</a></p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/32293/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/32293/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9966" title="sanity test_411: fail to trigger a memory allocation error" class="issue-link" data-issue-key="LU-9966"><del>LU-9966</del></a> tests: sanity-411 check LBUG direct<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 18637db7ca6fa92ec6ea494a353a5ec46700a30e</p> <p>Landed for 2.12</p> Related LU-10366 LU-8435 Development Rank 1|hzzjwv: Rank (Obsolete) 9223372036854775807 Severity