https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-4970 Lustre <p>This issue was created by maloo for Nathaniel Clark &lt;nathaniel.l.clark@intel.com&gt;</p> <p>This issue relates to the following test suite run:<br/> <a href="http://maloo.whamcloud.com/test_sets/33c0eb0c-cd6b-11e3-b548-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">http://maloo.whamcloud.com/test_sets/33c0eb0c-cd6b-11e3-b548-52540035b04c</a><br/> <a href="https://maloo.whamcloud.com/test_sets/1ca115d8-bb68-11e3-8ec1-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/1ca115d8-bb68-11e3-8ec1-52540035b04c</a><br/> <a href="https://maloo.whamcloud.com/test_sets/beed7496-c0a2-11e3-b5ea-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/beed7496-c0a2-11e3-b5ea-52540035b04c</a></p> <p>The sub-test test_14 failed with the following error:</p> <blockquote> <p>'ls' should success after layout LFSCK repairing<br/> ls: cannot access /mnt/lustre/d14.sanity-lfsck/f46: Cannot allocate memory<br/> ls: cannot access /mnt/lustre/d14.sanity-lfsck/f32: Cannot allocate memory<br/> sanity-lfsck test_14: @@@@@@ FAIL: (5) ls should success. </p></blockquote> <p>Info required for matching: sanity-lfsck 14</p> LU-4970

Test failure sanity-lfsck test_14: ls should success

Bug Major Resolved Fixed nasf Maloo Mon, 28 Apr 2014 17:49:08 +0000 Wed, 1 Oct 2014 07:35:26 +0000 Fri, 5 Sep 2014 04:23:05 +0000 Lustre 2.6.0 Lustre 2.7.0 0 4 <p>Nathaniel, it looks like there is a memory allocation problem during the running of test_14 on all three failures you reported.</p> <p>Could you please take a look to see if this is something specific to ZFS, or is it related to LFSCK?</p> <p>I don't see this on LFSCK</p> <p>OST debug_log:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>2306:0:(ldlm_lockd.c:1167:ldlm_handle_enqueue0()) ### server-side enqueue handler START 2305:0:(ldlm_resource.c:1154:ldlm_resource_get()) lustre-OST0000: lvbo_init failed for resource 0x171:0x0: rc = -2 2306:0:(ldlm_resource.c:1154:ldlm_resource_get()) lustre-OST0000: lvbo_init failed for resource 0x180:0x0: rc = -2 2305:0:(ldlm_lockd.c:1431:ldlm_handle_enqueue0()) ### server-side enqueue handler END (lock (null), rc -12) </pre> </div></div> <p>This bug doesn't date back much prior to Apr 1, 2014</p> <p>The earliest commit I can associate a failure to is:<br/> <a href="http://git.whamcloud.com/?p=fs%2Flustre-release.git;a=commit;h=588a29ba81c6c59652e24831c177348f87ad489b" class="external-link" target="_blank" rel="nofollow noopener">588a29b</a> Wed Mar 26 00:14:46 2014 +0000 <a href="https://jira.whamcloud.com/browse/LU-4462" title="ERR_PTR dereferences in mdt_reconstruct functions" class="issue-link" data-issue-key="LU-4462"><del>LU-4462</del></a> <a href="http://review.whamcloud.com/8788" class="external-link" target="_blank" rel="nofollow noopener">mdt: don't apply mdt_object_fid() to ERR_PTRs</a></p> <p>but this is the root of long patch chains. The earliest patch I can associate a direct failure to:<br/> <a href="http://git.whamcloud.com/?p=fs%2Flustre-release.git;a=commit;h=a38ac90d1a3681928e434de2b471f57470911d04" class="external-link" target="_blank" rel="nofollow noopener">a38ac90</a> Tue Apr 1 16:11:43 2014 +0000 <a href="https://jira.whamcloud.com/browse/LU-4805" title="Test failure sanity test_230c: stat /mnt/lustre/d230c.sanity/f5 failed" class="issue-link" data-issue-key="LU-4805"><del>LU-4805</del></a> <a href="http://review.whamcloud.com/9806" class="external-link" target="_blank" rel="nofollow noopener">lmv: lookup remote migrating object in LMV</a></p> <p>I don't have a suspect patch but it appears somewhere in here an issue was introduced.</p> <p>I see there is still the "ls" failure in the test logs:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>ls: cannot access /mnt/lustre/d14.sanity-lfsck/f46: Cannot allocate memory </pre> </div></div> <p>but nothing in the Lustre debug logs, since they are only running with the default debug level.</p> <p>It probably makes sense to bump the debug level to -1 for test_14, and make the ls (9) failure an error_ignore() for the short term. That would allow debugging the problem more easily than just disabling the test.</p> <p>Turn up debugging:</p> <p><a href="http://review.whamcloud.com/10472" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/10472</a></p> <p>EXCEPT test for zfs:<br/> <a href="http://review.whamcloud.com/10473" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/10473</a></p> <p>It appears that <tt>ls: cannot allocate memory</tt> is a bogus message caused by lvbo_init returning NULL if the object cannot be found, and NULL being interpreted by the higher layers as -ENOMEM. This would be fixed by the patch <a href="http://review.whamcloud.com/9004" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/9004</a> that returns the proper error in this case (-ENOENT). There would still a test failure, but at least it would be more understandable.</p> <p>Failures with full debugging turned on:<br/> <a href="https://maloo.whamcloud.com/test_sets/8d617942-cffa-11e3-9aa7-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/8d617942-cffa-11e3-9aa7-52540035b04c</a><br/> <a href="https://maloo.whamcloud.com/test_sets/c14112a2-e7d5-11e3-8bff-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/c14112a2-e7d5-11e3-8bff-52540035b04c</a></p> <p>OST debug_log:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>(ofd_objects.c:634:ofd_attr_get()) Process entered (ofd_objects.c:659:ofd_attr_get()) Process leaving (rc=18446744073709551614 : -2 : fffffffffffffffe) (service.c:1068:ptlrpc_update_export_timer()) Process entered (service.c:1081:ptlrpc_update_export_timer()) Process leaving (ofd_lvb.c:98:ofd_lvbo_init()) Process leaving via out_obj (rc=18446744073709551614 : -2 : 0xfffffffffffffffe) (lustre_fid.h:712:fid_flatten32()) Process leaving (rc=4278190179 : 4278190179 : ff000063) (lu_object.c:1643:keys_init()) kmalloced 'ctx-&gt;lc_value': 320 at ffff88006f9e9000. (lu_object.c:429:lu_global_key_fini()) kfreed 'info': 512 at ffff88006f9e2e00. (capa.c:416:lu_capainfo_key_init()) kmalloced 'value': 128 at ffff880071527ac0. (dt_object.c:56:dt_global_key_fini()) kfreed 'info': 1216 at ffff88006f4ef800. (lu_ucred.c:53:lu_ucred_key_init()) kmalloced 'value': 72 at ffff880072641cc0. (tgt_main.c:163:tgt_key_fini()) kfreed 'info': 1152 at ffff88006bace000. (fld_handler.c:60:fld_key_fini()) kfreed 'info': 72 at ffff88006fdac7c0. (tgt_main.c:187:tgt_ses_key_init()) kmalloced 'value': 144 at ffff880071461bc0. (fid_handler.c:406:seq_key_fini()) kfreed 'info': 48 at ffff88006e3b4800. (service.c:1678:ptlrpc_server_request_add()) Process entered (service.c:1579:ptlrpc_server_hpreq_init()) Process entered (lquota_lib.c:53:lquota_key_fini()) kfreed 'info': 240 at ffff880072539cc0. (qsd_lib.c:71:qsd_key_fini()) kfreed 'info': 376 at ffff88006e688a00. (service.c:1630:ptlrpc_server_hpreq_init()) Process leaving (rc=0 : 0 : 0) (service.c:1693:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) (lfsck_lib.c:55:lfsck_key_fini()) kfreed 'info': 1144 at ffff8800723f7000. (ofd_dev.c:2332:ofd_key_fini()) kfreed 'info': 656 at ffff880071e1e000. (service.c:2006:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) (lu_object.c:1593:keys_fini()) kfreed 'ctx-&gt;lc_value': 320 at ffff880071a58c00. (ldlm_resource.c:1161:ldlm_resource_get()) lustre-OST0000: lvbo_init failed for resource 0x163:0x0: rc = -2 </pre> </div></div> <p>So ENOENT is the correct error that should be being passed up.</p> <p>This has been EXCEPT'd for 2.6</p> <p>This seems to have shown up on review-dne-part-2:<br/> <a href="https://testing.hpdd.intel.com/test_sets/3db646c0-1433-11e4-88ed-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/3db646c0-1433-11e4-88ed-5254006e85c2</a></p> <p>Hi Clark, I think that it is my code issue caused the failure, I will take the bug and fix it.</p> <p>Here are the patches:</p> <p><a href="http://review.whamcloud.com/#/c/11276/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/11276/</a><br/> <a href="http://review.whamcloud.com/#/c/11590/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/11590/</a></p> <p>Patches have been landed to master.</p> Related LU-4524 LU-5184 Development Rank 1|hzwl9j: Rank (Obsolete) 13760 Severity