https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-4034 Lustre <p>One of our user noticed a strange problem during metadata operations, it looks like a memory allocation issue:<br/> <span class="error">&#91;root@XXX ~&#93;</span># ls -l /mnt/lustre/scratch/people/YYYY/SPE.SPIN/050524/28/temp.a438<br/> ls: cannot access /mnt/lustre/scratch/people/YYYY/SPE.SPIN/050524/28/temp.a438: Cannot allocate memory</p> <p>The client log says:<br/> Oct 1 16:20:11 zeus kernel: LustreError: 11-0: scratch-OST0013-osc-ffff8804925f1400: Communicating with 172.16.126.4@tcp, operation ldlm_enqueue failed with -12.<br/> Oct 1 16:20:11 zeus kernel: LustreError: 23207:0:(cl_lock.c:1420:cl_unuse_try()) result = -12, this is unlikely!</p> <p>OSS log has:<br/> Oct 1 16:20:11 scratch02 kernel: LustreError: 4630:0:(ldlm_resource.c:1165:ldlm_resource_get()) scratch-OST0013: lvbo_init failed for resource 0x40d9dcf:0x0: rc = -2</p> <p>Of course both servers and cients still have plenty of memory available. I've tried to look at similar issues in Jira, however I wasn't able to find a ticket with 1:1 relation to our issue.</p> SL6.4, 2.4.1 servers and clients with some patches, which have landed to b2_4 after 2.4.1 freeze. LU-4034

Cannot allocate memory on clients with 2.4.X

Bug Critical Resolved Cannot Reproduce WC Triage Marek Magrys Tue, 1 Oct 2013 14:22:53 +0000 Fri, 21 Jan 2022 01:25:07 +0000 Fri, 21 Jan 2022 01:25:07 +0000 Lustre 2.4.1 3 13 <p>This reminds me that the error reporting could be improved here. ldlm_resource_get() returned NULL because the resource could not be found. Then ldlm_lock_create() returns NULL. The ldlm_handle_enqueue0() misinterprets the returned NULL as being due to an allocation failure.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> /* The lock's callback data might be set in the policy function */ lock = ldlm_lock_create(ns, &amp;dlm_req-&gt;lock_desc.l_resource.lr_name, dlm_req-&gt;lock_desc.l_resource.lr_type, dlm_req-&gt;lock_desc.l_req_mode, cbs, NULL, 0, LVB_T_NONE); if (!lock) GOTO(out, rc = -ENOMEM); </pre> </div></div> <p>Any ideas anyone? I totally agree on the error reporting improvement idea, but it's a side problem here I guess. It still looks like that some object cannot be located. I've found a bunch of files with the same problem and I wonder if it's possible to find a solution other than taking the fs offline and running lfsck?</p> <p>I create <a href="https://jira.whamcloud.com/browse/LU-4524" title="ldlm_handle_enqueue0 incorrectly interprets ldlm_lock_create&#39;s NULL return value" class="issue-link" data-issue-key="LU-4524"><del>LU-4524</del></a> for the error code problem, as we have run into that too. </p> <p>Marek, it looks like the object is missing on the OSS. Basically you need to figure out why the objects are missing. Was there a hard crash or data corruption on the OSS? I would look for something earlier in the logs that might explain where the objects (0x40d9dcf in this case) went.</p> <p>AFAIK the lfsck in 2.4.1 does not do this kind of cleanup, that is going to be in a later phase of the lfsck rewrite. You should be able to do what the old lfsck would do and unlink the files.</p> <p>I'm getting similar error when trying to "ls" file which failed to create object (lustre 2.5.3). It looks like the error during "ls" is addressed at <a href="https://jira.whamcloud.com/browse/LU-4524" title="ldlm_handle_enqueue0 incorrectly interprets ldlm_lock_create&#39;s NULL return value" class="issue-link" data-issue-key="LU-4524"><del>LU-4524</del></a>.<br/> Should this ticket <a href="https://jira.whamcloud.com/browse/LU-4034" title="Cannot allocate memory on clients with 2.4.X" class="issue-link" data-issue-key="LU-4034"><del>LU-4034</del></a> be closed?</p> <p>Can we close this?</p> <p>We haven't seen this problem since upgrading to  2.5 and newer releases in Cyfronet.</p> <p>I'd vote for closing it and Marek will probably confirm</p> <p> </p> <p> </p> Related Development Rank 1|hzw4i7: Rank (Obsolete) 10838 Severity