https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-16497 Lustre <p>We're seeing quite a few errors on clients, OSSes and the MDS. </p> <p>For example on clients:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> Jan 16 09:53:09 juliet2 kernel: LustreError: 49499:0:(mdc_request.c:1441:mdc_read_page()) juliet-MDT0000-mdc-ffff99a3723aa800: [0x200001b3e:0x5f66:0x0] lock enqueue fails: rc = -4 Jan 16 21:30:41 juliet2 kernel: LustreError: 11-0: juliet-OST002a-osc-ffff99a3723aa800: operation ldlm_enqueue to node 10.29.22.93@tcp failed: rc = -107 Jan 16 21:30:41 juliet2 kernel: Lustre: juliet-OST002a-osc-ffff99a3723aa800: Connection to juliet-OST002a (at 10.29.22.93@tcp) was lost; in progress operations using <span class="code-keyword">this</span> service will wait <span class="code-keyword">for</span> recovery to complete Jan 16 21:30:41 juliet2 kernel: LustreError: 167-0: juliet-OST002a-osc-ffff99a3723aa800: This client was evicted by juliet-OST002a; in progress operations using <span class="code-keyword">this</span> service will fail. Jan 16 21:30:41 juliet2 kernel: Lustre: 4193:0:(llite_lib.c:2762:ll_dirty_page_discard_warn()) juliet: dirty page discard: 10.29.22.90@tcp:/juliet/fid: [0x20002dd8a:0x16daa:0x0]/ may get corrupted (rc -108) Jan 16 21:30:41 juliet2 kernel: Lustre: 4191:0:(llite_lib.c:2762:ll_dirty_page_discard_warn()) juliet: dirty page discard: 10.29.22.90@tcp:/juliet/fid: [0x20002dd8a:0x16cb1:0x0]/ may get corrupted (rc -108) </pre> </div></div> <p>OSS:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> Jan 16 06:17:54 joss1 kernel: LustreError: 6496:0:(events.c:455:server_bulk_callback()) event type 3, status -5, desc ffff92ef5dbb3000 Jan 16 06:17:54 joss1 kernel: LustreError: 16260:0:(ldlm_lib.c:3363:target_bulk_io()) @@@ network error on bulk WRITE req@ffff92f3dfcbb850 x1760556572171776/t0(0) o4-&gt;bd9b8fe9-b80f-7114-7b35-663a8e9d48db@10.29.22.97@tcp:446/0 lens 488/448 e 0 to 0 dl 1673867911 ref 1 fl Interpret:/0/0 rc 0/0 Jan 16 06:17:54 joss1 kernel: Lustre: juliet-OST0009: Client bd9b8fe9-b80f-7114-7b35-663a8e9d48db (at 10.29.22.97@tcp) reconnecting Jan 16 06:17:54 joss1 kernel: Lustre: juliet-OST0009: Connection restored to 3d01cce1-cfce-5103-0db6-32c1aa8f728c (at 10.29.22.97@tcp) Jan 16 06:17:54 joss1 kernel: Lustre: juliet-OST0009: Bulk IO write error with bd9b8fe9-b80f-7114-7b35-663a8e9d48db (at 10.29.22.97@tcp), client will retry: rc = -110 Jan 16 06:17:54 joss1 kernel: Lustre: Skipped 1 previous similar message Jan 16 06:17:54 joss1 kernel: LustreError: 16218:0:(ldlm_lib.c:3357:target_bulk_io()) @@@ Reconnect on bulk WRITE req@ffff92eb76e54050 x1760556572184448/t0(0) o4-&gt;bd9b8fe9-b80f-7114-7b35-663a8e9d48db@10.29.22.97@tcp:452/0 lens 488/448 e 0 to 0 dl 1673867917 ref 1 fl Interpret:/0/0 rc 0/0 </pre> </div></div> <p>MDS:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> Jan 16 19:52:10 jmds1 kernel: LustreError: 47609:0:(ldlm_lib.c:3357:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff995a6c544850 x1760579715652736/t0(0) o37-&gt;bd9b8fe9-b80f-7114-7b35-663a8e9d48db@10.29.22.97@tcp:220/0 lens 448/440 e 1 to 0 dl 1673916760 ref 1 fl Interpret:/0/0 rc 0/0 Jan 19 12:11:29 jmds1 kernel: LustreError: 15481:0:(mgs_handler.c:282:mgs_revoke_lock()) MGS: can't take cfg lock <span class="code-keyword">for</span> 0x736d61726170/0x3 : rc = -11 </pre> </div></div> <p>Is it possible to give us an idea of what these errors might indicate? e.g., network issues, misconfiguration, load etc, so we can narrow down the focus of investigation. Let us know what extra details (logs, cluster settings) you might need if further information is needed.</p> LU-16497

various lustre errors on clients and servers

Bug Minor Resolved Fixed Colin Faber Dneg Fri, 20 Jan 2023 15:52:38 +0000 Tue, 7 Mar 2023 15:01:27 +0000 Tue, 7 Mar 2023 15:01:27 +0000 Lustre 2.15.2 0 3 <p>Version of Lustre is 2.12.8_6_g5457c37 on all OSSes, the MDS and clients (apart from one, which is running 2.12.6)</p> <p>Hi,</p> <p>Can you please attach full logs around this incident? Thank you!</p> <p>(dmesg / syslogs, etc)</p> <p>Logs attached</p> <p>A bit more information. This cluster was upgraded to 2.12.8_6_g5457c37, to address issues seen in <a href="https://jira.whamcloud.com/browse/LU-15915" title="/bin/rm: fts_read failed: Cannot send after transport endpoint shutdown" class="issue-link" data-issue-key="LU-15915"><del>LU-15915</del></a> and <a href="https://jira.whamcloud.com/browse/LU-16343" title="soft lockups ptlrpcd" class="issue-link" data-issue-key="LU-16343"><del>LU-16343</del></a>. As a temporary mitigation, we lowered lru size to 128 from 10000. We set it back to 10000 just recently, and since that happened (around 19th of Jan), evictions have started occurring as well. Logs attached</p> <p>Hi <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=dneg" class="user-hover" rel="dneg">dneg</a> </p> <p>Based on what I'm seeing this does look very similar to <a href="https://jira.whamcloud.com/browse/LU-14644" title="IOR SSF PFL ill-formed I/O job aborted with EIO during automated FOFB testing" class="issue-link" data-issue-key="LU-14644"><del>LU-14644</del></a>, this has been addressed in 2.15.0 and 2.12.10. Can you try upgrading and see if you still experience?</p> <p>Thanks Colin, looking at the Whamcloud public repos, the 'latest-2.12-release' link points to 2.12.9-1. Has this got the required patches, or should we wait for 2.12.10 (and when will that be released?)</p> <p>Regards,</p> <p>Campbell</p> <p>Campbell</p> <p>There are no current plans to issue at 2.12.10. What Colin means is that the fix has been merged to the branch so that if we ever did decide to do one, it would include this fix. Using the latest 2.15.x LTS release (2.15.2) would be the most expedient option</p> <p> </p> <p>Peter</p> <p>Ok, thanks Peter</p> <p>Quick question Peter: since we're upgrading from EL7 to EL8 as well as from 2.12 to 2.15, is there any special steps to take?</p> <p>The one tip that I have heard is to be cautious to not inadvertently reformat your OSTs but, basically, this should be a standard OS upgrade. Based on the showing at the Lustre BOF at SC22, it seems like a number of people have already navigated this upgrade so you could always poll lustre-discuss to see if other community members have any experiences to share.</p> <p>Resolving as this is fixed in 2.15.2</p> Related LU-14644 Development Rank 1|i03aqf: Rank (Obsolete) 9223372036854775807 Severity