https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-14372 Lustre <p>We are seeing this error with hdf5 files that results in a job crash due to I/O error. </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> Wed Jan 27 10:51:01 2021 C r901i5n2 [1611773461.647472] LustreError: 38823:0:(vvp_io.c:1562:vvp_io_init()) nbp11: refresh file layout [0x2400498d9:0x1642:0x0] error -5. Wed Jan 27 10:51:01 2021 M r901i5n2 kernel: [1611773461.663472] LustreError: 38823:0:(vvp_io.c:1562:vvp_io_init()) nbp11: refresh file layout [0x2400498d9:0x1642:0x0] error -5. </pre> </div></div> <p>Here is the only thing relevant that I found in debug file. I have enable additional debugging hoping to gather additional info.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> 0000100 00080000 1.0 Wed Jan 27 10:32:40 PST 2021 0 4789 0 (<span class="code-keyword">import</span>.c 1078 ptlrpc_connect_interpret()) nbp11-OST0024-osc-ffff8f3f503c4000 connect to target with instance 4 00000100 00080000 1.0 Wed Jan 27 10:32:40 PST 2021 0 4789 0 (<span class="code-keyword">import</span>.c 939 ptlrpc_connect_set_flags()) nbp11-OST0024-osc-ffff8f3f503c4000 Resetting ns_connect_flags to server flags 0xa0425af2e3440478 00000080 00080000 1.0 Wed Jan 27 10:32:40 PST 2021 0 4789 0 (lcommon_misc.c 70 cl_init_ea_size()) updating def/max_easize 72/1704 00000100 00080000 1.0 Wed Jan 27 10:32:40 PST 2021 0 4789 0 (<span class="code-keyword">import</span>.c 1162 ptlrpc_connect_interpret()) connected to replayable target nbp11-OST0024_UUID 00000100 00080000 1.0 Wed Jan 27 10:32:40 PST 2021 0 4789 0 (<span class="code-keyword">import</span>.c 86 import_set_state_nolock()) ffff8f57200e7000 nbp11-OST0024_UUID changing <span class="code-keyword">import</span> state from CONNECTING to FULL 00000080 00080000 1.0 Wed Jan 27 10:32:40 PST 2021 0 4789 0 (lcommon_misc.c 70 cl_init_ea_size()) updating def/max_easize 72/1704 00000100 00080000 1.0 Wed Jan 27 10:32:40 PST 2021 0 4789 0 (recover.c 223 ptlrpc_wake_delayed()) @@@ waking (set ffff8f4de56df380) req@ffff8f533af8edc0 x1683453707600256/t0(0) o101-&gt;nbp11-OST0024-osc-ffff8f3f503c4000@10.151.26.168@o2ib 28/4 lens 328/400 e 0 to 0 dl 0 ref 2 fl Rpc W/0/ffffffff rc 0/-1 00000100 00080000 1.0 Wed Jan 27 10:41:58 PST 2021 0 4789 0 (<span class="code-keyword">import</span>.c 1078 ptlrpc_connect_interpret()) nbp11-OST0015-osc-ffff8f3f503c4000 connect to target with instance 5 00000100 00080000 1.0 Wed Jan 27 10:41:58 PST 2021 0 4789 0 (<span class="code-keyword">import</span>.c 939 ptlrpc_connect_set_flags()) nbp11-OST0015-osc-ffff8f3f503c4000 Resetting ns_connect_flags to server flags 0xa0425af2e3440478 00000080 00080000 1.0 Wed Jan 27 10:41:58 PST 2021 0 4789 0 (lcommon_misc.c 70 cl_init_ea_size()) updating def/max_easize 72/1704 00000100 00080000 1.0 Wed Jan 27 10:41:58 PST 2021 0 4789 0 (<span class="code-keyword">import</span>.c 1162 ptlrpc_connect_interpret()) connected to replayable target nbp11-OST0015_UUID 00000100 00080000 1.0 Wed Jan 27 10:41:58 PST 2021 0 4789 0 (<span class="code-keyword">import</span>.c 86 import_set_state_nolock()) ffff8f4faa569000 nbp11-OST0015_UUID changing <span class="code-keyword">import</span> state from CONNECTING to FULL 00000080 00080000 1.0 Wed Jan 27 10:41:58 PST 2021 0 4789 0 (lcommon_misc.c 70 cl_init_ea_size()) updating def/max_easize 72/1704 00000100 00080000 1.0 Wed Jan 27 10:41:58 PST 2021 0 4789 0 (recover.c 223 ptlrpc_wake_delayed()) @@@ waking (set ffff8f3bbd4ef380) req@ffff8f3f3491edc0 x1683453707789248/t0(0) o101-&gt;nbp11-OST0015-osc-ffff8f3f503c4000@10.151.26.167@o2ib 28/4 lens 328/400 e 0 to 0 dl 0 ref 2 fl Rpc W/0/ffffffff rc 0/-1 00000100 00080000 1.0 Wed Jan 27 10:41:58 PST 2021 0 4789 0 (<span class="code-keyword">import</span>.c 1078 ptlrpc_connect_interpret()) nbp11-OST001a-osc-ffff8f3f503c4000 connect to target with instance 8 00000100 00080000 1.0 Wed Jan 27 10:41:58 PST 2021 0 4789 0 (<span class="code-keyword">import</span>.c 939 ptlrpc_connect_set_flags()) nbp11-OST001a-osc-ffff8f3f503c4000 Resetting ns_connect_flags to server flags 0xa0425af2e3440478 00000080 00080000 1.0 Wed Jan 27 10:41:58 PST 2021 0 4789 0 (lcommon_misc.c 70 cl_init_ea_size()) updating def/max_easize 72/1704 00000100 00080000 1.0 Wed Jan 27 10:41:58 PST 2021 0 4789 0 (<span class="code-keyword">import</span>.c 1162 ptlrpc_connect_interpret()) connected to replayable target nbp11-OST001a_UUID 00000100 00080000 1.0 Wed Jan 27 10:41:58 PST 2021 0 4789 0 (<span class="code-keyword">import</span>.c 86 import_set_state_nolock()) ffff8f5764398800 nbp11-OST001a_UUID changing <span class="code-keyword">import</span> state from CONNECTING to FULL 00000080 00080000 1.0 Wed Jan 27 10:41:58 PST 2021 0 4789 0 (lcommon_misc.c 70 cl_init_ea_size()) updating def/max_easize 72/1704 00000100 00080000 1.0 Wed Jan 27 10:41:58 PST 2021 0 4789 0 (recover.c 223 ptlrpc_wake_delayed()) @@@ waking (set ffff8f3bbd4efb80) req@ffff8f36219ca000 x1683453707789888/t0(0) o101-&gt;nbp11-OST001a-osc-ffff8f3f503c4000@10.151.26.171@o2ib 28/4 lens 328/400 e 0 to 0 dl 0 ref 2 fl Rpc W/0/ffffffff rc 0/-1 00000100 00080000 1.0 Wed Jan 27 10:41:59 PST 2021 0 4789 0 (<span class="code-keyword">import</span>.c 1078 ptlrpc_connect_interpret()) nbp11-OST000a-osc-ffff8f3f503c4000 connect to target with instance 4 00000100 00080000 1.0 Wed Jan 27 10:41:59 PST 2021 0 4789 0 (<span class="code-keyword">import</span>.c 939 ptlrpc_connect_set_flags()) nbp11-OST000a-osc-ffff8f3f503c4000 Resetting ns_connect_flags to server flags 0xa0425af2e3440478 00000080 00080000 1.0 Wed Jan 27 10:41:59 PST 2021 0 4789 0 (lcommon_misc.c 70 cl_init_ea_size()) updating def/max_easize 72/1704 00000100 00080000 1.0 Wed Jan 27 10:41:59 PST 2021 0 4789 0 (<span class="code-keyword">import</span>.c 1162 ptlrpc_connect_interpret()) connected to replayable target nbp11-OST000a_UUID 00000100 00080000 1.0 Wed Jan 27 10:41:59 PST 2021 0 4789 0 (<span class="code-keyword">import</span>.c 86 import_set_state_nolock()) ffff8f53b9dde800 nbp11-OST000a_UUID changing <span class="code-keyword">import</span> state from CONNECTING to FULL 00000080 00080000 1.0 Wed Jan 27 10:41:59 PST 2021 0 4789 0 (lcommon_misc.c 70 cl_init_ea_size()) updating def/max_easize 72/1704 00000100 00080000 1.0 Wed Jan 27 10:41:59 PST 2021 0 4789 0 (recover.c 223 ptlrpc_wake_delayed()) @@@ waking (set ffff8f4de56df700) req@ffff8f57a8e38dc0 x1683453707790848/t0(0) o101-&gt;nbp11-OST000a-osc-ffff8f3f503c4000@10.151.26.169@o2ib 28/4 lens 328/400 e 0 to 0 dl 0 ref 2 fl Rpc W/0/ffffffff rc 0/-1 00000100 00080000 1.0 Wed Jan 27 10:51:01 PST 2021 0 4789 0 (<span class="code-keyword">import</span>.c 1078 ptlrpc_connect_interpret()) nbp16-OST0005-osc-ffff8f3d83ba4000 connect to target with instance 4 00000100 00080000 1.0 Wed Jan 27 10:51:01 PST 2021 0 4789 0 (<span class="code-keyword">import</span>.c 939 ptlrpc_connect_set_flags()) nbp16-OST0005-osc-ffff8f3d83ba4000 Resetting ns_connect_flags to server flags 0xa0425af2e3440478 00000080 00080000 1.0 Wed Jan 27 10:51:01 PST 2021 0 4789 0 (lcommon_misc.c 70 cl_init_ea_size()) updating def/max_easize 144/600 00000100 00080000 1.0 Wed Jan 27 10:51:01 PST 2021 0 4789 0 (<span class="code-keyword">import</span>.c 1162 ptlrpc_connect_interpret()) connected to replayable target nbp16-OST0005_UUID 00000100 00080000 1.0 Wed Jan 27 10:51:01 PST 2021 0 4789 0 (<span class="code-keyword">import</span>.c 86 import_set_state_nolock()) ffff8f57bb391800 nbp16-OST0005_UUID changing <span class="code-keyword">import</span> state from CONNECTING to FULL 00000080 00080000 1.0 Wed Jan 27 10:51:01 PST 2021 0 4789 0 (lcommon_misc.c 70 cl_init_ea_size()) updating def/max_easize 144/600 00000100 00080000 1.0 Wed Jan 27 10:51:01 PST 2021 0 4789 0 (recover.c 223 ptlrpc_wake_delayed()) @@@ waking (set ffff8f3f99b8cc00) req@ffff8f3c5228b240 x1683453707981184/t0(0) o101-&gt;nbp16-OST0005-osc-ffff8f3d83ba4000@10.151.26.195@o2ib 28/4 lens 328/400 e 0 to 0 dl 0 ref 2 fl Rpc W/0/ffffffff rc 0/-1 00000100 00080000 3.0 Wed Jan 27 11:05:33 PST 2021 0 4789 0 (<span class="code-keyword">import</span>.c 1753 ptlrpc_disconnect_idle_interpret()) @@@ inflight=1, refcount=5 rc = 0 req@ffff8f3fb2bb16c0 x1683453708172736/t0(0) o9-&gt;nbp11-OST0030-osc-ffff8f3f503c4000@10.151.26.168@o2ib 28/4 lens 224/192 e 0 to 0 d l 1611774613 ref 1 fl Interpret RN/0/0 rc 0/0 00000100 00080000 3.0 Wed Jan 27 11:05:33 PST 2021 0 4789 0 (<span class="code-keyword">import</span>.c 86 import_set_state_nolock()) ffff8f5764398000 nbp11-OST0030_UUID changing <span class="code-keyword">import</span> state from CONNECTING to IDLE 00000100 00080000 18.0 Wed Jan 27 11:06:00 PST 2021 0 4789 0 (<span class="code-keyword">import</span>.c 1753 ptlrpc_disconnect_idle_interpret()) @@@ inflight=1, refcount=5 rc = 0 req@ffff8f29b0455240 x1683453708177472/t0(0) o9-&gt;nbp11-OST0031-osc-ffff8f3f503c4000@10.151.26.170@o2ib 28/4 lens 224/192 e 0 to 0 dl 1611774640 ref 1 fl Interpret RN/0/0 rc 0/0 00000100 00080000 18.0 Wed Jan 27 11:06:00 PST 2021 0 4789 0 (<span class="code-keyword">import</span>.c 86 import_set_state_nolock()) ffff8f571ce55000 nbp11-OST0031_UUID changing <span class="code-keyword">import</span> state from CONNECTING to IDLE 00000002 00080000 45.0 Wed Jan 27 11:07:35 PST 2021 0 39214 0 (mdc_request.c 903 mdc_close()) @@@ matched open req@ffff8f348f370040 x1683453708196032/t41664090920(41664090920) o101-&gt;nbp11-MDT0002-mdc-ffff8f3f503c4000@10.151.26.171@o2ib 12/10 lens 976/1008 e 0 to 0 dl 1611775078 ref 1 fl Complete RP/4/ffffffff rc 0/-1 00000002 00080000 45.0 Wed Jan 27 11:07:35 PST 2021 0 39214 0 (mdc_request.c 903 mdc_close()) @@@ matched open req@ffff8f348f3716c0 x1683453708196416/t41664090924(41664090924) o101-&gt;nbp11-MDT0002-mdc-ffff8f3f503c4000@10.151.26.171@o2ib 12/10 lens 1080/1080 e 0 to 0 dl 161177507 8 ref 2 fl Complete RP/4/0 rc 0/0 </pre> </div></div> <p>I did see this error from MDT to OST around the same time. So this could be a network issue. </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> Jan 27 10:59:06 nbp11-srv1 kernel: [21636687.904930] Lustre: 24143:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out <span class="code-keyword">for</span> slow reply: [sent 1611773322/real 1611773322] req@ffff9e2aca8f6300 x1669611070093696/t0(0) o6-&gt;nbp11-OST0035-osc-MDT0000@10.151.26.172@o2ib:28/4 lens 544/432 e 1 to 1 dl 1611773946 ref 1 fl Rpc:X/2/ffffffff rc -11/-1 Jan 27 10:59:06 nbp11-srv1 kernel: [21636687.991757] Lustre: nbp11-OST0035-osc-MDT0000: Connection to nbp11-OST0035 (at 10.151.26.172@o2ib) was lost; in progress operations using <span class="code-keyword">this</span> service will wait <span class="code-keyword">for</span> recovery to complete Jan 27 10:59:06 nbp11-srv6 kernel: [21636637.466674] Lustre: nbp11-OST0035: Client nbp11-MDT0000-mdtlov_UUID (at 10.151.26.167@o2ib) reconnecting </pre> </div></div> Client running 2.12.5 <br/> server running 2.12.4/5 LU-14372

LustreError: 38823:0:(vvp_io.c:1562:vvp_io_init()) nbp11: refresh file layout [0x2400498d9:0x1642:0x0] error -5

Bug Critical Resolved Duplicate John Hammond Mahmoud Hanafi Wed, 27 Jan 2021 20:43:51 +0000 Fri, 7 Oct 2022 21:00:31 +0000 Fri, 7 Oct 2022 21:00:31 +0000 Lustre 2.12.5 0 5 <p>Serguei</p> <p>Does this seem like a network issue to you?</p> <p>Peter</p> <p>I'm not sure yet if this is caused by a network issue or OST fails to respond for some other reason.</p> <p>Would it be possible to collect net debug logs?</p> <p> </p> <p>This is not a network issue. This is comes from <tt>vvp_prune()</tt> failing to </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>$LUSTRE/tests/llmount.sh lfs setstripe \ -E 1M -c 4 \ -E 2M -c 4 \ -E 3M -c 4 \ -E 4M -c 4 \ -E 5M -c 4 \ -E 6M -c 4 \ -E 7M -c 4 \ -E 8M -c 4 \ -E 9M -c 4 \ -E 10M -c 4 \ -E 11M -c 4 \ -E 12M -c 4 \ -E 13M -c 4 \ -E 14M -c 4 \ -E 15M -c 4 \ -E eof -c 4 \ /mnt/lustre while dd if=/dev/zero of=f0 bs=1M count=16 &amp;&amp; rm f0; do true; done </pre> </div></div> <p>Eventually dd will fail with -EIO </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>dd: error writing 'f0': Input/output error 11+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.274785 s, 38.2 MB/s [20055.549945] LustreError: 2005822:0:(vvp_io.c:1820:vvp_io_init()) lustre: refresh file layout [0x200000bd1:0x1a67a:0x0] error -5. </pre> </div></div> <p>Tracing shows that the -EIO comes from:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-keyword">static</span> <span class="code-object">int</span> vvp_prune(<span class="code-keyword">const</span> struct lu_env *env, struct cl_object *obj) { struct inode *inode = vvp_object_inode(obj); <span class="code-object">int</span> rc; ENTRY; rc = cl_sync_file_range(inode, 0, OBD_OBJECT_EOF, CL_FSYNC_LOCAL, 1); <span class="code-keyword">if</span> (rc &lt; 0) { CDEBUG(D_VFSTRACE, DFID <span class="code-quote">": writeback failed: %d\n"</span>, PFID(lu_object_fid(&amp;obj-&gt;co_lu)), rc); RETURN(rc); } truncate_inode_pages(inode-&gt;i_mapping, 0); <span class="code-keyword">if</span> (inode-&gt;i_mapping-&gt;nrpages) { CDEBUG(D_VFSTRACE, DFID <span class="code-quote">": still has %lu pages remaining\n"</span>, PFID(lu_object_fid(&amp;obj-&gt;co_lu)), inode-&gt;i_mapping-&gt;nrpages); RETURN(-EIO); } RETURN(0); } </pre> </div></div> <p>This appears to be fixed in master by 7d5d004506 (<a href="https://jira.whamcloud.com/browse/LU-14644" title="IOR SSF PFL ill-formed I/O job aborted with EIO during automated FOFB testing" class="issue-link" data-issue-key="LU-14644"><del>LU-14644</del></a> vvp: wait for nrpages to be updated).</p> <p>"John L. Hammond &lt;jhammond@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/46948" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/46948</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14644" title="IOR SSF PFL ill-formed I/O job aborted with EIO during automated FOFB testing" class="issue-link" data-issue-key="LU-14644"><del>LU-14644</del></a> vvp: wait for nrpages to be updated<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: 1c73a734375a05fa75a9a55fa89494f743a10114</p> <p>Fix landed to b2_12</p> Duplicate LU-14644 Development Rank 1|i01khb: Rank (Obsolete) 9223372036854775807 Severity