https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-9825 Lustre <p>Running soak on <a href="https://jira.whamcloud.com/browse/LU-7899" title="osd_xattr_set() to batch actual EA update" class="issue-link" data-issue-key="LU-7899"><del>LU-7899</del></a> patch, seeing multiple repeated errors not tied to any system halts. <br/> First error, on OST, appeared after an OST failover, during LFSCK run. (LFSCK exceeded timeout and was aborted)<br/> There is a hiccup in network and simul job dies:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">sys-recov.today:/scratch/logs/syslog/soak-6.log:Aug 3 22:14:38 soak-6 kernel: Lustre: soaked-OST0016: Recovery over after 0:05, of 35 clients 35 recovered and 0 were evicted. sys-recov.today:/scratch/logs/syslog/soak-6.log:Aug 3 22:14:49 soak-6 kernel: Lustre: soaked-OST0004: Recovery over after 0:06, of 35 clients 35 recovered and 0 were evicted. </pre> </div></div> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">/scratch/logs/syslog/soak-5.log:Aug 3 22:17:08 soak-5 kernel: LustreError: 24771:0:(osd_index.c:224:__osd_xattr_load_by_oid()) Skipped 13 previous similar messages /scratch/logs/syslog/soak-5.log:Aug 3 22:17:08 soak-5 kernel: LustreError: 24771:0:(osd_index.c:224:__osd_xattr_load_by_oid()) soaked-OST0003: can't get bonus, rc = -17 /scratch/logs/syslog/soak-2.log:Aug 3 22:18:07 soak-2 kernel: LustreError: 24114:0:(osd_index.c:224:__osd_xattr_load_by_oid()) Skipped 534 previous similar messages /scratch/logs/syslog/soak-2.log:Aug 3 22:18:07 soak-2 kernel: LustreError: 24114:0:(osd_index.c:224:__osd_xattr_load_by_oid()) soaked-OST0012: can't get bonus, rc = -17 </pre> </div></div> <p>Second set of errors comes up on MDS</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">scratch/logs/syslog/soak-8.log:Aug 3 22:13:46 soak-8 kernel: LustreError: 4491:0:(client.c:3006:ptlrpc_replay_interpret()) @@@ status -2, old was 0 req@ffff8807d76d5700 x1574746919153872/t167504869448(167504869448) o6-&gt;soaked-OST0016-osc-MDT0000@192.168.1.107@o2ib:28/4 lens 664/400 e 0 to 0 dl 1501798434 ref 2 fl Interpret:R/4/0 rc -2/-2 /scratch/logs/syslog/soak-8.log:Aug 3 22:13:50 soak-8 kernel: LustreError: 4491:0:(client.c:3006:ptlrpc_replay_interpret()) @@@ status -2, old was 0 req@ffff8807f37a1500 x1574746919157120/t167504861127(167504861127) o6-&gt;soaked-OST0004-osc-MDT0000@192.168.1.107@o2ib:28/4 lens 664/400 e 0 to 0 dl 1501798438 ref 2 fl Interpret:R/4/0 rc -2/-2 /scratch/logs/syslog/soak-8.log:Aug 3 22:13:56 soak-8 kernel: LustreError: 4939:0:(mdt_lvb.c:163:mdt_lvbo_fill()) Skipped 1 previous similar message /scratch/logs/syslog/soak-8.log:Aug 3 22:13:56 soak-8 kernel: LustreError: 4939:0:(mdt_lvb.c:163:mdt_lvbo_fill()) soaked-MDT0000: expected 968 actual 344. /scratch/logs/syslog/soak-8.log:Aug 3 22:14:11 soak-8 kernel: LustreError: 11-0: soaked-OST0016-osc-MDT0000: operation ost_destroy to node 192.168.1.107@o2ib failed: rc = -107 /scratch/logs/syslog/soak-8.log:Aug 3 22:14:33 soak-8 kernel: LustreError: 11-0: soaked-OST0004-osc-MDT0000: operation ost_create to node 192.168.1.107@o2ib failed: rc = -107 /scratch/logs/syslog/soak-8.log:Aug 3 22:14:33 soak-8 kernel: LustreError: 4596:0:(osp_precreate.c:619:osp_precreate_send()) soaked-OST0004-osc-MDT0000: can't precreate: rc = -107 /scratch/logs/syslog/soak-8.log:Aug 3 22:14:40 soak-8 kernel: LustreError: 11-0: soaked-OST000a-osc-MDT0000: operation ost_create to node 192.168.1.107@o2ib failed: rc = -107 /scratch/logs/syslog/soak-8.log:Aug 3 22:14:40 soak-8 kernel: LustreError: 4609:0:(osp_precreate.c:619:osp_precreate_send()) soaked-OST000a-osc-MDT0000: can't precreate: rc = -107 /scratch/logs/syslog/soak-8.log:Aug 3 22:14:54 soak-8 kernel: LustreError: 11-0: soaked-OST0010-osc-MDT0000: operation ost_statfs to node 192.168.1.107@o2ib failed: rc = -107 /scratch/logs/syslog/soak-8.log:Aug 3 22:22:14 soak-8 kernel: LustreError: 4692:0:(mdt_lvb.c:163:mdt_lvbo_fill()) Skipped 23 previous similar messages /scratch/logs/syslog/soak-8.log:Aug 3 22:22:14 soak-8 kernel: LustreError: 4692:0:(mdt_lvb.c:163:mdt_lvbo_fill()) soaked-MDT0000: expected 416 actual 344. /scratch/logs/syslog/soak-8.log:Aug 3 22:22:15 soak-8 kernel: LustreError: 4853:0:(mdt_lvb.c:163:mdt_lvbo_fill()) Skipped 25 previous similar messages /scratch/logs/syslog/soak-8.log:Aug 3 22:22:15 soak-8 kernel: LustreError: 4853:0:(mdt_lvb.c:163:mdt_lvbo_fill()) soaked-MDT0000: expected 968 actual 416. /scratch/logs/syslog/soak-8.log:Aug 3 22:22:28 soak-8 kernel: LustreError: 4769:0:(mdt_lvb.c:163:mdt_lvbo_fill()) soaked-MDT0000: expected 968 actual 344. /scratch/logs/syslog/soak-8.log:Aug 3 22:25:20 soak-8 kernel: LustreError: 5065:0:(osp_object.c:582:osp_attr_get()) soaked-OST000b-osc-MDT0000:osp_attr_get update error [0x1000b0000:0x111cd18:0x0]: rc = -4 </pre> </div></div> <p>Servers are not dying, but this is new with the patch, so possibly of interest. </p> Soak cluster - testing <a href="https://jira.whamcloud.com/browse/LU-7899" title="osd_xattr_set() to batch actual EA update" class="issue-link" data-issue-key="LU-7899"><strike>LU-7899</strike></a> patch LU-9825

Multiple errors on OST/MDS

Bug Major Open Unresolved Lai Siyao Cliff White Thu, 3 Aug 2017 22:55:46 +0000 Mon, 2 Oct 2017 17:42:57 +0000 Lustre 2.10.1 0 5 <p>I can reproduce locally messages like the following:<br/> /scratch/logs/syslog/soak-5.log:Aug 3 22:17:08 soak-5 kernel: LustreError: 24771:0:(osd_index.c:224:__osd_xattr_load_by_oid()) Skipped 13 previous similar messages<br/> this is OI scrubber meeting objects being destroyed. <br/> in my local testing this don't lead to any test issues.</p> <p>The unexpected errors on the MDS are confusing for users, especially errors that appear during normal usage and do not indicate any kind of problem. </p> <p>Lai, the "mdt_lvbo_fill() expected N got M" messages are from PFL and should be quieted (2.11 and 2.10.1) since they are expected for composite files. </p> <p>Nasf, _osd_xattr_load_by_oid() are from LFSCK and should be investigated. I don't know if they indicate a problem or not. </p> <p>There are two callers in current osd-zfs for _osd_xattr_load_by_oid(): one is osd_zfs_otable_it_next(), the other is osd_get_fid_by_oid(). The former one is LFSCK related. I am not sure which one triggered the failure in test. But since LFSCK was triggered during the test, it is possible that the LFSCK called the _osd_xattr_load_by_oid() and caused the failure message. I checked the ZFS code, one suspected point is that:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>int dnode_hold_impl(objset_t *os, uint64_t object, int flag, int slots, void *tag, dnode_t **dnp) { ... mutex_enter(&amp;dn-&gt;dn_mtx); type = dn-&gt;dn_type; if (dn-&gt;dn_free_txg || ((flag &amp; DNODE_MUST_BE_FREE) &amp;&amp; !refcount_is_zero(&amp;dn-&gt;dn_holds))) { mutex_exit(&amp;dn-&gt;dn_mtx); zrl_remove(&amp;dnh-&gt;dnh_zrlock); dbuf_rele(db, FTAG); return (type == DMU_OT_NONE ? ENOENT : EEXIST); } ... } </pre> </div></div> <p>It seems that the object (that we are loading xattr) is removed. From current osd-zfs implementation, we do NOT hold reference on the object, instead, we hope to load the xattr via the oid directly. So that is normal race case.</p> <p>On the other hand, our zfs otable based iteration osd_zfs_otable_it_next() will ignore __osd_xattr_load_by_oid() failure, means that if it is the LFSCK triggered the __osd_xattr_load_by_oid() failure message, it is just some warning information, will NOT affect LFSCK running.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>static int osd_zfs_otable_it_next(const struct lu_env *env, struct dt_it *di) { ... rc = __osd_xattr_load_by_oid(dev, it-&gt;mit_pos, &amp;nvbuf); if (unlikely(rc != 0)) continue; ... } </pre> </div></div> <p>there is no point to hold a reference to object in this case, especially given this would be the same call to dmu_bonus_hold()..<br/> I agree we should avoid error messages in -ENOENT case at least - it's an object <em>being</em> removed.</p> Related LU-10055 Development Rank 1|hzzhqf: Rank (Obsolete) 9223372036854775807 Severity