https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-11422 Lustre <p>LNet Selftest post LNet Health landing loses backward compatibility, which means lnet-selftest cannot be run between cross-version peers (Lustre 2.12 and pre Lustre 2.12). We should fix that.</p> <p>In LNet Health feature, new health related stats have been added which changes the <tt>struct lnet_counters</tt> that we previously had (patch <a href="https://review.whamcloud.com/32949" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/32949</a> "<tt><a href="https://jira.whamcloud.com/browse/LU-9120" title="LNet Network Health Feature" class="issue-link" data-issue-key="LU-9120"><del>LU-9120</del></a> lnet: add global health statistics</tt>"). Due to this, <tt>struct srpc_stat_reply</tt> is changed as it looks like this -</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> struct srpc_stat_reply {         __u32                   str_status;         struct lst_sid          str_sid;         struct sfw_counters     str_fw;         struct srpc_counters    str_rpc;         struct lnet_counters    str_lnet; } WIRE_ATTR; struct lnet_counters { __u32 msgs_alloc; __u32 msgs_max; + __u32 rst_alloc; __u32 errors; __u32 send_count; __u32 recv_count; __u32 route_count; __u32 drop_count; + __u32 resend_count; + __u32 response_timeout_count; + __u32 local_interrupt_count; + __u32 local_dropped_count; + __u32 local_aborted_count; + __u32 local_no_route_count; + __u32 local_timeout_count; + __u32 local_error_count; + __u32 remote_dropped_count; + __u32 remote_error_count; + __u32 remote_timeout_count; + __u32 network_timeout_count; __u64 send_length; __u64 recv_length; __u64 route_length; __u64 drop_length; } WIRE_ATTR; </pre> </div></div> <p> </p> <p>Amir's idea - <br/> "What we can do is make a copy of the structure which is similar to the older one. And in the post health selftest we can have a translation function which takes the new structure and copies the relevant fields to the old one. This way selftest remains backwards compatible"</p> LU-11422

Make LNet Selftest post Health backward compatible

Bug Blocker Resolved Fixed Sonia Sharma Sonia Sharma Mon, 24 Sep 2018 19:28:17 +0000 Wed, 10 Oct 2018 02:21:29 +0000 Wed, 10 Oct 2018 02:21:29 +0000 Lustre 2.12.0 0 4 <p>To understand this issue better, it would be useful to firstly describe which structure is affected, whether this compatibility is between userspace and the kernel, or between peers over the network. Please link this ticket to the ticket/patch that introduced the compatibility issue. That would make it more clear between which versions the incompatibility exists, and how much effort is needed to maintain compatibility. </p> <p>Also, please fill in the Affects Version and Fix Version fields. If this incompatibility is introduced in current master, and we don't land the fix for this issue before 2.12 is released, does that mean cross-version LST usage is broken? That would make this issue a must-fix for 2.12 (ie. Critical or Blocker)</p> <p>Updated</p> <p>Having looked at this change, I'd say the current implementation is quite problematic.  It will require field-by-field data copying, and there isn't an easy way for older tools/clients to handle this easily.  Conversely, if all of the new fields are added to the end of the data structure, then it would be trivial for old clients to ignore the remaining fields, and new clients to just assume the new fields are zero of they are not present in the old struct. Is there any reason these fields were added in the middle of the struct, even being added into two disjoint places in the struct?</p> <p>I see in this patch that <tt>lnet_selftest_structure_assertion()</tt> had the code commented out that was telling you that this change should not be done in this manner. The whole point of these assertions is to warn the developer that they are making a change that will break the userspace API or the network protocol, or both. That kind of change might be temporarily OK during testing, but should basically never be landed.</p> <p>It appears that <tt>struct srpc_msg</tt> already has a mechanism to handle changes in a better manner - it has both a <tt>msg_magic</tt> and a <tt>msg_version</tt> field in the structure, that could be changed to indicate the presence of additional fields in the struct. New clients/tools would understand both the old/new version of the struct, and be able to (mostly) ignore the differences, except when accessing the <tt>stat_reply</tt> member. In the latter case, the new client would just avoid to access the added fields when the <tt>msg_version = 0x3</tt><span class="error">&#91;*&#93;</span>, and when communicating with an old client/tool it just copies the smaller struct and drops the rest of the stats.</p> <p><span class="error">&#91;*&#93;</span> NB - for the <tt>msg_version</tt> field, it is <b>much</b> better to make this a bitmap of features rather than a strict enumeration of versions. Having 2^32 versions is not useful as you will never have so many different features, and it is not clear to anyone what the difference between "v4" and "v7" of a struct/protocol would be. Conversely, having a <tt>SRPC_MSG_FEAT_LOCAL_COUNTER = 0x0002</tt> (or similar) feature flag is easy to check, and it makes it clear to any reader what feature is be present in the struct. If this flag is not set (for unrelated messages), or there is no need to send the extra stats (for older clients/tools), then there is no reason to break compatibility. If some other flag is set that the tool doesn't understand, then either it can return an error, or (better) parse the parts of the struct it understands, and ignore the rest.</p> <p>The best mechanism is if there are "compat" and "incompat" regions of the version (e.g. <tt>__u16 compat; __u16 incompat</tt>) and where a particular feature flag goes depends on what kind of changes are being made. Just having additional fields added to the end of the struct is typically a harmless "compat" change and is preferable. Changing the meaning of the fields (as was done in the original patch) would be an "incompat" change and should be avoided if at all possible (as it likely can be in this case).</p> <p>I also now see that there is also a <tt>msg_ses_feats</tt> field that appears to allow peers to negotiate which mutually compatible features they support, but I'm not sure if this would allow anything other than returning an error if an unknown feature is advertised. For Lustre RPC connections, the client sends a bitmask of features that it supports, and the server masks out any feature bits that it doesn't understand (using its own "feature supported" bitmask) and returns it to the client rather than returning an error. This feature mask mechanism has served us very well for about 15 years, and it was inherited from ext2/3/4 which had another 10 years of use for managing on-disk vs. user tools vs. kernel compatibility. It would be much better if this were changed to have a permissive connection that accepted any features in the request, but masked off the unknown ones and returned it the the client. Otherwise, we just have a gratuitous "request with all known features, get an error reply with supported features, retry with supported features, continue as if it had done it right the first time" dance.</p> <p>As for fixing this particular problem, I see some possible solutions, depending on how this is handled in the code:</p> <ul class="alternate" type="square"> <li>if the addition of the extra fields to <tt>lnet_counters</tt> makes the message size grow (i.e. it is (now) the largest member of <tt>msg_body</tt>) then this might need additional handling to maintain compatibility <ul class="alternate" type="square"> <li>if the tools/peer return an error for larger <tt>srpc_msg</tt> then that should be fixed in its own right <ul class="alternate" type="square"> <li>allowing a message size &gt;= the current <tt>sizeof(srpc_msg)</tt>, at least to avoid such issues in the future</li> <li>split the stats message into "old_stats" and "local_stats" and send them in two RPCs, assuming they are not constantly used)</li> </ul> </li> </ul> </li> <li>if this is accepted, then these flags should only be set in stats messages when they are needed</li> <li>if it is just a matter of determining whether the extra stat fields are present, then the <tt>srpc_msg</tt> version == feature field may be enough</li> </ul> <p>My main problem with lnet_selftest's use of the lnet_counters structure is that it creates a coupling where none should exist. We're taking an LNet internal structure and creating a wire protocol dependency. Now when we want to add extra statistics, then we have to jump through unnecessary hoops. IMO, there shouldn't have been this coupling between a test tool and LNet. That's why I'm proposing we break up this dependency in the first place. selftest wire protocol shouldn't be dependent on lnet's internal structure.</p> <p>I think it makes more sense to me to separate selftest from LNet by the use of APIs. selftest shouldn't be accessing LNet structures directly. There should be an API that pulls whatever information selftest needs. And then selftest can store this data in whatever form it needs.</p> <p>Sonia Sharma (sharmaso@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/33242" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33242</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11422" title="Make LNet Selftest post Health backward compatible" class="issue-link" data-issue-key="LU-11422"><del>LU-11422</del></a> lnet: Fix selftest backward compatibility post health<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: a1dae49acc92b48b1b3c463065a8d8ed0f30b155</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/33242/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33242/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11422" title="Make LNet Selftest post Health backward compatible" class="issue-link" data-issue-key="LU-11422"><del>LU-11422</del></a> lnet: Fix selftest backward compatibility post health<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 60f6f2b480b482f2022cbea416d8bea87f848bec</p> <p>Landed for 2.12</p> Related LU-9120 Development Rank 1|i002zj: Rank (Obsolete) 9223372036854775807 Severity