https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-7699 Lustre <p>There are numerous oddities about Lustre's system for applying version numbers in the configuration and packaging system. At this point, it is looking strongly like an overhaul is in order.</p> LU-7699

Overhaul lustre's versioning

Technical task LU-3953 Critical Closed Done Christopher Morrone Christopher Morrone Fri, 22 Jan 2016 20:06:11 +0000 Wed, 6 Nov 2019 06:02:40 +0000 Tue, 3 May 2016 17:45:15 +0000 Lustre 2.9.0 0 13 <p>Why do we have both lustre_ver.h and lustre_build_version.h? So silly.</p> <p>Christopher J. Morrone (morrone2@llnl.gov) uploaded a new patch: <a href="http://review.whamcloud.com/18107" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/18107</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7699" title="Overhaul lustre&#39;s versioning" class="issue-link" data-issue-key="LU-7699"><del>LU-7699</del></a> build: Replace version_tag.pl with LUSTRE-VERSION-GEN<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: e0a4a49958df15608615533c1e6f67bf3aca9f26</p> <p>I have a first rough version of the first overhaul patch. It is a net <em>reduction</em> in almost 300 lines, so I must be doing something right. <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>I think the patch is already decent, but I haven't really considered how lbuild is going to react to the changes, or looked at how the rpm or dpkg packaging systems are going to handle it. It may need further work to get along with all of that. But it is worth running this version through the build farm as a fortestonly patch to see what blows up first.</p> <p>There is quite a bit of additional cleanup that can be done, but I want to try to avoid putting too much change in one patch. I'll add additional cleanup in a follow-on patch.</p> <p>Some things that I'll look into (just so I don't forget):</p> <ul> <li>Get rid of lustre_build_version.h</li> <li>Fix how Lustre embeds the version into its binaries</li> <li>Further cleanup autoconf variables and m4 macros pertaining to versioning that are no longer necessary if we use standard ones like {AC_} <p>PACKAGE_VERSION</p></li> <li>git grep for "META" and deal with its obsolescence</li> </ul> <p>Two more revisions to fix easy problems. The first major issue is now what I expected: I need to teach lbuild about the new versioning scheme.</p> <p>It also looks like the ppc64 builder isn't happy either. I'm not sure that it provides enough information to do anything about it though...maybe I can make a good enough guess to get it working.</p> <p>But that will all have to wait until next week.</p> <p>Christopher J. Morrone (morrone2@llnl.gov) uploaded a new patch: <a href="http://review.whamcloud.com/18108" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/18108</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7699" title="Overhaul lustre&#39;s versioning" class="issue-link" data-issue-key="LU-7699"><del>LU-7699</del></a> build: Eliminate lustre_build_version.h<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 49e8ccfea6d6845ca13e9a2dd8d0a5635c9bc8c0</p> <p>Christopher J. Morrone (morrone2@llnl.gov) uploaded a new patch: <a href="http://review.whamcloud.com/18110" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/18110</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7699" title="Overhaul lustre&#39;s versioning" class="issue-link" data-issue-key="LU-7699"><del>LU-7699</del></a> build: Replace LUSTRE_VERSION_STRING with PACKAGE_VERSION<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 4ace235aef94e2182bc62a28e5cdb479ac3c2fcb</p> <p>Christopher J. Morrone (morrone2@llnl.gov) uploaded a new patch: <a href="http://review.whamcloud.com/18111" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/18111</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7699" title="Overhaul lustre&#39;s versioning" class="issue-link" data-issue-key="LU-7699"><del>LU-7699</del></a> build: Remove unnecessary AC_LUSTRE_</p> {MAJOR,MINOR,PATCH,FIX} <p>Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: b28a363a54d2b45a8f64b6ec81f0d0abedc220f7</p> <p>Christopher J. Morrone (morrone2@llnl.gov) uploaded a new patch: <a href="http://review.whamcloud.com/18112" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/18112</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7699" title="Overhaul lustre&#39;s versioning" class="issue-link" data-issue-key="LU-7699"><del>LU-7699</del></a> build: debug intel buildfarm<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 073eb66d36756f2f498d4c9aceb982a25c97e364</p> <p>OK, I think I'm ready for the first round of reviews on the current four patches:</p> <ul> <li><a href="http://review.whamcloud.com/18107" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/18107</a></li> <li><a href="http://review.whamcloud.com/18108" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/18108</a></li> <li><a href="http://review.whamcloud.com/18110" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/18110</a></li> <li><a href="http://review.whamcloud.com/18111" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/18111</a></li> </ul> <p>It is now just three patches:</p> <ul> <li><a href="http://review.whamcloud.com/18107" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/18107</a></li> <li><a href="http://review.whamcloud.com/18108" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/18108</a></li> <li><a href="http://review.whamcloud.com/18111" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/18111</a></li> </ul> <p>In a <a href="http://review.whamcloud.com/18107" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/18107</a> review, Oleg pointed out a potential issue with lack of a special-case for RC releases. I admit that I totally missed how RC releases are currently being done. To make a long story short, I am going to argue that we really shouldn't do it that way any more.</p> <p>Correct me if I am wrong, but as I now understand it, even though a tag might have a name of "2.8.0-RC1", the build system is currently stripping off the "-RC1" and building the code as if it is a final "2.8.0" release. As Oleg explained it, the intention here has been that when an RC is tested and agreed to be the final release, the packages from the RC build are already named with the final release version and can be released as-is. The advantage here is that we don't need to trust our build procedures to be consistent, we can simply release the packages from the most recent RC.</p> <p>The down side should be pretty clear though too: RC releases are things that we explicitly want to release and have the general community test and vet. So takeing 2.8.0 as an example, which to date has five RC releases (-RC1 through -RC5), there are exactly five <em>different</em> versions of the code now out there in the wild with no way to tell them apart.</p> <p>I would argue that this downside of having multiple different codes out in the wild with exactly the same version vastly out weighs the possibility that we might not compile and generate packages the same way twice.</p> <p>So I would argue that not special-casing RC releases is a feature of change 18017, not a defect. We will need to come up with a new way of versioning RC releases. I would suggest that we use the obvious method already established (but not actually fully employeed):</p> <p>Release candidates will use the previous release's number, but have the thrid number start at 90. For example, release candidates for the 2.8.0 release would have version numbers like: 2.7.90, 2.7.91, 2.7.92, 2.7.93, etc.</p> <p>This also has the advantage that it works well with packaging systems like RPM. Many rpm-based distros explicitly advise against using substrings like "RC1" in version strings, because rpm just doesn't handle them, and assiciated package upgrades in any particularly intelligent way. Granted, we somewhat avoided that problem by stripping off the RC altogether.</p> <p>But I think it is very bad form to have multiple different versions of the code to have the same version string.</p> <p>2.7.90, 2.7.91, ... are already used. they mean code drops in code freeze - i.e. when only blockers remain.</p> <p>RCs (release candidates) on the other hand are generated when there are no more blockers left (At the time of tagging, anyway). I guess it's possible to continue the versioning as is, but we might run our of digits quite soon in some cases.</p> <p>The "five versions of code" issue is real, I guess, but on the other hand it could be solved with other means. E.g. once we transition to signed packages, the real releases might be signed with a different key than the unsigned (or signed with a "build bot" only) key. Also we don't widely distribute these RCs and so nobody could get them. We can also destroy interim RCs from download site if needed (or it happens automatically over a relatively short time). And we can tell them apart by a build date too.<br/> The actual drawbacks I encountered are: The changelog could only list an estimated "release date" since who knows how long the testing and all the approvals would take.</p> <p>The "we don't need to trust our build procedures to be consistent" is a real risk on the other hand on many levels. There might have been a system update between the latest RC and the release that updated some library breaking something in process (updating gcc, whatever), a glitch during building for other reasons producing a dud (ram error, disk error), somebody might have compromised the build server meanwhile (or inserted some code into the git tree), ...<br/> All in all what it really means is we would really need to test the final release just like an another RC just to be sure it's still viable, and if it's not then what?</p> <p>In short, there's certain appeal to release code that was actually tested vs code that was only built even when it's supposedly the same as the one that was tested because it was supposedly built from the same source.</p> <blockquote><p>2.7.90, 2.7.91, ... are already used</p></blockquote> <p>OK, fine, then they could start at .120, or .500, or whatever. I don't care so much about the details as long as we follow some simple rules that make the versioning work well for the various packaging systems like RPM. For instance, each release (including release candidate releases) should have a unique version number. The version number should be mostly numerical, and atomically increasing. Letters should be used sparingly, and preferably not at all, in the main part of the version string. (Where main part is the first four period-separated numbers. We allow for alphanumeric strings for per-organization branding after that)</p> <blockquote><p>The "five versions of code" issue is real, I guess, but on the other hand it could be solved with other means. E.g. once we transition to signed packages, the real releases might be signed with a different key than the unsigned (or signed with a "build bot" only) key.</p></blockquote> <p>With all due respect, I do no think that will be an acceptable solution. The version string is specifically designed to make it easy to tell different builds and packages apart. Unique version numbers are something packaging systems like rpm, and dpkg understand. The relationship between different keys and how they effect install precedence, is not. You say that there are other reasonable methods, but I do not really think we are likely to find one. The version string is very explicitly the universal solution for this problem. We are introducing larger problems by trying to avoid that universal solution.</p> <blockquote><p>Also we don't widely distribute these RCs and so nobody could get them.</p></blockquote> <p>Right, and no doubt that is how this has flown under the radar for so long. Because this part of your development phase is kept pretty secret.</p> <p>But this is pretty much the opposite of how a good open source development project, with an open development process, should work. If we had a more open process with good communication, then every RC release would have an announcement on mailing lists letting people know where to get the packages and asking for help in testing. This announcement shouldn't just go to lustre-devel, it should also got to lustre-discuss. We <em>want</em> adventurous users/sysadmins that can carve out some time to help with testing to do so. They should not have to go to git to do that, they should be able to download packages.</p> <p>If we make those packages reasonably well known and easy to access (which we should definitely do for 2.9.0 RCs and into the future), then it will not be at all acceptable to have them all use exactly the same version number. They have to be uniquely and clearly versioned.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>The "we don't need to trust our build procedures to be consistent" is a real risk on the other hand on many levels. There might have been a system update between the latest RC and the release that updated some library breaking something in process (updating gcc, whatever)</pre> </div></div> <p>I agree, that is a more reasonable concern. But it is one that is pretty much entirely manageable with a small amount of process and good communication. When RC releases are about to start, we send an announcment out to the build farm admins saying "RCs are about to begin. Please freeze all changes to builders for branch X until further notice". If you are really worried, you call them on the phone and get a verbal confirmation that they understand. Many, many, many software projects are able to handle that level of release management in the real world. I believe that we can on Lustre as well. I'm putting my money where my mouth is too, by spending LLNL manpower on setting up a new buildfarm for Lustre. We'll present the prototype at LUG this year.</p> <p>Furthermore, there are those that would argue that if Lustre can't compile against multiple different libraries with the same ABI, then Lustre is just broken. Remember that Lustre does not own the entire OS. It really needs to work correctly anywhere the packages install without being forced.</p> <p>But I think keeping the builders static during the RC phase is totally reasonable.</p> <blockquote><p>a glitch during building for other reasons producing a dud (ram error, disk error),</p></blockquote> <p>Thats a one-in-a-billion type of problem. I don't think that in even remotely as much of a problem as releasing several packages with different contents, but versioning them identically.</p> <blockquote><p>somebody might have compromised the build server meanwhile (or inserted some code into the git tree), ...</p></blockquote> <p>That isn't a problem specific to release condidates, so I don't think it is relevant to the conversation.</p> <blockquote><p>All in all what it really means is we would really need to test the final release just like an another RC just to be sure it's still viable, and if it's not then what?</p></blockquote> <p>I actually think it is fairly reasonable to build the same commit with no other change but the version string and not do any additional testing. But if people want to do more testing, that is certainly fine with me. I would argue that 8-24 hours of automated-only testing would be more than sufficient as a sanity check that nothing went horribly wrong between the two tags (of the exact same commit).</p> <blockquote><p>In short, there's certain appeal to release code that was actually tested vs code that was only built even when it's supposedly the same as the one that was tested because it was supposedly built from the same source.</p></blockquote> <p>Yes, I understand the appeal and your points. But I don't think they are strong enough to justify the offsetting bad packing practice that you have introduced.</p> <p>We need to stop having multiple packages of different contents with the same exact version. That is simply not acceptable.</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/18107/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/18107/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7699" title="Overhaul lustre&#39;s versioning" class="issue-link" data-issue-key="LU-7699"><del>LU-7699</del></a> build: Replace version_tag.pl with LUSTRE-VERSION-GEN<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: ee813dbaa2a2b86f4873c4c289f62a0243aa9809</p> <p>I tried a spin with the latest lustre on Ubuntu 15.04 and I get this error:</p> <p>dpkg-buildpackage: warning: debian/changelog(l1): version '2.8.51_17_g5bd8f72-1' is invalid: version number contains illegal character '_'<br/> LINE: lustre (2.8.51_17_g5bd8f72-1) unstable; urgency=low<br/> dpkg-buildpackage: source package lustre<br/> dpkg-buildpackage: source version 2.8.51_17_g5bd8f72-1<br/> dpkg-buildpackage: error: version number contains illegal character '_'<br/> autoMakefile:1136: recipe for target 'debs' failed<br/> make: *** <span class="error">&#91;debs&#93;</span> Error 25</p> <p>So I did some digging and I found that '_' is actually an invalid character for fedora as well. See this link:</p> <p><a href="https://fedoraproject.org/wiki/Packaging:NamingGuidelines#Separators" class="external-link" target="_blank" rel="nofollow noopener">https://fedoraproject.org/wiki/Packaging:NamingGuidelines#Separators</a></p> <p>So basically we will need to fix the lustre version not to contain any '_' for proper packaging on rpm and deb systems.</p> <blockquote><p>So I did some digging and I found that '_' is actually an invalid character for fedora as well. See this link:</p></blockquote> <p>Actually, that page says the following:</p> <blockquote><p>When naming packages for Fedora, the maintainer must use the dash '-' as the delimiter for name parts. The maintainer must NOT use an underscore '_', a plus '+', or a period '.' as a delimiter.</p></blockquote> <p>We are not using underscores in the <em>name parts</em>, i.e. the Name field. Underscores are only used in the Version.</p> <p>But yes, it looks like we'll need to work something out for dpkg systems.</p> <p>The main reason that we use underscores in the version is because Lustre currently has a variable number of version fields, which makes it difficult to tell the delineation between the upstream version and the following part of the version, whether third-party or development (git describe) information.</p> <p>We can't use a dash (&#45;) in the version string on rpm systems. That pretty much leaves a period as the only other sane option if we decide to stop using underscores.</p> <p>But maybe we stick with underscores on rpm systems, and munge the string in some acceptable way for dpkg?</p> <p>I discovered the <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/forbidden.png" height="16" width="16" align="absmiddle" alt="" border="0"/> is needed on rpms systems for the version field the hard way today. For dpkgs both Name and Version fields can not have '_'. I need to rethink this. I did see many sites stating you should avoid '-' at all cost for rpms <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/> Also I found when you have a top level .git directory if you remove the LUSTRE-VERSION-FILE it will not regenerate and it causes make rpms to fail.</p> <blockquote><p>Also I found when you have a top level .git directory if you remove the LUSTRE-VERSION-FILE it will not regenerate and it causes make rpms to fail.</p></blockquote> <p>Yeah, that is true. For now, you will just need to rerun autogen.sh to make any version changes. This could be automated in the future, but we pretty much need to fix autoreconf (<a href="https://jira.whamcloud.com/browse/LU-7700" title="autoreconf does not work" class="issue-link" data-issue-key="LU-7700">LU-7700</a>) to do it sanely.</p> <p>I tracked down what needs to be changed for make debs. We have:</p> <p>lversion=$$(sed -ne 's/^#define VERSION "&#40;.*&#41;"$$/\1/p' config.h);</p> <p>The sed command needs to be modified to change '_' to '-'. I'm no sed master so Chris what needs to be done to make that so. </p> <p>Thanks for the legwork on that, James!</p> <p>We don't even need sed any more; I'll just use tr.</p> <p>Christopher J. Morrone (morrone2@llnl.gov) uploaded a new patch: <a href="http://review.whamcloud.com/19488" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/19488</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7699" title="Overhaul lustre&#39;s versioning" class="issue-link" data-issue-key="LU-7699"><del>LU-7699</del></a> build: Convert version underscores to dashes for dpkg<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 368150054a379f1da81dce8b2e5016016f126655</p> <p>Yeah more simplification. Debian platforms are back in business. Thanks Chris.</p> <p>Already mentioned this to Chris, but this may have broken SUSE builds on the Lustre Buildbot. The error I'm seeing on builds is "error: line 86: Empty tag: Release:". Example log is <a href="http://build.lustre.org/builders/SLES%2012.1%20x86_64%20%28BUILD%29/builds/878/steps/shell_2/logs/stdio" class="external-link" target="_blank" rel="nofollow noopener">here</a>.</p> <blockquote><p>Already mentioned this to Chris, but this may have broken SUSE builds on the Lustre Buildbot.</p></blockquote> <p>I don't think so. I think that this is a problem in the buildbot configuration. I will open a github ticket shortly.</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/18108/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/18108/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7699" title="Overhaul lustre&#39;s versioning" class="issue-link" data-issue-key="LU-7699"><del>LU-7699</del></a> build: Eliminate lustre_build_version.h<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 9c710162e5acebd860f1d3f0e1bf204ac1ba98c1</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/18111/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/18111/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7699" title="Overhaul lustre&#39;s versioning" class="issue-link" data-issue-key="LU-7699"><del>LU-7699</del></a> build: Convert lustre_ver.h.in into a static .h file<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 6e84dbce05959e854d886772841a5d6d0690eb65</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/19488/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/19488/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7699" title="Overhaul lustre&#39;s versioning" class="issue-link" data-issue-key="LU-7699"><del>LU-7699</del></a> build: Convert version underscores to dashes for dpkg<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: f2d28899a2c3b5a427f4322623ba138887fe6adc</p> <p>Chris is everything for this ticket landed. Can it be closed?</p> <p>Yes, I think now is a good time to close this one.</p> Blocker LU-7642 LU-7643 LU-7645 Related LU-7976 LU-474 Development Rank 1|hzxz5z: Rank (Obsolete) 9223372036854775807