Details
-
Technical task
-
Resolution: Fixed
-
Critical
-
None
-
9223372036854775807
Description
Right now lustre will fail to compile if one attempts to build from a source tree pulled from git, but not actually in a git working directory. The error will contain the string:
"I have no idea how to create a META file"
I can only guess at the reasons this was put in place originally, and none of the reasons that I am coming up with are particularly convincing.
There are, on the other hand, pretty strong counter examples for why this restriction should not exist. For instance, an automated build-and-test farm might reasonably want to use a command like the following to retrieve a tree for a specific commit:
git archive --format=tar.gz --remote=$URL $GIT_TREEISH
As it stands, that tar ball will not allow lustre to be built.
This is a long standing problem, and we really need to get it fixed.
I have already hacked the code to stop the problem for LLNL's local 2.5-llnl branch, but I think a little more extensive overhaul is needed to make things clean. I think it is pretty reasonable for me to get this done in time for Lustre 2.9.
Attachments
Issue Links
- is blocked by
-
LU-7699
Overhaul lustre's versioning
-
- Closed
-
Activity
morrone Yes, that makes sense. Care to submit your patch for master? Code talks louder than words. 
Prior to all of that, as you will recall we used to just have the version/build information in a file that somebody had to update, but it often got overlooked when doing version updates/releases, etc., so we automated getting that from git.
I don't think that is quite accurate. We still need to make commits to update the version information. It would be quite bad to fail to do so even now. It is mostly the release field that is effected by all of this.
And frankly, this use of the release field is a complete violation of what the release field is supposed to be for in rpm packaging. Anyone using a build system like koji (what fedora and related distros use), the release field must be reserved for packaging purposes. Lustre trying to reuse it for its own means causes a lot of complications. That just isn't what the Release field is for.
We are using the Release field for Versioning, of Lustre, whereas the Release field should only be used for packing purposes. Granted at the moment the concepts or building and packaging are pretty confused and incestuously interwined in Lustre, but I am working hard to fix that.
And further: There is minimal (no?) cost to you to allowing other people in the world to build without a working git repo. It is about as close to a free lunch as one can get. Other versioning issues will probably require tradeoffs, I agree, but I'm not going to go into those in this ticket.
rread Sure, we could throw any kinds of "placeholder" (i.e. like "custom") values into the missing metadata but the point of getting and using that metadata was to convey accurate information about exactly where the code came from.
Prior to all of that, as you will recall we used to just have the version/build information in a file that somebody had to update, but it often got overlooked when doing version updates/releases, etc., so we automated getting that from git. It also never conveyed "between release" information as well as a tag+fwd commits, or even a git hash could.
So yes, it does complicate and restrict, but there's no free lunch.
Robert: I don't think so, I think something like that should be an option. If you don't have the git working directory, you don't know what tag it came from. The code should just have the version included. It avoids soooooo many problems.
Now, I'm not at all saying that we can't also have Lustre add version info if it is in a git working directory. That part can certainly remain.
Right, and once I remove that restriction, we can support some completely reasonable build models that the current code prevents. Such as the one I listed in the description.
None of that git info is really required, and it is time to finally fix this well intentioned, but overly restrictive regulation.
I think we should probably also revisit how we do versioning in Lustre too. I was part of creating --downstream-release, but now I think that has outlived its usefullness, especially since the core Intel team has taken most of their branches and versions private. It probably time to rethink that and make it alot simpler. For instance, we probably just need to accept that it takes a commit to change the version string. I think that is reasonable. But I intend to open a separate ticket on that at some point. This one should be restricted to just allowing builds without a git working directory.
In the absence of the META file could we use a default for the build tag like "custom" or "$BUILDTAG"?
It is done to take advantage of the metadata that is in git, like the tag (plus how ever many commits forward of the last tag made) where the code was taken from.
The META file is supposed to be created before you remove the code from git, and should be done when you do make dist. The META file is used to store the git metadata and supply it to the next configure run (i.e. outside of git).
make dist is the supported way of exporting a copy of the code outside of git.
Absolutely! That is why I assigned this ticket to myself; I plan to do this work. This and other tickets will help me track what I need to get done, and also help begin socializing my vision for the build system in the 2.9 time frame.
And FYI, I opened
LU-7645to start planning what needs to change about our use of the RPM Release field.