https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-7643 Lustre <p>Right now when RPM packages are built, we insert into Lustre's release field the version string from the kernel against which Lustre was built. For instance:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>$ rpm -qpi lustre-2.7.0-2.6.32_504.8.1.el6_lustre.x86_64.x86_64.rpm Name : lustre Version : 2.7.0 Release : 2.6.32_504.8.1.el6_lustre.x86_64 </pre> </div></div> <p>Side note: A sysadmin is going to (and have in the past) think we messed up because of the ".x86_64.x86_64" in the file name, but the reason for it is that the first one is part of the Linux kernel version string, as we can see in the Release field above. The second .x86_64 is Lustre's.</p> <p>The reason for including the kernel's version string in Lustre's Release field because Lustre has traditionally been packaged to work with one, and only one, specific version of a kernel. If you have two very slightly different kernel versions "2.6.32_504.8.1.el6" and "2.6.32_504.8.2.el6", for instance, then you currently need to compile lustre against both kernels individually. While the "rpm -requires" should also list the specific required version number, because there are so many very closely compatible kernels for which we need to juggle lustre builds, it was simpler for sysadmins and developers alike to add the kernel's version string into Lustre's release field.</p> <p>But fortunately, this need to build lustre for every specific kernel is a self-imposed restriction, and work is under way to lift that restriction in <a href="https://jira.whamcloud.com/browse/LU-5614" title="use %kernel_module_package for weak-updates" class="issue-link" data-issue-key="LU-5614"><del>LU-5614</del></a>.</p> <p>For many years, it has been possible to compile kernel modules once and then use them with <em>any</em> kernel that is ABI compatible. The Linux distro mechanism that allows this is often called "weak modules". <a href="https://jira.whamcloud.com/browse/LU-5614" title="use %kernel_module_package for weak-updates" class="issue-link" data-issue-key="LU-5614"><del>LU-5614</del></a> should bring Lustre into the year 2006 and get it working with weak modules.</p> <p>Once that is done, we can finally drop the kernel version string.</p> <p>This is especially fortuitous for anyone using koji as a build system, because koji makes this sort of abuse of standard packaging practice pretty close to impossible. koji is used by fedora and its cousins, and it has also been adopted by LLNL for its RHEL-based TOSS distribution.</p> LU-7643

Remove kernel version string from Lustre release field

Technical task LU-3953 Critical Resolved Fixed Minh Diep Christopher Morrone llnl patch Fri, 8 Jan 2016 21:44:43 +0000 Tue, 6 Jun 2017 15:38:34 +0000 Tue, 6 Dec 2016 14:45:00 +0000 Lustre 2.9.0 0 13 <p>This will be an important step to support patchless servers. </p> <p>I have the patch for this waiting in the wings. I'm not going to bother pushing it to gerrit yet because it will need frequent refreshes until the <a href="https://jira.whamcloud.com/browse/LU-5614" title="use %kernel_module_package for weak-updates" class="issue-link" data-issue-key="LU-5614"><del>LU-5614</del></a> patch lands.</p> <p>Christopher J. Morrone (morrone2@llnl.gov) uploaded a new patch: <a href="http://review.whamcloud.com/19954" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/19954</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7643" title="Remove kernel version string from Lustre release field" class="issue-link" data-issue-key="LU-7643"><del>LU-7643</del></a> build: Remove Linux version string from RPM release field<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: b14c0b91f0623f301ea00fc7b2a053912e93c42d</p> <p>I decided to push change <a href="http://review.whamcloud.com/19954" class="external-link" target="_blank" rel="nofollow noopener">19954</a> after all. I just won't assign reviewers until it looks like <a href="https://jira.whamcloud.com/browse/LU-5614" title="use %kernel_module_package for weak-updates" class="issue-link" data-issue-key="LU-5614"><del>LU-5614</del></a> change <a href="http://review.whamcloud.com/12063" class="external-link" target="_blank" rel="nofollow noopener">12063</a> is closer to landing.</p> <p>With <a href="https://jira.whamcloud.com/browse/LU-5614" title="use %kernel_module_package for weak-updates" class="issue-link" data-issue-key="LU-5614"><del>LU-5614</del></a> close to landing, I rebased this issues' patch. It is ready to go through the normal review process.</p> <p><a href="https://jira.whamcloud.com/browse/LU-5614" title="use %kernel_module_package for weak-updates" class="issue-link" data-issue-key="LU-5614"><del>LU-5614</del></a> is done. Patch <a href="http://review.whamcloud.com/19954" class="external-link" target="_blank" rel="nofollow noopener">19954</a> for this ticket was already based on that ticket's patch, so no rebase should be necessary. We just need to get the review process under way on this one.</p> <p>Chris,</p> <p>while it's fine on client rpm where we can move to different kernel version, it's difficult for lustre server rpm to move to different kernel version since it requires patched kernel. Without kernel version string in the name, it's not easy to find out the kernel it's built on.</p> <p>Thanks<br/> -Minh</p> <p>Lustre server rpms do not require a patched kernel.</p> <p>The only thing that needs a patched kernel at this point is ldiskfs testing. That should be fixed soon in <a href="https://jira.whamcloud.com/browse/LU-684" title="replace dev_rdonly kernel patch with dm-flakey" class="issue-link" data-issue-key="LU-684"><del>LU-684</del></a>. If for some reason that fails, we can take the approach suggested by James Simmons in <a href="https://jira.whamcloud.com/browse/LU-20" title="patchless server kernel" class="issue-link" data-issue-key="LU-20"><del>LU-20</del></a>.</p> <p>I strongly disagree. <a href="https://jira.whamcloud.com/browse/LU-684" title="replace dev_rdonly kernel patch with dm-flakey" class="issue-link" data-issue-key="LU-684"><del>LU-684</del></a> doesn't eliminate the need for a lustre build with ldiskfs to be tightly tied to a specific linux kernel version. As long as we build ldiskfs by patching a particular upstream ext4 on the fly during a lustre server build we are subject to variations in the upstream ext4 that aren't represented by just having a compatible advertised kernel ABI that is constant. upstream ext4 changes occur unpredictably and in an unscheduled way in RHEL and SLES kernel updates.</p> <p>Compatibility problems that don't change the ABI won't necessarily be addressed by freshly applying patches and recompiling either. We have had problems in the past where ext4 internal semantics changed without changing the API and without breaking ldiskfs patch application. If you care that much, and since those problems have actually hit, you should probably stop using the ldiskfs-as-patches approach altogether.</p> <p>At least with the ldiskfs module fully compiled in the past, we eliminate the problem of overlooking ext4 internal semantic changes for a single version of the packages. The ldiskfs module is going to be in a known good frozen state. It will only be when larger semantic changes happen between the larger OS and filesystems that a recompile will be needed. And hopefully those types of changes are as rare as the issues inherent to the ldiskfs-as-patches approach within a stable OS kernel release series.</p> <p>But if folks still insist on ldiskfs being tied to a single kernel, we can certainly do that and while also removing the kernel string from the lustre packages. The kernel string does <em>not</em> belong in the lustre package name. That is incorrect packaging and needs to stop.</p> <p>The proper way would be to add a Requires to only the osd-ldiskfs subpackage. Is that going to be required to land this patch? It will probably cause us some trouble, but I'm willing to compromise and try adding that.</p> <p>I think it would be an acceptable solution if only the osd-ldiskfs rpm has a Requires for the particular and specific kernel version it was built on. That would directly enforce and tie it to the upstream ext4 version source it was built from. This is only my opinion. I think we need buy in from all concerned. Would really like to see comment from Minh, Andreas, Dmitry, or other experts.</p> <p>Not sure how that is properly carried thru with the Provides exported by osd-ldiskfs or the Requires in other lustre rpms.</p> <blockquote><p>Not sure how that is properly carried thru with the Provides exported by osd-ldiskfs or the Requires in other lustre rpms.</p></blockquote> <p>I'm not entirely sure what you mean, but I'll take a stab at explaining. If the various other parts are working now, they will continue to work. The "lustre-osd" requirement is currently supplied by either the osd-zfs or osd-ldiskfs packages. One or both of them must be installed to install in order to install the main "lustre" package. With the proposed new specific kernel requirement in the osd-ldiskfs package, if the osd-zfs package is selected, everthing will install fine with any kernel that supplies the required versions of various symbols. If osd-ldiskfs is selected, it can only be installed if the correct kernel is installed.</p> <p>Of course, multiple kernels can be installed at the same time, so there is no reason that the admin needs to boot the required kernel. Only that it be installed. But that can already happen now with the packages that contain the kernel version string.</p> <p>If you think that too is too much of a problem, you are basically arguing that weak modules can't ever be used with lustre.</p> <p>Your outline makes sense to me, but I'm a little worried about something unexpected creeping in if people start trying to mix and match pieces from different builds. Up until now with all lustre rpms tied to a specific kernel they all had to come from the same build. I'm not at all arguing that it was correct or even good that they were all tied that way, but it did have the beneficial side effect of blocking mix &amp; match.</p> <p>With osd-ldiskfs exporting only a generic "lustre-osd" Provides and other lustre packages having only that as Requires, there is now the possibility of taking an osd-ldiskfs from one build and installing it with lustre rpms from a different build. While I don't know that this would cause problems I'm very much worried that it might.</p> <blockquote><p>With osd-ldiskfs exporting only a generic "lustre-osd" Provides and other lustre packages having only that as Requires, there is now the possibility of taking an osd-ldiskfs from one build and installing it with lustre rpms from a different build.</p></blockquote> <p>If that issue exists it is independent of the topic in this ticket. You can open a new ticket for that if you like.</p> <p>It isn't independent. Without the changes proposed here it couldn't happen. With the changes proposed here it can happen. That makes it dependent on exactly this topic.</p> <p>I may agree that it could be covered as a separate ticket from this one in spite of that. Let me think about it a bit.</p> <p>I see the discuss is moving in the direction of ldiskfs &lt;<del>&gt; osd_ldiskfs &lt;</del>&gt; lustre having strong dependencies on each other which is needed. This is orthogonal to the dependency on the external linux kernel. What both lustre and ldiskfs depend on is that the kernel's VFS api staying stable. If that doesn't happen then weak module support is not possible. Am I missing something if this is not the discussion?</p> <p>This brings up interesting point. Do we need version hand shaking between ldiskfs and lustre? Anyways this is indeed a separate ticket.</p> <p>James,<br/> I think what you are saying is correct, but it isn't just the VFS api that must stay stable. It is all the internal kernel APIs that aren't part of the well defined ABI that must stay stable. lustre kernel modules both ldiskfs and not use lots of calls to symbols that are EXPORTs, but may or may not be part of the ABI and can and do occasionally change from time to time within the same major linux version from an upstream vendor.</p> <p>Now you are definitely arguing that one should never use weak modules. I don't even see why it would be acceptable use them on clients if you can't trust the semantics to stay constant with the same symbol version. If you can't trust them, you can't trust them.</p> <p>As to APIs that are not part of the ABI...in the kernel I don't believe that there is any such distinction. A linux distro vendor may choose to advertise a subset of the kernel ABI that it considers stable and safe. Red Hat has a kernel symbol whitelist. Suse, in contrast, does not.</p> <p>Yes, the osd-ldiskfs package has a long list of RHEL whitelist violations. Otherwise, the usage is pretty small. In a recent random build for master on CentOS 7.2, I see only the following non-osd-ldiskfs related off-whitelist symbols (in other words, osd-ldiskfs uses many off-whitelist symbols that I am not listing here):</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>PDE_DATA __fentry__ __free_pages __stack_chk_fail kernel_stack kstrtoull seq_lseek seq_open seq_read remove_wait_queue </pre> </div></div> <p>Some of those Red Hat might be amenable to adding to the whitelist. Some maybe we can choose a different symbol. Some we might not care and decide the level of risk is completely acceptable.</p> <p>I don't see an issue with weak-modules use with zfs. Sure, maybe the way ldiskfs is currently produced makes it more vulnerable. If you want to add extra restrictions and high barriers to usage to ldiskfs then, speaking as an all-zfs house, I don't have too much of a concern about that. <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/> Some concern though...we do have some labs that might still be using ldiskfs from TOSS's lustre.</p> <p>Anyhow, I think James is right about this getting off topic for this ticket.</p> <p>So the question related to the patch for this ticket is does having the kernel name string in the rpm provide any gain. I would say no. Not landing this patch will not change the kernel version dependency issues. Even when we resolve the dependency issues does having the kernel version string in the release field improve anything. Again I would say no. You can figure out kernel dependency using rpm queries instead.</p> <p>That's just it. These changes also delete Requires and Provides in packages too. Can't figure out dependencies using rpm queries either.</p> <p>Bob, I think you may be thinking of another change. Change <a href="http://review.whamcloud.com/19954" class="external-link" target="_blank" rel="nofollow noopener">19954</a> does not delete any Requires or Provides.</p> <p>Christopher,<br/> You are probably correct. I may have been thinking of some other change.</p> <p>If you go ahead and add a kernel version Requires for osd-ldiskfs as discussed in earlier comments I will be satisfied with that.</p> <p>Yeah, but as we got to later in the discussion, that isn't very useful. I think I have to retract that offer.</p> <p>Just having that one kernel installed doesn't stop the osd-ldiskfs package's modules from being used in any of the other kernels that are also installed. The weak-updates system will still symlink the modules into all kernels with compatible symbols.</p> <p>even if having a proper Requires permits (incorrect) weak-updates links of osd-ldiskfs in other not really matching kernels that happen to be installed, having the Requires at least gives better tracking of exactly which kernel version the particular osd-ldiskfs was derived from and is intended for.</p> <p>While I would really prefer enforcement I want to at least have visibility into the dependency.</p> <p>I'm not really on board with adding a dependency that won't function correctly.</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/19954/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/19954/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7643" title="Remove kernel version string from Lustre release field" class="issue-link" data-issue-key="LU-7643"><del>LU-7643</del></a> build: Remove Linux version string from RPM release field<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 28c17d40e5a597a3d2f10f1f43039ef92425954e</p> <p>reopening to fix fields to show up in 2.9.0 changelog</p> <p>FYI:<br/> <a href="http://cdn.opensfs.org/wp-content/uploads/2016/04/LUG2016D1_Improved-Versioning_Morrone_DiNatale.pdf" class="external-link" target="_blank" rel="nofollow noopener">http://cdn.opensfs.org/wp-content/uploads/2016/04/LUG2016D1_Improved-Versioning_Morrone_DiNatale.pdf</a><br/> <a href="https://www.youtube.com/watch?v=50VN9Q7BHYA&amp;index=5&amp;list=PLA5dHg1_l3V8r6eqrUnl8DiB-KH1K5yLH" class="external-link" target="_blank" rel="nofollow noopener">https://www.youtube.com/watch?v=50VN9Q7BHYA&amp;index=5&amp;list=PLA5dHg1_l3V8r6eqrUnl8DiB-KH1K5yLH</a></p> Blocker LU-7645 LU-5614 LU-7699 Related LU-8343 LU-20 Development End date Fri, 6 May 2016 21:44:43 +0000 Rank 1|hzxxon: Rank (Obsolete) 9223372036854775807 Start date Fri, 8 Jan 2016 21:44:43 +0000