https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-9067 Lustre <p>This problem has been seen on el6.8. The command 'lctl dl' fails.</p> <p>This appears to be due to the "devices" entry used by the command being missing.<br/> In older e6.x versions it is /proc/fs/lustre/devices.<br/> In some newer distros, for example el7.3, it is /sys/kernel/debug/lustre/devices.</p> <p>There isn't any lustre "devices" file anywhere in /proc or /sys. Don't know exactly why not.</p> LU-9067

lctl dl command fails on el6

Bug Minor Resolved Fixed James A Simmons Bob Glossman Mon, 30 Jan 2017 18:12:31 +0000 Wed, 1 Mar 2017 15:58:57 +0000 Wed, 1 Mar 2017 15:58:57 +0000 Lustre 2.10.0 0 8 <p>What version of Lustre is this? It must either be the upstream kernel client, or something from recent master, for it to be in <tt>/sys/&#42;</tt>, but they are using an old (pre 2.8) version of <tt>lctl</tt>. That was added in patch <a href="http://review.whamcloud.com/17468" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/17468</a> "<a href="https://jira.whamcloud.com/browse/LU-5030" title="&quot;lctl {get,set}_param&quot; should also check in /sys/fs/{lnet,lustre}" class="issue-link" data-issue-key="LU-5030"><del>LU-5030</del></a> util: migrate liblustreapi to use cfs_get_paths()", which was landed as commit v2_7_65_0-23-g8813fdf.</p> <p>this is the current tip of master. tag 2.9.52</p> <p>The lctl command is definitely looking in all the new locations, including /sys.<br/> The problem is that in el6.8 there doesn't seem to be any "devices" file anywhere.</p> <p>Could be fallout from patch <a href="https://review.whamcloud.com/23427" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/23427</a> "<a href="https://jira.whamcloud.com/browse/LU-8066" title="Move lustre procfs handling to sysfs and debugfs." class="issue-link" data-issue-key="LU-8066">LU-8066</a> obd: Add debugfs root" since part of the description is:</p> <blockquote> <p>Move /proc/fs/lustre/devices to debugfs. The devices file prints out<br/> status information about all obd devices in the system in human<br/> readable form.</p></blockquote> <p>Andreas,<br/> It could very well be true that <a href="https://review.whamcloud.com/23427" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/23427</a> breaks something. don't understand though why it works right in el7 but not in el6</p> <p>failure isn't only on prereleases<br/> Seen with latest master on el6.8 too.<br/> Strongly suspect it's broken on any el6, making it a pretty serious regression.</p> <p>Same symptom, no lustre "devices" file created anywhere in /sys or /proc.</p> <p>fwiw on failing systems there is a /sys/kernel/debug dir, there isn't any /sys/kernel/debug/lustre dir. Don't know if that's a useful clue or not.</p> <p>Bob,<br/> probably the debugfs is not mounted by default on RHEL 6.x. So, please use the following command to mount it.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">mount -t debugfs none /sys/kernel/debug </pre> </div></div> <p>You can add an equivalent /etc/fstab line to automatically mount it.</p> <p>Dmitry,<br/> You are correct. debugfs isn't mounted by default on el6. doing the manual mount command you suggest fixes the problem.</p> <p>This begs some questions though:<br/> 1) debugfs is mounted by default in el7. However it isn't in /etc/fstab there. How is it done in that case? Should el6 do the same?</p> <p>2) if mount of debugfs in now a requirement for lustre to operate correctly how do we ensure that it is always done on all distros?</p> <p>I agree with Bob - if this changes the behavior out of the box then it will be problematic for users. Either we need to automatically mount debugfs in lctl if "dl" (or other commands that need debugfs content) is used, revert the patch moving this over to debugfs until the problem is fixed, or figure out some other way to handle this. I'm surprised that this didn't cause any test failures during landing, but I guess that means there is no test that checks the output of "lctl dl" (yet).</p> <p>In ancient days we used to get the "dl" content via ioctl(), but for large filesystem that caused problems due to the size of the output. </p> <p>Ugh. I would suggest that we call mount() in lctl.c to handle this. To many patches have already landed for this to be reverted. Let me patch it up. We might need to back port this to earlier lustre versions as well.</p> <p>Hi James,</p> <p>Assigning to you per your commentary above.</p> <p>Thanks.<br/> Joe</p> <p>James Simmons (uja.ornl@yahoo.com) uploaded a new patch: <a href="https://review.whamcloud.com/25182" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/25182</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9067" title="lctl dl command fails on el6" class="issue-link" data-issue-key="LU-9067"><del>LU-9067</del></a> utils: ensure debugfs is mounted<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 8aac94c0e186bc55b8987df68343af0f56efa48d</p> <p>found the answer to my question 1) above.<br/> On el7 automount of debugfs is done by systemd, and controlled by the config file /usr/lib/systemd/system/sys-kernel-debug.mount</p> <p>For el6 the same thing could probably be done in an init.d file for debugfs</p> <p>James,<br/> Not sure I like your solution. The patch seems to work, but by calling mount() directly debugfs isn't added to mtab, Therefore it doesn't show up as visibly mounted in the 'mount' command. It does appear in /proc/mounts.</p> <p>I note that on el6 mount for /proc is accomplished with a line in /etc/fstab.<br/> Seems reasonable to do the same for debugfs.<br/> Maybe a little applet done during install of the lustre.rpm? only needed for el6.</p> <p>The init.d solution only handles node bring up. What happens if for some reason debugfs is umounted long after the node has been up? That is why I did my approach.</p> <p>afaik nothing prevents a root privileged user from unmounting /proc either.<br/> My feeling is if a superuser shoots themselves in the foot it serves them right.</p> <p>I just prefer an approach that is less intrusive than doing something extra on nearly every lfs or lctl invocation.</p> <p>I'm thinking of maybe altering /etc/fstab at installation time, and doing it only on el6 and only if not already done. If a debugfs line is added then also do a 'mount debugfs' right then. It then is mounted at boot time ever after. No impact on lustre libs or utils. Just a suggestion.</p> <p>True an admin can umount /proc either. Okay lets go with the special script at startup. I can test it on a RHEL6 client for you.</p> <p>Bob have you come up with a boot script yet?</p> <p>James,<br/> I thought you were going to do it. I only made a suggestion for an approach.</p> <p>My thinking is no boot script is needed. Once /etc/fstab is modified debugfs would always get mounted at boot time by current existing scripts.</p> <p>Okay. I misunderstood. I updated the lnet init.d startup script for RHEL6.8  to mount debugfs. Can you try the last version of my previous patch. </p> <p>James,<br/> Your change to the lnet script works, but I see 2 problems with it:</p> <p>1) not everybody uses the lnet script to startup lustre<br/> 2) if a permanent change to /etc/fstab has been done by any method then debugfs is already mounted. This leads to errors like:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># service lnet start mount: none already mounted or /sys/kernel/debug busy mount: according to mtab, debugfs is already mounted on /sys/kernel/debug LNET configured </pre> </div></div> <p>Does anyone use any of those scripts? Its just we can not replace fstab when installing lustre. fstab can be very site specific. Somewhere some how debugfs has to be mounted. Suggestions besides the whole idea of nuking a sites fstab file? Also we need debugfs available in the case of routers which only will have lnet installed. Perhaps my libcfs code is the best option.</p> <p>I favor the idea of editing /etc/fstab on the fly at install time, adding a line for debugfs. Do it only on el6, do it only if such a line is not already there. This preserves any local site fstab edits or changes. Just not sure how to accomplish that.</p> <p>I don't favor complete replacement of /etc/fstab ever.</p> <p>So most people don't use the provided startup scripts that come with lustre?</p> <p>I believe actual practice in real installations varies quite a bit. I have seen many sites that don't use them at all. Personally I don't in my own test setups. I think they were initially done as examples, not as required must use features. They have existed for a long time.</p> <p>afaik, the most common use of the lnet startup script is on routers. It's an easy way to get the needed kernel modules loaded reliably at boot time. On routers there are typically no mount or other lustre activities that would get modules loaded otherwise.</p> <p>Looking at our own systems we manage fstab with puppet so any changes at install time will be stomped on soon after. I don't think modifying fstab is going to work. If people don't use the startup script then we are going to have to go with the libcfs library mounting debugfs for us. We just need to do it one time. </p> <p>not sure what " manage fstab with puppet" means. if you have external methods to change and maintain fstab, how do you do other site specific changes, for example adding nfs client mounts? maybe in such cases a debugfs mount can be added by an admin.</p> <p>Does SLES11 have this issue also? I see my Cray system its mounted but I wonder in general. I have an idea!!! What about calling sys_mount when the libcfs modules loads? We can make it conditional only for RHEL6 and that way it only would happen at module load. Does that sound reasonable?</p> <p>not an issue on SLES11 or SLES12. debugfs mounted there. el6 is the only context I can find where it's not mounted by default.</p> <p>In sles11 it's mounted via fstab.<br/> In sles12 it's mounted via systemd, just like el7.</p> <p>James,<br/> Back to my original criticism of your original fix on your latest rev:</p> <blockquote> <p>The patch seems to work, but by calling mount() directly debugfs isn't added to mtab Therefore it doesn't show up as visibly mounted in the 'mount' command. It does appear in /proc/mounts.</p></blockquote> <p>I see nothing that restricts it to el6 or happening only at module load time, as mentioned in your comment (above).</p> <p>Newer distros symlink mtab to /proc/mounts but that is not the case for RHEL6. Luckly their is a function to add entries to mtab.</p> <p>yes, el6 is old school. Still maintains mtab as a real, separate file. Not linked to /proc/mounts. All distros used to be that way.</p> <p>I updated the patch so if mtab is a real file it will add a debugfs entry.</p> <p>more on master:<br/> <a href="https://testing.hpdd.intel.com/test_sets/ad39ea70-fc88-11e6-b542-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/ad39ea70-fc88-11e6-b542-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/test_sets/06f1473c-fd17-11e6-a77a-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/06f1473c-fd17-11e6-a77a-5254006e85c2</a></p> <p>I think this problem is blocking all el6 tests on master atm.<br/> The only reason it isn't causing more fallout is the fact that el7 is the default test distro for master. Not a problem on el7.</p> <p>Should be landing very soon <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/25182/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/25182/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9067" title="lctl dl command fails on el6" class="issue-link" data-issue-key="LU-9067"><del>LU-9067</del></a> utils: ensure debugfs is mounted<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: e53bbbc510f9ac96f2556131c405c7e5c749cc27</p> <p>Patch has landed <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> Related LU-8066 Development Rank 1|hzz21z: Rank (Obsolete) 9223372036854775807 Severity