https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-12529 Lustre <p>We have some bad config vlaues.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> Jul 9 16:13:23 testpbs systemd-udevd[98107]: <span class="code-object">Process</span> <span class="code-quote">'/usr/sbin/lctl set_param '</span>lov.nbp7-MDT0000-mdtlov.stripecount=1'' failed with exit code 2. Jul 9 16:13:23 testpbs systemd-udevd[98110]: <span class="code-object">Process</span> <span class="code-quote">'/usr/sbin/lctl set_param '</span>osc.nbp7-OST0026-osc-MDT0000.active=0'' failed with exit code 2. Jul 9 16:13:23 testpbs systemd-udevd[98110]: <span class="code-object">Process</span> <span class="code-quote">'/usr/sbin/lctl set_param '</span>osc.nbp7-OST002a-osc-MDT0000.active=0'' failed with exit code 2. Jul 9 16:13:23 testpbs systemd-udevd[98107]: <span class="code-object">Process</span> <span class="code-quote">'/usr/sbin/lctl set_param '</span>osc.nbp7-OST0046-osc-MDT0000.active=0'' failed with exit code 2. Jul 9 16:13:23 testpbs systemd-udevd[98107]: <span class="code-object">Process</span> <span class="code-quote">'/usr/sbin/lctl set_param '</span>osc.nbp7-OST003e-osc-MDT0000.active=0'' failed with exit code 2. Jul 9 16:13:23 testpbs systemd-udevd[98107]: <span class="code-object">Process</span> <span class="code-quote">'/usr/sbin/lctl set_param '</span>osc.nbp7-OST003a-osc-MDT0000.active=0'' failed with exit code 2. Jul 9 16:13:23 testpbs systemd-udevd[98107]: <span class="code-object">Process</span> <span class="code-quote">'/usr/sbin/lctl set_param '</span>osc.nbp7-OST002e-osc-MDT0000.active=0'' failed with exit code 2. Jul 9 16:13:23 testpbs systemd-udevd[98107]: <span class="code-object">Process</span> <span class="code-quote">'/usr/sbin/lctl set_param '</span>osc.nbp7-OST0032-osc-MDT0000.active=0'' failed with exit code 2. Jul 9 16:13:23 testpbs systemd-udevd[98107]: <span class="code-object">Process</span> <span class="code-quote">'/usr/sbin/lctl set_param '</span>osc.nbp7-OST004e-osc-MDT0000.active=0'' failed with exit code 2. Jul 9 16:13:23 testpbs systemd-udevd[98107]: <span class="code-object">Process</span> <span class="code-quote">'/usr/sbin/lctl set_param '</span>osc.nbp7-OST0042-osc-MDT0000.active=0'' failed with exit code 2. Jul 9 16:13:23 testpbs systemd-udevd[98107]: <span class="code-object">Process</span> <span class="code-quote">'/usr/sbin/lctl set_param '</span>osc.nbp7-OST0052-osc-MDT0000.active=0'' failed with exit code 2. Jul 9 16:13:23 testpbs systemd-udevd[98107]: <span class="code-object">Process</span> <span class="code-quote">'/usr/sbin/lctl set_param '</span>osc.nbp7-OST004a-osc-MDT0000.active=0'' failed with exit code 2. Jul 9 16:13:23 testpbs systemd-udevd[98107]: <span class="code-object">Process</span> <span class="code-quote">'/usr/sbin/lctl set_param '</span>osc.nbp7-OST0036-osc-MDT0000.active=0'' failed with exit code 2. </pre> </div></div> <p>How do we remove these I tried like this.<br/> lctl set_param -P -d "osc.nbp7-OST0036-osc-MDT0000.active" <br/> but didn't remove the entry from the config file.</p> lustre 2.10. LU-12529

How to remove bad config values from MGS logs

Improvement Minor Resolved Fixed Patrick Farrell Mahmoud Hanafi Tue, 9 Jul 2019 23:22:48 +0000 Wed, 7 Aug 2019 19:25:57 +0000 Wed, 7 Aug 2019 19:25:57 +0000 0 6 <p>Mahmoud,</p> <p>Did you run that on the MGS?  If not, you must.</p> <p>Additionally, I think you probably need something like:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>lctl set_param -P -d osc.*.active </pre> </div></div> <p>As you probably didn't specify each OST individually when setting this up?</p> <p>That doesn't work. Looks like there the log entry for one</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> nbp7-mds1 ~ # llog_reader /tmp/params | grep osc.nbp7-OST0026-osc-MDT0000.active #68 (224)marker 2 (flags=0x01, v2.7.3.0) general <span class="code-quote">'osc.nbp7-OST0026-osc-MDT0000.active'</span> Tue Jul 10 08:28:03 2018- #69 (128)set_param 0:general 1:osc.nbp7-OST0026-osc-MDT0000.active=0 2:lctl #70 (224)END marker 2 (flags=0x02, v2.7.3.0) general <span class="code-quote">'osc.nbp7-OST0026-osc-MDT0000.active'</span> Tue Jul 10 08:28:03 2018- #274 (224)SKIP START marker 5 (flags=0x05, v2.10.6.0) nbp7-OST-OST0026 <span class="code-quote">'osc.nbp7-OST0026-osc-MDT0000.active'</span> Tue Jul 9 15:48:34 2019-Tue Jul 9 16:02:23 2019 #275 (144)SKIP set_param 0:nbp7-OST-OST0026 1:osc.nbp7-OST0026-osc-MDT0000.active=0= 2:lctl #276 (224)SKIP END marker 5 (flags=0x06, v2.10.6.0) nbp7-OST-OST0026 <span class="code-quote">'osc.nbp7-OST0026-osc-MDT0000.active'</span> Tue Jul 9 15:48:34 2019-Tue Jul 9 16:02:23 2019 </pre> </div></div> <p>Running this has no effect.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> lctl set_param -P -d osc.nbp7-OST0026-osc-MDT0000.active </pre> </div></div> <p>Can you collect debug logs for this?</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>DEBUGMB=`lctl get_param -n debug_mb` lctl set_param *debug=-1 debug_mb=10000 lctl clear lctl mark "before" lctl set_param -P -d osc.nbp7-OST0026-osc-MDT0000.active #Write out the log lctl dk &gt; /tmp/log#Set debug back to defaults lctl set_param debug="super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck" lctl set_param debug_mb=$DEBUGMB</pre> </div></div> <p>And then also, separately, can you please collect an strace of the command?</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>strace -v lctl set_param -P -d osc.nbp7-OST0026-osc-MDT0000.active </pre> </div></div> <p>You could also run on the MGS "<tt>lctl --device MGS llog_print nbp7-client</tt>" or "<tt>... params</tt>" to dump the client config llogs while the system is mounted instead of the "<tt>params</tt>" log. You can use "<tt>lctl --device MGS llog_cancel params 69</tt>" to cancel that record number. </p> <p>Mahmoud,</p> <p>Even if Andreas' suggestion works (you should definitely try it), could you please gather those logs I asked for?  They may be helpful in identifying a problem or problems.</p> <p>On nbp7 I already ran Andreas command and it work. But we have 2 other filesystem that also have bad records. On those I get</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> nbp2-mds ~ # lctl --device MGS llog_print params OBD_IOC_LLOG_PRINT failed: Inappropriate ioctl <span class="code-keyword">for</span> device </pre> </div></div> <p>They both are running server versions.<br/> lctl --device MGS llog_print nbp2-client works.<br/> btw, with both param or client config if the number of records is large, they get truncated and we get this error</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> [3115652.139607] LustreError: 35937:0:(llog_ioctl.c:254:llog_print_cb()) not enough space <span class="code-keyword">for</span> print log records </pre> </div></div> <p>What do you mean by "server versions"?  These commands need to be run on the MGS.</p> <p>They are all running the same luster version.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> nbp7-mds1 ~ # modinfo lustre filename: /lib/modules/3.10.0-693.21.1.el7.20180508.x86_64.lustre2106/extra/lustre/fs/lustre.ko license: GPL version: 2.10.6 description: Lustre Client File <span class="code-object">System</span> author: OpenSFS, Inc. &lt;http:<span class="code-comment">//www.lustre.org/&gt; </span>retpoline: Y rhelversion: 7.4 srcversion: E459483EA54C83D0585ECA3 depends: obdclass,ptlrpc,libcfs,lnet,lmv,mdc,lov vermagic: 3.10.0-693.21.1.el7.20180508.x86_64.lustre2106 SMP mod_unload modversions nbp2-mds ~ # modinfo lustre filename: /lib/modules/3.10.0-693.21.1.el7.20180508.x86_64.lustre2106/extra/lustre/fs/lustre.ko license: GPL version: 2.10.6 description: Lustre Client File <span class="code-object">System</span> author: OpenSFS, Inc. &lt;http:<span class="code-comment">//www.lustre.org/&gt; </span>retpoline: Y rhelversion: 7.4 srcversion: E459483EA54C83D0585ECA3 depends: obdclass,ptlrpc,libcfs,lnet,lmv,mdc,lov vermagic: 3.10.0-693.21.1.el7.20180508.x86_64.lustre2106 SMP mod_unload modversions </pre> </div></div> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=mhanafi" class="user-hover" rel="mhanafi">mhanafi</a> the bad config records may appear in either the "<tt>params</tt>" log or in the "<tt>fsname-client</tt>" config log. The name of the logfile passed to "<tt>lctl llog_cancel</tt>" needs to match. </p> <p>If "<tt>llog_print</tt>" complains about the log file being too large, you can grab the lctl binary from 2.10.7 or later, or apply the patch <a href="https://review.whamcloud.com/3381" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/3381</a>� "<tt><a href="https://jira.whamcloud.com/browse/LU-11566" title="sanity test_60aa: llog_print_cb()) not enough space for print log records" class="issue-link" data-issue-key="LU-11566"><del>LU-11566</del></a> utils: fix lctl llog_print for large configs</tt>". This is purely a userspace problem and can be solved with an updated lctl binary, no need to update the kernel<br/> Modules or restart. </p> <p>What about the Inappropriate ioctl error.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> nbp2-mds ~ # lctl --device MGS llog_print params OBD_IOC_LLOG_PRINT failed: Inappropriate ioctl <span class="code-keyword">for</span> device </pre> </div></div> <p> </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> brk(0x67d000) = 0x67d000 brk(NULL) = 0x67d000 open(<span class="code-quote">"/dev/obd"</span>, O_RDWR) = 3 ioctl(3, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0x7f, 0x08), 0x7fffffffc9f0) = 0 ioctl(3, _IOC(_IOC_READ|_IOC_WRITE, 0x66, 0xc0, 0x08), 0x7fffffffc9f0) = -1 ENOTTY (Inappropriate ioctl <span class="code-keyword">for</span> device) write(2, <span class="code-quote">"OBD_IOC_LLOG_PRINT failed: Inapp"</span>..., 58OBD_IOC_LLOG_PRINT failed: Inappropriate ioctl <span class="code-keyword">for</span> device ) = 58 rt_sigaction(SIGINT, {0x405ef0, ~[RTMIN RT_1], SA_RESTORER|SA_RESTART, 0x7fffed6d65d0}, NULL, 8) = 0 exit_group(1) = ? +++ exited with 1 +++ </pre> </div></div> <blockquote> <p>What about the Inappropriate ioctl error.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>nbp2-mds ~ # lctl --device MGS llog_print params OBD_IOC_LLOG_PRINT failed: Inappropriate ioctl for device </pre> </div></div></blockquote> <p>I tested this out locally on my 2.10 system and got the same error. I <em>thought</em> that <tt>llog_print</tt> was working for the params file, and there is even a regression test for this (<tt>conf-sanity.sh test_123ab()</tt>, so it isn't clear why this isn't working.</p> <p>It looks like the MGS needs patch <a href="https://review.whamcloud.com/34250:" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34250:</a></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>LU-4939 obdclass: llog_print params file Allow llog_print to handle the params file in yaml </pre> </div></div> <p>which was landed to b2_10 as <tt>v2_10_6-45-gfb77f09ac8</tt>, so it would be included into 2.10.7 and later. My home/test server is running only 2.10.5.</p> <p>The various issues in this ticket have been resolved, or are fixed in a newer release.</p> Related LU-4939 Development Rank 1|i00jev: Rank (Obsolete) 9223372036854775807