https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-8145 Lustre <p>As we can not monitoring the debug trace all the time, especially when<br/> the error will happen. So if the dump trace can start once CERROR, that would be very helpful, and we can collect as more debug info as possible.</p> LU-8145

start dump trace thread once CERROR

Improvement Minor Resolved Won't Fix Joseph Gmitter Gu Zheng patch Mon, 16 May 2016 08:06:28 +0000 Thu, 7 Jun 2018 14:01:49 +0000 Thu, 7 Jun 2018 14:01:49 +0000 0 5 <p>Gu Zheng (gzheng@ddn.com) uploaded a new patch: <a href="http://review.whamcloud.com/20218" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/20218</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8145" title="start dump trace thread once CERROR" class="issue-link" data-issue-key="LU-8145"><del>LU-8145</del></a> libcfs: add dump debug trace on error support<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 831bd3adc5294a5d6a91bc46440062f9499f9bf6</p> <p>Have you been using this patch in production anywhere? It seems to me that virtually every time that Lustre is started, especially after a recovery, it will print a CERROR() to the console and start debug daemon. In that case, it will always be running on all of the nodes.</p> <p>Hi Andreas,<br/> Thanks for your comments.<br/> Just simple tests, it works well, but not used on production yet.<br/> “It seems to me that virtually every time that Lustre is started”<br/> Is there CERROR message in the dumped log?</p> <p>I thought that messages like:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Lustre: myth-OST0000: Will be in recovery for at least 5:00, or until 2 clients reconnect </pre> </div></div> <p>were printed with LustreError, but they are not. Maybe it will be OK. Best to check some of your customer systems for LustreError in the console logs starting at boot time to see if having debug_daemon enabled after CERROR() would be useful for debugging or just dumping the debug log once. I suspect that in most cases the valuable debugging information will have happened <em>before</em> the CERROR() and not after, so dumping the current logs would be enough.</p> <p>Note also that there are already module parameters like <tt>dump_on_eviction</tt>, <tt>dump_on_timeout</tt>, and <tt>dump_on_peer_timeout</tt> that will dump the logs once without having to enable debug_daemon to collect a large amount of debug information.</p> <p>Note that Fujitsu has a similar thing where all lustre messages are dumped into a debug-deamon-like buffer not to clog dmesg.</p> <p>But anyway the biggest problem here is such that we run with very lean debug mask by default, so there's hardly anything you get from it outside of CERROR/CWARN that is already logged in syslog/dmesg.</p> <p>Now you could increase the debug level, but if you do it by default, this suddenly makes your FS slower and nobody likes that. If you do it on first CERROR, it's kind of too late too.</p> <p>Yeah, it maybe not very helpful to production environment, but, IMO, it is useful to debug, especially when we hit error on production but hard to reproduce. </p> <p>Can we close this ticket.</p> Related LU-8980 Development End date Fri, 20 May 2016 08:06:28 +0000 Rank 1|hzybp3: Rank (Obsolete) 9223372036854775807 Start date Mon, 16 May 2016 08:06:28 +0000