https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-8980 Lustre <p>One of the major obstacles to the movement of the upstream client out of staging is our debug logging system. The libcfs debugging infrastructure was developed long before trace point existed.<br/> Today trace point + perf can provide far more functionally than what is offered by lctl debug. This ticket covers the work need to implement this functionality.</p> LU-8980

Add tracepoint support to Lustre

Technical task LU-9679 Critical Reopened Unresolved James A Simmons James A Simmons Wed, 28 Dec 2016 23:21:17 +0000 Sat, 28 Oct 2023 19:49:26 +0000 Lustre 2.10.0 Lustre 2.11.0 2 7 <p>I do have a first patch but I need to add the event filtering base on the debug flags. Here is the sample output I'm seeing.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-xml"># tracer: nop # # entries-in-buffer/entries-written: 2121/11382 #P:8 # # _-----=&gt; irqs-off # / _----=&gt; need-resched # | / _---=&gt; hardirq/softirq # || / _--=&gt; preempt-depth # ||| / delay # TASK-PID CPU# |||| TIMESTAMP FUNCTION # | | | |||| | | modprobe-10993 [007] .... 6584.386683: lnet_config: (linux-crypto.c:377:cfs_crypto_performance_test) Crypto hash algorithm crc32 speed = 1427 MB/s ..... </pre> </div></div> <p>James Simmons (uja.ornl@yahoo.com) uploaded a new patch: <a href="https://review.whamcloud.com/24554" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/24554</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8980" title="Add tracepoint support to Lustre" class="issue-link" data-issue-key="LU-8980">LU-8980</a> libcfs: add tracepoint support for libcfs<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: cd9d0fcbeddb0a0a00b58251bc315aff9b92ef39</p> <p>Also have llite in the works. Here is the output I have so far.</p> <p>kworker/u17:2-5181 <span class="error">&#91;001&#93;</span> .... 2135.478773: llite_vfstrace: (vvp_io.c:306:vvp_io_fini) <span class="error">&#91;0x200005222:0x13a4:0x0&#93;</span> ignore/verify layout 0/0, layout version 0 restore needed 0</p> <p>sendfile_groupl-23957 <span class="error">&#91;002&#93;</span> .... 2135.480731: llite_vfstrace: (file.c:3992:ll_layout_conf) <span class="error">&#91;0x200005222:0x13a4:0x0&#93;</span>: layout version change: 0 -&gt; 0</p> <p>James Simmons (uja.ornl@yahoo.com) uploaded a new patch: <a href="https://review.whamcloud.com/24560" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/24560</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8980" title="Add tracepoint support to Lustre" class="issue-link" data-issue-key="LU-8980">LU-8980</a> llite: add tracepoint support for llite layer<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 7fffac5a94243a03d92381cfd82340188d66684c</p> <p>Just pushed a version of tracepoint that handles printing messages to the console. Most of the infrastructure is now in place. Only a few issues to resolve.</p> <p>1) Currently ratelimting is per module. For the lustre debugging code it has a individual rate limit for every single message. I doubt that       would be acceptable upstream.</p> <p>2) Need to fine tune the printk rate limiting based on module parameters in libcfs.</p> <p>3) Some of the debug message mask are conditional i.e "CDEBUG(warn ? D_WARNING : D_INFO". Might need to create special               macros for this. Haven't figured a clean solution. Another option is break it up into two separate trace events.</p> <p>For the most part its ready for review. Peter who would be good choices to look at this work?</p> <p>James,</p> <p>This is not I expected to see when I vote for this feature. I expect to see correct function name in "FUNCTION" field instead of category like "lnet_info".</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># tracer: nop # # entries-in-buffer/entries-written: 58/61 #P:4 # # _-----=&gt; irqs-off # / _----=&gt; need-resched # | / _---=&gt; hardirq/softirq # || / _--=&gt; preempt-depth # ||| / delay # TASK-PID CPU# |||| TIMESTAMP FUNCTION # | | | |||| | | ldlm_bl_01-15147 [003] .... 189369.402891: lnet_info: (linux-crypto.c:119:cfs_crypto_hash_alloc) Using crypto hash: adler32 (adler32-zlib) speed 2454 MB/s ll_ost_io00_002-15366 [001] .... 189369.403051: lnet_info: (linux-crypto.c:119:cfs_crypto_hash_alloc) Using crypto hash: adler32 (adler32-zlib) speed 2454 MB/s ##### CPU 0 buffer started #### ptlrpcd_00_03-14400 [000] .... 189383.834528: lnet_info: (linux-crypto.c:119:cfs_crypto_hash_alloc) Using crypto hash: adler32 (adler32-zlib) speed 2454 MB/s ll_ost_io00_002-15366 [001] .... 189383.835278: lnet_info: (linux-crypto.c:119:cfs_crypto_hash_alloc) Using crypto hash: adler32 (adler32-zlib) speed 2454 MB/s ##### CPU 2 buffer started #### ptlrpcd_00_01-14398 [002] .... 189383.848857: lnet_info: (linux-crypto.c:119:cfs_crypto_hash_alloc) Using crypto hash: adler32 (adler32-zlib) speed 2454 MB/s </pre> </div></div> <p>It would be very beneficially to be able to select appropriate function name for tracing:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># perf list | grep lnet lnet:lnet_config [Tracepoint event] lnet:lnet_console [Tracepoint event] lnet:lnet_emerg [Tracepoint event] lnet:lnet_error [Tracepoint event] lnet:lnet_ha [Tracepoint event] lnet:lnet_info [Tracepoint event] lnet:lnet_ioctl [Tracepoint event] lnet:lnet_neterror [Tracepoint event] lnet:lnet_other [Tracepoint event] lnet:lnet_warning [Tracepoint event] </pre> </div></div> <p>Right now, I can select for tracing only category instead of exact function I interested in.</p> <p>Function in this case means the name the compiler generates from the TRACE_EVENT macro. For example DEFINE_EVENT(lnet_log_msg, lnet_config, ... --&gt; becomes lnet_config(...) in the source file. So they are not regular function but more like event markers. One of major goals of this project is to replace the current lctl debugging infrastructure since this is a requirement upstream. So I worked the code to map to the behavior lctl debug is doing which is category based. You will see CDEBUG(D_HA,...) maps to lnet_ha(...) etc. By the way the reason I used names like lnet_info, lnet_ioctl is to see clearly in the logs which subsys the event is showing up in. Perhaps</p> <p>ptlrpcd_00_01-14398 <span class="error">&#91;002&#93;</span> .... 189383.848857: info: lnet:(linux-crypto.c:119:cfs_crypto_hash_alloc) Using crypto hash: ...</p> <p>I have to look at the logs perf creates to see if it makes sense to the user.</p> <p>As for limiting debugging to an exact function you can do that with filtering.<br/> perf ... -filter cfs_hash_*. You can do this by module as well "-filter :mod:libcfs".<br/>  </p> <p>Unfortunately tracepoints don't allow to filter by "const char *" fields. I was not able to specify filter by file or function name because of it's interpreted as integer.</p> <p>Can we change the type to fixed array of char or dynamic array of char? I encountered that only such types allow to specify filter with wild card characters (*,?).</p> <p> </p> <p>This patch allow string filtering by file and function name.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>diff --git a/libcfs/include/libcfs/libcfs_trace.h b/libcfs/include/libcfs/libcfs_trace.h index c2aecdb..5d210cb 100644 --- a/libcfs/include/libcfs/libcfs_trace.h +++ b/libcfs/include/libcfs/libcfs_trace.h @@ -67,20 +67,20 @@ DECLARE_EVENT_CLASS(libcfs_log_msg, struct va_format *vaf), TP_ARGS(msg_file, msg_line, msg_fn, vaf), TP_STRUCT__entry( - __field(const char *, msg_file) + __string(msg_file, msg_file ? msg_file : "&lt;nofile&gt;") __field(int, msg_line) - __field(const char *, msg_fn) + __string(msg_func, msg_fn ? msg_fn : "&lt;nofunc&gt;") __dynamic_array(char, msg, CFS_TRACE_CONSOLE_BUFFER_SIZE) ), TP_fast_assign( - __entry-&gt;msg_file = msg_file; + __assign_str(msg_file, msg_file); __entry-&gt;msg_line = msg_line; - __entry-&gt;msg_fn = msg_fn; + __assign_str(msg_func, msg_fn); vsnprintf(__get_dynamic_array(msg), CFS_TRACE_CONSOLE_BUFFER_SIZE, vaf-&gt;fmt, *vaf-&gt;va); ), - TP_printk("(%s:%d:%s) %s", __entry-&gt;msg_file, __entry-&gt;msg_line, - __entry-&gt;msg_fn, __get_str(msg)) + TP_printk("(%s:%d:%s) %s", __get_str(msg_file), __entry-&gt;msg_line, + __get_str(msg_func), __get_str(msg)) ); DEFINE_EVENT(libcfs_log_msg, libcfs_config, @@ -118,20 +118,20 @@ DECLARE_EVENT_CLASS(libcfs_log_msg_limited, const char *msg_fn, struct va_format *vaf), TP_ARGS(rs, msg_file, msg_line, msg_fn, vaf), TP_STRUCT__entry( - __field(const char *, msg_file) + __string(msg_file, msg_file ? msg_file : "&lt;nofile&gt;") __field(int, msg_line) - __field(const char *, msg_fn) + __string(msg_func, msg_fn ? msg_fn : "&lt;nofunc&gt;") __dynamic_array(char, msg, CFS_TRACE_CONSOLE_BUFFER_SIZE) ), TP_fast_assign( - __entry-&gt;msg_file = msg_file; + __assign_str(msg_file, msg_file); __entry-&gt;msg_line = msg_line; - __entry-&gt;msg_fn = msg_fn; + __assign_str(msg_func, msg_fn); vsnprintf(__get_dynamic_array(msg), CFS_TRACE_CONSOLE_BUFFER_SIZE, vaf-&gt;fmt, *vaf-&gt;va); ), - TP_printk("(%s:%d:%s) %s", __entry-&gt;msg_file, __entry-&gt;msg_line, - __entry-&gt;msg_fn, __get_str(msg)) + TP_printk("(%s:%d:%s) %s", __get_str(msg_file), __entry-&gt;msg_line, + __get_str(msg_func), __get_str(msg)) ); DEFINE_EVENT_CONDITION(libcfs_log_msg_limited, libcfs_console, </pre> </div></div> <p>Probably with this patch the implementation will be suitable enough to replace current logging system.</p> <p>Oh now I understand what you mean by your filtering. Thanks for making it clearer. BTW its really  awesome to be able to filter that way.</p> <p>Dmitry how does it look to you now. BTW does anyone even know what the error codes for LCONSOLE_ERROR mean anymore?</p> <p>I pushed a new patch that has been updated and I fixed the build issue on RHEL6. Only the CERROR need to be ported but I like to have feedback on the patch before I complete  the remaining work.</p> <p>It lives. I have completed a functional patch with a complete port of all the trace point events. Please give it a review.</p> <p>"Timothy Day &lt;timday@amazon.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/52875" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/52875</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8980" title="Add tracepoint support to Lustre" class="issue-link" data-issue-key="LU-8980">LU-8980</a> debug: redefine CDEBUG to use tracing<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 7be979ac832dcafde22ca68737b4c37459dca83f</p> Related LU-14428 LU-9679 LU-8145 LU-7801 LU-8223 Development Epic/Theme upstream Rank 1|hzyzdr: Rank (Obsolete) 9223372036854775807