https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-4020 Lustre <p>This ticket is to track the work being done to add copytool event monitoring capabilities to liblustreapi. The end result will be that external monitoring agents are able to read an event stream out of a FIFO.</p> LU-4020

HSM copytool event monitoring capabilities

Improvement Minor Resolved Fixed Michael MacDonald Michael MacDonald HSM Fri, 27 Sep 2013 20:31:01 +0000 Fri, 20 Mar 2015 22:21:06 +0000 Wed, 5 Mar 2014 21:08:45 +0000 Lustre 2.6.0 Lustre 2.5.1 0 12 <p>Pushed a proof-of-concept for discussion: <a href="http://review.whamcloud.com/7790" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/7790</a></p> <p>To play with it, you'll need to create a fifo and then cat it before starting the copytool in a different shell. e.g.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">[root@europa ~]# mknod /<span class="code-keyword">var</span>/spool/hsm_events p &amp;&amp; cat /<span class="code-keyword">var</span>/spool/hsm_events {<span class="code-quote">"event_time"</span>: <span class="code-quote">"2013-09-27 17:26:32 -0400"</span>, <span class="code-quote">"event_type"</span>: <span class="code-quote">"REGISTER"</span>, <span class="code-quote">"archive"</span>: 0, <span class="code-quote">"mount_point"</span>: <span class="code-quote">"/mnt/jovian"</span>, <span class="code-quote">"uuid"</span>: <span class="code-quote">"ea370e22-b5ac-ab98-71bb-605d217071f7"</span>} {<span class="code-quote">"event_time"</span>: <span class="code-quote">"2013-09-27 17:26:39 -0400"</span>, <span class="code-quote">"event_type"</span>: <span class="code-quote">"RESTORE_START"</span>, <span class="code-quote">"total_bytes"</span>: 0, <span class="code-quote">"lustre_path"</span>: <span class="code-quote">"CentOS-6.4-x86_64-bin-DVD2.iso"</span>, <span class="code-quote">"fid"</span>: <span class="code-quote">"0x200000400:0x24:0x0"</span>} {<span class="code-quote">"event_time"</span>: <span class="code-quote">"2013-09-27 17:26:39 -0400"</span>, <span class="code-quote">"event_type"</span>: <span class="code-quote">"RESTORE_RUNNING"</span>, <span class="code-quote">"current_bytes"</span>: 0, <span class="code-quote">"total_bytes"</span>: 1452388352, <span class="code-quote">"lustre_path"</span>: <span class="code-quote">"CentOS-6.4-x86_64-bin-DVD2.iso"</span>, <span class="code-quote">"fid"</span>: <span class="code-quote">"0x200000400:0x24:0x0"</span>} {<span class="code-quote">"event_time"</span>: <span class="code-quote">"2013-09-27 17:27:09 -0400"</span>, <span class="code-quote">"event_type"</span>: <span class="code-quote">"RESTORE_RUNNING"</span>, <span class="code-quote">"current_bytes"</span>: 681574400, <span class="code-quote">"total_bytes"</span>: 1452388352, <span class="code-quote">"lustre_path"</span>: <span class="code-quote">"CentOS-6.4-x86_64-bin-DVD2.iso"</span>, <span class="code-quote">"fid"</span>: <span class="code-quote">"0x200000400:0x24:0x0"</span>} {<span class="code-quote">"event_time"</span>: <span class="code-quote">"2013-09-27 17:27:09 -0400"</span>, <span class="code-quote">"event_type"</span>: <span class="code-quote">"RESTORE_CANCEL"</span>, <span class="code-quote">"fid"</span>: <span class="code-quote">"0x200000400:0xa3:0x0"</span>} {<span class="code-quote">"event_time"</span>: <span class="code-quote">"2013-09-27 17:27:14 -0400"</span>, <span class="code-quote">"event_type"</span>: <span class="code-quote">"UNREGISTER"</span>, <span class="code-quote">"archive"</span>: 0, <span class="code-quote">"mount_point"</span>: <span class="code-quote">"/mnt/jovian"</span>, <span class="code-quote">"uuid"</span>: <span class="code-quote">"ea370e22-b5ac-ab98-71bb-605d217071f7"</span>} </pre> </div></div> <p>I guess you mean liblustreapi and not liblustre in the ticket description?</p> <p>Hmm, yes. Updated the description, thanks.</p> <p>I thought that it might be best to move discussion about this work from gerrit to this ticket. As I indicated in the commit message for the review I pushed, my intent was to prove the concept and get some feedback on the plan, so I am happy to see this conversation happening.</p> <p>I'll respond to general feedback on some of the higher-level topics here so that we can keep the discussion going:</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=jhammond" class="user-hover" rel="jhammond">jhammond</a>: I did consider a socket (unix, udp) based approach, but it seemed to add complexity to the implementation without really adding much benefit over the FIFO approach. My goal wasn't to make a completely reliable event stream &#8211; I was thinking more of making it best effort. If there is a reader to see the events, great. If not, life goes on and there's no negative impact on the copytool instance. I was careful to handle cases where the copytool started without a reader (works OK) or where the reader disappeared at various points (OK, in my testing).</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=adilger" class="user-hover" rel="adilger">adilger</a>: <a href="http://www.yaml.org/spec/1.2/spec.html#id2759572" class="external-link" target="_blank" rel="nofollow noopener">JSON is actually a subset of YAML</a>. YAML parsers can read JSON just fine, though the reverse isn't true. I decided to use JSON because the event format doesn't need all of YAML's capabilities, and it's much easier to generate correct JSON. JSON is also easier to validate on the reader side because it's simple. It's very easy to detect partial writes of JSON-formatted events, for example.</p> <p>All that having been said, I'm not opposed to the idea of using pure YAML, especially if someone else is writing or linking in a YAML library. In my opinion, though, JSON is probably good enough for 99% of what we need as far as structured output goes.</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=jcl" class="user-hover" rel="jcl">jcl</a>: I will certainly add tests for the final implementation, but thank you for calling it out. My initial focus was to get some code working in order to test ideas and generate discussion before committing to a final design. As far as libraries go, I am not opposed to the idea of using a well-tested library to generate JSON and/or YAML &#8211; I just wasn't sure what the reception would be to adding external dependencies like that. There is a MIT-licensed library called <a href="http://www.digip.org/jansson/" class="external-link" target="_blank" rel="nofollow noopener">Jansson</a> that seems mature and well-maintained.</p> <p>I think that covers most of the high-level topics. There's been a lot of really great feedback on implementation details too, and I appreciate that. I will certainly incorporate those improvements into the code as I make progress.</p> <p>Why not just get info from the policy agent? It knows the state of the entire FS. </p> <p>What do you mean by "policy agent"? The "coordinator" or the "policy engine" (ie RBH)? The initial idea was to provide a STD interface for external backend tools, as CT is running on STD Lustre client, implementing it in liblustreapi for the CT is the natural way.</p> <p>I mean Robinhood. I believe it is away of the state of the filesystem. It seems this plan is to capture the profile it exports in some other layer of Lustre. Why not just use Robinhood (or some other Policy Agent) directly to resolve the state of the filesystem? </p> <p>RBH runs on some client. This client may not be connected to external storage, like is the agent, so will have difficulties to communicate with it. Also there is a single instance of RBH so we may have a scalability issue. The agent count is easy to increase so even highly verbose CT will scale.</p> <p>I have pushed a new patch-set #6 for <a href="http://review.whamcloud.com/7790" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/7790</a>. Where after re-base, I tried to answer to the multiple comments from previous patch-sets.</p> <p>Andreas, is the new liblustreapi_json.c, what you wanted ? I am not really aware of this licensing protocols and thus about their packaging needs … What about the specific data-structures definitions being used, do they need to be in a separate .h file too with the appropriate header?</p> <p>Jinshan, I did not remove the head-list structure llapi_json_item_list, because I find code more easy to read than without.</p> <p>Attached a small proof-of-concept for generating valid JSON with PyYAML. The main takeaway is that the JSON we are currently generating is indeed valid YAML. Given that the DLC patches that bring in libyaml have not yet landed on master, and won't land for 2.5.x, I propose that we move forward with the existing simple JSON generator, but plan to replace it with libyaml when that becomes available.</p> <p>backport to b2_5;<br/> <a href="http://review.whamcloud.com/9512" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/9512</a></p> <p>Landed for 2.6</p> Related LU-4751 LU-3734 Development Rank 1|hzw49z: Rank (Obsolete) 10799