https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-10756 Lustre <p>For applications that manage / monitor Lustre, it would be useful if Lustre sent Uevents for interesting changes. An incomplete and non-exhaustive list:</p> <ul> <li>target mount / umount</li> <li>Lustre health</li> <li>nid / LNet changes</li> <li>evictions</li> <li>lbugs</li> <li>network timeouts</li> <li>recovery status changes<br/>  <br/>  <br/>  </li> </ul> LU-10756

Send Uevents for interesting Lustre changes

Improvement Minor Open Unresolved James A Simmons Joe Grund Fri, 2 Mar 2018 14:30:33 +0000 Tue, 13 Jul 2021 16:09:17 +0000 1 14 <p>Hi!</p> <p>I'm working on this right now. Please see <a href="https://jira.whamcloud.com/browse/LU-8066" title="Move lustre procfs handling to sysfs and debugfs." class="issue-link" data-issue-key="LU-8066">LU-8066</a>. Below are the sub-tickets</p> <p><a href="https://jira.whamcloud.com/browse/LU-8066" title="Move lustre procfs handling to sysfs and debugfs." class="issue-link" data-issue-key="LU-8066">LU-8066</a> : In order for this to work each Lustre subsystem needs a sysfs kobject.<br/> This is true for every subsystem except sptlrpc. The sptlrpc sysfs port <br/> will happen in 2.12. While we have a sysfs tree we don't have the actual<br/> tunables migrated over yet. That will happen in the 2.12 time frame. When<br/> that does happen you will be able to configure lustre using udev rules as well.<br/> Here is a example for what can be done with 2.12:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> SUBSYSTEM=="lustre", ACTION=="add", DEVPATH="*lov*", ATTR{stripecount}="4" </pre> </div></div> <p>Once LNet moves to sysfs in 2.12 you can if done right confgure LNet the<br/> same way.</p> <p><a href="https://jira.whamcloud.com/browse/LU-7004" title="fix &quot;lctl set_param -P&quot; to allow deprecation of &quot;lctl conf_param&quot;" class="issue-link" data-issue-key="LU-7004"><del>LU-7004</del></a> : This is to make lctl set_param -P to work. This combined with <a href="https://jira.whamcloud.com/browse/LU-9431" title="class_process_proc_param can&#39;t handle sysfs" class="issue-link" data-issue-key="LU-9431"><del>LU-9431</del></a><br/> send out udev events for when tunables i.e procfs/sysfs/debugfs changes<br/> are requested in mass via the MSG server.</p> <p><a href="https://jira.whamcloud.com/browse/LU-10756" title="Send Uevents for interesting Lustre changes" class="issue-link" data-issue-key="LU-10756">LU-10756</a> - send udev for client import state changes. I have a patch at <br/> <a href="https://review.whamcloud.com/#/c/31407" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/31407</a>. This cover client evictions<br/> plus other client state events.</p> <p><a href="https://jira.whamcloud.com/browse/LU-9431" title="class_process_proc_param can&#39;t handle sysfs" class="issue-link" data-issue-key="LU-9431"><del>LU-9431</del></a> - send udev events when changing tunables via sysfs/debugfs. This is<br/> set to land for lustre 2.11 in the next batch. This only works with<br/> <tt>lctl set_param -P ...</tt> on the MGS. You can see the udev rule:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> SUBSYSTEM=="lustre", ACTION=="change", ENV{PARAM}=="?*", RUN+="/usr/sbin/lctl set_param $env{PARAM}=$env{SETTING}" </pre> </div></div> <p>This is the default but in reality you can run anything for RUN <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p><a href="https://jira.whamcloud.com/browse/LU-9667" title="LNet Kernel/Userspace Interface" class="issue-link" data-issue-key="LU-9667">LU-9667</a> - This covers the move of LNet to sysfs. I have discussed<br/> already with the LNet developers about using uevents.</p> <p>This is what is on the table so far. I expected more things to be requested once this functionality<br/> started to show up. Feel free to try out my patches <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>I got a working client state patch going ; <a href="https://review.whamcloud.com/#/c/31407/)." class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/31407.</a> The question is what do we want transmitted in the uevent. So far we have for example:</p> <p>change@/fs/lustre/mdc</p> <p>ACTION=change</p> <p>DEVPATH=/fs/lustre/mdc</p> <p>SUBSYSTEM=lustre</p> <p>IMPORT=lustre-MDT0001_UUID</p> <p>STATE=REPLAY_WAIT</p> <p>SEQNUM=4622</p> <p>Anything else to add. Perhaps the obd device such as lustre-MDT0000-mdc-ffff88105dbc1000 being transmitted as well.</p> <p>^^</p> <p>timestamp?</p> <p>That is a good idea to add a timestamp for the import state. Currently the only uevents sent are for lctl set_param -P which don't include a time stamp. Should we? Also is a timestamp in seconds good enough?</p> <p>timestamp for all records; can just be the time it shows up at the server.</p> <p>Can all lctl's generate a uevent? Might be a good way to audit changes.</p> <p>James Simmons (jsimmons@infradead.org) uploaded a new patch: <a href="https://review.whamcloud.com/35463" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35463</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10756" title="Send Uevents for interesting Lustre changes" class="issue-link" data-issue-key="LU-10756">LU-10756</a> ptlrpc: change IMPORT_SET_* macros into real functions<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: f26400387d1150005486ad70762dc767d71a303e</p> <p>James Simmons (jsimmons@infradead.org) uploaded a new patch: <a href="https://review.whamcloud.com/35464" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35464</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10756" title="Send Uevents for interesting Lustre changes" class="issue-link" data-issue-key="LU-10756">LU-10756</a> osp: properly order sysfs registeration<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: ceebf58486dea07f9fe3e4de03f301a69c96cd84</p> <p>Started this work back up. Sorry I didn't reply earlier Nathan. I did add a second precision timestamp to the lctl conf_param events. For the upcoming import change state events second precision timestamps are also available. If nanosecond timestamps are needed let me know. To honest uevents are not designed to be sent by the thousands per second so I doubt nanoseconds are needed. By lctl what commands are you thinking of? Also at this point the sysfs LNet work under <a href="https://jira.whamcloud.com/browse/LU-9667" title="LNet Kernel/Userspace Interface" class="issue-link" data-issue-key="LU-9667">LU-9667</a> will provide the framework to send network events. Especially now that LNet health has landed.</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/35463/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35463/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10756" title="Send Uevents for interesting Lustre changes" class="issue-link" data-issue-key="LU-10756">LU-10756</a> ptlrpc: change IMPORT_SET_* macros into real functions<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: cf78502e48d6dbbc0d6c113e573ba9c68c5c311e</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/35464/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35464/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10756" title="Send Uevents for interesting Lustre changes" class="issue-link" data-issue-key="LU-10756">LU-10756</a> osp: properly order sysfs registration<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 2f16681d68650c0c834c9af3e05c8ed98f481d1d</p> <p>Minh Diep (mdiep@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/35795" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35795</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10756" title="Send Uevents for interesting Lustre changes" class="issue-link" data-issue-key="LU-10756">LU-10756</a> ptlrpc: change IMPORT_SET_* macros into real functions<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: 62fb72ce1955d7086c3555ddc41d9cf75441f67b</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/35795/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35795/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10756" title="Send Uevents for interesting Lustre changes" class="issue-link" data-issue-key="LU-10756">LU-10756</a> ptlrpc: change IMPORT_SET_* macros into real functions<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: <br/> Commit: 1cdc366dca7cf2a97e02de68f865f539dd58da85</p> <p>Mike Pershin (mpershin@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/37405" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37405</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10756" title="Send Uevents for interesting Lustre changes" class="issue-link" data-issue-key="LU-10756">LU-10756</a> ptlrpc: fix IMP_CLOSED state is being never set<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: baca205ac7c8f4cf8a859af84d2b5862059d2e32</p> <p>Explanation about new patch, original code in IMPORT_SET_STATE_NOLOCK() was checking imp-&gt;imp_state != LUSTRE_IMP_CLOSED before applying new state, therefore preventing closed import from changing its closed state. Meanwhile the new code checks 'state' parameter which is not current import state but new state to be set. So new code does opposite thing - instead of keeping 'closed' state forever it prevents import state to become LUSTRE_IMP_CLOSED, so import stays in FULL state until destroyed, I suppose. The patch above restores original logic.</p> <p>I've found that by noticing the following errors shortly after client remount:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> [ 1139.774868] LustreError: 25570:0:(ldlm_lockd.c:716:ldlm_handle_ast_error()) ### client (nid 10.9.3.117@tcp) returned error from blocking AST (req@ffff960abad43180 x1657226243021504 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: ffff960aba9a2240/0x60032f478e4387c8 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT flags: 0x60200400000020 nid: 10.9.3.117@tcp remote: 0xb16e71cea23ecf65 expref: 18639 pid: 13354 timeout: 2039 lvb_type: 0 [ 1139.783656] LustreError: 138-a: lustre-MDT0000: A client on nid 10.9.3.117@tcp was evicted due to a lock blocking callback time out: rc -107 [ 1139.791100] LustreError: 13344:0:(ldlm_lockd.c:259:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 10.9.3.117@tcp ns: mdt-lustre-MDT0000_UUID lock: ffff960aba9a2240/0x60032f478e4387c8 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT flags: 0x60200400000020 nid: 10.9.3.117@tcp remote: 0xb16e71cea23ecf65 expref: 18601 pid: 13354 timeout: 0 lvb_type: 0 </pre> </div></div> <p>After remount there is an old stale export on server which has a lot of locks to be canceled in background, some of them can be a blocking lock for a new locks from just mounted client. Normally such locks shouldn't cause AST be sent to a client, but while stale export is disconnected, its reverse import was not set to LUSTRE_IMP_CLOSED as needed and remains in FULL state, so AST was sent causing all these errors. I don't know about other possible side effects, but there can be.</p> <p>That was my bad. I was attempting to collect debug info even when the import entered a close state.</p> <p>I see, but that will be in debug - <tt>imp_state</tt> is not yet <tt>IMP_CLOSED</tt>, so setting the 'CLOSED' state will be in debug. As I see, only skipped cases when someone is trying to change closed import state. That case can be added in debug by separate message under the same check <tt>imp-&gt;imp_state == LUSTRE_IMP_CLOSED</tt>, e.g.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-keyword">if</span> (imp-&gt;imp_state == LUSTRE_IMP_CLOSED) { CDEBUG(D_HA, <span class="code-quote">"%p %s: attempt to change closed <span class="code-keyword">import</span> state to %s\n"</span>, imp, obd2cli_tgt(imp-&gt;imp_obd), ptlrpc_import_state_name(state)); </pre> </div></div> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/37405/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37405/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10756" title="Send Uevents for interesting Lustre changes" class="issue-link" data-issue-key="LU-10756">LU-10756</a> ptlrpc: fix IMP_CLOSED state is being never set<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 43dddbd0785d4da14714390d802bf6ec65567350</p> <p>Sebastien Piechurski (sebastien.piechurski@atos.net) uploaded a new patch: <a href="https://review.whamcloud.com/38621" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38621</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10756" title="Send Uevents for interesting Lustre changes" class="issue-link" data-issue-key="LU-10756">LU-10756</a> ptlrpc: fix IMP_CLOSED state is being never set<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: 34581dc976dbfaef708287898da4b1fb2fb4b44b</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/38621/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38621/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10756" title="Send Uevents for interesting Lustre changes" class="issue-link" data-issue-key="LU-10756">LU-10756</a> ptlrpc: fix IMP_CLOSED state is being never set<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: <br/> Commit: d22ae9251fe04d717aa0e323312879ba7e2ae3ae</p> Blocker LU-12362 Related LU-9431 LU-9667 LU-9120 LU-8066 LU-7004 LU-12564 LU-12362 LUDOC-420 LU-8609 LU-10599 Development Rank 1|hzzto7: Rank (Obsolete) 9223372036854775807