https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-8017 Lustre <p>Current build installed; <a href="https://build.hpdd.intel.com/job/lustre-reviews/38245/" class="external-link" target="_blank" rel="nofollow noopener">https://build.hpdd.intel.com/job/lustre-reviews/38245/</a><br/> This issue has persisted for the last two builds. <br/> After mounting the filesystem, all nodes report NOT HEALTHY in /proc/fs/lustre/health_check. </p> <ol> <li>pdsh -g server 'lctl get_param health_check' |dshbak -c<br/> ----------------<br/> lola-<span class="error">&#91;2-11&#93;</span><br/> ----------------<br/> health_check=healthy<br/> NOT HEALTHY</li> </ol> <p>The filesystem otherwise operates normally, jobs run, results are created.<br/> We were using the health_check as part of our monitoring - this has been discontinued. <br/> We are uncertain as to the cause, as all operations we can test work fine, and no errors are reported. </p> LU-8017

All Nodes report NOT HEALTHY, system is healthy

Bug Critical Resolved Fixed James A Simmons Cliff White soak Wed, 13 Apr 2016 17:51:15 +0000 Wed, 7 Dec 2016 20:51:31 +0000 Tue, 7 Jun 2016 21:07:54 +0000 Lustre 2.9.0 Lustre 2.9.0 0 6 <p>This looks like a bug introduced by <a href="http://review.whamcloud.com/16933" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/16933</a> "<a href="https://jira.whamcloud.com/browse/LU-6215" title="Sync Lustre external tree with lustre linux kernel client" class="issue-link" data-issue-key="LU-6215"><del>LU-6215</del></a> lprocfs: handle seq_printf api change".</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">lustre/obdclass/linux/linux-module.c @@ -275,7 +277,7 @@ <span class="code-keyword">static</span> <span class="code-object">int</span> obd_proc_health_seq_show(struct seq_file *m, void *data) read_unlock(&amp;obd_dev_lock); <span class="code-keyword">if</span> (healthy) - <span class="code-keyword">return</span> seq_printf(m, <span class="code-quote">"healthy\n"</span>); + seq_puts(m, <span class="code-quote">"healthy\n"</span>); seq_printf(m, <span class="code-quote">"NOT HEALTHY\n"</span>); <span class="code-keyword">return</span> 0; </pre> </div></div> <p>It should still have returned after printing "healthy" instead of continuing to "NOT HEALTHY".</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-keyword">if</span> (healthy) { seq_puts(m, <span class="code-quote">"healthy\n"</span>); <span class="code-keyword">return</span>; } </pre> </div></div> <p>Even in a healthy environment, it still show "NOT HEALTHY"</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@testnode tests]# MDSCOUNT=4 sh llmount.sh Stopping clients: testnode /mnt/lustre (opts:) Stopping clients: testnode /mnt/lustre2 (opts:) Loading modules from /work/lustre-new/lustre-release/lustre/tests/.. ........ [root@testnode tests]# cat /proc/fs/lustre/health_check healthy NOT HEALTHY </pre> </div></div> <p>I checked the code, and it looks like a typo in the code</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>static int obd_proc_health_seq_show(struct seq_file *m, void *data) { ............ if (healthy) seq_puts(m, "healthy\n"); ---------------------------&gt; probably else is missing here. seq_printf(m, "NOT HEALTHY\n"); </pre> </div></div> <p>This also indicates a test miss, since we <em>should</em> check health_check in auto test. </p> <p>I'd be fine with an "else" also.</p> <p>Please also add a test that this is working properly.</p> <p>I think QA team can do this. </p> <p>Oops, missed fixing up a sed change. Will fix. Sorry I didn't add a test with this patch but it is a really good idea. This way we can see if the upstream client will also behave properly. Which test should it go into?</p> <p>James Simmons (uja.ornl@yahoo.com) uploaded a new patch: <a href="http://review.whamcloud.com/19537" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/19537</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8017" title="All Nodes report NOT HEALTHY, system is healthy" class="issue-link" data-issue-key="LU-8017"><del>LU-8017</del></a> obd: report correct health state of a node<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: ad8fb18517e31bc3878b13319c428b27c7279c16</p> <p>The test can go into sanity.sh. It should be enough to have a simple test that checks for "healthy" visible on all nodes and all services, AND that "NOT HEALTHY" is not present. </p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/19537/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/19537/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8017" title="All Nodes report NOT HEALTHY, system is healthy" class="issue-link" data-issue-key="LU-8017"><del>LU-8017</del></a> obd: report correct health state of a node<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: c28933602a6971739cb5ec3a1e920409ff19b01e</p> <p>Patch has landed.</p> <p>running a client build that has the new additional check added in test-framework.sh by this fix on a server that has the old problem about returning incorrect health status causes failures from test-framework nearly everywhere. do we need some version test around the health check in test-framework to avoid phony failures in interop?</p> <p>Do you have a example log of the failure? Also what does lctl get_param health_check show?</p> <p>Bob, which specific versions are you testing? I thought this only affected 2.8.51 and was fixed in 2.8.52, and we typically do not test interop between point releases during development? Was the 16933 patch backported to some maintenance branch? In that case, the right answer is to also backport 19537 to that same branch, or any HA system that checks <tt>health_check</tt> will fail.</p> <p>Andreas,<br/> client is from master, v2.8.52 built yesterday. It has the <a href="https://jira.whamcloud.com/browse/LU-8017" title="All Nodes report NOT HEALTHY, system is healthy" class="issue-link" data-issue-key="LU-8017"><del>LU-8017</del></a> fix.<br/> servers are also from master, but older. also v2.8.52, but don't have the fix.</p> <p>James,<br/> example error from runtests:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>subsystem_debug=all -lnet -lnd -pinger Setup mgs, mdt, osts Starting mds1: /dev/sdb /mnt/mds1 runtests test_1: @@@@@@ FAIL: mds1 is in a unhealthy state Trace dump: = /usr/lib64/lustre/tests/test-framework.sh:4769:error() = /usr/lib64/lustre/tests/test-framework.sh:1281:mount_facet() = /usr/lib64/lustre/tests/test-framework.sh:1344:start() = /usr/lib64/lustre/tests/test-framework.sh:3649:setupall() = /usr/lib64/lustre/tests/runtests:90:test_1() = /usr/lib64/lustre/tests/test-framework.sh:5033:run_one() = /usr/lib64/lustre/tests/test-framework.sh:5072:run_one_logged() = /usr/lib64/lustre/tests/test-framework.sh:4919:run_test() = /usr/lib64/lustre/tests/runtests:135:main() Dumping lctl log to /tmp/test_logs/2016-05-10/154232/runtests.test_1.*.1462920188.log Resetting fail_loc on all nodes...done. FAIL 1 (32s) </pre> </div></div> <p>lctl get_param health_check on servers show:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># lctl get_param health_check health_check=healthy NOT HEALTHY </pre> </div></div> <p>Bob, in that case it wouldn't even be <em>possible</em> to have a version check, even if we did that for development versions, since they both have the same version number. I would just update the old nodes and move on. </p> <p>I can avoid the failure by commenting or deleting out or deleting the new health test check in test-framework.sh on the client.<br/> Pretty sure I could also avoid the fail by installing a newer build on servers too.<br/> Wasn't sure how exposed we are to hitting this sort of fail in general, in interop between master and other versions.</p> <p>Andreas, I will take your advice.</p> <p>Thankfully the window of failure with broken server version is very small.</p> <p>Shall we close this ticket again?</p> Related LU-8066 Development Rank 1|hzy7xb: Rank (Obsolete) 9223372036854775807 Severity