[LU-8017] All Nodes report NOT HEALTHY, system is healthy

• Status: Resolved
• Project: Lustre
• Affects Version/s: Lustre 2.9.0
• Fix Version/s: Lustre 2.9.0
• Type: Bug
• Priority: Critical
• Reporter: Cliff White (Inactive)
• Assignee: James A Simmons
• Resolution: Fixed
• Labels: soak
• Severity: 3

Description

Current build installed; https://build.hpdd.intel.com/job/lustre-reviews/38245/
This issue has persisted for the last two builds.
After mounting the filesystem, all nodes report NOT HEALTHY in /proc/fs/lustre/health_check.

1. pdsh -g server 'lctl get_param health_check' |dshbak -c
   ----------------
   lola-[2-11]
   ----------------
   health_check=healthy
   NOT HEALTHY

The filesystem otherwise operates normally, jobs run, results are created.
We were using the health_check as part of our monitoring - this has been discontinued.
We are uncertain as to the cause, as all operations we can test work fine, and no errors are reported.

Issue Links

• is related to: LU-8066 Move lustre procfs handling to sysfs ...

Comments

[Andreas Dilger]

This looks like a bug introduced by http://review.whamcloud.com/16933 "LU-6215 lprocfs: handle seq_printf api change".

lustre/obdclass/linux/linux-module.c
@@ -275,7 +277,7 @@ static int obd_proc_health_seq_show(struct seq_file *m, void *data)
        read_unlock(&obd_dev_lock);
 
        if (healthy)
-               return seq_printf(m, "healthy\n");
+               seq_puts(m, "healthy\n");
 
        seq_printf(m, "NOT HEALTHY\n");
        return 0;

It should still have returned after printing "healthy" instead of continuing to "NOT HEALTHY".

        if (healthy) {
               seq_puts(m, "healthy\n");
               return;
        }

[Di Wang]

Even in a healthy environment, it still show "NOT HEALTHY"

[root@testnode tests]# MDSCOUNT=4 sh llmount.sh 
Stopping clients: testnode /mnt/lustre (opts:)
Stopping clients: testnode /mnt/lustre2 (opts:)
Loading modules from /work/lustre-new/lustre-release/lustre/tests/..
........
[root@testnode tests]# cat /proc/fs/lustre/health_check 
healthy
NOT HEALTHY

I checked the code, and it looks like a typo in the code

static int obd_proc_health_seq_show(struct seq_file *m, void *data)
{
    ............
        if (healthy)
                seq_puts(m, "healthy\n");
                                     ---------------------------> probably else is missing here.
        seq_printf(m, "NOT HEALTHY\n");

[Cliff White (Inactive)]

This also indicates a test miss, since we should check health_check in auto test.

[Andreas Dilger]

I'd be fine with an "else" also.

Please also add a test that this is working properly.

[Cliff White (Inactive)]

I think QA team can do this.

[James A Simmons]

Oops, missed fixing up a sed change. Will fix. Sorry I didn't add a test with this patch but it is a really good idea. This way we can see if the upstream client will also behave properly. Which test should it go into?

[Gerrit Updater]

James Simmons ([e-mail]) uploaded a new patch: http://review.whamcloud.com/19537
Subject: LU-8017 obd: report correct health state of a node
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: ad8fb18517e31bc3878b13319c428b27c7279c16

[Andreas Dilger]

The test can go into sanity.sh. It should be enough to have a simple test that checks for "healthy" visible on all nodes and all services, AND that "NOT HEALTHY" is not present.

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch http://review.whamcloud.com/19537/
Subject: LU-8017 obd: report correct health state of a node
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: c28933602a6971739cb5ec3a1e920409ff19b01e

[James A Simmons]

Patch has landed.

[Bob Glossman (Inactive)]

running a client build that has the new additional check added in test-framework.sh by this fix on a server that has the old problem about returning incorrect health status causes failures from test-framework nearly everywhere. do we need some version test around the health check in test-framework to avoid phony failures in interop?

[James A Simmons]

Do you have a example log of the failure? Also what does lctl get_param health_check show?

[Andreas Dilger]

Bob, which specific versions are you testing? I thought this only affected 2.8.51 and was fixed in 2.8.52, and we typically do not test interop between point releases during development? Was the 16933 patch backported to some maintenance branch? In that case, the right answer is to also backport 19537 to that same branch, or any HA system that checks health_check will fail.

[Bob Glossman (Inactive)]

Andreas,
client is from master, v2.8.52 built yesterday. It has the LU-8017 fix.
servers are also from master, but older. also v2.8.52, but don't have the fix.

James,
example error from runtests:

subsystem_debug=all -lnet -lnd -pinger
Setup mgs, mdt, osts
Starting mds1:   /dev/sdb /mnt/mds1
 runtests test_1: @@@@@@ FAIL: mds1 is in a unhealthy state 
  Trace dump:
  = /usr/lib64/lustre/tests/test-framework.sh:4769:error()
  = /usr/lib64/lustre/tests/test-framework.sh:1281:mount_facet()
  = /usr/lib64/lustre/tests/test-framework.sh:1344:start()
  = /usr/lib64/lustre/tests/test-framework.sh:3649:setupall()
  = /usr/lib64/lustre/tests/runtests:90:test_1()
  = /usr/lib64/lustre/tests/test-framework.sh:5033:run_one()
  = /usr/lib64/lustre/tests/test-framework.sh:5072:run_one_logged()
  = /usr/lib64/lustre/tests/test-framework.sh:4919:run_test()
  = /usr/lib64/lustre/tests/runtests:135:main()
Dumping lctl log to /tmp/test_logs/2016-05-10/154232/runtests.test_1.*.1462920188.log
Resetting fail_loc on all nodes...done.
FAIL 1 (32s)

lctl get_param health_check on servers show:

# lctl get_param health_check
health_check=healthy
NOT HEALTHY

[Andreas Dilger]

Bob, in that case it wouldn't even be possible to have a version check, even if we did that for development versions, since they both have the same version number. I would just update the old nodes and move on.

[Bob Glossman (Inactive)]

I can avoid the failure by commenting or deleting out or deleting the new health test check in test-framework.sh on the client.
Pretty sure I could also avoid the fail by installing a newer build on servers too.
Wasn't sure how exposed we are to hitting this sort of fail in general, in interop between master and other versions.

Andreas, I will take your advice.

[James A Simmons]

Thankfully the window of failure with broken server version is very small.

[James A Simmons]

Shall we close this ticket again?