[LU-5674] Maloo test report should include zfs debugging data when when FSTYPE=zfs

• Status: Resolved
• Project: Lustre
• Fix Version/s: Lustre 2.7.0, Lustre 2.5.4
• Type: Improvement
• Priority: Critical
• Reporter: Isaac Huang (Inactive)
• Assignee: Minh Diep
• Resolution: Fixed
• Labels: prz, triaged

Description

If I haven't missed something, zfs debugging data hasn't been included in test reports, e.g.:
https://maloo.whamcloud.com/test_sets/42573266-9f17-11e3-934b-52540035b04c

It'd be very useful to have a tarball of /proc/spl/. Lots of useful data to troubleshoot ZFS problems can be found under that directory, e.g. dmu_tx_assign delay histogram.

Comments

[Isaac Huang (Inactive)]

Some of the test failures (e.g. LU-4716) we've seen could be the known ZFS IO starvation issue with the Linux cfq scheduler. Although ZFS automatically set IO scheduler to noop on whole disks, the host OS could still be using cfq for the disks behind the guess OS disks. It seemed that ZFS pools on OSS were set up:
zpool import -f -o cachefile=none -d /dev/lvm-Role_OSS lustre-ost3
It was not clear to me how devices under /dev/lvm-Role_OSS were setup and used. I think it makes sense to make sure that our test system:
1. Use whole disks for zfs pools on guest VMs.
2. Use noop IO scheduler for corresponding disks on host OS.

[Mike Stok (Inactive)]

I would like to understand this a little better.

At the end of an autotest run with ZFS autotest could just include the content of /proc/spl/... in the tar ball it sends over to maloo, and maloo could make it available to download. (this is how I read the original request - Mike) How much data is involved, and would that data in isolation be useful?

Would this be sufficient for a start, or do we need to include more information (for example the kernel which was running)?

[Isaac Huang (Inactive)]

Yes, a tgz of /proc/spl/ should be sufficient. And it's necessary only for failed tests. Everything under /proc/spl/ is text, when compressed the total size shouldn't be large.

[Minh Diep]

My take on this is we should start this in Lustre test-framework and generate such file under ZFS tests. After that we can see if autotest automatically grab the files send to maloo.

[Isaac Huang (Inactive)]

Another piece of vital debug information to collect is outputs from "zpool events -v", available since zfs 0.6.3.

Recently in debugging a few ZFS issues (e.g. LU-4950 and LU-5242) I found it very difficult to troubleshoot without such information - basically the only thing I had was just some stack dumps.

[Isaac Huang (Inactive)]

Please also set ZFS module option on MDS and OSS: zfs_txg_history=3

Without the option, some debug information will not be exported in the proc file.

[Minh Diep]

http://review.whamcloud.com/#/c/11580/

[Peter Jones]

Landed for 2.7

[Jian Yu]

Here is the back-ported patch for Lustre b2_5 branch: http://review.whamcloud.com/12590

[Isaac Huang (Inactive)]

Looks like ZFS info is missing from this report:
https://testing.hpdd.intel.com/test_sets/9e3a7c26-769b-11e4-ad19-5254006e85c2

Or have I missed something?

[Minh Diep]

the test that you mentioned doesn't follow the test-framework way of start test. this results in zfs log was not called. additional, the test timed out which could also mean that the log would not be collect at the end of the client crashed.

[Isaac Huang (Inactive)]

Did you mean that even if the test I mentioned had failed a different way (i.e. not a timeout, so it'd be possible to collect to logs) the zfs logs would still not be collected? If yes, does it apply to all Maloo tests triggered from Gerrit?

[Minh Diep]

sorry for not being clear. after looking at this I think it likely that the test timed out caused the zfs log to not collected.

Please if you find a case where a test failed but not log, please open a new ticket instead of reopen this. I believe this enhancement is completed.

[Isaac Huang (Inactive)]

When you said "the test timed out caused the zfs log to not collected", did you mean:

• on test time out, the scripts would not try to collect the ZFS logs, or
• it tries to get the logs, but servers wouldn't respond

In the report I mentioned above, the OSS was in good state; there was a deadlock on the MDS, which made some service threads unresponsive, but user space process should still work. In addition, dmesg and Lustre debug logs were all available for both the OSS and the MDS, then why wasn't the ZFS logs available as well?

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch http://review.whamcloud.com/12590/
Subject: LU-5674 test: print spl debug info
Project: fs/lustre-release
Branch: b2_5
Current Patch Set:
Commit: f3ecfa69ecbfaa3e28b50c2849ffc99ca6bebf6a

[Isaac Huang (Inactive)]

https://testing.hpdd.intel.com/test_sets/deca9712-9bc1-11e4-857a-5254006e85c2

In the test report above, I couldn't find any ZFS data requested here. Since dmesg and other data that'd require a working user space were all there, I'd believe that the ZFS data should be available as well. Please take a look - the missing of such data made it harder to debug. Thanks!

[Minh Diep]

since the test has timed out, there isn't any way to collect the zfs log

[Minh Diep]

this ticket already in 2.7..etc. we can not include any log if a node is crashed (ie timeout). this ticket should be closed.