https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-5674 Lustre <p>If I haven't missed something, zfs debugging data hasn't been included in test reports, e.g.:<br/> <a href="https://maloo.whamcloud.com/test_sets/42573266-9f17-11e3-934b-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/42573266-9f17-11e3-934b-52540035b04c</a></p> <p>It'd be very useful to have a tarball of /proc/spl/. Lots of useful data to troubleshoot ZFS problems can be found under that directory, e.g. dmu_tx_assign delay histogram.</p> LU-5674

Maloo test report should include zfs debugging data when when FSTYPE=zfs

Improvement Critical Resolved Fixed Minh Diep Isaac Huang prz triaged Fri, 14 Mar 2014 21:31:47 +0000 Wed, 16 Mar 2016 13:37:03 +0000 Wed, 15 Jul 2015 15:41:18 +0000 Lustre 2.7.0 Lustre 2.5.4 0 7 <p>Some of the test failures (e.g. <a href="https://jira.whamcloud.com/browse/LU-4716" title="replay-ost-single test_5: stuck in dbuf_read-&gt;zio_wait" class="issue-link" data-issue-key="LU-4716"><del>LU-4716</del></a>) we've seen could be the known ZFS IO starvation issue with the Linux cfq scheduler. Although ZFS automatically set IO scheduler to noop on whole disks, the host OS could still be using cfq for the disks behind the guess OS disks. It seemed that ZFS pools on OSS were set up:<br/> zpool import -f -o cachefile=none -d /dev/lvm-Role_OSS lustre-ost3<br/> It was not clear to me how devices under /dev/lvm-Role_OSS were setup and used. I think it makes sense to make sure that our test system:<br/> 1. Use whole disks for zfs pools on guest VMs.<br/> 2. Use noop IO scheduler for corresponding disks on host OS.</p> <p>I would like to understand this a little better.</p> <p>At the end of an autotest run with ZFS autotest could just include the content of /proc/spl/... in the tar ball it sends over to maloo, and maloo could make it available to download. (this is how I read the original request - Mike) <b>How much data is involved, and would that data in isolation be useful?</b></p> <p>Would this be sufficient for a start, or do we need to include more information (for example the kernel which was running)? </p> <p>Yes, a tgz of /proc/spl/ should be sufficient. And it's necessary only for failed tests. Everything under /proc/spl/ is text, when compressed the total size shouldn't be large.</p> <p>My take on this is we should start this in Lustre test-framework and generate such file under ZFS tests. After that we can see if autotest automatically grab the files send to maloo.</p> <p>Another piece of vital debug information to collect is outputs from "zpool events -v", available since zfs 0.6.3.</p> <p>Recently in debugging a few ZFS issues (e.g. <a href="https://jira.whamcloud.com/browse/LU-4950" title="sanity-benchmark test fsx hung: txg_sync was stuck on OSS" class="issue-link" data-issue-key="LU-4950"><del>LU-4950</del></a> and <a href="https://jira.whamcloud.com/browse/LU-5242" title="Test hang sanity test_132, test_133: umount ost" class="issue-link" data-issue-key="LU-5242"><del>LU-5242</del></a>) I found it very difficult to troubleshoot without such information - basically the only thing I had was just some stack dumps.</p> <p>Please also set ZFS module option on MDS and OSS: zfs_txg_history=3</p> <p>Without the option, some debug information will not be exported in the proc file.</p> <p><a href="http://review.whamcloud.com/#/c/11580/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/11580/</a></p> <p>Landed for 2.7</p> <p>Here is the back-ported patch for Lustre b2_5 branch: <a href="http://review.whamcloud.com/12590" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/12590</a></p> <p>Looks like ZFS info is missing from this report:<br/> <a href="https://testing.hpdd.intel.com/test_sets/9e3a7c26-769b-11e4-ad19-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/9e3a7c26-769b-11e4-ad19-5254006e85c2</a></p> <p>Or have I missed something?</p> <p>the test that you mentioned doesn't follow the test-framework way of start test. this results in zfs log was not called. additional, the test timed out which could also mean that the log would not be collect at the end of the client crashed.</p> <p>Did you mean that even if the test I mentioned had failed a different way (i.e. not a timeout, so it'd be possible to collect to logs) the zfs logs would still not be collected? If yes, does it apply to all Maloo tests triggered from Gerrit?</p> <p>sorry for not being clear. after looking at this I think it likely that the test timed out caused the zfs log to not collected.</p> <p>Please if you find a case where a test failed but not log, please open a new ticket instead of reopen this. I believe this enhancement is completed.</p> <p>When you said "the test timed out caused the zfs log to not collected", did you mean:</p> <ul class="alternate" type="square"> <li>on test time out, the scripts would not try to collect the ZFS logs, or</li> <li>it tries to get the logs, but servers wouldn't respond</li> </ul> <p>In the report I mentioned above, the OSS was in good state; there was a deadlock on the MDS, which made some service threads unresponsive, but user space process should still work. In addition, dmesg and Lustre debug logs were all available for both the OSS and the MDS, then why wasn't the ZFS logs available as well?</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/12590/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/12590/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-5674" title="Maloo test report should include zfs debugging data when when FSTYPE=zfs" class="issue-link" data-issue-key="LU-5674"><del>LU-5674</del></a> test: print spl debug info<br/> Project: fs/lustre-release<br/> Branch: b2_5<br/> Current Patch Set: <br/> Commit: f3ecfa69ecbfaa3e28b50c2849ffc99ca6bebf6a</p> <p><a href="https://testing.hpdd.intel.com/test_sets/deca9712-9bc1-11e4-857a-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/deca9712-9bc1-11e4-857a-5254006e85c2</a></p> <p>In the test report above, I couldn't find any ZFS data requested here. Since dmesg and other data that'd require a working user space were all there, I'd believe that the ZFS data should be available as well. Please take a look - the missing of such data made it harder to debug. Thanks!</p> <p>since the test has timed out, there isn't any way to collect the zfs log</p> <p>this ticket already in 2.7..etc. we can not include any log if a node is crashed (ie timeout). this ticket should be closed.</p> Development Rank 1|hzupy7: Rank (Obsolete) 1854