Whamcloud Community JIRA

Whamcloud Community JIRA https://jira.whamcloud.com This file is an XML representation of an issue en-us

9.4.14

940014

05-12-2023

[LU-1926] Reboots during test runs https://jira.whamcloud.com/browse/LU-1926 Lustre <p>See </p> <p><a href="https://maloo.whamcloud.com/test_sessions/9ddf73ba-f23d-11e1-87d6-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sessions/9ddf73ba-f23d-11e1-87d6-52540035b04c</a><br/> <a href="https://maloo.whamcloud.com/test_sessions/94e74180-f296-11e1-807d-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sessions/94e74180-f296-11e1-807d-52540035b04c</a></p> LU-1926

Reboots during test runs

Bug Blocker Resolved Duplicate Jian Yu Peter Jones releases Thu, 30 Aug 2012 12:34:03 +0000 Wed, 31 Jul 2013 15:19:49 +0000 Thu, 13 Sep 2012 08:31:02 +0000 0 4 <p>I have no idea, we don't even know for sure if some other entity caused the reboots. Nothing in the logs suggests that autotest rebooted the nodes.</p> <p>We shall just have to monitor the situation.</p> <p>Didn't you isolate and fix this one?</p> <p>Fixed by TT-852 and upgrade of the weekend 1/9/2012</p> <p>It looks like the MDS reboot for some reason<br/> <a href="https://maloo.whamcloud.com/test_sessions/f50e5332-f6d3-11e1-b320-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sessions/f50e5332-f6d3-11e1-b320-52540035b04c</a></p> <p>Could you provide details of where the reboot was, the traces are long and the test looks not particularly successful. As you have already found the unexpected MDS reboots in the logs it would be help to pass that information on to me.</p> <p>Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b2_3/16" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_3/16</a></p> <p>parallel-scale-nfsv4 test compilebench timed out: <a href="https://maloo.whamcloud.com/test_sets/f7a291b0-fa72-11e1-a03c-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/f7a291b0-fa72-11e1-a03c-52540035b04c</a></p> <p>There is no error message in the test output.</p> <p>Console log on Client 2 (client-28vm2) showed that:</p> <div class="panel" style="border-width: 1px;"><div class="panelContent"> <p>02:18:51:Lustre: DEBUG MARKER: == parallel-scale-nfsv4 test compilebench: compilebench == 02:18:50 (1347182330)<br/> 02:18:51:Lustre: DEBUG MARKER: /usr/sbin/lctl mark .\/compilebench -D \/mnt\/lustre\/d0.compilebench -i 2 -r 2 --makej<br/> 02:18:51:Lustre: DEBUG MARKER: ./compilebench -D /mnt/lustre/d0.compilebench -i 2 -r 2 --makej<br/> <font color="red">02:30:43:nfs: server client-28vm7 not responding, still trying</font><br/> 03:18:59:********** Timeout by autotest system **********03:19:54:</p> </div></div> <p>Console log on MDS (client-28vm7) showed that:</p> <div class="panel" style="border-width: 1px;"><div class="panelContent"> <p>02:18:56:Lustre: DEBUG MARKER: == parallel-scale-nfsv4 test compilebench: compilebench == 02:18:50 (1347182330)<br/> 02:18:56:Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400):0:mdt<br/> 02:18:56:Lustre: DEBUG MARKER: /usr/sbin/lctl mark .\/compilebench -D \/mnt\/lustre\/d0.compilebench -i 2 -r 2 --makej<br/> 02:18:56:Lustre: DEBUG MARKER: ./compilebench -D /mnt/lustre/d0.compilebench -i 2 -r 2 --makej<br/> 03:18:59:********** Timeout by autotest system **********03:20:39:</p> </div></div> <p>The MDS (client-28vm7) was not responding during the test, however there was no error message in its console log. Now, let's take a look at its syslong from /scratch/logs/syslog/client-28vm7.log:</p> <div class="panel" style="border-width: 1px;"><div class="panelContent"> <p>Sep 9 02:18:51 client-28vm7 kernel: Lustre: DEBUG MARKER: ./compilebench -D /mnt/lustre/d0.compilebench -i 2 -r 2 --makej<br/> Sep 9 02:18:51 client-28vm7 rshd<span class="error">[5258]</span>: pam_unix(rsh:session): session closed for user root<br/> Sep 9 02:18:51 client-28vm7 xinetd<span class="error">[1661]</span>: EXIT: shell status=0 pid=5258 duration=0(sec)<br/> <font color="red">Sep 9 02:19:38 client-28vm7 ntpd<span class="error">[1669]</span>: synchronized to 10.10.0.2, stratum 3</font><br/> Sep 9 02:19:38 client-28vm7 ntpd<span class="error">[1669]</span>: time reset -0.393295 s<br/> Sep 9 02:19:38 client-28vm7 ntpd<span class="error">[1669]</span>: kernel time sync status change 2001<br/> Sep 9 02:24:25 client-28vm7 ntpd<span class="error">[1669]</span>: synchronized to 10.10.0.1, stratum 2<br/> <font color="red">Sep 9 03:21:09 client-28vm7 kernel: MTRR variable ranges enabled:</font><br/> Sep 9 03:21:09 client-28vm7 kernel: 0 base 00E0000000 mask FFE0000000 uncachable<br/> Sep 9 03:21:09 client-28vm7 kernel: 1 disabled<br/> Sep 9 03:21:09 client-28vm7 kernel: 2 disabled<br/> Sep 9 03:21:09 client-28vm7 kernel: 3 disabled<br/> Sep 9 03:21:09 client-28vm7 kernel: 4 disabled<br/> Sep 9 03:21:09 client-28vm7 kernel: 5 disabled<br/> Sep 9 03:21:09 client-28vm7 kernel: 6 disabled<br/> Sep 9 03:21:09 client-28vm7 kernel: 7 disabled<br/> Sep 9 03:21:09 client-28vm7 kernel: PAT not supported by CPU.<br/> Sep 9 03:21:09 client-28vm7 kernel: initial memory mapped : 0 - 20000000</p> </div></div> <p>As we can see, before the MDS node was rebooted on 03:21:09, there was no error message on both the console and syslog. The only culprit is the ntpd run on MDS node during the test. Will check more test reports associated with this ticket.</p> <p>More instances on b2_3 with the same issue:<br/> <a href="https://maloo.whamcloud.com/test_sets/dc68d638-fa73-11e1-887d-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/dc68d638-fa73-11e1-887d-52540035b04c</a><br/> <a href="https://maloo.whamcloud.com/test_sets/8ec57d46-fa73-11e1-887d-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/8ec57d46-fa73-11e1-887d-52540035b04c</a><br/> <a href="https://maloo.whamcloud.com/test_sets/891ac004-fa73-11e1-887d-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/891ac004-fa73-11e1-887d-52540035b04c</a><br/> <a href="https://maloo.whamcloud.com/test_sets/7c320096-fa73-11e1-887d-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/7c320096-fa73-11e1-887d-52540035b04c</a></p> <p>performance-sanity: <a href="https://maloo.whamcloud.com/test_sets/7c320096-fa73-11e1-887d-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/7c320096-fa73-11e1-887d-52540035b04c</a></p> <p>The above report showed that the MDS (client-30vm3) was out of service during the test. Its syslog showed that:</p> <div class="panel" style="border-width: 1px;"><div class="panelContent"> <p><font color="red">Sep 8 20:30:43 client-30vm3 kernel: Lustre: DEBUG MARKER: ===== mdsrate-create-small.sh</font><br/> Sep 8 20:30:43 client-30vm3 rshd<span class="error">[23422]</span>: pam_unix(rsh:session): session closed for user root<br/> Sep 8 20:30:43 client-30vm3 xinetd<span class="error">[1549]</span>: EXIT: shell status=0 pid=23422 duration=0(sec)<br/> <font color="red">Sep 8 20:31:33 client-30vm3 kernel: MTRR variable ranges enabled:</font><br/> Sep 8 20:31:33 client-30vm3 kernel: 0 base 00E0000000 mask FFE0000000 uncachable<br/> Sep 8 20:31:33 client-30vm3 kernel: 1 disabled<br/> Sep 8 20:31:33 client-30vm3 kernel: 2 disabled<br/> Sep 8 20:31:33 client-30vm3 kernel: 3 disabled<br/> Sep 8 20:31:33 client-30vm3 kernel: 4 disabled<br/> Sep 8 20:31:33 client-30vm3 kernel: 5 disabled<br/> Sep 8 20:31:33 client-30vm3 kernel: 6 disabled<br/> Sep 8 20:31:33 client-30vm3 kernel: 7 disabled<br/> Sep 8 20:31:33 client-30vm3 kernel: PAT not supported by CPU.<br/> Sep 8 20:31:33 client-30vm3 kernel: initial memory mapped : 0 - 20000000</p> </div></div> <p>The MDS node was really rebooted with unknown reason during the test.</p> <p>I'm looking at somemore that look similar, we now have stack traces etc.</p> <p>1////</p> <p><a href="https://maloo.whamcloud.com/test_sessions/0b9ab03c-fbed-11e1-a4a6-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sessions/0b9ab03c-fbed-11e1-a4a6-52540035b04c</a></p> <p>Conf Sanity reboots MDS </p> <p><a href="https://maloo.whamcloud.com/test_logs/7b2c562e-fbef-11e1-a4a6-52540035b04c/show_text" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_logs/7b2c562e-fbef-11e1-a4a6-52540035b04c/show_text</a></p> <p>but prior to the reboot we see an assertion</p> <p>21:29:47:LustreError: 32529:0:(obd_class.h:527:obd_set_info_async()) obd_set_info_async: dev 0 no operation<br/> 21:29:48:LustreError: 402:0:(mgs_internal.h:283:mgs_env_info()) ASSERTION( info != ((void *)0) ) failed: <br/> 21:29:48:LustreError: 402:0:(mgs_internal.h:283:mgs_env_info()) LBUG<br/> 21:29:48:Pid: 402, comm: llog_process_th</p> <p>2////</p> <p><a href="https://maloo.whamcloud.com/test_sessions/a41f27a2-fbde-11e1-bde2-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sessions/a41f27a2-fbde-11e1-bde2-52540035b04c</a></p> <p>same on conf-sanity</p> <p><a href="https://maloo.whamcloud.com/test_logs/2e3dc9cc-fbe2-11e1-bde2-52540035b04c/show_text" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_logs/2e3dc9cc-fbe2-11e1-bde2-52540035b04c/show_text</a></p> <p>again a reboot, but an assertion before that</p> <p>20:16:48:Lustre: MGS: Client 9711f524-c291-d894-f430-704c69e8e03d (at 0@lo) reconnecting<br/> 20:16:48:LustreError: 32139:0:(obd_class.h:527:obd_set_info_async()) obd_set_info_async: dev 0 no operation<br/> 20:16:48:LustreError: 32478:0:(mgs_internal.h:288:mgs_env_info()) ASSERTION( info != ((void *)0) ) failed: <br/> 20:16:48:LustreError: 32478:0:(mgs_internal.h:288:mgs_env_info()) LBUG<br/> 20:16:48:Pid: 32478, comm: llog_process_th</p> <p>same with the other failures this morning, I can't see why the previous did not have the LBUG but I can see any that replicate it.</p> <p>I'm going to create a ticket for the bugs I found this morning.</p> <p>Of the 4 examples of Yu Jian,</p> <p>1st<br/> <a href="https://maloo.whamcloud.com/test_sets/dc68d638-fa73-11e1-887d-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/dc68d638-fa73-11e1-887d-52540035b04c</a></p> <p>Timeouts correctly and reboot, the timeout message can be seen. Strangly the reboot is still captured after the timeout but the timeout is correct.</p> <p>02:10:44:Lustre: DEBUG MARKER: == parallel-scale-nfsv4 test compilebench: compilebench == 02:10:36 (1347181836)<br/> 02:10:44:Lustre: DEBUG MARKER: /usr/sbin/lctl mark .\/compilebench -D \/mnt\/lustre\/d0.compilebench -i 2 -r 2 --makej<br/> 02:10:44:Lustre: DEBUG MARKER: ./compilebench -D /mnt/lustre/d0.compilebench -i 2 -r 2 --makej<br/> 02:21:56:nfs: server client-30vm3 not responding, still trying<br/> 03:10:59:********** Timeout by autotest system **********03:11:08:<br/> 03:11:08:<ConMan> Console <span class="error">[client-30vm6]</span> disconnected from <client-30:6005> at 09-09 03:11.<br/> 03:11:29:<br/> 03:11:29:<ConMan> Console <span class="error">[client-30vm6]</span> connected to <client-30:6005> at 09-09 03:11.<br/> 03:11:29:<br/> Press any key to continue.<br/> 03:11:29:</p> <p>2nd<br/> <a href="https://maloo.whamcloud.com/test_sets/8ec57d46-fa73-11e1-887d-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/8ec57d46-fa73-11e1-887d-52540035b04c</a></p> <p>Timeouts correctly and reboot, the timeout message can be seen. Strangly the reboot is still captured after the timeout but the timeout is correct.</p> <p>00:02:38:Lustre: 2964:0:(client.c:1917:ptlrpc_expire_one_request()) Skipped 151 previous similar messages<br/> 00:12:49:Lustre: 2964:0:(client.c:1917:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: <span class="error">[sent 1347174744/real 1347174747]</span> req@ffff880076483000 x1412612131656402/t0(0) o250->MGC10.10.4.182@tcp@10.10.4.182@tcp:26/25 lens 400/544 e 0 to 1 dl 1347174769 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1<br/> 00:12:49:Lustre: 2964:0:(client.c:1917:ptlrpc_expire_one_request()) Skipped 146 previous similar messages<br/> 00:23:01:Lustre: 2964:0:(client.c:1917:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: <span class="error">[sent 1347175374/real 1347175374]</span> req@ffff880076483000 x1412612131657436/t0(0) o250->MGC10.10.4.182@tcp@10.10.4.182@tcp:26/25 lens 400/544 e 0 to 1 dl 1347175399 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1<br/> 00:23:01:Lustre: 2964:0:(client.c:1917:ptlrpc_expire_one_request()) Skipped 151 previous similar messages<br/> 00:26:58:********** Timeout by autotest system **********00:28:03:<br/> 00:28:03:<ConMan> Console <span class="error">[client-30vm6]</span> disconnected from <client-30:6005> at 09-09 00:27.<br/> 00:28:24:<br/> 00:28:24:<ConMan> Console <span class="error">[client-30vm6]</span> connected to <client-30:6005> at 09-09 00:28.<br/> 00:28:24:<br/> Press any key to continue.<br/> 00:28:24:</p> <p>3rd <br/> <a href="https://maloo.whamcloud.com/test_sets/dc68d638-fa73-11e1-887d-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/dc68d638-fa73-11e1-887d-52540035b04c</a></p> <p>Timeouts correctly and reboot, the timeout message can be seen. Strangly the reboot is still captured after the timeout but the timeout is correct.</p> <p>02:10:44:Lustre: DEBUG MARKER: == parallel-scale-nfsv4 test compilebench: compilebench == 02:10:36 (1347181836)<br/> 02:10:44:Lustre: DEBUG MARKER: /usr/sbin/lctl mark .\/compilebench -D \/mnt\/lustre\/d0.compilebench -i 2 -r 2 --makej<br/> 02:10:44:Lustre: DEBUG MARKER: ./compilebench -D /mnt/lustre/d0.compilebench -i 2 -r 2 --makej<br/> 02:21:56:nfs: server client-30vm3 not responding, still trying<br/> 03:10:59:********** Timeout by autotest system **********03:11:08:<br/> 03:11:08:<ConMan> Console <span class="error">[client-30vm6]</span> disconnected from <client-30:6005> at 09-09 03:11.<br/> 03:11:29:<br/> 03:11:29:<ConMan> Console <span class="error">[client-30vm6]</span> connected to <client-30:6005> at 09-09 03:11.<br/> 03:11:29:<br/> Press any key to continue.</p> <p>4th<br/> <a href="https://maloo.whamcloud.com/test_sessions/757ba820-fb85-11e1-8e05-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sessions/757ba820-fb85-11e1-8e05-52540035b04c</a></p> <p>Timeouts correctly and reboot, the timeout message can be seen. Strangly the reboot is still captured after the timeout but the timeout is correct.</p> <p>02:10:44:Lustre: DEBUG MARKER: == parallel-scale-nfsv4 test compilebench: compilebench == 02:10:36 (1347181836)<br/> 02:10:44:Lustre: DEBUG MARKER: /usr/sbin/lctl mark .\/compilebench -D \/mnt\/lustre\/d0.compilebench -i 2 -r 2 --makej<br/> 02:10:44:Lustre: DEBUG MARKER: ./compilebench -D /mnt/lustre/d0.compilebench -i 2 -r 2 --makej<br/> 02:21:56:nfs: server client-30vm3 not responding, still trying<br/> 03:10:59:********** Timeout by autotest system **********03:11:08:<br/> 03:11:08:<ConMan> Console <span class="error">[client-30vm6]</span> disconnected from <client-30:6005> at 09-09 03:11.<br/> 03:11:29:<br/> 03:11:29</p> <p>An issue did exist for a short while where the console log was not being shut before the report. This lead to the reboot appearing in the test log when the test was being correctly timedout.</p> <p>I'm closing this issue as this is I beleive not resolved.</p> <p>we need collect all relevent logs before rebooting the node. Or else we know nothing about the issue.</p> <blockquote><p>same with the other failures this morning, I can't see why the previous did not have the LBUG but I can see any that replicate it.</p></blockquote> <p>The two conf-sanity failures are caused by <a href="http://review.whamcloud.com/#change,3671" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,3671</a> and <a href="http://review.whamcloud.com/#change,3670" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,3670</a>, which are still under development.</p> <p>The failure instances I reported in this ticket are on the main b2_3 branch. There is no such LBUG.</p> <p>The failures are real timeouts is my point. The reboot happens after the test has timed out.</p> <p>and we do capture the logs, autotest was restarted because it had stopped capturing logs.</p> <p>Hi Chris,<br/> Please take a look at my above comments on the syslog of performance-sanity: <a href="https://maloo.whamcloud.com/test_sets/7c320096-fa73-11e1-887d-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/7c320096-fa73-11e1-887d-52540035b04c</a>, the MDS (client-30vm3) was rebooted before timeout.</p> <p>BTW, autotest did not gather syslogs for the above reports, I had to find out the syslogs on brent:/scratch/logs/syslog.</p> <p>Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b2_3/17" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_3/17</a></p> <p>The issue still exists:<br/> parallel-scale-nfsv4 test_compilebench: <a href="https://maloo.whamcloud.com/test_sets/f2b8c2b8-fc85-11e1-a4a6-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/f2b8c2b8-fc85-11e1-a4a6-52540035b04c</a><br/> parallel-scale-nfsv3 test_compilebench: <a href="https://maloo.whamcloud.com/test_sets/d241d4ca-fc85-11e1-a4a6-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/d241d4ca-fc85-11e1-a4a6-52540035b04c</a><br/> parallel-scale test_compilebench: <a href="https://maloo.whamcloud.com/test_sets/3b4a8f4e-fc85-11e1-a4a6-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/3b4a8f4e-fc85-11e1-a4a6-52540035b04c</a><br/> large-scale test_3a: <a href="https://maloo.whamcloud.com/test_sets/733bf24e-fc85-11e1-a4a6-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/733bf24e-fc85-11e1-a4a6-52540035b04c</a></p> <p>BTW, all of the syslogs in the above reports are empty. I checked the syslogs from brent node but still found nothing useful for debugging.</p> <p>However, comparing to the results on b2_3 build #16, although performance-saniy test_3 and sanity test_32n also hit MDS reboot issue, there are error messages on the MDS console logs on build #17 (no such messages on build #16), please refer to <a href="https://jira.whamcloud.com/browse/LU-1906" title="performance-sanity subtest test_3: Oops: RIP: __find_get_block_slow+0x87/0x130" class="issue-link" data-issue-key="LU-1906"><del>LU-1906</del></a>, <a href="https://jira.whamcloud.com/browse/LU-1909" title="performance-sanity subtest test_3: Oops: RIP: __do_softirq+0x73/0x1e0" class="issue-link" data-issue-key="LU-1909"><del>LU-1909</del></a> and <a href="https://jira.whamcloud.com/browse/LU-1863" title="Test failure with MDS spontaneous rebooting (test suite sanity, subtest test_32n)" class="issue-link" data-issue-key="LU-1863"><del>LU-1863</del></a>.</p> <p>So, I'm not sure whether the above parallel-scale* and large-scale failures were caused by Lustre issues or not although there were no specific error messages on their logs.</p> <p>Oleg, Bobi, do you have any ideas about whether the above MDS reboot issue is a Lustre issue or test environment issue? Thanks.<br/> This is really blocking the Lustre b2_3 testing now.</p> <p>From Skype</p> <p><span class="error">[13/09/2012 08:21:29]</span> Yu Jian</p> <p>aha, parallel-scale test compilebench hung in manual run, the MDS is being rebooted<br/> more logs occurr<br/> ok, Chris, <a href="https://jira.whamcloud.com/browse/LU-1926" title="Reboots during test runs" class="issue-link" data-issue-key="LU-1926"><del>TT-851</del></a> is not a test environment issue, it's a Lustre issue, I'll gather logs, file new ticket and update <a href="https://jira.whamcloud.com/browse/LU-1926" title="Reboots during test runs" class="issue-link" data-issue-key="LU-1926"><del>TT-851</del></a><br/> thank you, Chris</p> <p>Just assigning this to you for clarity.</p> <p>Chris, I'm still creating new LU tickets per the manual test runs. After all of the tests are done and the corresponding LU tickets are created, we can close this one.</p> <p>The remaining issue for TT is that the Oops on consoles were not gathered, and syslogs on Maloo were empty.</p> <p>After running those failed tests manually, it turned out to be Lustre failures:</p> <p>Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b2_3/17" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_3/17</a></p> <p>parallel-scale-nfsv4 test_compilebench: <a href="https://jira.whamcloud.com/browse/LU-1928" title="parallel-scale-nfsv3 subtest test_compilebench: Oops: RIP: put_page+0x9/0x40" class="issue-link" data-issue-key="LU-1928"><del>LU-1928</del></a><br/> parallel-scale-nfsv3 test_compilebench: <a href="https://jira.whamcloud.com/browse/LU-1928" title="parallel-scale-nfsv3 subtest test_compilebench: Oops: RIP: put_page+0x9/0x40" class="issue-link" data-issue-key="LU-1928"><del>LU-1928</del></a><br/> parallel-scale test_compilebench: <a href="https://jira.whamcloud.com/browse/LU-1925" title="kernel tried to execute NX-protected page - exploit attempt? (uid: 0)" class="issue-link" data-issue-key="LU-1925"><del>LU-1925</del></a><br/> large-scale test_3a: <a href="https://jira.whamcloud.com/browse/LU-1927" title="large-scale subtest test_3a: Oops: RIP: _spin_lock+0xe/0x30" class="issue-link" data-issue-key="LU-1927"><del>LU-1927</del></a><br/> performance-saniy test_3: <a href="https://jira.whamcloud.com/browse/LU-1906" title="performance-sanity subtest test_3: Oops: RIP: __find_get_block_slow+0x87/0x130" class="issue-link" data-issue-key="LU-1906"><del>LU-1906</del></a>, <a href="https://jira.whamcloud.com/browse/LU-1909" title="performance-sanity subtest test_3: Oops: RIP: __do_softirq+0x73/0x1e0" class="issue-link" data-issue-key="LU-1909"><del>LU-1909</del></a>, <a href="https://jira.whamcloud.com/browse/LU-1929" title="performance-sanity subtest test_3: list_add corruption" class="issue-link" data-issue-key="LU-1929"><del>LU-1929</del></a><br/> sanity test_32n: <a href="https://jira.whamcloud.com/browse/LU-1863" title="Test failure with MDS spontaneous rebooting (test suite sanity, subtest test_32n)" class="issue-link" data-issue-key="LU-1863"><del>LU-1863</del></a><br/> Console log and syslog missing issue: TT-875</p> Duplicate LU-1863 Business Value 0.1 Development Rank 1|hzus4v: Rank (Obsolete) 2220 Severity Story Points 1.0