Whamcloud Community JIRA

Whamcloud Community JIRA https://jira.whamcloud.com This file is an XML representation of an issue en-us

9.4.14

940014

05-12-2023

[LU-457] MDS crash under DDN file stress tests https://jira.whamcloud.com/browse/LU-457 Lustre <p>Running the DDN file stress tests with tests spread across 60 clients, MDS panic'd, <br/> panic output attached. </p> Hyperion - LLNL Running chaos-4 , 2.6.18-238.12.1.el5_lustre.g266a955<br/> 1.8.6-RC3 build LU-457

MDS crash under DDN file stress tests

Bug Major Resolved Cannot Reproduce Hongchao Zhang Cliff White Thu, 23 Jun 2011 18:39:33 +0000 Fri, 13 Apr 2012 09:35:45 +0000 Fri, 13 Apr 2012 09:35:45 +0000 Lustre 1.8.6 0 4 <p>2011-06-23 14:55:52 Kernel BUG at mm/slab.c:3482<br/> 2011-06-23 14:55:52 Pid: 9565, comm: ldlm_cn_02 Tainted: G 2.6.18-238.12.1.el5_lustre.g266a955 #1<br/> 2011-06-23 14:55:52 RIP: 0010:<span class="error">[<ffffffff8000751c>]</span> <span class="error">[<ffffffff8000751c>]</span> kmem_cache_free+0x89/0x1dd<br/> 2011-06-23 14:55:52 RSP: 0018:ffff814026f15c80 EFLAGS: 00010207<br/> 2011-06-23 14:55:52 RAX: 0000000000000080 RBX: ffff81100104a840 RCX: 0000000000000200<br/> 2011-06-23 14:55:52 RDX: ffff810138039030 RSI: 0000000038039030 RDI: 0000000000000001<br/> 2011-06-23 14:55:52 RBP: ffff81102a4a9a00 R08: 0000000000000000 R09: 0000000000000000<br/> 2011-06-23 14:55:52 R10: ffff81100104a840 R11: 0000000000000158 R12: ffff811028ee5040<br/> 2011-06-23 14:55:52 R13: ffff81100104a840 R14: ffff81302530a0c0 R15: 00000000000000f6<br/> 2011-06-23 14:55:52 FS: 00002aaaab01f6e0(0000) GS:ffff811038155e40(0000) knlGS:0000000000000000<br/> 2011-06-23 14:55:52 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b<br/> 2011-06-23 14:55:52 CR2: 00002aaaaad626e0 CR3: 0000002f43886000 CR4: 00000000000006e0<br/> 2011-06-23 14:55:52 Process ldlm_cn_02 (pid: 9565, threadinfo ffff814026f14000, task ffff814037bab040)<br/> 2011-06-23 14:55:52 Call Trace:<br/> 2011-06-23 14:55:52 <span class="error">[<ffffffff886ca3eb>]</span> :ptlrpc:ldlm_resource_putref_internal+0x3ab/0x460<br/> 2011-06-23 14:55:52 LustreError: 3297:0:(mds_open.c:1693:mds_close()) @@@ no handle for file close ino 81431268: cookie 0x0 req@ffff813fd3b6b000 x1372347904393553/t0 o35->c339861d-0c0a-b9ee-a39d-9cfe591452c1@NET_0x50000c0a87223_UUID:0/0 lens 408/864 e 0 to 0 dl 1308866169 ref 1 fl Interpret:/0/0 rc 0/0<br/> 2011-06-23 14:55:52 LustreError: 3297:0:(mds_open.c:1693:mds_close()) Skipped 3269 previous similar messages<br/> 2011-06-23 14:55:52 <span class="error">[<ffffffff886c58d2>]</span> :ptlrpc:ldlm_lock_put+0x372/0x3d0<br/> 2011-06-23 14:55:52 <span class="error">[<ffffffff886e58a2>]</span> :ptlrpc:ldlm_request_cancel+0x222/0x330<br/> 2011-06-23 14:55:52 <span class="error">[<ffffffff8870bd69>]</span> :ptlrpc:lustre_pack_reply+0x29/0xb0<br/> 2011-06-23 14:55:52 <span class="error">[<ffffffff886e5bcd>]</span> :ptlrpc:ldlm_handle_cancel+0x21d/0x2e0<br/> 2011-06-23 14:55:52 <span class="error">[<ffffffff88707ef5>]</span> :ptlrpc:lustre_msg_get_opc+0x35/0xf0<br/> 2011-06-23 14:55:52 <span class="error">[<ffffffff886e5f98>]</span> :ptlrpc:ldlm_cancel_handler+0x308/0x690<br/> 2011-06-23 14:55:52 <span class="error">[<ffffffff88707ef5>]</span> :ptlrpc:lustre_msg_get_opc+0x35/0xf0<br/> 2011-06-23 14:55:52 <span class="error">[<ffffffff887176d9>]</span> :ptlrpc:ptlrpc_server_handle_request+0x989/0xe00<br/> 2011-06-23 14:55:52 <span class="error">[<ffffffff88717e35>]</span> :ptlrpc:ptlrpc_wait_event+0x2e5/0x310<br/> 2011-06-23 14:55:52 <span class="error">[<ffffffff8008c85d>]</span> __wake_up_common+0x3e/0x68<br/> 2011-06-23 14:55:52 <span class="error">[<ffffffff88718dc6>]</span> :ptlrpc:ptlrpc_main+0xf66/0x1120<br/> 2011-06-23 14:55:52 <span class="error">[<ffffffff8005dfb1>]</span> child_rip+0xa/0x11<br/> 2011-06-23 14:55:52 <span class="error">[<ffffffff88717e60>]</span> :ptlrpc:ptlrpc_main+0x0/0x1120<br/> 2011-06-23 14:55:52 <span class="error">[<ffffffff8005dfa7>]</span> child_rip+0x0/0x11</p> <p>3478 void kmem_cache_free(struct kmem_cache *cachep, void *objp)<br/> 3479 {<br/> 3480 unsigned long flags;<br/> 3481 <br/> 3482 BUG_ON(virt_to_cache(objp) != cachep);</p> <p>It looks like a slab corruption. Cliff, is this easy to reproduce?</p> <p>And we have the second crash, but in a different location.<br/> Data attached, will reboot Hyperion servers onto 1.8.5-56 at 9 or so PDT unless otherwise advised</p> <p>It seems I can make it crash again, but not the same location.</p> <p>HongChao</p> <p>Please can you look into this bug as your top priority today and make sure to provide a status update as to your findings (or just what you have looked into) before you leave for the day.</p> <p>Thanks</p> <p>Peter</p> <p>A second crash in a diiferent part of the code makes me think that there is memory corruption happening.</p> <p>Do we have any ability to enable slab debug, list debug, etc to try and isolate this problem earlier?</p> <p>On very tricky cases, Ricardo had also written a patch that made all allocations "use on once" and any thread that accessed old memory would Oops and dump a stack. </p> <p>> A second crash in a diiferent part of the code makes me think that there is memory corruption happening.</p> <p>Yes, that's what i initially suspected.</p> <p>> Do we have any ability to enable slab debug, list debug, etc to try and isolate this problem earlier?</p> <p>I think the chaos kernel has list debug but no slab debug.<br/> Also, it would be great to get a crash dump.</p> <p>Reverted to 1.8.5 GA, ext4 version lustre-modules-1.8.5-2.6.18_194.17.1.el5_lustre.1.8.5 lustre-tests-1.8.5-2.6.18_194.17.1.el5_lustre.1.8.5 kernel-2.6.18-194.17.1.el5_lustre.1.8.5 lustre-tools-llnl-1.2-6.ch4.4 lustre-1.8.5-2.6.18_194.17.1.el5_lustre.1.8.5 lustre-ldiskfs-3.1.4-2.6.18_194.17.1.el5_lustre.1.8.5 Have run fsstress for > 45 minutes, no crash, few errors - RAID is still rebuiding. I am attaching the full console log from the last crash, from start of MDT to crash.</p> <p>Spoke (literally) about two minutes too soon, MDT just panic'd - Log attached, previous mds log not attached <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>Crash running 1.8.5 GA -ext4</p> <p>All crashes are in different area of the code. Those kind of problem are hard to diagnose.<br/> We really need to recompile the kernel with slab debugging and collect a crash dump.<br/> If not possible on hyperion, can we try on toro?<br/> Cliff, can you reproduce it at a smaller scale (small = something that could fit in toro)?</p> <p>it's hard to diagnose the problem. in 1.8.5 GA+ext4, the NULL pointer reference in RCU callback context should be a very rare<br/> case, what kind of problems can cause it? <br/> can we do this stress test with ext3 (<8TB) under 1.8.6/1.8.5 to check whether it still panics, then isolate the problem to be<br/> an ext4-related issue?</p> <p>btw, I have discussed with Yujian, and she said the fsstress was only tested against 2.X before, and the scale was about<br/> 120 clients and no problem, but she was not clear whether the ldiskfs in those tests was against ext4.</p> <p>This is the MDS console log from the second 1.8.6, showing everything from mount to crash</p> <p>Second 1.8.6 crash</p> <p>Replicate 1.8.5 GA crash w/10 clients </p> <blockquote> <p>btw, I have discussed with Yujian, and she said the fsstress was only tested against 2.X before, and the scale was about<br/> 120 clients and no problem, but she was not clear whether the ldiskfs in those tests was against ext4.</p></blockquote> <p>After doing a search on Bugzilla, I found the latest run of this test on Lustre master branch was against 2.0.0 Build 41 (v1_10_0_41):</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Branch: v1_10_0_41(client and servers) - ext3 Distro/Arch: RHEL5/x86_64(client and servers) Cluster: Hyperion MDS Node: hyperion-mds7 OSS Node: hyperion-ost[5,6,7] Client Nodes: 224 (patchless) Network Type: o2ib Test: client mount 4 mounts but run fs_test only on 1 Result: bug 22802 </pre> </div></div> <p>And it was also run on Lustre b1_8 branch several times, the latest run was against 1.8.2 and passed, please refer to <a href="https://bugzilla.lustre.org/show_bug.cgi?id=21292" class="external-link" target="_blank" rel="nofollow noopener">bug 21292</a>.</p> <p>System crashed after 2 hours, but immediately rebooted - no crash dump. Log attached</p> <p>Friday night fsstress crash</p> <p>in the corrupted slab cache, the block size is 128, and both the previous and next block of the corrupted one are used<br/> by end IO routine of RAID10, maybe this issue be related to RAID10? but nothing found at the related code in 2.6.18-238.12.1.</p> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-b1_8/./arch=x86_64,build_type=client,distro=el6,ib_stack=inkernel/93/" class="external-link" target="_blank" rel="nofollow noopener">lustre-b1_8 » x86_64,client,el6,inkernel #93</a><br/> <a href="https://jira.whamcloud.com/browse/LU-457" title="MDS crash under DDN file stress tests" class="issue-link" data-issue-key="LU-457"><del>LU-457</del></a> Enable slab debug for RHEL5/x86_64</p> <p>Johann Lombardi : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/b1_8&a=commit&h=529529a3dfbb8849f479c15f75e8f875ba5427b4" class="external-link" target="_blank" rel="nofollow noopener">529529a3dfbb8849f479c15f75e8f875ba5427b4</a><br/> Files : </p> <ul> <li>lustre/kernel_patches/kernel_configs/kernel-2.6.18-2.6-rhel5-x86_64.config</li> <li>lustre/mds/mds_open.c</li> <li>lustre/kernel_patches/kernel_configs/kernel-2.6.18-2.6-rhel5-x86_64-smp.config</li> <li>lustre/kernel_patches/patches/md-avoid-bug_on-when-bmc-overflow.patch</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-b1_8/./arch=i686,build_type=client,distro=el6,ib_stack=inkernel/93/" class="external-link" target="_blank" rel="nofollow noopener">lustre-b1_8 » i686,client,el6,inkernel #93</a><br/> <a href="https://jira.whamcloud.com/browse/LU-457" title="MDS crash under DDN file stress tests" class="issue-link" data-issue-key="LU-457"><del>LU-457</del></a> Enable slab debug for RHEL5/x86_64</p> <p>Johann Lombardi : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/b1_8&a=commit&h=529529a3dfbb8849f479c15f75e8f875ba5427b4" class="external-link" target="_blank" rel="nofollow noopener">529529a3dfbb8849f479c15f75e8f875ba5427b4</a><br/> Files : </p> <ul> <li>lustre/kernel_patches/kernel_configs/kernel-2.6.18-2.6-rhel5-x86_64-smp.config</li> <li>lustre/kernel_patches/kernel_configs/kernel-2.6.18-2.6-rhel5-x86_64.config</li> <li>lustre/mds/mds_open.c</li> <li>lustre/kernel_patches/patches/md-avoid-bug_on-when-bmc-overflow.patch</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-b1_8/./arch=i686,build_type=client,distro=el5,ib_stack=ofa/93/" class="external-link" target="_blank" rel="nofollow noopener">lustre-b1_8 » i686,client,el5,ofa #93</a><br/> <a href="https://jira.whamcloud.com/browse/LU-457" title="MDS crash under DDN file stress tests" class="issue-link" data-issue-key="LU-457"><del>LU-457</del></a> Enable slab debug for RHEL5/x86_64</p> <p>Johann Lombardi : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/b1_8&a=commit&h=529529a3dfbb8849f479c15f75e8f875ba5427b4" class="external-link" target="_blank" rel="nofollow noopener">529529a3dfbb8849f479c15f75e8f875ba5427b4</a><br/> Files : </p> <ul> <li>lustre/kernel_patches/kernel_configs/kernel-2.6.18-2.6-rhel5-x86_64-smp.config</li> <li>lustre/kernel_patches/kernel_configs/kernel-2.6.18-2.6-rhel5-x86_64.config</li> <li>lustre/mds/mds_open.c</li> <li>lustre/kernel_patches/patches/md-avoid-bug_on-when-bmc-overflow.patch</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-b1_8/./arch=x86_64,build_type=client,distro=ubuntu1004,ib_stack=inkernel/93/" class="external-link" target="_blank" rel="nofollow noopener">lustre-b1_8 » x86_64,client,ubuntu1004,inkernel #93</a><br/> <a href="https://jira.whamcloud.com/browse/LU-457" title="MDS crash under DDN file stress tests" class="issue-link" data-issue-key="LU-457"><del>LU-457</del></a> Enable slab debug for RHEL5/x86_64</p> <p>Johann Lombardi : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/b1_8&a=commit&h=529529a3dfbb8849f479c15f75e8f875ba5427b4" class="external-link" target="_blank" rel="nofollow noopener">529529a3dfbb8849f479c15f75e8f875ba5427b4</a><br/> Files : </p> <ul> <li>lustre/kernel_patches/kernel_configs/kernel-2.6.18-2.6-rhel5-x86_64.config</li> <li>lustre/kernel_patches/kernel_configs/kernel-2.6.18-2.6-rhel5-x86_64-smp.config</li> <li>lustre/mds/mds_open.c</li> <li>lustre/kernel_patches/patches/md-avoid-bug_on-when-bmc-overflow.patch</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-b1_8/./arch=i686,build_type=client,distro=el5,ib_stack=inkernel/93/" class="external-link" target="_blank" rel="nofollow noopener">lustre-b1_8 » i686,client,el5,inkernel #93</a><br/> <a href="https://jira.whamcloud.com/browse/LU-457" title="MDS crash under DDN file stress tests" class="issue-link" data-issue-key="LU-457"><del>LU-457</del></a> Enable slab debug for RHEL5/x86_64</p> <p>Johann Lombardi : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/b1_8&a=commit&h=529529a3dfbb8849f479c15f75e8f875ba5427b4" class="external-link" target="_blank" rel="nofollow noopener">529529a3dfbb8849f479c15f75e8f875ba5427b4</a><br/> Files : </p> <ul> <li>lustre/mds/mds_open.c</li> <li>lustre/kernel_patches/kernel_configs/kernel-2.6.18-2.6-rhel5-x86_64.config</li> <li>lustre/kernel_patches/patches/md-avoid-bug_on-when-bmc-overflow.patch</li> <li>lustre/kernel_patches/kernel_configs/kernel-2.6.18-2.6-rhel5-x86_64-smp.config</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-b1_8/./arch=i686,build_type=server,distro=el5,ib_stack=ofa/93/" class="external-link" target="_blank" rel="nofollow noopener">lustre-b1_8 » i686,server,el5,ofa #93</a><br/> <a href="https://jira.whamcloud.com/browse/LU-457" title="MDS crash under DDN file stress tests" class="issue-link" data-issue-key="LU-457"><del>LU-457</del></a> Enable slab debug for RHEL5/x86_64</p> <p>Johann Lombardi : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/b1_8&a=commit&h=529529a3dfbb8849f479c15f75e8f875ba5427b4" class="external-link" target="_blank" rel="nofollow noopener">529529a3dfbb8849f479c15f75e8f875ba5427b4</a><br/> Files : </p> <ul> <li>lustre/kernel_patches/patches/md-avoid-bug_on-when-bmc-overflow.patch</li> <li>lustre/mds/mds_open.c</li> <li>lustre/kernel_patches/kernel_configs/kernel-2.6.18-2.6-rhel5-x86_64.config</li> <li>lustre/kernel_patches/kernel_configs/kernel-2.6.18-2.6-rhel5-x86_64-smp.config</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-b1_8/./arch=i686,build_type=server,distro=el5,ib_stack=inkernel/93/" class="external-link" target="_blank" rel="nofollow noopener">lustre-b1_8 » i686,server,el5,inkernel #93</a><br/> <a href="https://jira.whamcloud.com/browse/LU-457" title="MDS crash under DDN file stress tests" class="issue-link" data-issue-key="LU-457"><del>LU-457</del></a> Enable slab debug for RHEL5/x86_64</p> <p>Johann Lombardi : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/b1_8&a=commit&h=529529a3dfbb8849f479c15f75e8f875ba5427b4" class="external-link" target="_blank" rel="nofollow noopener">529529a3dfbb8849f479c15f75e8f875ba5427b4</a><br/> Files : </p> <ul> <li>lustre/kernel_patches/kernel_configs/kernel-2.6.18-2.6-rhel5-x86_64-smp.config</li> <li>lustre/kernel_patches/kernel_configs/kernel-2.6.18-2.6-rhel5-x86_64.config</li> <li>lustre/kernel_patches/patches/md-avoid-bug_on-when-bmc-overflow.patch</li> <li>lustre/mds/mds_open.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-b1_8/./arch=x86_64,build_type=server,distro=el5,ib_stack=ofa/93/" class="external-link" target="_blank" rel="nofollow noopener">lustre-b1_8 » x86_64,server,el5,ofa #93</a><br/> <a href="https://jira.whamcloud.com/browse/LU-457" title="MDS crash under DDN file stress tests" class="issue-link" data-issue-key="LU-457"><del>LU-457</del></a> Enable slab debug for RHEL5/x86_64</p> <p>Johann Lombardi : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/b1_8&a=commit&h=529529a3dfbb8849f479c15f75e8f875ba5427b4" class="external-link" target="_blank" rel="nofollow noopener">529529a3dfbb8849f479c15f75e8f875ba5427b4</a><br/> Files : </p> <ul> <li>lustre/kernel_patches/kernel_configs/kernel-2.6.18-2.6-rhel5-x86_64-smp.config</li> <li>lustre/kernel_patches/patches/md-avoid-bug_on-when-bmc-overflow.patch</li> <li>lustre/mds/mds_open.c</li> <li>lustre/kernel_patches/kernel_configs/kernel-2.6.18-2.6-rhel5-x86_64.config</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-b1_8/./arch=x86_64,build_type=client,distro=el5,ib_stack=inkernel/93/" class="external-link" target="_blank" rel="nofollow noopener">lustre-b1_8 » x86_64,client,el5,inkernel #93</a><br/> <a href="https://jira.whamcloud.com/browse/LU-457" title="MDS crash under DDN file stress tests" class="issue-link" data-issue-key="LU-457"><del>LU-457</del></a> Enable slab debug for RHEL5/x86_64</p> <p>Johann Lombardi : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/b1_8&a=commit&h=529529a3dfbb8849f479c15f75e8f875ba5427b4" class="external-link" target="_blank" rel="nofollow noopener">529529a3dfbb8849f479c15f75e8f875ba5427b4</a><br/> Files : </p> <ul> <li>lustre/kernel_patches/kernel_configs/kernel-2.6.18-2.6-rhel5-x86_64-smp.config</li> <li>lustre/mds/mds_open.c</li> <li>lustre/kernel_patches/patches/md-avoid-bug_on-when-bmc-overflow.patch</li> <li>lustre/kernel_patches/kernel_configs/kernel-2.6.18-2.6-rhel5-x86_64.config</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-b1_8/./arch=x86_64,build_type=client,distro=el5,ib_stack=ofa/93/" class="external-link" target="_blank" rel="nofollow noopener">lustre-b1_8 » x86_64,client,el5,ofa #93</a><br/> <a href="https://jira.whamcloud.com/browse/LU-457" title="MDS crash under DDN file stress tests" class="issue-link" data-issue-key="LU-457"><del>LU-457</del></a> Enable slab debug for RHEL5/x86_64</p> <p>Johann Lombardi : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/b1_8&a=commit&h=529529a3dfbb8849f479c15f75e8f875ba5427b4" class="external-link" target="_blank" rel="nofollow noopener">529529a3dfbb8849f479c15f75e8f875ba5427b4</a><br/> Files : </p> <ul> <li>lustre/kernel_patches/kernel_configs/kernel-2.6.18-2.6-rhel5-x86_64.config</li> <li>lustre/mds/mds_open.c</li> <li>lustre/kernel_patches/patches/md-avoid-bug_on-when-bmc-overflow.patch</li> <li>lustre/kernel_patches/kernel_configs/kernel-2.6.18-2.6-rhel5-x86_64-smp.config</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-b1_8/./arch=x86_64,build_type=server,distro=el5,ib_stack=inkernel/93/" class="external-link" target="_blank" rel="nofollow noopener">lustre-b1_8 » x86_64,server,el5,inkernel #93</a><br/> <a href="https://jira.whamcloud.com/browse/LU-457" title="MDS crash under DDN file stress tests" class="issue-link" data-issue-key="LU-457"><del>LU-457</del></a> Enable slab debug for RHEL5/x86_64</p> <p>Johann Lombardi : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/b1_8&a=commit&h=529529a3dfbb8849f479c15f75e8f875ba5427b4" class="external-link" target="_blank" rel="nofollow noopener">529529a3dfbb8849f479c15f75e8f875ba5427b4</a><br/> Files : </p> <ul> <li>lustre/kernel_patches/kernel_configs/kernel-2.6.18-2.6-rhel5-x86_64-smp.config</li> <li>lustre/mds/mds_open.c</li> <li>lustre/kernel_patches/kernel_configs/kernel-2.6.18-2.6-rhel5-x86_64.config</li> <li>lustre/kernel_patches/patches/md-avoid-bug_on-when-bmc-overflow.patch</li> </ul> <p>This is suspected to be unrelated to Lustre. The Hyperion configuration has been fixed and Cliff will re-test next week to confirm</p> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/red.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-b1_8/./arch=x86_64,build_type=client,distro=el5,ib_stack=inkernel/95/" class="external-link" target="_blank" rel="nofollow noopener">lustre-b1_8 » x86_64,client,el5,inkernel #95</a><br/> Revert "<a href="https://jira.whamcloud.com/browse/LU-457" title="MDS crash under DDN file stress tests" class="issue-link" data-issue-key="LU-457"><del>LU-457</del></a> Enable slab debug for RHEL5/x86_64"</p> <p>Johann Lombardi : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/b1_8&a=commit&h=a0375b393ef34c457bec68835ed6ac2f37b5a537" class="external-link" target="_blank" rel="nofollow noopener">a0375b393ef34c457bec68835ed6ac2f37b5a537</a><br/> Files : </p> <ul> <li>lustre/mds/mds_open.c</li> <li>lustre/kernel_patches/kernel_configs/kernel-2.6.18-2.6-rhel5-x86_64-smp.config</li> <li>lustre/kernel_patches/patches/md-avoid-bug_on-when-bmc-overflow.patch</li> <li>lustre/kernel_patches/kernel_configs/kernel-2.6.18-2.6-rhel5-x86_64.config</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/red.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-b1_8/./arch=x86_64,build_type=client,distro=el5,ib_stack=ofa/95/" class="external-link" target="_blank" rel="nofollow noopener">lustre-b1_8 » x86_64,client,el5,ofa #95</a><br/> Revert "<a href="https://jira.whamcloud.com/browse/LU-457" title="MDS crash under DDN file stress tests" class="issue-link" data-issue-key="LU-457"><del>LU-457</del></a> Enable slab debug for RHEL5/x86_64"</p> <p>Johann Lombardi : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/b1_8&a=commit&h=a0375b393ef34c457bec68835ed6ac2f37b5a537" class="external-link" target="_blank" rel="nofollow noopener">a0375b393ef34c457bec68835ed6ac2f37b5a537</a><br/> Files : </p> <ul> <li>lustre/kernel_patches/kernel_configs/kernel-2.6.18-2.6-rhel5-x86_64-smp.config</li> <li>lustre/mds/mds_open.c</li> <li>lustre/kernel_patches/kernel_configs/kernel-2.6.18-2.6-rhel5-x86_64.config</li> <li>lustre/kernel_patches/patches/md-avoid-bug_on-when-bmc-overflow.patch</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-b1_8/./arch=x86_64,build_type=server,distro=el5,ib_stack=ofa/95/" class="external-link" target="_blank" rel="nofollow noopener">lustre-b1_8 » x86_64,server,el5,ofa #95</a><br/> Revert "<a href="https://jira.whamcloud.com/browse/LU-457" title="MDS crash under DDN file stress tests" class="issue-link" data-issue-key="LU-457"><del>LU-457</del></a> Enable slab debug for RHEL5/x86_64"</p> <p>Johann Lombardi : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/b1_8&a=commit&h=a0375b393ef34c457bec68835ed6ac2f37b5a537" class="external-link" target="_blank" rel="nofollow noopener">a0375b393ef34c457bec68835ed6ac2f37b5a537</a><br/> Files : </p> <ul> <li>lustre/mds/mds_open.c</li> <li>lustre/kernel_patches/kernel_configs/kernel-2.6.18-2.6-rhel5-x86_64.config</li> <li>lustre/kernel_patches/patches/md-avoid-bug_on-when-bmc-overflow.patch</li> <li>lustre/kernel_patches/kernel_configs/kernel-2.6.18-2.6-rhel5-x86_64-smp.config</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-b1_8/./arch=i686,build_type=client,distro=el6,ib_stack=inkernel/95/" class="external-link" target="_blank" rel="nofollow noopener">lustre-b1_8 » i686,client,el6,inkernel #95</a><br/> Revert "<a href="https://jira.whamcloud.com/browse/LU-457" title="MDS crash under DDN file stress tests" class="issue-link" data-issue-key="LU-457"><del>LU-457</del></a> Enable slab debug for RHEL5/x86_64"</p> <p>Johann Lombardi : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/b1_8&a=commit&h=a0375b393ef34c457bec68835ed6ac2f37b5a537" class="external-link" target="_blank" rel="nofollow noopener">a0375b393ef34c457bec68835ed6ac2f37b5a537</a><br/> Files : </p> <ul> <li>lustre/kernel_patches/kernel_configs/kernel-2.6.18-2.6-rhel5-x86_64.config</li> <li>lustre/mds/mds_open.c</li> <li>lustre/kernel_patches/patches/md-avoid-bug_on-when-bmc-overflow.patch</li> <li>lustre/kernel_patches/kernel_configs/kernel-2.6.18-2.6-rhel5-x86_64-smp.config</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-b1_8/./arch=x86_64,build_type=client,distro=el6,ib_stack=inkernel/95/" class="external-link" target="_blank" rel="nofollow noopener">lustre-b1_8 » x86_64,client,el6,inkernel #95</a><br/> Revert "<a href="https://jira.whamcloud.com/browse/LU-457" title="MDS crash under DDN file stress tests" class="issue-link" data-issue-key="LU-457"><del>LU-457</del></a> Enable slab debug for RHEL5/x86_64"</p> <p>Johann Lombardi : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/b1_8&a=commit&h=a0375b393ef34c457bec68835ed6ac2f37b5a537" class="external-link" target="_blank" rel="nofollow noopener">a0375b393ef34c457bec68835ed6ac2f37b5a537</a><br/> Files : </p> <ul> <li>lustre/kernel_patches/patches/md-avoid-bug_on-when-bmc-overflow.patch</li> <li>lustre/kernel_patches/kernel_configs/kernel-2.6.18-2.6-rhel5-x86_64-smp.config</li> <li>lustre/kernel_patches/kernel_configs/kernel-2.6.18-2.6-rhel5-x86_64.config</li> <li>lustre/mds/mds_open.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-b1_8/./arch=i686,build_type=client,distro=el5,ib_stack=inkernel/95/" class="external-link" target="_blank" rel="nofollow noopener">lustre-b1_8 » i686,client,el5,inkernel #95</a><br/> Revert "<a href="https://jira.whamcloud.com/browse/LU-457" title="MDS crash under DDN file stress tests" class="issue-link" data-issue-key="LU-457"><del>LU-457</del></a> Enable slab debug for RHEL5/x86_64"</p> <p>Johann Lombardi : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/b1_8&a=commit&h=a0375b393ef34c457bec68835ed6ac2f37b5a537" class="external-link" target="_blank" rel="nofollow noopener">a0375b393ef34c457bec68835ed6ac2f37b5a537</a><br/> Files : </p> <ul> <li>lustre/kernel_patches/patches/md-avoid-bug_on-when-bmc-overflow.patch</li> <li>lustre/mds/mds_open.c</li> <li>lustre/kernel_patches/kernel_configs/kernel-2.6.18-2.6-rhel5-x86_64.config</li> <li>lustre/kernel_patches/kernel_configs/kernel-2.6.18-2.6-rhel5-x86_64-smp.config</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-b1_8/./arch=x86_64,build_type=server,distro=el5,ib_stack=inkernel/95/" class="external-link" target="_blank" rel="nofollow noopener">lustre-b1_8 » x86_64,server,el5,inkernel #95</a><br/> Revert "<a href="https://jira.whamcloud.com/browse/LU-457" title="MDS crash under DDN file stress tests" class="issue-link" data-issue-key="LU-457"><del>LU-457</del></a> Enable slab debug for RHEL5/x86_64"</p> <p>Johann Lombardi : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/b1_8&a=commit&h=a0375b393ef34c457bec68835ed6ac2f37b5a537" class="external-link" target="_blank" rel="nofollow noopener">a0375b393ef34c457bec68835ed6ac2f37b5a537</a><br/> Files : </p> <ul> <li>lustre/mds/mds_open.c</li> <li>lustre/kernel_patches/kernel_configs/kernel-2.6.18-2.6-rhel5-x86_64-smp.config</li> <li>lustre/kernel_patches/kernel_configs/kernel-2.6.18-2.6-rhel5-x86_64.config</li> <li>lustre/kernel_patches/patches/md-avoid-bug_on-when-bmc-overflow.patch</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-b1_8/./arch=x86_64,build_type=client,distro=ubuntu1004,ib_stack=inkernel/95/" class="external-link" target="_blank" rel="nofollow noopener">lustre-b1_8 » x86_64,client,ubuntu1004,inkernel #95</a><br/> Revert "<a href="https://jira.whamcloud.com/browse/LU-457" title="MDS crash under DDN file stress tests" class="issue-link" data-issue-key="LU-457"><del>LU-457</del></a> Enable slab debug for RHEL5/x86_64"</p> <p>Johann Lombardi : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/b1_8&a=commit&h=a0375b393ef34c457bec68835ed6ac2f37b5a537" class="external-link" target="_blank" rel="nofollow noopener">a0375b393ef34c457bec68835ed6ac2f37b5a537</a><br/> Files : </p> <ul> <li>lustre/kernel_patches/kernel_configs/kernel-2.6.18-2.6-rhel5-x86_64-smp.config</li> <li>lustre/kernel_patches/kernel_configs/kernel-2.6.18-2.6-rhel5-x86_64.config</li> <li>lustre/kernel_patches/patches/md-avoid-bug_on-when-bmc-overflow.patch</li> <li>lustre/mds/mds_open.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-b1_8/./arch=i686,build_type=client,distro=el5,ib_stack=ofa/95/" class="external-link" target="_blank" rel="nofollow noopener">lustre-b1_8 » i686,client,el5,ofa #95</a><br/> Revert "<a href="https://jira.whamcloud.com/browse/LU-457" title="MDS crash under DDN file stress tests" class="issue-link" data-issue-key="LU-457"><del>LU-457</del></a> Enable slab debug for RHEL5/x86_64"</p> <p>Johann Lombardi : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/b1_8&a=commit&h=a0375b393ef34c457bec68835ed6ac2f37b5a537" class="external-link" target="_blank" rel="nofollow noopener">a0375b393ef34c457bec68835ed6ac2f37b5a537</a><br/> Files : </p> <ul> <li>lustre/mds/mds_open.c</li> <li>lustre/kernel_patches/kernel_configs/kernel-2.6.18-2.6-rhel5-x86_64-smp.config</li> <li>lustre/kernel_patches/patches/md-avoid-bug_on-when-bmc-overflow.patch</li> <li>lustre/kernel_patches/kernel_configs/kernel-2.6.18-2.6-rhel5-x86_64.config</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-b1_8/./arch=i686,build_type=server,distro=el5,ib_stack=ofa/95/" class="external-link" target="_blank" rel="nofollow noopener">lustre-b1_8 » i686,server,el5,ofa #95</a><br/> Revert "<a href="https://jira.whamcloud.com/browse/LU-457" title="MDS crash under DDN file stress tests" class="issue-link" data-issue-key="LU-457"><del>LU-457</del></a> Enable slab debug for RHEL5/x86_64"</p> <p>Johann Lombardi : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/b1_8&a=commit&h=a0375b393ef34c457bec68835ed6ac2f37b5a537" class="external-link" target="_blank" rel="nofollow noopener">a0375b393ef34c457bec68835ed6ac2f37b5a537</a><br/> Files : </p> <ul> <li>lustre/mds/mds_open.c</li> <li>lustre/kernel_patches/kernel_configs/kernel-2.6.18-2.6-rhel5-x86_64.config</li> <li>lustre/kernel_patches/patches/md-avoid-bug_on-when-bmc-overflow.patch</li> <li>lustre/kernel_patches/kernel_configs/kernel-2.6.18-2.6-rhel5-x86_64-smp.config</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-b1_8/./arch=i686,build_type=server,distro=el5,ib_stack=inkernel/95/" class="external-link" target="_blank" rel="nofollow noopener">lustre-b1_8 » i686,server,el5,inkernel #95</a><br/> Revert "<a href="https://jira.whamcloud.com/browse/LU-457" title="MDS crash under DDN file stress tests" class="issue-link" data-issue-key="LU-457"><del>LU-457</del></a> Enable slab debug for RHEL5/x86_64"</p> <p>Johann Lombardi : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/b1_8&a=commit&h=a0375b393ef34c457bec68835ed6ac2f37b5a537" class="external-link" target="_blank" rel="nofollow noopener">a0375b393ef34c457bec68835ed6ac2f37b5a537</a><br/> Files : </p> <ul> <li>lustre/kernel_patches/patches/md-avoid-bug_on-when-bmc-overflow.patch</li> <li>lustre/mds/mds_open.c</li> <li>lustre/kernel_patches/kernel_configs/kernel-2.6.18-2.6-rhel5-x86_64-smp.config</li> <li>lustre/kernel_patches/kernel_configs/kernel-2.6.18-2.6-rhel5-x86_64.config</li> </ul> <p>Has not happened for a long time so let's close and reopen if it reoccurs</p> Development Rank 1|hzvi6f: Rank (Obsolete) 6594 Severity