https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-1823 Lustre <p>Lustre: DEBUG MARKER: == sanity test 103: acl test ========================================================================= 19:57:07 (1346774227)<br/> /work/lustre/head/clean/lustre/utils/l_getidentity<br/> Slab corruption (Tainted: P --------------- ): size-2048 start=dac6c470, len=2048<br/> Redzone: 0x9f911029d74e35b/0x9f911029d74e35b.<br/> Last user: <span class="error">&#91;&lt;dff39e58&gt;&#93;</span>(cfs_free+0x8/0x10 <span class="error">&#91;libcfs&#93;</span>)<br/> 310: 02 00 00 00 01 00 07 00 ff ff ff ff 02 00 05 00<br/> 320: 01 00 00 00 02 00 07 00 02 00 00 00 04 00 07 00<br/> 330: ff ff ff ff 10 00 07 00 ff ff ff ff 20 00 05 00<br/> 340: ff ff ff ff 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b<br/> Next obj: start=dac6cc88, len=2048<br/> Redzone: 0x9f911029d74e35b/0x9f911029d74e35b.<br/> Last user: <span class="error">&#91;&lt;dff39e58&gt;&#93;</span>(cfs_free+0x8/0x10 <span class="error">&#91;libcfs&#93;</span>)<br/> 000: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b<br/> 010: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b</p> <p>02000000:00000010:1.0:1346774231.327841:1804:3373:0:(sec_null.c:217:null_alloc_repbuf()) kmalloced 'req-&gt;rq_repbuf': 2048 at dac6c470.<br/> ...</p> <p>02000000:00000010:1.0:1346774231.328361:836:3373:0:(sec_null.c:231:null_free_repbuf()) kfreed 'req-&gt;rq_repbuf': 2048 at dac6c470.</p> CONFIG_DEBUG_SLAB=y LU-1823

sanity/103: slab corruption

Bug Blocker Resolved Fixed Oleg Drokin Alex Zhuravlev Tue, 4 Sep 2012 13:00:39 +0000 Mon, 29 May 2017 03:35:02 +0000 Thu, 13 Sep 2012 01:39:22 +0000 Lustre 2.3.0 Lustre 2.4.0 Lustre 2.3.0 Lustre 2.4.0 0 10 <p>Keith is going to try and reproduce this with a debug kernel</p> <p>Are there more logs from the rest of the systems? Is there anything special needed to reproduce this? </p> <p>I'm able to reproduce this almost 100% with REFORMAT=y ONLY=103 sh sanity.sh, within single vbox instance.</p> <p>I know in the past, Oleg, Johann, and I have wanted to run early development kernels with various debug options enabled for all kernel builds, so that this kind of problem can be flushed out when patches land instead of only at users who run these debug kernels, or hitting silent corruption problems. This previously was suggested by Johann in TT-359, but I think it could be done with a patch to the kernel config options during the development cycle instead of via the test environment (which would need more effort/complexity).</p> <p>I think since we are early in the 2.4 release cycle that it makes sense to enable these config options for all our server kernels (so they will be seen for servers and clients running the server kernel). We can leave this as a blocker bug for the 2.4 release to remember to revert the debug kernel config changes.</p> <p>Given the relatively small number of patches that have landed on master compared to 2.3, it probably also makes sense to submit a patch to b2_3 to enable CONFIG_DEBUG_SLAB, CONFIG_DEBUG_SPINLOCK and possibly some others, with:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Test-Parameters: fortestonly testgroup=full </pre> </div></div> <p>to see if there is a similar failure for b2_3.</p> <p>probably it'd make sense to add a trivial (dmesg|grep 'Slab corruption' &amp;&amp; error) to t-f so that we don't miss it.</p> <p>I submitted a few config changes for b2_3 as suggested. <a href="http://review.whamcloud.com/3875" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/3875</a> . </p> <p>I have been able to get some local testing done today. I tried an older 2.2.59 code base (I had it setup on one of my build servers) and I don't seem to see the problem there but I do see it with Master. I will work to narrow down the window of possible changes. Sorting out if b2_3 is effected is my next step. </p> <p>My config test didn't make it though build on the first pass but Yu has a very nice patch/test here I am watching <a href="http://review.whamcloud.com/#change,3876" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,3876</a></p> <p>Hi Keith,</p> <p>By using the build <a href="http://build.whamcloud.com/job/lustre-reviews/8904/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-reviews/8904/</a> in <a href="http://review.whamcloud.com/#change,3876" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,3876</a>, I can manually reproduce the slab corruption issue on RHEL6 distro by only running sanity test 103:<br/> <a href="https://maloo.whamcloud.com/test_sets/2c479ade-f7d3-11e1-8b95-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/2c479ade-f7d3-11e1-8b95-52540035b04c</a></p> <p>The autotest run for the above build skipped sanity test 103 because it's in the EXCEPT_SLOW list. I'm updating the commit message to add SLOW=yes into the test parameters.</p> <p>Hi Keith,</p> <p>FYI, with the build for patch set 5 of <a href="http://review.whamcloud.com/#change,3876" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,3876</a>, I reproduced the issue with PTLDEBUG=-1 manually:<br/> <a href="https://maloo.whamcloud.com/test_sets/59a5ca46-f832-11e1-b114-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/59a5ca46-f832-11e1-b114-52540035b04c</a></p> <p>If there are no obvious sources of this corruption, it probably makes sense to submit this test patch as several separate changes, each based on one of the recent 2.2.* tags, to see if we can isolate when this corruption started. After that, it is hopefully possible to do a (manual?) git-bisect to find which patch is the culprit, or at least narrow down the range of patches that need to be examined manually. It is also important to check in each of the failure cases what node type the corruption is seen on (MDS, OSS, client), since that will also reduce the number of changes which might have introduced the problem.</p> <p>It would make sense to include a check for the <a href="https://jira.whamcloud.com/browse/LU-1844" title="sanityn, subtest test_16: list_del corruption when run ofd + ldiskfs" class="issue-link" data-issue-key="LU-1844"><del>LU-1844</del></a> list_add/list_del corruption messages as well, since I suspect that is also a sign of random memory corruption.</p> <p>I have started a git bisect to narrow down the code change but I fear it is not realiable data. I am not sure what has happened on my local vms (I shuffled some vms around yesterday) but I am no longer able to reproduce the core issue. I am running Lustre: 2.3.50 (from Master) with kernel-2.6.32-279.5.2 an not triggering the issue. I am moving back to kernel-2.6.32-279.1.1 (confirmed failed with Yu's test run) to see if the issue reappears. </p> <p>I will update when I know more. </p> <p>Per the above test report, the slab corruption issue occurred only on the MDS (fat-intel-2):</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>fat-intel-2: Slab corruption (Not tainted): size-2048 start=ffff8802e1b534f8, len=2048 fat-intel-2: Slab corruption (Not tainted): size-2048 start=ffff8802e1d776f8, len=2048 fat-intel-2: Slab corruption (Not tainted): size-2048 start=ffff8802e13ca4c8, len=2048 sanity test_103: @@@@@@ FAIL: slab corruption detected </pre> </div></div> <p>Moving kernels does not seem to reproduce the issue so it is not a lead. I am going to try some client nodes tomorrow. I saw the error on the MDS as well on my initial Master run but have not see it since. </p> <p>I acquired some torro nods today and am starting to setup. My mds vm crashed while running " REFORMAT=y ONLY=103 sh sanity.sh", it took about 30 hours to trigger). This could be the bad cfs_free path that is corrupting the slab. </p> <p>I will try and attach the whole dmesg. </p> <p>This was master + kernel-2.6.32-279 on the MDS vm node. </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> Lustre: DEBUG MARKER: == sanity test 103: acl test =========================================== 06:06:43 (1347109603) kfree_debugcheck: out of range ptr 6000100000002h. ------------[ cut here ]------------ kernel BUG at mm/slab.c:2911! invalid opcode: 0000 [#1] SMP last sysfs file: /sys/devices/LNXSYSTM:00/LNXSYBUS:00/PNP0A03:00/PNP0C0A:00/power_supply/BAT0/energy_full CPU 0 Modules linked in: cmm(U) osd_ldiskfs(U) mdt(U) mdd(U) mds(U) fsfilt_ldiskfs(U) exportfs mgs(U) mgc(U) ldiskfs(U) lustre(U) lquota(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) autofs4 sunrpc ipv6 ppdev parport_pc parport microcode i2c_piix4 i2c_core snd_intel8x0 snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm snd_timer snd soundcore snd_page_alloc e1000 sg ext4 mbcache jbd2 sd_mod crc_t10dif sr_mod cdrom ahci pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: speedstep_lib] Pid: 24218, comm: jbd2/dm-2-8 Not tainted 2.6.32.masterDEBUG11A #1 innotek GmbH VirtualBox RIP: 0010:[&lt;ffffffff81162530&gt;] [&lt;ffffffff81162530&gt;] kfree_debugcheck+0x30/0x40 RSP: 0018:ffff88002733dba0 EFLAGS: 00010082 RAX: 0000000000000039 RBX: 0006000100000002 RCX: 0000000000007a74 RDX: 0000000000000000 RSI: 0000000000000046 RDI: 0000000000000046 RBP: ffff88002733dbb0 R08: 0000000000000000 R09: ffffffff8163acc0 R10: 0000000000000001 R11: 0000000000000000 R12: 0000000000000202 R13: 0006000100000002 R14: ffff880024d9d298 R15: ffff880024d9d298 FS: 0000000000000000(0000) GS:ffff880002200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b CR2: 0000003ac2ef5170 CR3: 000000003d0e0000 CR4: 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process jbd2/dm-2-8 (pid: 24218, threadinfo ffff88002733c000, task ffff88003d640ae0) Stack: ffff880000000020 ffffffffa035ebae ffff88002733dc00 ffffffff8116594b &lt;d&gt; ffff88002851f720 ffff88003f810080 ffff88002733dc20 0006000100000002 &lt;d&gt; ffff880024d9d240 0000000000000000 ffff880024d9d298 ffff880024d9d298 Call Trace: [&lt;ffffffffa035ebae&gt;] ? cfs_free+0xe/0x10 [libcfs] [&lt;ffffffff8116594b&gt;] kfree+0x5b/0x2a0 [&lt;ffffffffa035ebae&gt;] cfs_free+0xe/0x10 [libcfs] [&lt;ffffffffa04ceb73&gt;] lu_global_key_fini+0xa3/0xf0 [obdclass] [&lt;ffffffffa04cf380&gt;] key_fini+0x60/0x190 [obdclass] [&lt;ffffffffa04cf4df&gt;] keys_fini+0x2f/0x120 [obdclass] [&lt;ffffffffa04cf5fd&gt;] lu_context_fini+0x2d/0xc0 [obdclass] [&lt;ffffffffa0b86aa2&gt;] osd_trans_commit_cb+0xe2/0x2b0 [osd_ldiskfs] [&lt;ffffffffa0a3f21a&gt;] ldiskfs_journal_commit_callback+0x8a/0xc0 [ldiskfs] [&lt;ffffffffa00a18af&gt;] jbd2_journal_commit_transaction+0x110f/0x1530 [jbd2] [&lt;ffffffff810096f0&gt;] ? __switch_to+0xd0/0x320 [&lt;ffffffff8107eabb&gt;] ? try_to_del_timer_sync+0x7b/0xe0 [&lt;ffffffffa00a7128&gt;] kjournald2+0xb8/0x220 [jbd2] [&lt;ffffffff81091d66&gt;] kthread+0x96/0xa0 [&lt;ffffffff8100c14a&gt;] child_rip+0xa/0x20 [&lt;ffffffff81091cd0&gt;] ? kthread+0x0/0xa0 [&lt;ffffffff8100c140&gt;] ? child_rip+0x0/0x20 Code: 48 83 ec 08 0f 1f 44 00 00 48 89 fb e8 7a 67 ee ff 84 c0 74 07 48 83 c4 08 5b c9 c3 48 89 de 48 c7 c7 c8 0b 7a 81 e8 ed cc 39 00 &lt;0f&gt; 0b eb fe 66 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 41 RIP [&lt;ffffffff81162530&gt;] kfree_debugcheck+0x30/0x40 RSP &lt;ffff88002733dba0&gt; ---[ end trace ff4011ce2a20c79c ]--- Kernel panic - not syncing: Fatal exception Pid: 24218, comm: jbd2/dm-2-8 Tainted: G D --------------- 2.6.32.masterDEBUG11A #1 Call Trace: [&lt;ffffffff814ff155&gt;] ? panic+0xa0/0x168 [&lt;ffffffff815032e4&gt;] ? oops_end+0xe4/0x100 [&lt;ffffffff8100f26b&gt;] ? die+0x5b/0x90 [&lt;ffffffff81502bb4&gt;] ? do_trap+0xc4/0x160 [&lt;ffffffff8100ce35&gt;] ? do_invalid_op+0x95/0xb0 [&lt;ffffffff81162530&gt;] ? kfree_debugcheck+0x30/0x40 [&lt;ffffffffa036def3&gt;] ? libcfs_debug_vmsg2+0x4e3/0xb60 [libcfs] [&lt;ffffffff8100bedb&gt;] ? invalid_op+0x1b/0x20 [&lt;ffffffff81162530&gt;] ? kfree_debugcheck+0x30/0x40 [&lt;ffffffffa035ebae&gt;] ? cfs_free+0xe/0x10 [libcfs] [&lt;ffffffff8116594b&gt;] ? kfree+0x5b/0x2a0 [&lt;ffffffffa035ebae&gt;] ? cfs_free+0xe/0x10 [libcfs] [&lt;ffffffffa04ceb73&gt;] ? lu_global_key_fini+0xa3/0xf0 [obdclass] [&lt;ffffffffa04cf380&gt;] ? key_fini+0x60/0x190 [obdclass] [&lt;ffffffffa04cf4df&gt;] ? keys_fini+0x2f/0x120 [obdclass] [&lt;ffffffffa04cf5fd&gt;] ? lu_context_fini+0x2d/0xc0 [obdclass] [&lt;ffffffffa0b86aa2&gt;] ? osd_trans_commit_cb+0xe2/0x2b0 [osd_ldiskfs] [&lt;ffffffffa0a3f21a&gt;] ? ldiskfs_journal_commit_callback+0x8a/0xc0 [ldiskfs] [&lt;ffffffffa00a18af&gt;] ? jbd2_journal_commit_transaction+0x110f/0x1530 [jbd2] [&lt;ffffffff810096f0&gt;] ? __switch_to+0xd0/0x320 [&lt;ffffffff8107eabb&gt;] ? try_to_del_timer_sync+0x7b/0xe0 [&lt;ffffffffa00a7128&gt;] ? kjournald2+0xb8/0x220 [jbd2] [&lt;ffffffff810920d0&gt;] ? autoremove_wake_function+0x0/0x40 [&lt;ffffffffa00a7070&gt;] ? kjournald2+0x0/0x220 [jbd2] [&lt;ffffffff81091d66&gt;] ? kthread+0x96/0xa0 [&lt;ffffffff8100c14a&gt;] ? child_rip+0xa/0x20 [&lt;ffffffff81091cd0&gt;] ? kthread+0x0/0xa0 [&lt;ffffffff8100c140&gt;] ? child_rip+0x0/0x20 </pre> </div></div> <p>Keith local vm MDS panic -v1 dmesg</p> <p>Hi Keith,</p> <p>I created several test patches per the following comments from Andreas:</p> <blockquote><p>If there are no obvious sources of this corruption, it probably makes sense to submit this test patch as several separate changes, each based on one of the recent 2.2.* tags, to see if we can isolate when this corruption started.</p></blockquote> <p>Patch on tag 2.2.94: <a href="http://review.whamcloud.com/#change,3921" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,3921</a><br/> Patch on tag 2.3.50: <a href="http://review.whamcloud.com/#change,3918" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,3918</a><br/> Patch on tag 2.2.93: <a href="http://review.whamcloud.com/#change,3919" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,3919</a><br/> Patch on tag 2.2.92: <a href="http://review.whamcloud.com/#change,3920" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,3920</a></p> <p>Hope we can isolate the issue.</p> <p>Keith, can you please fix Yu Jian's patches that hit build failures.</p> <p>The 2.3.50 patch failed to build due to built-in version checks, so it needs to be rebased one patch later (git hash 388111848489ef99b1fa31ce8fef255ab9c08e84). I haven't investigated the other failure, but hopefully it is similarly trivial. Please get to this ASAP so that the testing can be started on these patches, and hopefully we can isolate this serious defect more quickly.</p> <p>Ok, I enabled debug pagealloc and caught the offender I think:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">[ 3201.251050] Lustre: 9532:0:(mdt_lproc.c:418:lprocfs_wr_identity_upcall()) lus tre-MDT0000: identity upcall set to /home/green/git/lustre-release/lustre/utils/ l_getidentity [ 3204.127208] BUG: unable to handle kernel paging request at ffff88021851ab48 [ 3204.129164] IP: [&lt;ffffffff811d9dd8&gt;] posix_acl_to_xattr+0x28/0x80 [ 3204.130098] PGD 1a26063 PUD 2766067 PMD 2829067 PTE 800000021851a160 [ 3204.130673] Oops: 0002 [#1] SMP DEBUG_PAGEALLOC [ 3204.130673] last sysfs file: /sys/devices/system/cpu/possible [ 3204.130673] CPU 1 [ 3204.130673] Modules linked in: lustre obdfilter ost cmm mdt osd_ldiskfs fsfil t_ldiskfs ldiskfs mdd mds mgs lquota obdecho mgc lov osc mdc lmv fid fld ptlrpc obdclass lvfs ksocklnd lnet libcfs ext2 exportfs jbd sha512_generic sha256_gener ic sunrpc ipv6 microcode virtio_balloon virtio_net i2c_piix4 i2c_core ext4 mbcac he jbd2 virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] [ 3204.130673] [ 3204.130673] Pid: 8603, comm: mdt00_001 Not tainted 2.6.32-debug #3 Bochs Boch s [ 3204.130673] RIP: 0010:[&lt;ffffffff811d9dd8&gt;] [&lt;ffffffff811d9dd8&gt;] posix_acl_to _xattr+0x28/0x80 [ 3204.130673] RSP: 0018:ffff88027a9459d0 EFLAGS: 00010287 [ 3204.130673] RAX: 0000000000000034 RBX: ffff88021851ab48 RCX: 0000000000000034 [ 3204.130673] RDX: 0000000000000060 RSI: ffff88021851ab48 RDI: ffff88023a694e50 [ 3204.130673] RBP: ffff88027a9459d0 R08: ffffffffa055f3cd R09: ffffffffa05f8e36 [ 3204.130673] R10: 0000000000000000 R11: ffffffffa05fee98 R12: ffff88023a694e50 [ 3204.130673] R13: 0000000000000060 R14: ffffffffa055f3cd R15: ffff8802470b5b98 [ 3204.130673] FS: 00007fcc8f724700(0000) GS:ffff880028240000(0000) knlGS:0000000000000000 [ 3204.130673] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 3204.130673] CR2: ffff88021851ab48 CR3: 00000002653cb000 CR4: 00000000000006e0 [ 3204.130673] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 3204.130673] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 3204.130673] <span class="code-object">Process</span> mdt00_001 (pid: 8603, threadinfo ffff88027a944000, task ffff88027a942540) [ 3204.130673] Stack: [ 3204.130673] ffff88027a945a10 ffffffffa05ac598 ffff88027a9459f0 ffffffffa067b661 [ 3204.130673] &lt;d&gt; 0000000000000010 ffff880222a70ef0 ffff880202a25bb8 ffff88027848b928 [ 3204.130673] &lt;d&gt; ffff88027a945a20 ffffffffa05ac612 ffff88027a945a30 ffffffff811a0147 [ 3204.130673] Call Trace: [ 3204.130673] [&lt;ffffffffa05ac598&gt;] ldiskfs_xattr_get_acl+0x78/0xa0 [ldiskfs] [ 3204.130673] [&lt;ffffffffa067b661&gt;] ? libcfs_debug_msg+0x41/0x50 [libcfs] [ 3204.130673] [&lt;ffffffffa05ac612&gt;] ldiskfs_xattr_get_acl_access+0x22/0x30 [ldiskfs] [ 3204.130673] [&lt;ffffffff811a0147&gt;] generic_getxattr+0x87/0x90 [ 3204.130673] [&lt;ffffffffa074af64&gt;] osd_xattr_get+0x284/0x360 [osd_ldiskfs] [ 3204.130673] [&lt;ffffffffa05356d0&gt;] mdd_attr_set+0x15b0/0x2250 [mdd] [ 3204.130673] [&lt;ffffffffa067b661&gt;] ? libcfs_debug_msg+0x41/0x50 [libcfs] [ 3204.130673] [&lt;ffffffffa0888a26&gt;] cml_attr_set+0x66/0x1a0 [cmm] [ 3204.130673] [&lt;ffffffffa07bc768&gt;] mdt_attr_set+0x2a8/0x590 [mdt] [ 3204.130673] [&lt;ffffffffa07bd096&gt;] mdt_reint_setattr+0x646/0x13f0 [mdt] [ 3204.130673] [&lt;ffffffffa067b661&gt;] ? libcfs_debug_msg+0x41/0x50 [libcfs] [ 3204.130673] [&lt;ffffffffa07b6081&gt;] mdt_reint_rec+0x41/0xe0 [mdt] [ 3204.130673] [&lt;ffffffffa07af42a&gt;] mdt_reint_internal+0x50a/0x810 [mdt] [ 3204.130673] [&lt;ffffffffa07af774&gt;] mdt_reint+0x44/0xe0 [mdt] [ 3204.130673] [&lt;ffffffffa07a1d42&gt;] mdt_handle_common+0x922/0x1740 [mdt] [ 3204.130673] [&lt;ffffffffa07a2c35&gt;] mdt_regular_handle+0x15/0x20 [mdt] [ 3204.130673] [&lt;ffffffffa0f7486f&gt;] ptlrpc_server_handle_request+0x44f/0xee0 [ptlrpc] [ 3204.130673] [&lt;ffffffffa066b66e&gt;] ? cfs_timer_arm+0xe/0x10 [libcfs] [ 3204.130673] [&lt;ffffffffa0f6d711&gt;] ? ptlrpc_wait_event+0xb1/0x2a0 [ptlrpc] [ 3204.130673] [&lt;ffffffffa067b661&gt;] ? libcfs_debug_msg+0x41/0x50 [libcfs] [ 3204.130673] [&lt;ffffffff81051f73&gt;] ? __wake_up+0x53/0x70 [ 3204.130673] [&lt;ffffffffa0f773de&gt;] ptlrpc_main+0xaee/0x1800 [ptlrpc] [ 3204.130673] [&lt;ffffffffa0f768f0&gt;] ? ptlrpc_main+0x0/0x1800 [ptlrpc] [ 3204.130673] [&lt;ffffffff8100c14a&gt;] child_rip+0xa/0x20 [ 3204.130673] [&lt;ffffffffa0f768f0&gt;] ? ptlrpc_main+0x0/0x1800 [ptlrpc] [ 3204.130673] [&lt;ffffffffa0f768f0&gt;] ? ptlrpc_main+0x0/0x1800 [ptlrpc] [ 3204.130673] [&lt;ffffffff8100c140&gt;] ? child_rip+0x0/0x20 [ 3204.130673] Code: c3 90 90 55 48 89 e5 0f 1f 44 00 00 48 63 47 04 48 85 f6 8d 04 c5 04 00 00 00 75 07 c9 c3 0f 1f 44 00 00 48 63 c8 48 39 d1 77 48 &lt;c7&gt; 06 02 00 00 00 8b 57 04 85 d2 74 e4 48 83 c6 04 31 d2 0f 1f [ 3204.130673] RIP [&lt;ffffffff811d9dd8&gt;] posix_acl_to_xattr+0x28/0x80 [ 3204.130673] RSP &lt;ffff88027a9459d0&gt; [ 3204.130673] CR2: ffff88021851ab48 [ 3204.130673] ---[ end trace f9aa2ef75b2f20bf ]--- </pre> </div></div> <p>Ok, my current theory is we have a problem coming from patch in <a href="https://jira.whamcloud.com/browse/LU-998" title="Test failure on test suite sanity, subtest test_103" class="issue-link" data-issue-key="LU-998"><del>LU-998</del></a>.</p> <p>mdd_declare_attr_set:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-keyword">if</span> (ma-&gt;ma_attr.la_valid &amp; LA_MODE) { mdd_read_lock(env, obj, MOR_TGT_CHILD); rc = mdo_xattr_get(env, obj, buf, XATTR_NAME_ACL_ACCESS, BYPASS_CAPA); mdd_read_unlock(env, obj); </pre> </div></div> <p>here we'll just use whatever happens to be in buf which sometimes ends up as garbage.</p> <p>I am not exactly clear on how it's supposed to be used, but it looks like the buffer should be reinitialized every time it's about to be used afresh (frankly I am not even sure why is it needed because it seems to be always duplicated in some local vars that we pass around). Other cases in that same function do initialize the buffer address.</p> <p>My reading of the code is such that we are just trying to make sure the xattrs exist at all, but if the buf happens to contain some pointer from before and declared size is big enough to fit the xattr, we read them to the buffer overwriting whatever important data was there (and it seems it's frequently points to freed and then possibly reallocated data too).</p> <p>The fix for this particular problem is to probably pass in &amp;LU_BUF_NULL instead of buf argument. (patch at <a href="http://review.whamcloud.com/3928" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/3928</a> and seems to be holding well in my testing)<br/> The wider question if the buf is supposed to point to freed memory at any time still remains to be open and I need somebody's help to determine this.<br/> If that is not supposed to happen, we can track down offending user (and I imagine it's probably not super valid because occasionally on shutdown we do try to free that memory that buffer points to and if it's invalid, that's a problem, even though nobody has ever seen this before for some reason).</p> <p>Oleg are you testing with DEBUG_SLAB as well? child_rip is the only thing that seems common there is no mention of the slab code path and cfs_free in your back trace. </p> <p>Can you confirm that the slab errors have gone away with your patch? I have put your patch in my test queue on the bisect setup as well. </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Keith, can you please fix Yu Jian's patches that hit build failures. The 2.3.50 patch failed to build due to built-in version checks, so it needs to be rebased one patch later (git hash 388111848489ef99b1fa31ce8fef255ab9c08e84). I haven't investigated the other failure, but hopefully it is similarly trivial. Please get to this ASAP so that the testing can be started on these patches, and hopefully we can isolate this serious defect more quickly. </pre> </div></div> <p>I have started a fresh build/bisect setup with client-7 client-17 client-18. I am able to reliability re-create here. This is the exact build that does not trigger on my system and the same config changes at Yu's. I fear that running on a random setups (via autotest) may lead to false negatives since some setups produce the error and some don't. I will kick the builds along but I would not trust any single run passes as "Not Broken". </p> <p>Keith, PAGEALLOC debug is a more powerful (and slow) kernel feature to track random memory writes. What it does is when you free a page instead of merely writing it with some poison value to check later (what you are hitting, and once hit - no way to know who made the write) it actually unmaps the page from address space, so when the offender comes to do the write, the page is not there and we get the nice oops pointing at the red-handed writer to the place they are not supposed to be writing.</p> <p>Oleg I am familiar with PAGEALLOC. I am just wondering if it is a separate issue or not. I didn't get a chance to test your patch but I will do it first thing tomorrow. </p> <p>On my nodes can produce the issue I was able to test 2.2.59 and I don't see the issue but lu-998 code should be present it the release. </p> <p>I am building on the client nodes now to speed thing up I hope tomorrow I can get thing greatly narrowed down if Oleg's patch is not the root issue. </p> <p>Sorry I didn't get a change to push Yu's build errors yet.</p> <p>Hi Keith,<br/> I submitted a patch which contained Oleg's patch and enabled the kernel debug options to verify whether the issue is resolved or not on master branch: <a href="http://review.whamcloud.com/3936" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/3936</a>.</p> <p>Keith, ok. Well, the original issue my patch fixes is there since long ago, it might have been hidden to because nobody was leaving incorrect pointers in the mti_info buf, and then later somebody started to do it and that's why everything started to fall apart lately.<br/> Granted I did not try to go back all the way to see if I can reproduce it on just the original commit</p> <p>I been running <a href="http://review.whamcloud.com/3928" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/3928</a> with Master for a few hours now and I do not seem to be able to reproduce the errors. It looks to be the fix to me. </p> <p>If Yu's latest build agrees I will cherry pick it to b2_3 and b2_2. </p> <p>Thanks for finding this Oleg. </p> <p>Don't create separate patches for other branches, please/ I'll cherry-pick this patch directly since that part of the code have not changed since landing.</p> <blockquote><p>I submitted a patch which contained Oleg's patch and enabled the kernel debug options to verify whether the issue is resolved or not on master branch: <a href="http://review.whamcloud.com/3936" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/3936</a>.</p></blockquote> <p>sanity test 103 passed without detecting slab corruption issue.</p> <p>I have almost 24 hours of testing and still no recreation. </p> <p>Oleg I will leave the cherry-pick to you. </p> <p>Landed for 2.3 and 2.4</p> <p>review-dne failed as:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>12:47:05:[ 3488.673687] Lustre: 3067:0:(client.c:1903:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1386362637/real 1386362637] req@ffff880036623bb8 x1453703367244336/t0(0) o38-&gt;lustre-MDT0002-osp-MDT0000@10.10.16.232@tcp:24/4 lens 400/544 e 0 to 1 dl 1386362662 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 12:47:06:[ 3488.676610] Lustre: 3067:0:(client.c:1903:ptlrpc_expire_one_request()) Skipped 305 previous similar messages 12:47:06:[ 3673.675126] BUG: unable to handle kernel paging request at ffff880022a6ae30 12:47:06:[ 3673.675968] IP: [&lt;ffffffffa09ef395&gt;] ksocknal_send+0x165/0x450 [ksocklnd] 12:47:06:[ 3673.676583] PGD 1a26063 PUD 1a2a063 PMD 215067 PTE 22a6a060 12:47:06:[ 3673.677103] Oops: 0000 [#1] SMP DEBUG_PAGEALLOC 12:47:06:[ 3673.677103] last sysfs file: /sys/devices/system/cpu/possible 12:47:07:[ 3673.677103] CPU 1 12:47:07:[ 3673.677103] Modules linked in: osp(U) mdd(U) lfsck(U) lod(U) mdt(U) mgs(U) mgc(U) osd_ldiskfs(U) lquota(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ksocklnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic sha256_generic libcfs(U) ldiskfs(U) jbd2 nfsd exportfs autofs4 nfs lockd fscache auth_rpcgss nfs_acl sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: speedstep_lib] 12:47:07:[ 3673.677103] 12:47:07:[ 3673.677103] Pid: 3069, comm: ptlrpcd_1 Not tainted 2.6.32-358.23.2.el6_lustre.g3ddc521.x86_64 #1 Red Hat KVM 12:47:07:[ 3673.677103] RIP: 0010:[&lt;ffffffffa09ef395&gt;] [&lt;ffffffffa09ef395&gt;] ksocknal_send+0x165/0x450 [ksocklnd] 12:47:07:[ 3673.677103] RSP: 0018:ffff88005f7bf980 EFLAGS: 00010286 12:47:07:[ 3673.677103] RAX: 0000000000000000 RBX: ffff88000eb08ef0 RCX: 0000000000000009 12:47:07:[ 3673.677103] RDX: ffff880002300000 RSI: ffff880002311b20 RDI: ffffffffa09feb38 12:47:07:[ 3673.677103] RBP: ffff88005f7bf9f0 R08: 0000000000000000 R09: 0000000000000000 12:47:07:[ 3673.677103] R10: ffff8800497d37e0 R11: 0000000000000000 R12: ffff880022a6adf0 12:47:07:[ 3673.677103] R13: 0000000000000001 R14: ffff880064cd64e0 R15: 00000000000000e0 12:47:07:[ 3673.677103] FS: 0000000000000000(0000) GS:ffff880002300000(0000) knlGS:0000000000000000 12:47:07:[ 3673.677103] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b 12:47:07:[ 3673.677103] CR2: ffff880022a6ae30 CR3: 0000000001a25000 CR4: 00000000000006e0 12:47:07:[ 3673.677103] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 12:47:07:[ 3673.677103] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 12:47:07:[ 3673.677103] Process ptlrpcd_1 (pid: 3069, threadinfo ffff88005f7be000, task ffff88005c46c340) 12:47:07:[ 3673.677103] Stack: 12:47:07:[ 3673.677103] ffff880015f79c01 0000000122a6adf0 0000000000000000 ffff880052054df0 12:47:07:[ 3673.677103] &lt;d&gt; 0000000000000000 000000e800000000 000200000a0a10ed 0000000000003039 12:47:07:[ 3673.677103] &lt;d&gt; ffff88005f7bfa10 ffff880052054df0 ffff880022a6adf0 ffff880022a6adf0 12:47:07:[ 3673.677103] Call Trace: 12:47:08:[ 3673.677103] [&lt;ffffffffa0532dfb&gt;] lnet_ni_send+0x4b/0xf0 [lnet] 12:47:08:[ 3673.677103] [&lt;ffffffffa0537005&gt;] lnet_send+0x655/0xb80 [lnet] 12:47:08:[ 3673.677103] [&lt;ffffffffa053806a&gt;] LNetPut+0x31a/0x860 [lnet] 12:47:08:[ 3673.677103] [&lt;ffffffffa0808dc0&gt;] ptl_send_buf+0x1e0/0x550 [ptlrpc] 12:47:08:[ 3673.677103] [&lt;ffffffff8103b82c&gt;] ? kvm_clock_read+0x1c/0x20 12:47:08:[ 3673.677103] [&lt;ffffffffa080c2bd&gt;] ptl_send_rpc+0x4dd/0xcc0 [ptlrpc] 12:47:08:[ 3673.677103] [&lt;ffffffffa0800994&gt;] ptlrpc_send_new_req+0x454/0x7c0 [ptlrpc] 12:47:08:[ 3673.677103] [&lt;ffffffffa0804728&gt;] ptlrpc_check_set+0x898/0x1da0 [ptlrpc] 12:47:08:[ 3673.677103] [&lt;ffffffffa083061b&gt;] ptlrpcd_check+0x55b/0x590 [ptlrpc] 12:47:08:[ 3673.677103] [&lt;ffffffffa0830bdd&gt;] ptlrpcd+0x2ad/0x3f0 [ptlrpc] 12:47:08:[ 3673.677103] [&lt;ffffffff8105bca0&gt;] ? default_wake_function+0x0/0x20 12:47:08:[ 3673.677103] [&lt;ffffffffa0830930&gt;] ? ptlrpcd+0x0/0x3f0 [ptlrpc] 12:47:08:[ 3673.677103] [&lt;ffffffff81095696&gt;] kthread+0x96/0xa0 12:47:08:[ 3673.677103] [&lt;ffffffff8100c10a&gt;] child_rip+0xa/0x20 12:47:08:[ 3673.677103] [&lt;ffffffff81095600&gt;] ? kthread+0x0/0xa0 12:47:09:[ 3673.677103] [&lt;ffffffff8100c100&gt;] ? child_rip+0x0/0x20 12:47:09:[ 3673.677103] Code: 00 c7 43 60 c1 00 00 00 48 89 de 48 c7 43 70 00 00 00 00 48 c7 43 68 00 00 00 00 48 8b 55 c0 8b 4d c8 48 8b 7d a8 e8 7b fa ff ff &lt;41&gt; f6 44 24 40 08 74 1b 8b 4d b4 85 c9 0f 84 c8 00 00 00 65 48 12:47:09:[ 3673.677103] RIP [&lt;ffffffffa09ef395&gt;] ksocknal_send+0x165/0x450 [ksocklnd] 12:47:09:[ 3673.677103] RSP &lt;ffff88005f7bf980&gt; 12:47:09:[ 3673.677103] CR2: ffff880022a6ae30 12:47:09:[ 0.000000] Initializing cgroup subsys cpuset 12:47:09:[ 0.000000] Initializing cgroup subsys cpu </pre> </div></div> Duplicate LU-1235 Related LU-1877 LU-1844 Development Rank 1|hzv4cf: Rank (Obsolete) 4237 Severity