https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-1513 Lustre <p>This issue was created by maloo for Li Wei &lt;liwei@whamcloud.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://maloo.whamcloud.com/test_sets/9d879d58-b4de-11e1-bad8-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/9d879d58-b4de-11e1-bad8-52540035b04c</a>.</p> <p>The sub-test lustre-initialization_1 failed with the following error:</p> <blockquote> <p>Test system failed to start single suite, so abandoning all hope and giving up</p></blockquote> <p>Info required for matching: lustre-initialization-1 lustre-initialization_1</p> <p>From the OSS console log:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>14:52:23:LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: 14:52:23:LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: 14:52:23:------------[ cut here ]------------ 14:52:23:kernel BUG at mm/slab.c:2833! 14:52:23:invalid opcode: 0000 [#1] SMP 14:52:23:last sysfs file: /sys/devices/pci0000:00/0000:00:05.0/virtio1/block/vda/queue/max_sectors_kb 14:52:24:CPU 1 14:52:24:Modules linked in: obdfilter(U) fsfilt_ldiskfs(U) ost(U) mgc(U) lustre(U) lquota(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ksocklnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) sha512_generic sha256_generic libcfs(U) ldiskfs(U) jbd2 nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode 8139too 8139cp mii i2c_piix4 i2c_core virtio_balloon ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: speedstep_lib] 14:52:24: 14:52:25:Pid: 5368, comm: llog_process_th Not tainted 2.6.32-220.17.1.el6_lustre.x86_64 #1 Red Hat KVM 14:52:25:RIP: 0010:[&lt;ffffffff8115e7b3&gt;] [&lt;ffffffff8115e7b3&gt;] cache_grow+0x313/0x320 14:52:25:RSP: 0018:ffff88007c1d1990 EFLAGS: 00010002 14:52:25:RAX: ffff88007f821800 RBX: ffff88007f910440 RCX: 0000000000000000 14:52:25:RDX: 0000000000000000 RSI: 0000000000041252 RDI: ffff88007f910440 14:52:25:RBP: ffff88007c1d19f0 R08: 0000000000000246 R09: 00000000fffffffe 14:52:25:R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000041252 14:52:25:R13: ffff88007f8217c0 R14: 000000000000000c R15: 0000000000000000 14:52:25:FS: 0000000000000000(0000) GS:ffff880002300000(0000) knlGS:0000000000000000 14:52:25:CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b 14:52:25:CR2: 00007f0a50fde450 CR3: 000000007c173000 CR4: 00000000000006e0 14:52:25:DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 14:52:25:DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 14:52:25:Process llog_process_th (pid: 5368, threadinfo ffff88007c1d0000, task ffff8800760ca080) 14:52:25:Stack: 14:52:25: 000000000000008e 0100000400000020 4fd7ba1600000001 0000000000062549 14:52:29:&lt;0&gt; 000014f800000000 000000af00000000 ffff88000002bb08 ffff88007f910440 14:52:29:&lt;0&gt; ffff88007fa7d480 ffff88007f8217c0 000000000000000c ffff88007f8217e0 14:52:29:Call Trace: 14:52:29: [&lt;ffffffff8115e9c2&gt;] cache_alloc_refill+0x202/0x240 14:52:29: [&lt;ffffffffa043bbf0&gt;] ? cfs_alloc+0x30/0x60 [libcfs] 14:52:29: [&lt;ffffffff8115f6e9&gt;] __kmalloc+0x1a9/0x220 14:52:29: [&lt;ffffffffa043bbf0&gt;] cfs_alloc+0x30/0x60 [libcfs] 14:52:29: [&lt;ffffffffa0b6a94e&gt;] filter_common_setup+0xde/0x13f0 [obdfilter] 14:52:29: [&lt;ffffffffa0b6c280&gt;] filter_setup+0x620/0xa20 [obdfilter] 14:52:29: [&lt;ffffffffa059d974&gt;] obd_setup+0x1b4/0x2f0 [obdclass] 14:52:29: [&lt;ffffffffa058939b&gt;] ? class_new_export+0x73b/0x970 [obdclass] 14:52:29: [&lt;ffffffffa059dcb8&gt;] class_setup+0x208/0x890 [obdclass] 14:52:29: [&lt;ffffffffa05a4e1c&gt;] class_process_config+0xbec/0x1c20 [obdclass] 14:52:29: [&lt;ffffffffa043bbf0&gt;] ? cfs_alloc+0x30/0x60 [libcfs] 14:52:29: [&lt;ffffffffa059f623&gt;] ? lustre_cfg_new+0x353/0x7e0 [obdclass] 14:52:29: [&lt;ffffffffa05a6efb&gt;] class_config_llog_handler+0x9bb/0x1610 [obdclass] 14:52:29: [&lt;ffffffffa05765b0&gt;] ? llog_lvfs_next_block+0x2d0/0x650 [obdclass] 14:52:29: [&lt;ffffffffa0570940&gt;] ? llog_process_thread+0x0/0xd00 [obdclass] 14:52:31: [&lt;ffffffffa05711c8&gt;] llog_process_thread+0x888/0xd00 [obdclass] 14:52:31: [&lt;ffffffffa0570940&gt;] ? llog_process_thread+0x0/0xd00 [obdclass] 14:52:31: [&lt;ffffffff8100c14a&gt;] child_rip+0xa/0x20 14:52:31: [&lt;ffffffffa0570940&gt;] ? llog_process_thread+0x0/0xd00 [obdclass] 14:52:31: [&lt;ffffffffa0570940&gt;] ? llog_process_thread+0x0/0xd00 [obdclass] 14:52:31: [&lt;ffffffff8100c140&gt;] ? child_rip+0x0/0x20 14:52:31:Code: 0f 1f 84 00 00 00 00 00 49 8d 54 24 30 48 c7 c0 fc ff ff ff 48 89 55 c8 e9 e1 fe ff ff 0f 0b eb fe ba 01 00 00 00 e9 2a fe ff ff &lt;0f&gt; 0b eb fe 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 41 57 41 56 14:52:31:RIP [&lt;ffffffff8115e7b3&gt;] cache_grow+0x313/0x320 14:52:31: RSP &lt;ffff88007c1d1990&gt; 14:52:31:---[ end trace edf3800813592bec ]--- 14:52:31:Kernel panic - not syncing: Fatal exception 14:52:31:Pid: 5368, comm: llog_process_th Tainted: G D ---------------- 2.6.32-220.17.1.el6_lustre.x86_64 #1 14:52:31:Call Trace: 14:52:31: [&lt;ffffffff814eccea&gt;] ? panic+0x78/0x143 14:52:31: [&lt;ffffffff814f0e84&gt;] ? oops_end+0xe4/0x100 14:52:31: [&lt;ffffffff8100f26b&gt;] ? die+0x5b/0x90 14:52:31: [&lt;ffffffff814f0754&gt;] ? do_trap+0xc4/0x160 14:52:32: [&lt;ffffffff8100ce35&gt;] ? do_invalid_op+0x95/0xb0 14:52:32: [&lt;ffffffff8115e7b3&gt;] ? cache_grow+0x313/0x320 14:52:32: [&lt;ffffffff81275b56&gt;] ? vsnprintf+0x2b6/0x5f0 14:52:32: [&lt;ffffffff8100bedb&gt;] ? invalid_op+0x1b/0x20 14:52:32: [&lt;ffffffff8115e7b3&gt;] ? cache_grow+0x313/0x320 14:52:32: [&lt;ffffffff8115e9c2&gt;] ? cache_alloc_refill+0x202/0x240 14:52:32: [&lt;ffffffffa043bbf0&gt;] ? cfs_alloc+0x30/0x60 [libcfs] 14:52:32: [&lt;ffffffff8115f6e9&gt;] ? __kmalloc+0x1a9/0x220 14:52:32: [&lt;ffffffffa043bbf0&gt;] ? cfs_alloc+0x30/0x60 [libcfs] 14:52:32: [&lt;ffffffffa0b6a94e&gt;] ? filter_common_setup+0xde/0x13f0 [obdfilter] 14:52:32: [&lt;ffffffffa0b6c280&gt;] ? filter_setup+0x620/0xa20 [obdfilter] 14:52:32: [&lt;ffffffffa059d974&gt;] ? obd_setup+0x1b4/0x2f0 [obdclass] 14:52:32: [&lt;ffffffffa058939b&gt;] ? class_new_export+0x73b/0x970 [obdclass] 14:52:32: [&lt;ffffffffa059dcb8&gt;] ? class_setup+0x208/0x890 [obdclass] 14:52:32: [&lt;ffffffffa05a4e1c&gt;] ? class_process_config+0xbec/0x1c20 [obdclass] 14:52:32: [&lt;ffffffffa043bbf0&gt;] ? cfs_alloc+0x30/0x60 [libcfs] 14:52:32: [&lt;ffffffffa059f623&gt;] ? lustre_cfg_new+0x353/0x7e0 [obdclass] 14:52:32: [&lt;ffffffffa05a6efb&gt;] ? class_config_llog_handler+0x9bb/0x1610 [obdclass] 14:52:33: [&lt;ffffffffa05765b0&gt;] ? llog_lvfs_next_block+0x2d0/0x650 [obdclass] 14:52:33: [&lt;ffffffffa0570940&gt;] ? llog_process_thread+0x0/0xd00 [obdclass] 14:52:33: [&lt;ffffffffa05711c8&gt;] ? llog_process_thread+0x888/0xd00 [obdclass] 14:52:33: [&lt;ffffffffa0570940&gt;] ? llog_process_thread+0x0/0xd00 [obdclass] 14:52:33: [&lt;ffffffff8100c14a&gt;] ? child_rip+0xa/0x20 14:52:33: [&lt;ffffffffa0570940&gt;] ? llog_process_thread+0x0/0xd00 [obdclass] 14:52:33: [&lt;ffffffffa0570940&gt;] ? llog_process_thread+0x0/0xd00 [obdclass] 14:52:33: [&lt;ffffffff8100c140&gt;] ? child_rip+0x0/0x20 </pre> </div></div> LU-1513

Test failure on test lustre-initialization-1, sanity, others: "kernel BUG at mm/slab.c:2833"

Bug Blocker Closed Fixed nasf Maloo Tue, 12 Jun 2012 23:41:52 +0000 Mon, 18 Jun 2012 18:12:24 +0000 Sat, 16 Jun 2012 17:38:48 +0000 Lustre 2.3.0 Lustre 2.3.0 0 1 <p>This is the same kernel assertion with the one in <a href="https://jira.whamcloud.com/browse/LU-1510" title="Test failure on test suite sanity-quota, subtest test_18c" class="issue-link" data-issue-key="LU-1510"><del>LU-1510</del></a>, though the stacks differ.</p> <p>This looks like kernel memory corruption, possibly due to double free, or use of freed memory. Are there any patches which have landed that appear suspicious? It might make sense to start submitting some of the recent builds back to autotest to see where the problem was first introduced. </p> <p>This is randomly affecting a large fraction of tests, so I'm bumping this to a blocker.</p> <p>All of the test failures appear to be on the OSS while doing initial configuration, so it is very likely some patch that has landed to that code recently.</p> <p>It appears that the first test failure is seen in:</p> <p><a href="https://maloo.whamcloud.com/test_sets/5219062a-b3e1-11e1-a2dd-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/5219062a-b3e1-11e1-a2dd-52540035b04c</a></p> <p>This is running a "full" test on git commit 220c6d92200792c318f5248e8e44dbcfc712c788 "<a href="https://jira.whamcloud.com/browse/LU-1402" title="cfs_alloc_flags_to_gfp should translate CFS_ALLOC_HIGH to __GFP_HIGHMEM instead of __GFP_HIGH" class="issue-link" data-issue-key="LU-1402"><del>LU-1402</del></a> libcfs: CFS_ALLOC_HIGH is __GFP_HIGHMEM", but I don't see many users of CFS_ALLOC_HIGHMEM, so I'm not sure this is the culprit. That said, it would have to be one of the patches committed shortly before this one, or we would have seen the failure earlier (about 1-in-5 tests fail).</p> <p>The reason is found. The failure was triggered here:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">lustre/obdfilter/filter.c <span class="code-keyword">static</span> <span class="code-object">int</span> filter_iobuf_pool_init(struct filter_obd *filter) { void **pool; ENTRY; ===&gt; OBD_ALLOC_GFP(filter-&gt;fo_iobuf_pool, OSS_THREADS_MAX * sizeof(*pool), GFP_KERNEL); <span class="code-keyword">if</span> (filter-&gt;fo_iobuf_pool == NULL) RETURN(-ENOMEM); filter-&gt;fo_iobuf_count = OSS_THREADS_MAX; RETURN(0); } mm/slab.c <span class="code-keyword">static</span> <span class="code-object">int</span> cache_grow(struct kmem_cache *cachep, gfp_t flags, <span class="code-object">int</span> nodeid, void *objp) { struct slab *slabp; size_t offset; gfp_t local_flags; struct kmem_list3 *l3; /* * Be lazy and only check <span class="code-keyword">for</span> valid flags here, keeping it out of the * critical path in kmem_cache_alloc(). */ ===&gt; BUG_ON(flags &amp; GFP_SLAB_BUG_MASK); ... } include/linux/gfp.h ===&gt; #define __GFP_HIGHMEM ((__force gfp_t)0x02u) ... ===&gt; #define __GFP_WAIT ((__force gfp_t)0x10u) <span class="code-comment">/* Can wait and reschedule? */</span> #define __GFP_HIGH ((__force gfp_t)0x20u) <span class="code-comment">/* Should access emergency pools? */</span> ===&gt; #define __GFP_IO ((__force gfp_t)0x40u) <span class="code-comment">/* Can start physical IO? */</span> ===&gt; #define __GFP_FS ((__force gfp_t)0x80u) <span class="code-comment">/* Can call down to low-level FS? */</span> ... ===&gt; #define GFP_KERNEL (__GFP_WAIT | __GFP_IO | __GFP_FS) ... <span class="code-comment">/* Do not use these with a slab allocator */</span> #define GFP_SLAB_BUG_MASK (__GFP_DMA32|__GFP_HIGHMEM|~__GFP_BITS_MASK) </pre> </div></div> <p>We expect "GFP_KERNEL" to be transferred into kernel slab allocator. Unfortunately, the flag "GFP_KERNEL" is converted by Lustre code as following:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">libcfs/include/libcfs/libcfs.h /* * Universal memory allocator API */ <span class="code-keyword">enum</span> cfs_alloc_flags { <span class="code-comment">/* allocation is not allowed to block */</span> CFS_ALLOC_ATOMIC = 0x1, <span class="code-comment">/* allocation is allowed to block */</span> CFS_ALLOC_WAIT = 0x2, <span class="code-comment">/* allocation should <span class="code-keyword">return</span> zeroed memory */</span> CFS_ALLOC_ZERO = 0x4, /* allocation is allowed to call file-system code to free/clean * memory */ CFS_ALLOC_FS = 0x8, <span class="code-comment">/* allocation is allowed to <span class="code-keyword">do</span> io to free/clean memory */</span> ===&gt; CFS_ALLOC_IO = 0x10, <span class="code-comment">/* don't report allocation failure to the console */</span> CFS_ALLOC_NOWARN = 0x20, <span class="code-comment">/* standard allocator flag combination */</span> CFS_ALLOC_STD = CFS_ALLOC_FS | CFS_ALLOC_IO, CFS_ALLOC_USER = CFS_ALLOC_WAIT | CFS_ALLOC_FS | CFS_ALLOC_IO, }; <span class="code-comment">/* flags <span class="code-keyword">for</span> cfs_page_alloc() in addition to <span class="code-keyword">enum</span> cfs_alloc_flags */</span> <span class="code-keyword">enum</span> cfs_alloc_page_flags { /* allow to <span class="code-keyword">return</span> page beyond KVM. It has to be mapped into KVM by * cfs_kmap() and unmapped with cfs_kunmap(). */ ===&gt; CFS_ALLOC_HIGHMEM = 0x40, CFS_ALLOC_HIGHUSER = CFS_ALLOC_WAIT | CFS_ALLOC_FS | CFS_ALLOC_IO | CFS_ALLOC_HIGHMEM, }; libcfs/libcfs/linux/linux-mem.c <span class="code-keyword">static</span> unsigned <span class="code-object">int</span> cfs_alloc_flags_to_gfp(u_int32_t flags) { unsigned <span class="code-object">int</span> mflags = 0; <span class="code-keyword">if</span> (flags &amp; CFS_ALLOC_ATOMIC) mflags |= __GFP_HIGH; <span class="code-keyword">else</span> mflags |= __GFP_WAIT; <span class="code-keyword">if</span> (flags &amp; CFS_ALLOC_NOWARN) mflags |= __GFP_NOWARN; ===&gt; <span class="code-keyword">if</span> (flags &amp; CFS_ALLOC_IO) mflags |= __GFP_IO; <span class="code-keyword">if</span> (flags &amp; CFS_ALLOC_FS) mflags |= __GFP_FS; ===&gt; <span class="code-keyword">if</span> (flags &amp; CFS_ALLOC_HIGHMEM) mflags |= __GFP_HIGHMEM; <span class="code-keyword">return</span> mflags; } </pre> </div></div> <p>Firstly, "GFP_KERNEL" was explained as "__GFP_WAIT | __GFP_IO | __GFP_FS", means "0x10 | 0x40 | 0x80".<br/> Then they are regarded as Lustre flags "CFS_ALLOC_IO | CFS_ALLOC_HIGHMEM | 0x80".<br/> And then, after the conversion of "cfs_alloc_flags_to_gfp()", they become "_<em>GFP_IO | __GFP_HIGHMEM". Those are the real flags transferred to kernel slab allocator. But the "</em>_GFP_HIGHMEM" is NOT what we expected.</p> <p>So <a href="https://jira.whamcloud.com/browse/LU-1402" title="cfs_alloc_flags_to_gfp should translate CFS_ALLOC_HIGH to __GFP_HIGHMEM instead of __GFP_HIGH" class="issue-link" data-issue-key="LU-1402"><del>LU-1402</del></a> patch triggered the existing issues. I will make patch to fix it.</p> <p>This is the patch:</p> <p><a href="http://review.whamcloud.com/#change,3118" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,3118</a></p> <p>Patch landed.</p> <p>Patch landed.</p> Duplicate LU-1529 LU-1530 Related LU-1402 LU-1510 Development Rank 1|hzv2xb: Rank (Obsolete) 3973 Severity