https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-3598 Lustre <p>This issue was created by maloo for sarah &lt;sarah@whamcloud.com&gt;</p> <p>This issue relates to the following test suite run: <a href="http://maloo.whamcloud.com/test_sets/02803404-ed9b-11e2-8e3a-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">http://maloo.whamcloud.com/test_sets/02803404-ed9b-11e2-8e3a-52540035b04c</a>.</p> <p>The sub-test test_iozone failed with the following error:</p> <blockquote> <p>test failed to respond and timed out</p></blockquote> <p>client console:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>12:52:41:Lustre: DEBUG MARKER: == sanity-benchmark test iozone: iozone == 12:52:37 (1373831557) 12:52:41:Lustre: DEBUG MARKER: /usr/sbin/lctl mark min OST has 1850592kB available, using 3844936kB file size 12:52:41:Lustre: DEBUG MARKER: min OST has 1850592kB available, using 3844936kB file size 12:54:03:ptlrpcd_1: page allocation failure. order:1, mode:0x40 12:54:03:Pid: 28900, comm: ptlrpcd_1 Not tainted 2.6.32-358.11.1.el6.x86_64 #1 12:54:03:Call Trace: 12:54:03: [&lt;ffffffff8112c157&gt;] ? __alloc_pages_nodemask+0x757/0x8d0 12:54:03: [&lt;ffffffffa06627df&gt;] ? ptlrpc_set_add_new_req+0xcf/0x150 [ptlrpc] 12:54:03: [&lt;ffffffff81166b02&gt;] ? kmem_getpages+0x62/0x170 12:54:03: [&lt;ffffffff8116771a&gt;] ? fallback_alloc+0x1ba/0x270 12:54:03: [&lt;ffffffff8116716f&gt;] ? cache_grow+0x2cf/0x320 12:54:03: [&lt;ffffffff81167499&gt;] ? ____cache_alloc_node+0x99/0x160 12:54:03: [&lt;ffffffffa048fd57&gt;] ? LNetMDAttach+0x157/0x5a0 [lnet] 12:54:03: [&lt;ffffffff81168269&gt;] ? __kmalloc+0x189/0x220 12:54:03: [&lt;ffffffffa048fd57&gt;] ? LNetMDAttach+0x157/0x5a0 [lnet] 12:54:03: [&lt;ffffffffa0670b55&gt;] ? ptlrpc_register_bulk+0x265/0x9d0 [ptlrpc] 12:54:03: [&lt;ffffffffa069ea48&gt;] ? sptlrpc_import_sec_ref+0x28/0x50 [ptlrpc] 12:54:03: [&lt;ffffffffa0671fa2&gt;] ? ptl_send_rpc+0x232/0xc40 [ptlrpc] 12:54:03: [&lt;ffffffffa06a65b4&gt;] ? sptlrpc_req_refresh_ctx+0x164/0x970 [ptlrpc] 12:54:03: [&lt;ffffffffa0890971&gt;] ? osc_io_unplug+0x11/0x20 [osc] 12:54:03: [&lt;ffffffffa06676bb&gt;] ? ptlrpc_send_new_req+0x45b/0x7a0 [ptlrpc] 12:54:03: [&lt;ffffffff81055ab3&gt;] ? __wake_up+0x53/0x70 12:54:03: [&lt;ffffffffa066b3a8&gt;] ? ptlrpc_check_set+0x878/0x1b20 [ptlrpc] 12:54:03: [&lt;ffffffffa06976cb&gt;] ? ptlrpcd_check+0x53b/0x560 [ptlrpc] 12:54:03: [&lt;ffffffff8109705c&gt;] ? remove_wait_queue+0x3c/0x50 12:54:03: [&lt;ffffffffa0697b50&gt;] ? ptlrpcd+0x190/0x380 [ptlrpc] 12:54:03: [&lt;ffffffff81063310&gt;] ? default_wake_function+0x0/0x20 12:54:03: [&lt;ffffffffa06979c0&gt;] ? ptlrpcd+0x0/0x380 [ptlrpc] 12:54:03: [&lt;ffffffff81096936&gt;] ? kthread+0x96/0xa0 12:54:03: [&lt;ffffffff8100c0ca&gt;] ? child_rip+0xa/0x20 12:54:03: [&lt;ffffffff810968a0&gt;] ? kthread+0x0/0xa0 12:54:03: [&lt;ffffffff8100c0c0&gt;] ? child_rip+0x0/0x20 12:54:03:Mem-Info: 12:54:03:Node 0 DMA per-cpu: 12:54:03:CPU 0: hi: 0, btch: 1 usd: 0 12:54:03:Node 0 DMA32 per-cpu: 12:54:03:CPU 0: hi: 186, btch: 31 usd: 159 </pre> </div></div> server and client: lustre-master build #1560 LU-3598

Failure on test suite sanity-benchmark test_iozone: page allocation failure

Bug Minor Resolved Duplicate WC Triage Maloo Tue, 16 Jul 2013 18:06:01 +0000 Thu, 13 Feb 2014 22:05:11 +0000 Thu, 13 Feb 2014 22:05:11 +0000 Lustre 2.5.0 Lustre 2.5.1 0 6 <p>The failure is not fatal is is how the rest of the code deals with it. </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>12:54:03:128789 pages non-shared 12:54:03:LNetError: 28900:0:(lib-lnet.h:457:lnet_md_alloc()) LNET: out of memory at /var/lib/jenkins/workspace/lustre-master/arch/x86_64/build_type/client/distro/el6/ib_stack/inkernel/BUILD/BUILD/lustre-2.4.52/lnet/include/lnet/lib-lnet.h:457 (tried to alloc '(md)' = 4208) 12:54:03:LNetError: 28900:0:(lib-lnet.h:457:lnet_md_alloc()) LNET: 10894919 total bytes allocated by lnet 12:54:03:BUG: unable to handle kernel NULL pointer dereference at 00000000000000b8 12:54:03:IP: [&lt;ffffffffa0670d5a&gt;] ptlrpc_register_bulk+0x46a/0x9d0 [ptlrpc] 12:54:03:PGD 5e7df067 PUD 38cce067 PMD 0 12:54:03:Oops: 0000 [#1] SMP </pre> </div></div> <p>This looks to be the same as: <a href="https://jira.whamcloud.com/browse/LU-3585" title="Client panic during IOR single file per process: Lnet out of Memory" class="issue-link" data-issue-key="LU-3585"><del>LU-3585</del></a> Client panic during IOR single file per process. Lnet is not failing allocation very well it seems.</p> <p>Same Lnet error message. </p> <p>Lets just track this in the earlier LU. </p> <p>Hit this error again in tag-2.4.93 testing<br/> <a href="https://maloo.whamcloud.com/test_sets/9eeff4f8-2727-11e3-88c6-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/9eeff4f8-2727-11e3-88c6-52540035b04c</a></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>22:09:10:Lustre: DEBUG MARKER: == sanity-benchmark test iozone: iozone == 22:09:03 (1380172143) 22:09:10:Lustre: DEBUG MARKER: /usr/sbin/lctl mark min OST has 1850320kB available, using 3844920kB file size 22:09:11:Lustre: DEBUG MARKER: min OST has 1850320kB available, using 3844920kB file size 22:11:24:ptlrpcd_1: page allocation failure. order:1, mode:0x40 22:11:25:Pid: 2459, comm: ptlrpcd_1 Not tainted 2.6.32-358.18.1.el6.x86_64 #1 22:11:25:Call Trace: 22:11:26: [&lt;ffffffff8112c257&gt;] ? __alloc_pages_nodemask+0x757/0x8d0 22:11:27: [&lt;ffffffff81166d92&gt;] ? kmem_getpages+0x62/0x170 22:11:27: [&lt;ffffffff811679aa&gt;] ? fallback_alloc+0x1ba/0x270 22:11:27: [&lt;ffffffff811673ff&gt;] ? cache_grow+0x2cf/0x320 22:11:27: [&lt;ffffffff81167729&gt;] ? ____cache_alloc_node+0x99/0x160 22:11:28: [&lt;ffffffffa048ced7&gt;] ? LNetMDAttach+0x157/0x5a0 [lnet] 22:11:28: [&lt;ffffffff811684f9&gt;] ? __kmalloc+0x189/0x220 22:11:28: [&lt;ffffffffa048ced7&gt;] ? LNetMDAttach+0x157/0x5a0 [lnet] 22:11:28: [&lt;ffffffffa066dae5&gt;] ? ptlrpc_register_bulk+0x265/0x9d0 [ptlrpc] 22:11:29: [&lt;ffffffffa069a608&gt;] ? sptlrpc_import_sec_ref+0x28/0x50 [ptlrpc] 22:11:30: [&lt;ffffffffa066ef42&gt;] ? ptl_send_rpc+0x232/0xc40 [ptlrpc] 22:11:30: [&lt;ffffffffa06a1ebc&gt;] ? sptlrpc_req_refresh_ctx+0x15c/0x930 [ptlrpc] 22:11:30: [&lt;ffffffffa0664674&gt;] ? ptlrpc_send_new_req+0x454/0x790 [ptlrpc] 22:11:30: [&lt;ffffffffa06683e8&gt;] ? ptlrpc_check_set+0x888/0x1b40 [ptlrpc] 22:11:31: [&lt;ffffffffa069322b&gt;] ? ptlrpcd_check+0x53b/0x560 [ptlrpc] 22:11:31: [&lt;ffffffffa069374b&gt;] ? ptlrpcd+0x20b/0x370 [ptlrpc] 22:11:32: [&lt;ffffffff81063410&gt;] ? default_wake_function+0x0/0x20 22:11:33: [&lt;ffffffffa0693540&gt;] ? ptlrpcd+0x0/0x370 [ptlrpc] 22:11:33: [&lt;ffffffff81096a36&gt;] ? kthread+0x96/0xa0 22:11:33: [&lt;ffffffff8100c0ca&gt;] ? child_rip+0xa/0x20 22:11:33: [&lt;ffffffff810969a0&gt;] ? kthread+0x0/0xa0 22:11:33: [&lt;ffffffff8100c0c0&gt;] ? child_rip+0x0/0x20 22:11:33:Mem-Info: 22:11:33:Node 0 DMA per-cpu: 22:11:34:CPU 0: hi: 0, btch: 1 usd: 0 22:11:34:Node 0 DMA32 per-cpu: 22:11:35:CPU 0: hi: 186, btch: 31 usd: 33 22:11:35:active_anon:6072 inactive_anon:6123 isolated_anon:0 22:11:35: active_file:29973 inactive_file:288526 isolated_file:0 22:11:35: unevictable:0 dirty:36357 writeback:16384 unstable:0 22:11:35: free:16750 slab_reclaimable:9827 slab_unreclaimable:110251 22:11:36: mapped:3569 shmem:38 pagetables:1027 bounce:0 22:11:36:Node 0 DMA free:8272kB min:332kB low:412kB high:496kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:5748kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15324kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:40kB slab_unreclaimable:1672kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 22:11:36:lowmem_reserve[]: 0 2003 2003 2003 22:11:37:Node 0 DMA32 free:58728kB min:44720kB low:55900kB high:67080kB active_anon:24288kB inactive_anon:24492kB active_file:119892kB inactive_file:1148356kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:2052064kB mlocked:0kB dirty:145428kB writeback:65536kB mapped:14276kB shmem:152kB slab_reclaimable:39268kB slab_unreclaimable:439332kB kernel_stack:1648kB pagetables:4108kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 22:11:38:lowmem_reserve[]: 0 0 0 0 22:11:38:Node 0 DMA: 2*4kB 3*8kB 1*16kB 1*32kB 4*64kB 4*128kB 5*256kB 2*512kB 1*1024kB 2*2048kB 0*4096kB = 8272kB 22:11:38:Node 0 DMA32: 13044*4kB 25*8kB 11*16kB 3*32kB 3*64kB 0*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 1*4096kB = 58728kB 22:11:38:318554 total pagecache pages 22:11:38:0 pages in swap cache 22:11:38:Swap cache stats: add 0, delete 0, find 0/0 22:11:38:Free swap = 4128760kB 22:11:39:Total swap = 4128760kB 22:11:39:524284 pages RAM 22:11:40:43669 pages reserved 22:11:40:342147 pages shared 22:11:41:154367 pages non-shared 22:11:41:LNetError: 2459:0:(lib-lnet.h:457:lnet_md_alloc()) LNET: out of memory at /var/lib/jenkins/workspace/lustre-master/arch/x86_64/build_type/client/distro/el6/ib_stack/inkernel/BUILD/BUILD/lustre-2.4.93/lnet/include/lnet/lib-lnet.h:457 (tried to alloc '(md)' = 4208) 22:11:41:LNetError: 2459:0:(lib-lnet.h:457:lnet_md_alloc()) LNET: 10962695 total bytes allocated by lnet 22:11:41:LustreError: 2459:0:(niobuf.c:376:ptlrpc_register_bulk()) lustre-OST0004-osc-ffff88005dc12000: LNetMDAttach failed x1447210203467200/0: rc = -12 </pre> </div></div> <p>similar error in SLES11 SP2 client</p> <p><a href="https://maloo.whamcloud.com/test_sets/0194542a-2915-11e3-b598-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/0194542a-2915-11e3-b598-52540035b04c</a></p> <p>Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b2_5/9/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_5/9/</a><br/> Distro/Arch: RHEL6.4/x86_64(server), SLES11SP3/x86_64(client)</p> <p>The same failure occurred:<br/> <a href="https://maloo.whamcloud.com/test_sets/2b6e3350-7826-11e3-9a46-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/2b6e3350-7826-11e3-9a46-52540035b04c</a></p> <p>More instance on Lustre b2_5 branch:<br/> <a href="https://maloo.whamcloud.com/test_sets/4bc449ba-8fe5-11e3-8d88-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/4bc449ba-8fe5-11e3-8d88-52540035b04c</a></p> <p>Also shows "mode:0x40" when allocation fails.</p> Duplicate LU-4357 Related LU-3585 Development Rank 1|hzvviv: Rank (Obsolete) 9130 Severity