https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-4432 Lustre <p>While running recovery-mds-scale test failover_ost, tar operation on one of the client nodes failed as follows:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>tar: etc/libreport/plugins/rhtsupport.conf: Cannot write: Cannot allocate memory tar: Exiting with failure status due to previous errors </pre> </div></div> <p>Maloo report: <a href="https://maloo.whamcloud.com/test_sets/e8a2857a-7529-11e3-936d-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/e8a2857a-7529-11e3-936d-52540035b04c</a></p> <br/> Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b2_5/5/">http://build.whamcloud.com/job/lustre-b2_5/5/</a><br/> Distro/Arch: RHEL6.4/x86_64<br/> FSTYPE=zfs<br/> TEST_GROUP=failover<br/> LU-4432

recovery-mds-scale test_failover_ost: tar: Cannot write: Cannot allocate memory

Bug Major Resolved Duplicate WC Triage Jian Yu Sat, 4 Jan 2014 12:33:49 +0000 Thu, 13 Feb 2014 22:09:18 +0000 Thu, 13 Feb 2014 22:09:18 +0000 Lustre 2.5.1 0 5 <p>Lustre client build: <a href="http://build.whamcloud.com/job/lustre-b2_4/70/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_4/70/</a> (2.4.2)<br/> Lustre server build: <a href="http://build.whamcloud.com/job/lustre-b2_5/8/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_5/8/</a></p> <p>performance-sanity test 8 failed as follows:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>rank 0: stat(f173313) error: Cannot allocate memory </pre> </div></div> <p>Console log on OSS:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>08:33:51:Lustre: DEBUG MARKER: ===== mdsrate-stat-large.sh 08:33:51:ldlm_cn00_006: page allocation failure. order:1, mode:0x40 08:33:51:Pid: 640, comm: ldlm_cn00_006 Not tainted 2.6.32-358.18.1.el6_lustre.g6093be6.x86_64 #1 08:33:52:Call Trace: 08:33:52: [&lt;ffffffff8112c257&gt;] ? __alloc_pages_nodemask+0x757/0x8d0 08:33:52: [&lt;ffffffff8127f72c&gt;] ? put_dec+0x10c/0x110 08:33:53: [&lt;ffffffff81166d92&gt;] ? kmem_getpages+0x62/0x170 08:33:53: [&lt;ffffffff811679aa&gt;] ? fallback_alloc+0x1ba/0x270 08:33:53: [&lt;ffffffff811673ff&gt;] ? cache_grow+0x2cf/0x320 08:33:54: [&lt;ffffffff81167729&gt;] ? ____cache_alloc_node+0x99/0x160 08:33:54: [&lt;ffffffff811688f0&gt;] ? kmem_cache_alloc_node_trace+0x90/0x200 08:33:54: [&lt;ffffffff81168b0d&gt;] ? __kmalloc_node+0x4d/0x60 08:33:54: [&lt;ffffffffa0457651&gt;] ? cfs_cpt_malloc+0x31/0x60 [libcfs] 08:33:54: [&lt;ffffffffa0a42b48&gt;] ? ptlrpc_alloc_rqbd+0x1e8/0x670 [ptlrpc] 08:33:54: [&lt;ffffffffa0a430b5&gt;] ? ptlrpc_grow_req_bufs+0xe5/0x2a0 [ptlrpc] 08:33:54: [&lt;ffffffff81063410&gt;] ? default_wake_function+0x0/0x20 08:33:55: [&lt;ffffffffa0a474bd&gt;] ? ptlrpc_main+0xb5d/0x1740 [ptlrpc] 08:33:55: [&lt;ffffffffa0a46960&gt;] ? ptlrpc_main+0x0/0x1740 [ptlrpc] 08:33:55: [&lt;ffffffff81096a36&gt;] ? kthread+0x96/0xa0 08:33:55: [&lt;ffffffff8100c0ca&gt;] ? child_rip+0xa/0x20 08:33:56: [&lt;ffffffff810969a0&gt;] ? kthread+0x0/0xa0 08:33:56: [&lt;ffffffff8100c0c0&gt;] ? child_rip+0x0/0x20 </pre> </div></div> <p>Maloo report: <a href="https://maloo.whamcloud.com/test_sets/51d6d872-78ed-11e3-a27b-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/51d6d872-78ed-11e3-a27b-52540035b04c</a></p> <p>More instance on Lustre b2_5 branch:<br/> <a href="https://maloo.whamcloud.com/test_sets/2df52e18-7ab4-11e3-8b19-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/2df52e18-7ab4-11e3-8b19-52540035b04c</a></p> <p>I think I did report a similar issue some time ago: <a href="https://jira.whamcloud.com/browse/LU-4034" title="Cannot allocate memory on clients with 2.4.X" class="issue-link" data-issue-key="LU-4034"><del>LU-4034</del></a></p> <p>Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b2_5/13/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_5/13/</a><br/> Distro/Arch: RHEL6.4/x86_64<br/> TEST_GROUP=failover</p> <p>The similar issue occurred on client while running recovery-double-scale test:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>10:54:02:Lustre: DEBUG MARKER: mount -t lustre -o user_xattr,flock client-27vm3:client-27vm7:/lustre /mnt/lustre 10:54:02:mount.lustre: page allocation failure. order:2, mode:0x40 10:54:02:Pid: 10206, comm: mount.lustre Not tainted 2.6.32-358.18.1.el6.x86_64 #1 10:54:02:Call Trace: 10:54:02: [&lt;ffffffff8112c257&gt;] ? __alloc_pages_nodemask+0x757/0x8d0 10:54:03: [&lt;ffffffff81166d92&gt;] ? kmem_getpages+0x62/0x170 10:54:03: [&lt;ffffffff811679aa&gt;] ? fallback_alloc+0x1ba/0x270 10:54:03: [&lt;ffffffff811673ff&gt;] ? cache_grow+0x2cf/0x320 10:54:03: [&lt;ffffffff81167729&gt;] ? ____cache_alloc_node+0x99/0x160 10:54:03: [&lt;ffffffffa0706bc6&gt;] ? null_alloc_repbuf+0x66/0x3b0 [ptlrpc] 10:54:03: [&lt;ffffffff811684f9&gt;] ? __kmalloc+0x189/0x220 10:54:03: [&lt;ffffffffa0706bc6&gt;] ? null_alloc_repbuf+0x66/0x3b0 [ptlrpc] 10:54:03: [&lt;ffffffffa06f4f25&gt;] ? sptlrpc_cli_alloc_repbuf+0x175/0x220 [ptlrpc] 10:54:03: [&lt;ffffffffa06c88ec&gt;] ? ptl_send_rpc+0x93c/0xc40 [ptlrpc] 10:54:03: [&lt;ffffffff81281734&gt;] ? snprintf+0x34/0x40 10:54:04: [&lt;ffffffffa03e77b1&gt;] ? libcfs_debug_msg+0x41/0x50 [libcfs] 10:54:04: [&lt;ffffffffa06bd894&gt;] ? ptlrpc_send_new_req+0x454/0x790 [ptlrpc] 10:54:04: [&lt;ffffffffa06c2e7e&gt;] ? ptlrpc_set_wait+0x5be/0x860 [ptlrpc] 10:54:04: [&lt;ffffffffa053d8ec&gt;] ? lustre_get_jobid+0xcc/0x380 [obdclass] 10:54:04: [&lt;ffffffffa06cc316&gt;] ? lustre_msg_set_jobid+0xb6/0x140 [ptlrpc] 10:54:04: [&lt;ffffffffa06c31a7&gt;] ? ptlrpc_queue_wait+0x87/0x220 [ptlrpc] 10:54:04: [&lt;ffffffffa06e12d8&gt;] ? llog_client_read_header+0xd8/0x5e0 [ptlrpc] 10:54:04: [&lt;ffffffffa0533d2c&gt;] ? llog_init_handle+0xcc/0x960 [obdclass] 10:54:04: [&lt;ffffffffa0565683&gt;] ? class_config_parse_llog+0x1a3/0x330 [obdclass] 10:54:04: [&lt;ffffffffa09f0302&gt;] ? mgc_process_log+0xd22/0x18e0 [mgc] 10:54:04: [&lt;ffffffffa09f1630&gt;] ? config_recover_log_add+0x150/0x280 [mgc] 10:54:04: [&lt;ffffffffa09ea360&gt;] ? mgc_blocking_ast+0x0/0x810 [mgc] 10:54:04: [&lt;ffffffffa06aa530&gt;] ? ldlm_completion_ast+0x0/0x920 [ptlrpc] 10:54:05: [&lt;ffffffffa09f24a5&gt;] ? mgc_process_config+0x645/0x11d0 [mgc] 10:54:05: [&lt;ffffffffa0574626&gt;] ? lustre_process_log+0x256/0xa60 [obdclass] 10:54:05: [&lt;ffffffff8128ca66&gt;] ? __percpu_counter_init+0x56/0x70 10:54:05: [&lt;ffffffffa0a732d8&gt;] ? ll_fill_super+0xaa8/0x14d0 [lustre] 10:54:05: [&lt;ffffffffa057993d&gt;] ? lustre_fill_super+0x34d/0x510 [obdclass] 10:54:05: [&lt;ffffffffa05795f0&gt;] ? lustre_fill_super+0x0/0x510 [obdclass] 10:54:05: [&lt;ffffffff811845cf&gt;] ? get_sb_nodev+0x5f/0xa0 10:54:05: [&lt;ffffffffa0571545&gt;] ? lustre_get_sb+0x25/0x30 [obdclass] 10:54:05: [&lt;ffffffff81183beb&gt;] ? vfs_kern_mount+0x7b/0x1b0 10:54:05: [&lt;ffffffff81183d92&gt;] ? do_kern_mount+0x52/0x130 10:54:05: [&lt;ffffffff811a3ef2&gt;] ? do_mount+0x2d2/0x8d0 10:54:05: [&lt;ffffffff811a4580&gt;] ? sys_mount+0x90/0xe0 10:54:05: [&lt;ffffffff8100b072&gt;] ? system_call_fastpath+0x16/0x1b 10:54:05:Mem-Info: </pre> </div></div> <p>Maloo report: <a href="https://maloo.whamcloud.com/sub_tests/d75a7160-7f3d-11e3-94f3-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/sub_tests/d75a7160-7f3d-11e3-94f3-52540035b04c</a></p> <p>Shows mode:0x40 == __GFP_IO, but missing __GFP_WAIT from <a href="https://jira.whamcloud.com/browse/LU-4357" title="page allocation failure. mode:0x40 caused by missing __GFP_WAIT flag" class="issue-link" data-issue-key="LU-4357"><del>LU-4357</del></a>.</p> Duplicate LU-4357 Development Rank 1|hzwc6n: Rank (Obsolete) 12176 Severity