https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-1261 Lustre <p>While running recovery-*-scale tests on VMs with RHEL5.7/x86_64 clients and RHEL6.2/x86_64 servers, oom issue kept occurring on one of the client nodes:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>init invoked oom-killer: gfp_mask=0x200d2, order=0, oomkilladj=0 Call Trace: [&lt;ffffffff800c962a&gt;] out_of_memory+0x8e/0x2f3 [&lt;ffffffff8000f625&gt;] __alloc_pages+0x27f/0x308 [&lt;ffffffff80032903&gt;] read_swap_cache_async+0x45/0xd8 [&lt;ffffffff800cf3e3&gt;] swapin_readahead+0x60/0xd3 [&lt;ffffffff800092cb&gt;] __handle_mm_fault+0xb62/0x1039 [&lt;ffffffff8008e430&gt;] default_wake_function+0x0/0xe [&lt;ffffffff8006720b&gt;] do_page_fault+0x4cb/0x874 [&lt;ffffffff800a4931&gt;] ktime_get_ts+0x1a/0x4e [&lt;ffffffff800bfe9c&gt;] delayacct_end+0x5d/0x86 [&lt;ffffffff8005dde9&gt;] error_exit+0x0/0x84 [&lt;ffffffff80061e0e&gt;] copy_user_generic_unrolled+0x86/0xac [&lt;ffffffff800eb7f9&gt;] core_sys_select+0x1f9/0x265 [&lt;ffffffff8002cc16&gt;] mntput_no_expire+0x19/0x89 [&lt;ffffffff8001b007&gt;] cp_new_stat+0xe5/0xfd [&lt;ffffffff80016a40&gt;] sys_select+0x153/0x17c [&lt;ffffffff8005d116&gt;] system_call+0x7e/0x83 Node 0 DMA per-cpu: cpu 0 hot: high 0, batch 1 used:0 cpu 0 cold: high 0, batch 1 used:0 Node 0 DMA32 per-cpu: cpu 0 hot: high 186, batch 31 used:48 cpu 0 cold: high 62, batch 15 used:61 Node 0 Normal per-cpu: empty Node 0 HighMem per-cpu: empty Free pages: 8656kB (0kB HighMem) Active:6 inactive:486961 dirty:0 writeback:675 unstable:0 free:2164 slab:12585 mapped-file:1064 mapped-anon:596 pagetables:1241 Node 0 DMA free:3032kB min:24kB low:28kB high:36kB active:0kB inactive:0kB present:9736kB pages_scanned:0 all_unreclaimable? yes lowmem_reserve[]: 0 2003 2003 2003 Node 0 DMA32 free:5624kB min:5712kB low:7140kB high:8568kB active:24kB inactive:1947844kB present:2052068kB pages_scanned:5192088 all_unreclaimable? yes lowmem_reserve[]: 0 0 0 0 Node 0 Normal free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no lowmem_reserve[]: 0 0 0 0 Node 0 HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no lowmem_reserve[]: 0 0 0 0 Node 0 DMA: 4*4kB 5*8kB 2*16kB 4*32kB 4*64kB 2*128kB 1*256kB 0*512kB 2*1024kB 0*2048kB 0*4096kB = 3032kB Node 0 DMA32: 0*4kB 1*8kB 1*16kB 1*32kB 1*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 1*4096kB = 5624kB Node 0 Normal: empty Node 0 HighMem: empty 486967 pagecache pages Swap cache: add 9174, delete 8578, find 337/499, race 0+0 Free swap = 4072784kB Total swap = 4104596kB Out of memory: Killed process 2072, UID 51, (sendmail). </pre> </div></div> <p>Maloo report: <a href="https://maloo.whamcloud.com/test_sessions/b3c52910-77de-11e1-841d-5254004bbbd3" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sessions/b3c52910-77de-11e1-841d-5254004bbbd3</a></p> <p>Another instance: <a href="https://maloo.whamcloud.com/test_sessions/1eaca93a-7800-11e1-841d-5254004bbbd3" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sessions/1eaca93a-7800-11e1-841d-5254004bbbd3</a></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>syslogd invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0 Call Trace: [&lt;ffffffff800c962a&gt;] out_of_memory+0x8e/0x2f3 [&lt;ffffffff8000f625&gt;] __alloc_pages+0x27f/0x308 [&lt;ffffffff8001300a&gt;] __do_page_cache_readahead+0x96/0x179 [&lt;ffffffff80013945&gt;] filemap_nopage+0x14c/0x360 [&lt;ffffffff80008964&gt;] __handle_mm_fault+0x1fb/0x1039 [&lt;ffffffff800a28fb&gt;] autoremove_wake_function+0x0/0x2e [&lt;ffffffff8000ebd4&gt;] find_get_pages_tag+0x34/0x89 [&lt;ffffffff8006720b&gt;] do_page_fault+0x4cb/0x874 [&lt;ffffffff800f5a22&gt;] sync_inode+0x24/0x33 [&lt;ffffffff8804c370&gt;] :ext3:ext3_sync_file+0xcc/0xf8 [&lt;ffffffff8005dde9&gt;] error_exit+0x0/0x84 Node 0 DMA per-cpu: cpu 0 hot: high 0, batch 1 used:0 cpu 0 cold: high 0, batch 1 used:0 Node 0 DMA32 per-cpu: cpu 0 hot: high 186, batch 31 used:55 cpu 0 cold: high 62, batch 15 used:47 Node 0 Normal per-cpu: empty Node 0 HighMem per-cpu: empty Free pages: 8624kB (0kB HighMem) Active:6 inactive:486329 dirty:0 writeback:2652 unstable:0 free:2156 slab:13169 mapped-file:1064 mapped-anon:596 pagetables:1232 Node 0 DMA free:3032kB min:24kB low:28kB high:36kB active:0kB inactive:0kB present:9736kB pages_scanned:0 all_unreclaimable? yes lowmem_reserve[]: 0 2003 2003 2003 Node 0 DMA32 free:5592kB min:5712kB low:7140kB high:8568kB active:24kB inactive:1945316kB present:2052068kB pages_scanned:4943222 all_unreclaimable? yes lowmem_reserve[]: 0 0 0 0 Node 0 Normal free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no lowmem_reserve[]: 0 0 0 0 Node 0 HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no lowmem_reserve[]: 0 0 0 0 Node 0 DMA: 4*4kB 5*8kB 2*16kB 4*32kB 4*64kB 2*128kB 1*256kB 0*512kB 2*1024kB 0*2048kB 0*4096kB = 3032kB Node 0 DMA32: 0*4kB 1*8kB 1*16kB 0*32kB 1*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 1*4096kB = 5592kB Node 0 Normal: empty Node 0 HighMem: empty 486367 pagecache pages Swap cache: add 8131, delete 7535, find 108/139, race 0+0 Free swap = 4072828kB Total swap = 4104596kB Out of memory: Killed process 2075, UID 51, (sendmail). </pre> </div></div> <p>The total memory size on each VM is about 2GB.</p> <p>BTW, the same tests passed on the same VMs with RHEL6.2/x86_64 distro/arch both on clients and servers:<br/> <a href="https://maloo.whamcloud.com/test_sessions/f4dd044e-7708-11e1-a169-5254004bbbd3" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sessions/f4dd044e-7708-11e1-a169-5254004bbbd3</a></p> <br/> Lustre Tag: v2_2_0_0_RC2<br/> Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b2_2/17/">http://build.whamcloud.com/job/lustre-b2_2/17/</a><br/> Distro/Arch: RHEL5.7/x86_64(client), RHEL6.2/x86_64(server)<br/> Network: TCP (1GigE)<br/> ENABLE_QUOTA=yes<br/> FAILURE_MODE=HARD<br/> <br/> MGS/MDS Nodes: client-32vm5(active), client-32vm6(passive)<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;\ /<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;1 combined MGS/MDT<br/> <br/> OSS Nodes: client-32vm7(active), client-32vm8(active)<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;\ /<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;OST1 (active in client-32vm7)<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;OST2 (active in client-32vm8)<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;OST3 (active in client-32vm7)<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;OST4 (active in client-32vm8)<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;OST5 (active in client-32vm7)<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;OST6 (active in client-32vm8)<br/> <br/> Client Nodes: client-32vm[1-4]<br/> LU-1261

oom-killer was invoked while running recovery-*-scale tests on VMs

Bug Minor Resolved Cannot Reproduce WC Triage Jian Yu Tue, 27 Mar 2012 08:43:04 +0000 Mon, 29 May 2017 04:13:04 +0000 Mon, 29 May 2017 04:13:04 +0000 Lustre 2.2.0 0 4 <p>While running the tests with async journal commit disabled on OSSs, the above issue did not occur.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">lctl set_param obdfilter.${FSNAME}-*.sync_journal=1 </pre> </div></div> <p>Maloo reports:<br/> <a href="https://maloo.whamcloud.com/test_sessions/d309cc0c-79a2-11e1-9d2a-5254004bbbd3" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sessions/d309cc0c-79a2-11e1-9d2a-5254004bbbd3</a><br/> <a href="https://maloo.whamcloud.com/test_sessions/ce61e5e0-79a2-11e1-9d2a-5254004bbbd3" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sessions/ce61e5e0-79a2-11e1-9d2a-5254004bbbd3</a><br/> <a href="https://maloo.whamcloud.com/test_sessions/abd0317a-79a8-11e1-9d2a-5254004bbbd3" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sessions/abd0317a-79a8-11e1-9d2a-5254004bbbd3</a></p> <p>Close old ticket.</p> Development Rank 1|hzvyhb: Rank (Obsolete) 9770 Severity