https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-17151 Lustre <p>This issue was created by maloo for Serguei Smirnov &lt;ssmirnov@ddn.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://testing.whamcloud.com/test_sets/2a3fbe0b-f784-4875-bd67-6ab32aa223a3" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/2a3fbe0b-f784-4875-bd67-6ab32aa223a3</a></p> <p>Test session details:<br/> clients: <a href="https://build.whamcloud.com/job/lustre-reviews/99011" class="external-link" target="_blank" rel="nofollow noopener">https://build.whamcloud.com/job/lustre-reviews/99011</a> - 4.18.0-425.10.1.el8_7.aarch64<br/> servers: <a href="https://build.whamcloud.com/job/lustre-reviews/99011" class="external-link" target="_blank" rel="nofollow noopener">https://build.whamcloud.com/job/lustre-reviews/99011</a> - 4.18.0-477.21.1.el8_lustre.x86_64</p> <p>sanity test_411b: @@@@@@ FAIL: (3) failed to write successfully <br/> Trace dump:<br/> = /usr/lib64/lustre/tests/test-framework.sh:6700:error()<br/> = /usr/lib64/lustre/tests/sanity.sh:27545:test_411b()<br/> = /usr/lib64/lustre/tests/test-framework.sh:7040:run_one()<br/> = /usr/lib64/lustre/tests/test-framework.sh:7096:run_one_logged()<br/> = /usr/lib64/lustre/tests/test-framework.sh:6926:run_test()<br/> = /usr/lib64/lustre/tests/sanity.sh:27592:main()<br/> Dumping lctl log to /autotest/autotest-2/2023-09-27/lustre-reviews_review-ldiskfs-dne-arm_99011_29_0b05909e-9d3c-46a3-9f81-125f5c37cc5d//sanity.test_411b.*.1695797605.log<br/> CMD: trevis-108vm17.trevis.whamcloud.com,trevis-108vm18,trevis-72vm4,trevis-72vm5,trevis-72vm6 /usr/sbin/lctl dk &gt; /autotest/autotest-2/2023-09-27/lustre-reviews_review-ldiskfs-dne-arm_99011_29_0b05909e-9d3c-46a3-9f81-125f5c37cc5d//sanity.test_411b.debug_log.\$(hostname -s).1695797605.log;<br/> dmesg &gt; /autotest/autotest-2/2023-09-27/lustre-reviews_review-ldiskfs-dne-arm_99011_29_0b05909e-9d3c-46a3-9f81-125f5c37cc5d//sanity.test_411b.dmesg.\$(hostname -s).1695797605.log<br/> cache 19660800<br/> rss 0<br/> rss_huge 0<br/> shmem 0<br/> mapped_file 0<br/> dirty 2883584<br/> writeback 0<br/> swap 2949120<br/> pgpgin 21206<br/> pgpgout 20906<br/> pgfault 4034<br/> pgmajfault 417<br/> inactive_anon 0<br/> active_anon 0<br/> inactive_file 18022400<br/> active_file 1638400<br/> unevictable 0<br/> hierarchical_memory_limit 268435456<br/> hierarchical_memsw_limit 9223372036854710272<br/> total_cache 19660800<br/> total_rss 0<br/> total_rss_huge 0<br/> total_shmem 0<br/> total_mapped_file 0<br/> total_dirty 2883584<br/> total_writeback 0<br/> total_swap 2949120<br/> total_pgpgin 21206<br/> total_pgpgout 20906<br/> total_pgfault 4034<br/> total_pgmajfault 417<br/> total_inactive_anon 0<br/> total_active_anon 0<br/> total_inactive_file 18022400<br/> total_active_file 1638400<br/> total_unevictable 0</p> LU-17151

sanity: test_411b Error: '(3) failed to write successfully'

Bug Minor Reopened Unresolved Qian Yingjin Maloo Wed, 27 Sep 2023 14:46:11 +0000 Sun, 28 Jan 2024 04:29:46 +0000 Lustre 2.16.0 0 9 <p>cgroup limit causes OOM which in turn kills dd</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> [16546.136239] oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(<span class="code-keyword">null</span>),cpuset=/,mems_allowed=0,oom_memcg=/f411b.sanity,task_memcg=/f411b.sanity,task=dd,pid=1071239,uid=0 [16546.146364] Memory cgroup out of memory: Killed process 1071239 (dd) total-vm:36672kB, anon-rss:0kB, file-rss:2048kB, shmem-rss:0kB, UID:0 pgtables:384kB oom_score_adj:0 [16546.861389] dd invoked oom-killer: gfp_mask=0x6200ca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0 </pre> </div></div> <p>+<ins>1 On Master +<a href="https://testing.whamcloud.com/sub_tests/bfbca56f-69b4-4c4c-94e1-f2b801dbc276" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/sub_tests/bfbca56f-69b4-4c4c-94e1-f2b801dbc276</a></ins></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>========================================================== 10:28:52 (1696328932) [19651.231511] Lustre: DEBUG MARKER: == sanity test 411b: confirm Lustre can avoid OOM with reasonable cgroups limits ========================================================== 10:28:52 (1696328932) [19669.199217] dd invoked oom-killer: gfp_mask=0x6200ca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0 [19669.208834] CPU: 1 PID: 1254796 Comm: dd Kdump: loaded Tainted: G OE --------- - - 4.18.0-425.10.1.el8_7.aarch64 #1 [19669.216852] Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0 02/06/2015 [19669.221748] Call trace: [19669.223597] dump_backtrace+0x0/0x178 [19669.226811] show_stack+0x28/0x38 [19669.229191] dump_stack+0x5c/0x74 [19669.231641] dump_header+0x4c/0x1e0 [19669.234205] oom_kill_process+0x18c/0x190 [19669.237056] out_of_memory+0x188/0x518 [19669.239728] mem_cgroup_out_of_memory+0xf8/0x108 [19669.243134] try_charge+0x6a8/0x6f0 [19669.245659] __mem_cgroup_charge+0x48/0xe8 [19669.248531] mem_cgroup_swapin_charge_page+0x68/0x170 [19669.252071] __read_swap_cache_async+0x1d8/0x280 [19669.255316] swap_cluster_readahead+0x184/0x300 [19669.258578] swapin_readahead+0x2a8/0x3cc [19669.261400] do_swap_page+0x53c/0x828 [19669.264147] __handle_mm_fault+0x260/0x698 [19669.267037] handle_mm_fault+0xe4/0x188 [19669.269721] do_page_fault+0x164/0x488 [19669.272493] do_translation_fault+0xa0/0xb0 [19669.275595] do_mem_abort+0x54/0xb0 [19669.278156] el1_abort+0x28/0x30 [19669.280633] el1_sync_handler+0x70/0x120 [19669.283516] el1_sync+0x7c/0x100 [19669.285854] fault_in_readable+0x9c/0x188 [19669.288769] fault_in_iov_iter_readable+0x80/0xd0 [19669.292460] generic_perform_write+0x7c/0x1b8 [19669.295657] __generic_file_write_iter+0x104/0x1e8 [19669.299209] vvp_io_write_start+0x4d8/0x1058 [lustre] [19669.303501] cl_io_start+0x98/0x1d8 [obdclass] [19669.307477] cl_io_loop+0xcc/0x2a0 [obdclass] [19669.310836] ll_file_io_generic+0x3dc/0xfe0 [lustre] [19669.314709] ll_file_write_iter+0x7d4/0xa58 [lustre] [19669.318471] new_sync_write+0x104/0x158 [19669.321468] __vfs_write+0x78/0x90 [19669.324000] vfs_write+0xb0/0x1b8 [19669.326565] ksys_write+0x70/0xd8 [19669.328989] __arm64_sys_write+0x28/0x38 [19669.331899] do_el0_svc+0xb4/0x188 [19669.334521] el0_sync_handler+0x88/0xac [19669.337306] el0_sync+0x140/0x180 [19669.340071] memory: usage 262144kB, limit 262144kB, failcnt 15083 [19669.344527] memory+swap: usage 328128kB, limit 9007199254740928kB, failcnt 0 [19669.349159] kmem: usage 192kB, limit 9007199254740928kB, failcnt 0 [19669.353303] Memory cgroup stats for /f411b.sanity: [19669.353714] anon 0 file 163250176 kernel_stack 0 pagetables 0 percpu 0 sock 0 shmem 0 file_mapped 0 file_dirty 3866624 file_writeback 0 swapcached 606339072 anon_thp 0 file_thp 0 shmem_thp 0 inactive_anon 131072 active_anon 0 inactive_file 265682944 active_file 2424832 unevictable 0 slab_reclaimable 70840 slab_unreclaimable 13632 slab 84472 workingset_refault_anon 8219 workingset_refault_file 23 workingset_activate_anon 8177 workingset_activate_file 3 workingset_restore_anon 7464 workingset_restore_file 2 workingset_nodereclaim 0 pgfault 13248 pgmajfault 1427 pgrefill 32282 pgscan 2242586 pgsteal 26364 pgactivate 31809 pgdeactivate 32273 pglazyfree 0 pglazyfreed 0 thp_fault_alloc 0 thp_collapse_alloc 0 [19669.422320] Tasks state (memory values in pages): [19669.425648] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name [19669.431758] [1254796] 0 1254796 1085 32 393216 1068 0 dd [19669.437512] oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=/,mems_allowed=0,oom_memcg=/f411b.sanity,task_memcg=/f411b.sanity,task=dd,pid=1254796,uid=0 [19669.447727] Memory cgroup out of memory: Killed process 1254796 (dd) total-vm:69440kB, anon-rss:0kB, file-rss:2048kB, shmem-rss:0kB, UID:0 pgtables:384kB oom_score_adj:0 [19670.119273] Lustre: DEBUG MARKER: /usr/sbin/lctl mark sanity test_411b: @@@@@@ FAIL: (3) failed to write successfully [19670.662747] Lustre: DEBUG MARKER: sanity test_411b: @@@@@@ FAIL: (3) failed to write successfully [19671.323379] Lustre: DEBUG MARKER: /usr/sbin/lctl dk &gt; /autotest/autotest-2/2023-10-03/lustre-reviews_review-ldiskfs-dne-arm_99080_29_acee458f-e73b-455d-b9a7-38fbc1e13ed6//sanity.test_411b.debug_log.$(hostname -s).1696328953.log; dmesg &gt; /autotest/autotest-2/2023-10-03/lustre-reviews_rev</pre> </div></div> <p>+1 on Master <a href="https://testing.whamcloud.com/test_sets/a2c9650c-efdc-49bf-80bb-7770e72cd53a" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/a2c9650c-efdc-49bf-80bb-7770e72cd53a</a></p> <p>Hi Yingjin, can you please take a look at this. The sanity.sh test_411b added from patch <a href="https://review.whamcloud.com/50544" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/50544</a> "<tt><a href="https://jira.whamcloud.com/browse/LU-16713" title="Writeback and commit pages under memory pressure to avoid OOM" class="issue-link" data-issue-key="LU-16713"><del>LU-16713</del></a> llite: writeback/commit pages under memory pressure</tt>" is failing regularly since it landed (and once before landing on <tt>master-next</tt>). There is also a regular error "<tt>error writing to file from 1249987</tt>".</p> <p>It looks like almost all of the failures are on <tt>aarch64</tt> clients (48/58 failures).</p> <p>"Timothy Day &lt;timday@amazon.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/52610" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/52610</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-17151" title="sanity: test_411b Error: &#39;(3) failed to write successfully&#39; " class="issue-link" data-issue-key="LU-17151">LU-17151</a> tests: increase 411b mem limit for arm<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 8444e3f8d5d467ec7653c713529806d6123a449c</p> <p>Can we land this since this issue is very common.</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/52610/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/52610/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-17151" title="sanity: test_411b Error: &#39;(3) failed to write successfully&#39; " class="issue-link" data-issue-key="LU-17151">LU-17151</a> tests: increase sanity/411b memory limit<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 0e878390e1c8c5883bccd01758392eaa16a67f31</p> <p>Landed for 2.16</p> <p>There is another failure on x86_64, on the first dd invocation in the test, on latest master that includes this fix at <a href="https://testing.whamcloud.com/test_sets/36fdecc1-7e58-410d-b1eb-7345e901b7e5" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/36fdecc1-7e58-410d-b1eb-7345e901b7e5</a>.</p> <p>It looks like this is still failing fairly regularly in the past week:</p> <p><a href="https://testing.whamcloud.com/search?status%5B%5D=FAIL&amp;test_set_script_id=f9516376-32bc-11e0-aaee-52540025f9ae&amp;sub_test_script_id=e877b2b8-1ca3-4655-836b-81fb1693f948&amp;start_date=2023-11-13&amp;end_date=2023-11-20&amp;source=sub_tests#redirect" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/search?status%5B%5D=FAIL&amp;test_set_script_id=f9516376-32bc-11e0-aaee-52540025f9ae&amp;sub_test_script_id=e877b2b8-1ca3-4655-836b-81fb1693f948&amp;start_date=2023-11-13&amp;end_date=2023-11-20&amp;source=sub_tests#redirect</a></p> <p>It looks like those failures are despite patch <a href="https://review.whamcloud.com/52610" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/52610</a> "<tt><a href="https://jira.whamcloud.com/browse/LU-17151" title="sanity: test_411b Error: &#39;(3) failed to write successfully&#39; " class="issue-link" data-issue-key="LU-17151">LU-17151</a> tests: increase sanity/411b memory limit</tt>" having landed.</p> <p>This is failing quite a bit on master right now.</p> <p>Should we increase the memory limit value in the test script or enforce the test only running on real environment not VM?</p> <p>do I understand correctly that the all last runs were tmpfs-based?</p> <p>Yes, it is using tmpfs on the VM host for the MDT since August or so. It is exported to the VM guest as a block device. The OSTs are still HDD I believe. </p> <p>my observation with tmpfs-based ZFS is that it tends do not reuse blocks instead allocate new ones ("inner") and this leads to memory overuse. can be fixed running "zfs trim .." once few minutes/subtests.</p> <p>Does trim get passed from the VM down to the host to free the memory? You could try adding a patch to this subtest to run <tt>fstrim</tt> or <tt>zfs trim</tt> on all of the targets before or during the test. However, if only the MDT is on tmpfs then I don't think these tests are using much memory there.</p> <p>It would be useful to add some debugging to see where all of the memory <b>is</b> used (slabinfo/meminfo) so that we can reduce the size. Some of the internal data structures are allocated/limited at mount time based on the total RAM size and not limited by the cgroup size (e.g. lu cache, max_dirty_mb, etc), and the client needs to do a better job to free this memory under pressure (e.g. registered shrinker). Should we do anything to flush the cache at the <b>start</b> of the test, so that the process in the cgroup is not penalized by previous allocations outside its control?</p> <p>It would also be good to improve the test scripts slightly to match proper test script style:</p> <ul class="alternate" type="square"> <li>no need for "<tt>trap 0</tt>" in <tt>cleanup_test411_cgroup()</tt> since this would clobber any other registered <tt>stack_trap</tt> calls</li> <li>in <tt>test_411a</tt> use "<tt>stack_trap cleanup_test411_cgroup</tt>" to do the cleanup instead of calling it explicitly</li> <li>in <tt>test_411b</tt> add "<tt>stack_trap 'rm -f $DIR/$tfile.&#42;'</tt>" to clean up the files even if the test fails</li> </ul> <blockquote><p>Does trim get passed from the VM down to the host to free the memory?</p></blockquote> <p>yes, here are my local runs:</p> <ul> <li>sanity on ZFS w/o trim: 9785 MBs allocated from the host by sanity's completion</li> <li>sanity on ZFS w/ zfs trim in run_one_logged: 8159 MBs</li> </ul> <p>22 failures in the past week</p> Related LU-16713 LU-17183 Development Rank 1|i03wxj: Rank (Obsolete) 9223372036854775807 Severity