https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-2795 Lustre <p>At our center, we are running a Lustre 2.1.2 file system with Lustre 2.1.2 clients on all of the compute nodes of our Penguin cluster. Recently, a user has been performing WRF runs where he uses a special feature of WRF to offload all of the I/O onto a single node, which improves his I/O performance dramatically, but results in the node losing ~1 GB of memory to "Inactive" after each run. In our epilogue, we have a script checking for available free memory above a specified percentage, and every job that this user runs results in the node being set to offline due to this 1 GB of Inactive memory. </p> <p>Here is an example of the output from drop_caches showing before and after the epilogue starts on one of these nodes: </p> <ol> <li>Before:</li> <li>MemTotal: 15.681 GB</li> <li>MemFree: 6.495 GB</li> <li>Cached: 6.206 GB</li> <li>Active: 1.395 GB</li> <li>Inactive: 6.247 GB</li> <li>Dirty: 0.000 GB</li> <li>Mapped: 0.003 GB</li> <li>Slab: 1.391 GB</li> <li>After:</li> <li>MemTotal: 15.681 GB</li> <li>MemFree: 14.003 GB</li> <li>Cached: 0.007 GB</li> <li>Active: 0.134 GB</li> <li>Inactive: 1.309 GB</li> <li>Dirty: 0.000 GB</li> <li>Mapped: 0.003 GB</li> <li>Slab: 0.082 GB</li> </ol> <p>While looking for possible solutions to this problem, I stumbled upon a recent HPDD-Discuss question that was entitled "Possible file page leak in Lustre 2.1.2" which was very similar to our problem. It was suggested that the issue had already been discovered and resolved in <a href="http://jira.whamcloud.com/browse/LU-1576" class="external-link" rel="nofollow">http://jira.whamcloud.com/browse/LU-1576</a> </p> <p>This ticket suggests that the resolution was included as part of Lustre 2.1.3, so we tested this by installing the Lustre 2.1.3 client packages on some of our compute nodes and allowing the WRF job to run on these nodes. However, even after the upgrade to Lustre 2.1.3, we still saw the inactive memory at the end of the job. Do we need to upgrade our Lustre installation on the OSSes and MDS to Lustre 2.1.3 to fix this problem, or do you have any other suggestions? </p> <p>Any help that you could provide us with would be appreciated! </p> LU-2795

WRF runs causing Lustre clients to lose memory

Bug Minor Resolved Not a Bug Oleg Drokin Archie Dizon Mon, 11 Feb 2013 15:59:24 +0000 Tue, 10 Apr 2018 17:11:07 +0000 Tue, 10 Apr 2018 17:11:07 +0000 0 8 <p> It is possible you are seeing uncommitted writes to the OSTs - if you wait 5 - 10 minutes, can the cache be cleaned? It is possible you are seeing a leak, however this does not appear to match <a href="https://jira.whamcloud.com/browse/LU-1576" title="client sluggish after running lpurge" class="issue-link" data-issue-key="LU-1576"><del>LU-1576</del></a>. The 'slabtop' tool may provide some additional data on memory consumption.<br/> If this is a leak, is there any way you can run this on a single node, and/or provide us with the workload? We may need to reproduce this one locally to fix the issue. </p> <p>Can you update us on your status?</p> <p>In regards to the question of waiting for a few minutes, the answer is no.<br/> Even if we wait for hours, the inactive memory is never given back to the<br/> system, we are forced to reboot these nodes to return them with their full<br/> memory again. However, as you can see from the output in my last message,<br/> we start off with &gt; 6 GB of inactive memory at the beginning of the<br/> epilogue and ~ 1 GB of inactive memory after the epilogue has waited<br/> approximately 30 seconds. Although, no matter how long we wait, that 1 GB<br/> of memory is never returned to the system</p> <p>We had planned to set up a run of WRF to test the memory usage on our test<br/> cluster, but this has gotten delayed as all of us were busy during the<br/> week. We will have to wait until next week to get you some data on memory<br/> usage.</p> <p>Having talked with someone much more familiar with WRF and its dependencies<br/> than myself, it sounds like running the WRF software the way that is being<br/> done here, it may be a fairly big hassle. In other words, getting it<br/> running for you locally may be fairly difficult. We will have to see if<br/> going down that road is necessary when we give you some more data.</p> <p>In the meantime, I'm curious as to how WhamCloud has determined that our<br/> problem does not match up with <a href="http://jira.whamcloud.com/browse/LU-1576" class="external-link" rel="nofollow">http://jira.whamcloud.com/browse/LU-1576</a>.<br/> The symptoms are identical, and it was suggested in the HPDD discussion<br/> list that this was an occurrence in Lustre 2.1.2 for some irregular I/O<br/> patterns. What do they see as different between our problem and the one<br/> described by LLNL on the list? For my future reference, I would be<br/> interested to know how they determined that so I could use their methods<br/> for better diagnosing Lustre problems in the future.</p> <p>I'll have more to share with you next week.</p> <p>Thanks</p> <p>You indicated that you had installed 2.1.3, which contains the fix for <a href="https://jira.whamcloud.com/browse/LU-1576" title="client sluggish after running lpurge" class="issue-link" data-issue-key="LU-1576"><del>LU-1576</del></a>, this was our main indication. The <a href="https://jira.whamcloud.com/browse/LU-1576" title="client sluggish after running lpurge" class="issue-link" data-issue-key="LU-1576"><del>LU-1576</del></a> fix mostly deals with readdir pages, so unless your workload includes a lot of readdirs your have likely a different problem. </p> <p>Are you saying that dropping cache does not free the 1GB of memory? </p> <p>Yes, we had tested installing 2.1.3 on a couple of our client systems to<br/> see if that would fix the problem, but we were still seeing the issue on<br/> those nodes with the Lustre 2.1.3 client installed on them. Thanks for<br/> clarifying that, and it doesn't appear that this code would<br/> be performing a great deal of readdirs, probably not the same memory leak.</p> <p>Correct, dropping cache does not free the 1 GB of memory. Our epilogue<br/> script attempts to drop cache twice, and after the second time it compares<br/> the amount of free memory before determining if it can return the compute<br/> node to service.</p> <p>We are going to run the WRF job with Lustre at a higher logging level and<br/> using the leak_finder.pl script provided by WhamCloud. We will send<br/> whatever we find along to you.</p> <p>Thanks, let us know how it goes.</p> <p>Customer ran there WRF job with the Lustre debugging set to gather malloc information, and it does appear that we have found a leak in Lustre. Here were the steps we followed:<br/> 1) sudo lctl set_param debug=+malloc<br/> 2) sudo lctl set_param debug_mb=512<br/> 3) * let the WRF job run *<br/> Epilogue sets the node offline (1.62 GB of memory set to inactive)<br/> 4) sudo lctl dk /tmp/lustre_debug<br/> 5) perl leak_finder.pl /tmp/lustre_debug 2&gt;&amp;1 | grep "Leak"</p> <p>From that last command, here is what we found:</p> <ul> <li> <ul> <li> <ul> <li>Leak: 1080 bytes allocated at ffff8101d5eae140 (super25.c:ll_alloc_inode:56, debug file line 1745506)</li> <li>Leak: 104 bytes allocated at ffff8101cecdbdc0 (dcache.c:ll_set_dd:192, debug file line 1745508)</li> <li>Leak: 1080 bytes allocated at ffff810214eb3ac0 (super25.c:ll_alloc_inode:56, debug file line 1745551)</li> <li>Leak: 104 bytes allocated at ffff8101d7523840 (dcache.c:ll_set_dd:192, debug file line 1745553)</li> </ul> </li> </ul> </li> </ul> <p>The Lustre documentation states that this is a cyclical log so I would imagine that if there was a small leak shown here, that throughout the run small amounts of memory could have been lost, resulting in our overall large leak.</p> <p>We will attach the lustre_debug log to this case for you to analyze as well. It does look as though we may be narrowing down on the problem now though.</p> <p>Additionally, I am going to attach the /proc/slabinfo for the end of the<br/> WRF run as you had previously requested, along with the /proc/meminfo<br/> before, during, and after the WRF run.</p> <p>cat /proc/slabinfo<br/> slabinfo - version: 2.1</p> <ol> <li>name &lt;active_objs&gt; &lt;num_objs&gt; &lt;objsize&gt; &lt;objperslab&gt;<br/> &lt;pagesperslab&gt; : tunables &lt;limit&gt; &lt;batchcount&gt; &lt;sharedfactor&gt; : slabdata<br/> &lt;active_slabs&gt; &lt;num_slabs&gt; &lt;sharedavail&gt;<br/> ll_qunit_cache 0 0 112 34 1 : tunables 120 60 8<br/> : slabdata 0 0 0<br/> lmv_objects 0 0 96 40 1 : tunables 120 60 8<br/> : slabdata 0 0 0<br/> ccc_req_kmem 0 0 40 92 1 : tunables 120 60 8<br/> : slabdata 0 0 0<br/> ccc_session_kmem 105 132 176 22 1 : tunables 120 60 8<br/> : slabdata 6 6 0<br/> ccc_thread_kmem 112 121 336 11 1 : tunables 54 27 8<br/> : slabdata 11 11 0<br/> ccc_object_kmem 0 0 256 15 1 : tunables 120 60 8<br/> : slabdata 0 0 0<br/> ccc_lock_kmem 0 0 40 92 1 : tunables 120 60 8<br/> : slabdata 0 0 0<br/> vvp_session_kmem 105 148 104 37 1 : tunables 120 60 8<br/> : slabdata 4 4 0<br/> vvp_thread_kmem 112 126 440 9 1 : tunables 54 27 8<br/> : slabdata 14 14 0<br/> vvp_page_kmem 0 0 80 48 1 : tunables 120 60 8<br/> : slabdata 0 0 0<br/> ll_rmtperm_hash_cache 0 0 256 15 1 : tunables 120 60<br/> 8 : slabdata 0 0 0<br/> ll_remote_perm_cache 0 0 40 92 1 : tunables 120 60<br/> 8 : slabdata 0 0 0<br/> ll_file_data 0 0 192 20 1 : tunables 120 60 8<br/> : slabdata 0 0 0<br/> lustre_inode_cache 3 21 1088 7 2 : tunables 24 12 8<br/> : slabdata 3 3 0<br/> lov_oinfo 0 0 320 12 1 : tunables 54 27 8<br/> : slabdata 0 0 0<br/> lov_lock_link_kmem 0 0 32 112 1 : tunables 120 60 8<br/> : slabdata 0 0 0<br/> lovsub_req_kmem 0 0 40 92 1 : tunables 120 60 8<br/> : slabdata 0 0 0<br/> lovsub_object_kmem 0 0 240 16 1 : tunables 120 60 8<br/> : slabdata 0 0 0<br/> lovsub_lock_kmem 0 0 64 59 1 : tunables 120 60 8<br/> : slabdata 0 0 0<br/> lovsub_page_kmem 0 0 40 92 1 : tunables 120 60 8<br/> : slabdata 0 0 0<br/> lov_req_kmem 0 0 40 92 1 : tunables 120 60 8<br/> : slabdata 0 0 0<br/> lov_session_kmem 105 120 384 10 1 : tunables 54 27 8<br/> : slabdata 12 12 0<br/> lov_thread_kmem 112 121 336 11 1 : tunables 54 27 8<br/> : slabdata 11 11 0<br/> lov_object_kmem 0 0 200 19 1 : tunables 120 60 8<br/> : slabdata 0 0 0<br/> lov_lock_kmem 0 0 104 37 1 : tunables 120 60 8<br/> : slabdata 0 0 0<br/> lov_page_kmem 0 0 48 77 1 : tunables 120 60 8<br/> : slabdata 0 0 0<br/> osc_req_kmem 0 0 40 92 1 : tunables 120 60 8<br/> : slabdata 0 0 0<br/> osc_session_kmem 105 130 296 13 1 : tunables 54 27 8<br/> : slabdata 10 10 0<br/> osc_thread_kmem 112 126 216 18 1 : tunables 120 60 8<br/> : slabdata 7 7 0<br/> osc_object_kmem 0 0 136 28 1 : tunables 120 60 8<br/> : slabdata 0 0 0<br/> osc_lock_kmem 0 0 184 21 1 : tunables 120 60 8<br/> : slabdata 0 0 0<br/> osc_page_kmem 0 0 264 15 1 : tunables 54 27 8<br/> : slabdata 0 0 0<br/> llcd_cache 0 0 3952 1 1 : tunables 24 12 8<br/> : slabdata 0 0 0<br/> interval_node 22 90 128 30 1 : tunables 120 60 8<br/> : slabdata 3 3 0<br/> ldlm_locks 43 63 576 7 1 : tunables 54 27 8<br/> : slabdata 9 9 0<br/> ldlm_resources 41 72 320 12 1 : tunables 54 27 8<br/> : slabdata 6 6 0<br/> cl_page_kmem 0 0 184 21 1 : tunables 120 60 8<br/> : slabdata 0 0 0<br/> cl_lock_kmem 0 0 216 18 1 : tunables 120 60 8<br/> : slabdata 0 0 0<br/> cl_env_kmem 105 132 176 22 1 : tunables 120 60 8<br/> : slabdata 6 6 0<br/> capa_cache 0 0 184 21 1 : tunables 120 60 8<br/> : slabdata 0 0 0<br/> ll_import_cache 0 0 1424 5 2 : tunables 24 12 8<br/> : slabdata 0 0 0<br/> ll_obdo_cache 0 0 208 19 1 : tunables 120 60 8<br/> : slabdata 0 0 0<br/> ll_obd_dev_cache 17 17 7048 1 2 : tunables 8 4 0<br/> : slabdata 17 17 0<br/> SDP 0 0 1792 2 1 : tunables 24 12 8<br/> : slabdata 0 0 0<br/> fib6_nodes 7 118 64 59 1 : tunables 120 60 8<br/> : slabdata 2 2 0<br/> ip6_dst_cache 7 36 320 12 1 : tunables 54 27 8<br/> : slabdata 3 3 0<br/> ndisc_cache 1 15 256 15 1 : tunables 120 60 8<br/> : slabdata 1 1 0<br/> RAWv6 11 12 960 4 1 : tunables 54 27 8<br/> : slabdata 3 3 0<br/> UDPv6 0 0 896 4 1 : tunables 54 27 8<br/> : slabdata 0 0 0<br/> tw_sock_TCPv6 0 0 192 20 1 : tunables 120 60 8<br/> : slabdata 0 0 0<br/> request_sock_TCPv6 0 0 192 20 1 : tunables 120 60 8<br/> : slabdata 0 0 0<br/> TCPv6 0 0 1728 4 2 : tunables 24 12 8<br/> : slabdata 0 0 0<br/> nfs_direct_cache 0 0 136 28 1 : tunables 120 60 8<br/> : slabdata 0 0 0<br/> nfs_write_data 36 36 832 9 2 : tunables 54 27 8<br/> : slabdata 4 4 0<br/> nfs_read_data 32 36 832 9 2 : tunables 54 27 8<br/> : slabdata 4 4 0<br/> nfs_inode_cache 123 195 1032 3 1 : tunables 24 12 8<br/> : slabdata 65 65 0<br/> nfs_page 0 0 128 30 1 : tunables 120 60 8<br/> : slabdata 0 0 0<br/> rpc_buffers 8 8 2048 2 1 : tunables 24 12 8<br/> : slabdata 4 4 0<br/> rpc_tasks 20 20 384 10 1 : tunables 54 27 8<br/> : slabdata 2 2 0<br/> rpc_inode_cache 30 30 768 5 1 : tunables 54 27 8<br/> : slabdata 6 6 0<br/> scsi_cmd_cache 5 10 384 10 1 : tunables 54 27 8<br/> : slabdata 1 1 2<br/> sgpool-128 32 32 4096 1 1 : tunables 24 12 8<br/> : slabdata 32 32 0<br/> sgpool-64 32 32 2048 2 1 : tunables 24 12 8<br/> : slabdata 16 16 0<br/> sgpool-32 32 32 1024 4 1 : tunables 54 27 8<br/> : slabdata 8 8 0<br/> sgpool-16 32 32 512 8 1 : tunables 54 27 8<br/> : slabdata 4 4 0<br/> sgpool-8 32 60 256 15 1 : tunables 120 60 8<br/> : slabdata 3 4 0<br/> scsi_io_context 0 0 112 34 1 : tunables 120 60 8<br/> : slabdata 0 0 0<br/> ib_mad 2048 2296 448 8 1 : tunables 54 27 8<br/> : slabdata 287 287 0<br/> ip_fib_alias 14 59 64 59 1 : tunables 120 60 8<br/> : slabdata 1 1 0<br/> ip_fib_hash 14 59 64 59 1 : tunables 120 60 8<br/> : slabdata 1 1 0<br/> UNIX 9 33 704 11 2 : tunables 54 27 8<br/> : slabdata 3 3 0<br/> flow_cache 0 0 128 30 1 : tunables 120 60 8<br/> : slabdata 0 0 0<br/> msi_cache 9 59 64 59 1 : tunables 120 60 8<br/> : slabdata 1 1 0<br/> cfq_ioc_pool 13 60 128 30 1 : tunables 120 60 8<br/> : slabdata 2 2 0<br/> cfq_pool 11 54 216 18 1 : tunables 120 60 8<br/> : slabdata 3 3 0<br/> crq_pool 4 96 80 48 1 : tunables 120 60 8<br/> : slabdata 1 2 0<br/> deadline_drq 0 0 80 48 1 : tunables 120 60 8<br/> : slabdata 0 0 0<br/> as_arq 0 0 96 40 1 : tunables 120 60 8<br/> : slabdata 0 0 0<br/> mqueue_inode_cache 1 4 896 4 1 : tunables 54 27 8<br/> : slabdata 1 1 0<br/> isofs_inode_cache 0 0 608 6 1 : tunables 54 27 8<br/> : slabdata 0 0 0<br/> hugetlbfs_inode_cache 1 7 576 7 1 : tunables 54 27<br/> 8 : slabdata 1 1 0<br/> ext2_inode_cache 91 145 720 5 1 : tunables 54 27 8<br/> : slabdata 29 29 0<br/> ext2_xattr 0 0 88 44 1 : tunables 120 60 8<br/> : slabdata 0 0 0<br/> dnotify_cache 0 0 40 92 1 : tunables 120 60 8<br/> : slabdata 0 0 0<br/> dquot 0 0 256 15 1 : tunables 120 60 8<br/> : slabdata 0 0 0<br/> eventpoll_pwq 5 106 72 53 1 : tunables 120 60 8<br/> : slabdata 2 2 0<br/> eventpoll_epi 5 40 192 20 1 : tunables 120 60 8<br/> : slabdata 2 2 0<br/> inotify_event_cache 0 0 40 92 1 : tunables 120 60<br/> 8 : slabdata 0 0 0<br/> inotify_watch_cache 0 0 72 53 1 : tunables 120 60<br/> 8 : slabdata 0 0 0<br/> kioctx 0 0 320 12 1 : tunables 54 27 8<br/> : slabdata 0 0 0<br/> kiocb 0 0 256 15 1 : tunables 120 60 8<br/> : slabdata 0 0 0<br/> fasync_cache 0 0 24 144 1 : tunables 120 60 8<br/> : slabdata 0 0 0<br/> shmem_inode_cache 1360 1370 768 5 1 : tunables 54 27 8<br/> : slabdata 274 274 0<br/> posix_timers_cache 0 0 128 30 1 : tunables 120 60 8<br/> : slabdata 0 0 0<br/> uid_cache 2 30 128 30 1 : tunables 120 60 8<br/> : slabdata 1 1 0<br/> ip_mrt_cache 0 0 128 30 1 : tunables 120 60 8<br/> : slabdata 0 0 0<br/> tcp_bind_bucket 28 448 32 112 1 : tunables 120 60 8<br/> : slabdata 4 4 0<br/> inet_peer_cache 0 0 128 30 1 : tunables 120 60 8<br/> : slabdata 0 0 0<br/> secpath_cache 0 0 64 59 1 : tunables 120 60 8<br/> : slabdata 0 0 0<br/> xfrm_dst_cache 0 0 384 10 1 : tunables 54 27 8<br/> : slabdata 0 0 0<br/> ip_dst_cache 107 180 384 10 1 : tunables 54 27 8<br/> : slabdata 18 18 0<br/> arp_cache 53 75 256 15 1 : tunables 120 60 8<br/> : slabdata 5 5 0<br/> RAW 9 10 768 5 1 : tunables 54 27 8<br/> : slabdata 2 2 0<br/> UDP 10 15 768 5 1 : tunables 54 27 8<br/> : slabdata 3 3 0<br/> tw_sock_TCP 20 40 192 20 1 : tunables 120 60 8<br/> : slabdata 1 2 0<br/> request_sock_TCP 0 0 128 30 1 : tunables 120 60 8<br/> : slabdata 0 0 0<br/> TCP 32 35 1600 5 2 : tunables 24 12 8<br/> : slabdata 7 7 0<br/> blkdev_ioc 13 118 64 59 1 : tunables 120 60 8<br/> : slabdata 2 2 0<br/> blkdev_queue 17 20 1576 5 2 : tunables 24 12 8<br/> : slabdata 4 4 0<br/> blkdev_requests 7 14 272 14 1 : tunables 54 27 8<br/> : slabdata 1 1 2<br/> biovec-256 7 7 4096 1 1 : tunables 24 12 8<br/> : slabdata 7 7 0<br/> biovec-128 7 8 2048 2 1 : tunables 24 12 8<br/> : slabdata 4 4 0<br/> biovec-64 7 8 1024 4 1 : tunables 54 27 8<br/> : slabdata 2 2 0<br/> biovec-16 7 30 256 15 1 : tunables 120 60 8<br/> : slabdata 2 2 0<br/> biovec-4 7 118 64 59 1 : tunables 120 60 8<br/> : slabdata 2 2 0<br/> biovec-1 7 404 16 202 1 : tunables 120 60 8<br/> : slabdata 2 2 0<br/> bio 262 300 128 30 1 : tunables 120 60 8<br/> : slabdata 10 10 2<br/> utrace_engine_cache 0 0 64 59 1 : tunables 120 60<br/> 8 : slabdata 0 0 0<br/> utrace_cache 0 0 64 59 1 : tunables 120 60 8<br/> : slabdata 0 0 0<br/> sock_inode_cache 90 108 640 6 1 : tunables 54 27 8<br/> : slabdata 18 18 0<br/> skbuff_fclone_cache 14 14 512 7 1 : tunables 54 27<br/> 8 : slabdata 2 2 0<br/> skbuff_head_cache 2847 3060 256 15 1 : tunables 120 60 8<br/> : slabdata 204 204 0<br/> file_lock_cache 1 22 176 22 1 : tunables 120 60 8<br/> : slabdata 1 1 0<br/> Acpi-Operand 1848 2360 64 59 1 : tunables 120 60 8<br/> : slabdata 40 40 0<br/> Acpi-ParseExt 0 0 64 59 1 : tunables 120 60 8<br/> : slabdata 0 0 0<br/> Acpi-Parse 0 0 40 92 1 : tunables 120 60 8<br/> : slabdata 0 0 0<br/> Acpi-State 0 0 80 48 1 : tunables 120 60 8<br/> : slabdata 0 0 0<br/> Acpi-Namespace 839 896 32 112 1 : tunables 120 60 8<br/> : slabdata 8 8 0<br/> delayacct_cache 379 531 64 59 1 : tunables 120 60 8<br/> : slabdata 9 9 0<br/> taskstats_cache 19 53 72 53 1 : tunables 120 60 8<br/> : slabdata 1 1 0<br/> proc_inode_cache 146 180 592 6 1 : tunables 54 27 8<br/> : slabdata 30 30 0<br/> sigqueue 53 96 160 24 1 : tunables 120 60 8<br/> : slabdata 4 4 0<br/> radix_tree_node 9320 15316 536 7 1 : tunables 54 27 8<br/> : slabdata 2188 2188 0<br/> bdev_cache 6 12 832 4 1 : tunables 54 27 8<br/> : slabdata 3 3 0<br/> sysfs_dir_cache 5366 5412 88 44 1 : tunables 120 60 8<br/> : slabdata 123 123 0<br/> mnt_cache 42 60 256 15 1 : tunables 120 60 8<br/> : slabdata 4 4 0<br/> inode_cache 1231 1274 560 7 1 : tunables 54 27 8<br/> : slabdata 182 182 0<br/> dentry_cache 3139 4140 216 18 1 : tunables 120 60 8<br/> : slabdata 230 230 0<br/> filp 200 570 256 15 1 : tunables 120 60 8<br/> : slabdata 38 38 0<br/> names_cache 9 9 4096 1 1 : tunables 24 12 8<br/> : slabdata 9 9 0<br/> avc_node 30 106 72 53 1 : tunables 120 60 8<br/> : slabdata 2 2 0<br/> selinux_inode_security 3124 4032 80 48 1 : tunables 120 60<br/> 8 : slabdata 84 84 0<br/> key_jar 4 20 192 20 1 : tunables 120 60 8<br/> : slabdata 1 1 0<br/> idr_layer_cache 199 238 528 7 1 : tunables 54 27 8<br/> : slabdata 34 34 0<br/> buffer_head 148 320 96 40 1 : tunables 120 60 8<br/> : slabdata 8 8 0<br/> mm_struct 24 32 896 4 1 : tunables 54 27 8<br/> : slabdata 8 8 0<br/> vm_area_struct 428 1430 176 22 1 : tunables 120 60 8<br/> : slabdata 65 65 1<br/> fs_cache 50 177 64 59 1 : tunables 120 60 8<br/> : slabdata 3 3 0<br/> files_cache 35 60 768 5 1 : tunables 54 27 8<br/> : slabdata 12 12 0<br/> signal_cache 367 378 832 9 2 : tunables 54 27 8<br/> : slabdata 42 42 0<br/> sighand_cache 357 360 2112 3 2 : tunables 24 12 8<br/> : slabdata 120 120 0<br/> task_struct 368 370 1920 2 1 : tunables 24 12 8<br/> : slabdata 185 185 0<br/> anon_vma 294 1008 24 144 1 : tunables 120 60 8<br/> : slabdata 7 7 0<br/> pid 393 531 64 59 1 : tunables 120 60 8<br/> : slabdata 9 9 0<br/> shared_policy_node 0 0 48 77 1 : tunables 120 60 8<br/> : slabdata 0 0 0<br/> numa_policy 72 432 24 144 1 : tunables 120 60 8<br/> : slabdata 3 3 0<br/> size-131072(DMA) 0 0 131072 1 32 : tunables 8 4 0<br/> : slabdata 0 0 0<br/> size-131072 2 2 131072 1 32 : tunables 8 4 0<br/> : slabdata 2 2 0<br/> size-65536(DMA) 0 0 65536 1 16 : tunables 8 4 0<br/> : slabdata 0 0 0<br/> size-65536 6 6 65536 1 16 : tunables 8 4 0<br/> : slabdata 6 6 0<br/> size-32768(DMA) 0 0 32768 1 8 : tunables 8 4 0<br/> : slabdata 0 0 0<br/> size-32768 7 7 32768 1 8 : tunables 8 4 0<br/> : slabdata 7 7 0<br/> size-16384(DMA) 0 0 16384 1 4 : tunables 8 4 0<br/> : slabdata 0 0 0<br/> size-16384 2070 2070 16384 1 4 : tunables 8 4 0<br/> : slabdata 2070 2070 0<br/> size-8192(DMA) 0 0 8192 1 2 : tunables 8 4 0<br/> : slabdata 0 0 0<br/> size-8192 2026 2026 8192 1 2 : tunables 8 4 0<br/> : slabdata 2026 2026 0<br/> size-4096(DMA) 0 0 4096 1 1 : tunables 24 12 8<br/> : slabdata 0 0 0<br/> size-4096 911 911 4096 1 1 : tunables 24 12 8<br/> : slabdata 911 911 0<br/> size-2048(DMA) 0 0 2048 2 1 : tunables 24 12 8<br/> : slabdata 0 0 0<br/> size-2048 1080 1120 2048 2 1 : tunables 24 12 8<br/> : slabdata 560 560 83<br/> size-1024(DMA) 0 0 1024 4 1 : tunables 54 27 8<br/> : slabdata 0 0 0<br/> size-1024 1429 1756 1024 4 1 : tunables 54 27 8<br/> : slabdata 439 439 83<br/> size-512(DMA) 0 0 512 8 1 : tunables 54 27 8<br/> : slabdata 0 0 0<br/> size-512 1607 2024 512 8 1 : tunables 54 27 8<br/> : slabdata 253 253 2<br/> size-256(DMA) 0 0 256 15 1 : tunables 120 60 8<br/> : slabdata 0 0 0<br/> size-256 3144 3495 256 15 1 : tunables 120 60 8<br/> : slabdata 233 233 0<br/> size-128(DMA) 0 0 128 30 1 : tunables 120 60 8<br/> : slabdata 0 0 0<br/> size-64(DMA) 0 0 64 59 1 : tunables 120 60 8<br/> : slabdata 0 0 0<br/> size-64 8683 22243 64 59 1 : tunables 120 60 8<br/> : slabdata 377 377 0<br/> size-32(DMA) 0 0 32 112 1 : tunables 120 60 8<br/> : slabdata 0 0 0<br/> size-128 3423 7410 128 30 1 : tunables 120 60 8<br/> : slabdata 247 247 1<br/> size-32 54883 59024 32 112 1 : tunables 120 60 8<br/> : slabdata 527 527 0<br/> kmem_cache 182 182 2688 1 1 : tunables 24 12 8<br/> : slabdata 182 182 0</li> </ol> <p>cat /proc/meminfo &lt;before job begins&gt;</p> <p>MemTotal: 16442916 kB<br/> MemFree: 15650204 kB<br/> Buffers: 200 kB<br/> Cached: 303428 kB<br/> SwapCached: 0 kB<br/> Active: 331180 kB<br/> Inactive: 206008 kB<br/> HighTotal: 0 kB<br/> HighFree: 0 kB<br/> LowTotal: 16442916 kB<br/> LowFree: 15650204 kB<br/> SwapTotal: 4225084 kB<br/> SwapFree: 4225084 kB<br/> Dirty: 0 kB<br/> Writeback: 0 kB<br/> AnonPages: 235168 kB<br/> Mapped: 8452 kB<br/> Slab: 77148 kB<br/> PageTables: 2740 kB<br/> NFS_Unstable: 0 kB<br/> Bounce: 0 kB<br/> CommitLimit: 12446540 kB<br/> Committed_AS: 857620 kB<br/> VmallocTotal: 34359738367 kB<br/> VmallocUsed: 80412 kB<br/> VmallocChunk: 34359657895 kB<br/> HugePages_Total: 0<br/> HugePages_Free: 0<br/> HugePages_Rsvd: 0<br/> Hugepagesize: 2048 kB</p> <p>cat /proc/meminfo &lt;during the WRF run&gt;<br/> MemTotal: 16442916 kB<br/> MemFree: 360168 kB<br/> Buffers: 160 kB<br/> Cached: 5678292 kB<br/> SwapCached: 2230028 kB<br/> Active: 8199640 kB<br/> Inactive: 6118380 kB<br/> HighTotal: 0 kB<br/> HighFree: 0 kB<br/> LowTotal: 16442916 kB<br/> LowFree: 360168 kB<br/> SwapTotal: 4225084 kB<br/> SwapFree: 1557472 kB<br/> Dirty: 9940 kB<br/> Writeback: 7072 kB<br/> AnonPages: 6545728 kB<br/> Mapped: 9612 kB<br/> Slab: 1349160 kB<br/> PageTables: 19704 kB<br/> NFS_Unstable: 0 kB<br/> Bounce: 0 kB<br/> CommitLimit: 12446540 kB<br/> Committed_AS: 9478760 kB<br/> VmallocTotal: 34359738367 kB<br/> VmallocUsed: 80412 kB<br/> VmallocChunk: 34359657895 kB<br/> HugePages_Total: 0<br/> HugePages_Free: 0<br/> HugePages_Rsvd: 0<br/> Hugepagesize: 2048 kB</p> <p>cat /proc/meminfo &lt;after the WRF run&gt;<br/> MemTotal: 16442916 kB<br/> MemFree: 14180204 kB<br/> Buffers: 208 kB<br/> Cached: 14928 kB<br/> SwapCached: 1788636 kB<br/> Active: 122928 kB<br/> Inactive: 1682064 kB<br/> HighTotal: 0 kB<br/> HighFree: 0 kB<br/> LowTotal: 16442916 kB<br/> LowFree: 14180204 kB<br/> SwapTotal: 4225084 kB<br/> SwapFree: 2213112 kB<br/> Dirty: 68 kB<br/> Writeback: 0 kB<br/> AnonPages: 26628 kB<br/> Mapped: 2668 kB<br/> Slab: 86572 kB<br/> PageTables: 672 kB<br/> NFS_Unstable: 0 kB<br/> Bounce: 0 kB<br/> CommitLimit: 12446540 kB<br/> Committed_AS: 279580 kB<br/> VmallocTotal: 34359738367 kB<br/> VmallocUsed: 80412 kB<br/> VmallocChunk: 34359657895 kB<br/> HugePages_Total: 0<br/> HugePages_Free: 0<br/> HugePages_Rsvd: 0<br/> Hugepagesize: 2048 kB</p> <p>NOTE: The debug log is to large to attach to this case. Here is a link instead.</p> <p><a href="https://www.dropbox.com/s/vwuklioioytcl7e/lustre_debug" class="external-link" target="_blank" rel="nofollow noopener">https://www.dropbox.com/s/vwuklioioytcl7e/lustre_debug</a></p> <p>Thanks</p> <p>Thank you, I am engaging further engineering resources now.</p> <p>So, just to reconfirm, when you run this app several times on the same client, it adds 1G more of inactive data, right, so eventually the node will die with OOM?<br/> If you unmount lustre fs after the run on this client and then mount it back instead of the reboot, is the memory reclaimed?</p> <p>I would nto put too much into the leaks you see reported as those are useless unless you take the reading after unmount since every bit of memory allocated but not yet freed because it is in use will show as leaked.</p> <p>Closing this old ticket.</p> <p>Just because memory is not "Free" doesn't mean that it is "leaked". The kernel will cache pages even if they are unused, until all of the free memory is consumed, and then old data will be freed.</p> <p>The main concern would be if the node actually runs out of memory and applications start failing (OOM killer, or <tt>-ENOMEM=-12</tt> memory allocation errors).</p> Development Epic/Theme Performance Rank 1|hzviyv: Rank (Obsolete) 6768 Severity