https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-5179 Lustre <p>We have been seeing our SLES11SP2 and SLES11SP3 clients have stuck anonymous memory that cannot be cleared up without a reboot. We have three test cases which can replicate the problem reliably. We have been able to replicate the problem on different clients on all of our lustre file systems. We have not been able to reproduce the problem when using NFS, ext3, CXFS, or tmpfs.</p> <p>We have been working with SGI on tracking down this problem. Unfortunately, they have been unable to reproduce the problem on their systems. On our systems, they have simplified the test case to mmaping a file along with an equally sized anonymous region, and reading the contents of the mmaped file into the anonymous mmaped region. This test case can be provided to see if you can reproduce this problem.</p> <p>To determine if the problem is occurring, reboot the system to ensure that memory is clean. Check /proc/meminfo for the amount of Active(anon) memory being used. Run the test case. During the test case, the amount of anonymous memory will increase. At the end of the test case, it would be expected for the amount to drop back to pre-test case levels. </p> <p>To confirm that the anonymous memory is stuck, we have been using memhog to attempt to allocate memory. If the node has 32Gb of memory, with 2Gb of anonymous memory used, we attempt to allocate 31Gb of memory. If memhog completes and you then have only 1Gb of anonymous memory, you have not reproduced the problem. If memhog is killed, you have.</p> <p>SGI would like to get information about how to get debug information to track down this problem.</p> Clients:<br/> Endeavour: 2.4.3, ldan: 2.4.1, Pleiades compute nodes: 2.1.5 or 2.4.1<br/> Servers:<br/> 2.1.5, 2.4.1, 2.4.3 LU-5179

Reading files from lustre results in stuck anonymous memory when JOBID is enabled

Bug Critical Resolved Fixed Oleg Drokin Herbert Yeung HB Thu, 12 Jun 2014 01:57:19 +0000 Tue, 12 Aug 2014 19:46:40 +0000 Fri, 20 Jun 2014 18:58:00 +0000 Lustre 2.5.0 Lustre 2.4.3 Lustre 2.6.0 Lustre 2.5.3 0 10 <p>Please upload your testcase.</p> <p>You list multiple client node versions, is this observed in all of them?</p> <p>Yes, we have been able to reproduce the problem on all of the clients. Though I am not positive that we have tested every permutation, I believe that all of the lustre versions have been reproduced on all of the clients.</p> <p>From the call:This was also reproduced on rhel6.4 kernel with the same testcase.<br/> unmounting the fs either fails (fs busy) or when succeeding, does not free memory either.</p> <p>The reproducer only works on NASA systems and it's not 100% reliable, but a high frequency is still reported.</p> <p>This originally was investigated from strange OOM issues.</p> <p>Second test case that reproduces the problem.</p> <p>Normally, we run the test script through PBS. To do so, you can use My_run.ivy. If you want to run interactively or without PBS, use My_run_I_ivy. Sample output from PBS is at kdgordon.o328785, though some of the data has been sanitized.</p> <p>The script prints out a variety of information, including the amount of anonymous memory being used by the system. The runit script is called which produces the stuck anonymous memory. After that, the amount of used anonymous memory is checked. memhog is called to try to clear up the memory and the amount of anonymous memory is displayed again.</p> <p>You will need to tweak the amount of memory that memhog attempts allocate based on how much memory your test system has. Generally, about 800MB of memory gets stuck after running the job. runit can be called several times to increase the amount of stuck memory.</p> <p>After running the test, the file checkpoint.mtcp is created. On some of our systems, the file may need to be deleted before being able to reproduce the problem again.</p> <p>After running memhog, the memory can go to swap instead and remains there used. This still indicates that the memory is not being freed.</p> <p>I would just like to note that there's no source to the timeout and test_ckpt so it's kind of hard to see what they are doing.</p> <p>Source for test_ckpt uploaded. timeout comes from coreutils-8.22</p> <p>can you show me the output of /proc/meminfo when you see this problem? </p> <p>Below is an exmaple of a system that stuck in this memory. The system has 4TB memry and 1.5TB stuck in Acitve(anon) that can not be released. There are 126 nodes in that systems and application would request a number of nodes for their testing. After the memory leak, those nodes would have not enough memory for other jobs. The application would fail and resubmision of the job would then fail to start because the requested notes did not have enough memory.</p> <ol> <li>cat /proc/meminfo<br/> MemTotal: 4036524872 kB<br/> MemFree: 2399583516 kB<br/> Buffers: 243504 kB<br/> Cached: 5204560 kB<br/> SwapCached: 678520 kB<br/> Active: 1544908812 kB<br/> Inactive: 56619188 kB<br/> Active(anon): 1543105636 kB<br/> Inactive(anon): 53018772 kB<br/> Active(file): 1803176 kB<br/> Inactive(file): 3600416 kB<br/> Unevictable: 0 kB<br/> Mlocked: 0 kB<br/> SwapTotal: 10239996 kB<br/> SwapFree: 0 kB<br/> Dirty: 554504 kB<br/> Writeback: 26128 kB<br/> AnonPages: 1595359296 kB<br/> Mapped: 143708 kB<br/> Shmem: 98772 kB<br/> Slab: 11485844 kB<br/> SReclaimable: 161660 kB<br/> SUnreclaim: 11324184 kB<br/> KernelStack: 87016 kB<br/> PageTables: 6747560 kB<br/> NFS_Unstable: 31856 kB<br/> Bounce: 0 kB<br/> WritebackTmp: 0 kB<br/> CommitLimit: 2027403680 kB<br/> Committed_AS: 1262704572 kB<br/> VmallocTotal: 34359738367 kB<br/> VmallocUsed: 18824052 kB<br/> VmallocChunk: 25954600480 kB<br/> HardwareCorrupted: 0 kB<br/> AnonHugePages: 1264787456 kB<br/> HugePages_Total: 1073<br/> HugePages_Free: 468<br/> HugePages_Rsvd: 468<br/> HugePages_Surp: 1073<br/> Hugepagesize: 2048 kB<br/> DirectMap4k: 335872 kB<br/> DirectMap2M: 134963200 kB<br/> DirectMap1G: 3958374400 kB</li> </ol> <p>just to narrow down the problem, will you please try it again with huge page disabled?</p> <p>I really meant to say transparent huge page.</p> <p>ldan2 and ldan3 below are fairly ordinary self-contained systems running<br/> Lustre 2.4.1-6nas_ofed154 client. This problem has been reproduced on several versions of the NAS Lustre client and server software.</p> <p>Log into ldan2. (I've mostly used a qsub session, but I have reproduced the problem outside of PBS)</p> <p>Log into ldan3. ( I have special permission to log into an ldan I don't<br/> have a PBS job running on)</p> <p>On both systems cd to the test (lustre) directory<br/> In this directory exist the following:<br/> 1. a copy of hedi's test, I've tested with the first one he wrote and a fairly<br/> late version(mmap4.c). The later version (attached) is more flexible.</p> <p>2. a 1g file created with:<br/> dd if=/dev/zero of=1g bs=4096 count=262144</p> <p>#</p> <ol> <li>set up your favorite Anonymous Memory monitor on ldan2</li> <li>I've been using nodeinfo in a seperate window<br/> #</li> </ol> <p>On ldan3 run:<br/> dd count=1 bs=1 conv=notrunc of=1g if=/dev/zero</p> <p>On ldan2 run:<br/> ./mmap 1g</p> <p>After the mmap program terminates, notice that the anonymous memory<br/> used by mmap remains in memory. I've never been able to force <br/> persistent Anonymous memory out of the greater Linux virtual memory<br/> system (memory + swap). Anonymous memory swapped to disk is not<br/> accounted for as "Anonymous Memory", but it is accounted for as swap.</p> <p>The problem does not reproduce if the dd "interference" is not run.<br/> The problem does not reproduce if the dd "interference" is run and then<br/> lflush is run, or the file is read (w/o mmap) from a third system. The problem does not reproduce if the dd "interference" is run on ldan2, then the mmap test is run on ldan2.</p> <p>we may have found the problem, Oleg will push a patch soon.</p> <p>I did audit of all mmput calls in the code, there are only two functions that use it.<br/> One of them has a mm_struct leak that would introduce symptoms very similar to what you see.</p> <p>My proposed patch is at <a href="http://review.whamcloud.com/10759" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/10759</a> please give it a try</p> <p>Also additional info.<br/> For the problem to manifest you must have jobid functionality enabled in the "read env variable" mode. Once I enabled that, I can immediately reproduce and so I tested the patch and it fixes the problem for me.</p> <p>The patch affects both server and client. If I only update the client, would it solve the problem at the client side?</p> <p>yes, applying it on the client side only will fix the problem.</p> <p>Jay built a copy of the client for the ldan test system. Initial testing is positive, this fixes the test cases reported in this LU on the type of system I am using to test.</p> <p>Patch landed to Master. Please reopen ticket if more work is needed.</p> <p>b2_4 backport: <a href="http://review.whamcloud.com/10868" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/10868</a></p> Related Development Rank 1|hzwo8n: Rank (Obsolete) 14374 Severity