https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-9977 Lustre <p>Systems were imaged to el7.3/lustre2.10.0. A zfs mount point (v6.5.9) was <br/> created on the lustre file system. A 2GB file was then copied to a directory on <br/> the zfs mount point.</p> <p>After systems were imaged to el7.4/2.10.52, an import of the 6.5.9 zpool was <br/> performed. A 2GB file was then copied onto the zfs mount point (same file as<br/> above - different directory). Diff was then used to compare the two files.</p> <p>While diff was running, top showed it consuming 80-90% of memory. At some<br/> point close to 90%, the client killed the diff process.</p> <p>I've found two ways to avoid this: </p> <p>1) Keep everything above the same except work with a newly created zfs pool<br/> rather than an imported pool.</p> <p>2) Instead of diffing two 2GB files, diff two 2GB sets of several smaller <br/> files (largest file in set &lt;60MB).</p> <p>Note: When diff is used to compare several smaller files, it uses much less <br/> memory (&lt;10%).</p> <p>Note: This has also been seen with ldiskfs, but is easier to repro with zfs.</p> clients: trevis-60vm1 &amp; 2<br/> mds: trevis-62<br/> ost: trevis-65<br/> <br/> before upgrade: el7.3, zfs 6.5.9, b2_10 branch, v2.10.0, b5 <br/> after upgrade: el7.4, zfs 7.1, master branch, v2.10.52, b3631 <br/> LU-9977

client ran out of memory when diffing two 2GB files

Bug Critical Resolved Not a Bug Zhenyu Xu James Casper Tue, 12 Sep 2017 16:00:30 +0000 Thu, 7 Dec 2017 09:44:40 +0000 Thu, 7 Dec 2017 09:44:40 +0000 Lustre 2.11.0 Lustre 2.10.2 0 4 <p>My last successful clean/zfs upgrade was from 2.9.0 to 2.10.0 RC1.</p> <p>Nathaniel</p> <p>Could you please advise</p> <p>Thanks</p> <p>Peter</p> <p>Just tried my diff again on an imported 6.5.9 zpool. This time it was with 2.10.51 b3624, which also uses zfs 6.5.9 (and has an el7.3 kernel). It killed the diff process again. So this is not a zfs 7.1 issue.</p> <p>Could you include the stack traces from the console when the OOM was hit? That should be recorded by conman. </p> <p>Also, for slabtop (and to a lesser extent top), the output should be sorted by total memory usage rather than number of objects (and CPU, respectively). It looks like "too" is being charged for most of the memory usage (it shows in top as 89%), but it isn't clear what that memory is. </p> <p>It is definitely strange that ZFS has anything to do with this, because that is running on the OSS, and the memory used is on the client. </p> <p>Stack and new slabtop output attached.</p> <p>FYI: This is not zfs related. I finally saw the same results with ldiskfs. </p> <p>I was able to repro and get a crash dump.</p> <p>I verified the output from top. The memory usage during diff is dramatically different between reads of a couple 2GB files and reads of several smaller files (60MB or less, total read still ~4GB):</p> <p>two large files: ~85.0%<br/> many small files: ~00.5%</p> <p>Bobijam</p> <p>Could you please advise on this? It seems to be a CLIO issue that has been introduced since 2.10</p> <p>Peter</p> <p>Still seeing 80-90% of memory used during a diff of large files. This is also happening on the latest b2_10 (2.10.2 RC1, b50). The message sent to the console when the memory killer kicks in is "diff: memory exhausted".</p> <p>Looking at the slabtop output it doesn’t seem like any slabs are taking up much space, the largest is tens of MB. If all the memory is accounts by diff, then either there is a significant memory leak in diff itself, or the file pages are being accounted against diff but cannot be released while the files are in use? </p> <p>I ran the diff again with the two directories on / rather than /mnt/lustre. The same behavior was seen (diff: memory exhausted), so I believe this is not a lustre issue.</p> Related LU-9601 Development Rank 1|hzzk0f: Rank (Obsolete) 9223372036854775807 Severity