[LU-9977] client ran out of memory when diffing two 2GB files

• Status: Resolved
• Project: Lustre
• Affects Version/s: Lustre 2.11.0, Lustre 2.10.2
• Type: Bug
• Priority: Critical
• Reporter: James Casper
• Assignee: Zhenyu Xu
• Resolution: Not a Bug

Environment

clients: trevis-60vm1 & 2
mds: trevis-62
ost: trevis-65

before upgrade: el7.3, zfs 6.5.9, b2_10 branch, v2.10.0, b5
after upgrade: el7.4, zfs 7.1, master branch, v2.10.52, b3631

• Severity: 3

Description

Systems were imaged to el7.3/lustre2.10.0. A zfs mount point (v6.5.9) was
created on the lustre file system. A 2GB file was then copied to a directory on
the zfs mount point.

After systems were imaged to el7.4/2.10.52, an import of the 6.5.9 zpool was
performed. A 2GB file was then copied onto the zfs mount point (same file as
above - different directory). Diff was then used to compare the two files.

While diff was running, top showed it consuming 80-90% of memory. At some
point close to 90%, the client killed the diff process.

I've found two ways to avoid this:

1) Keep everything above the same except work with a newly created zfs pool
rather than an imported pool.

2) Instead of diffing two 2GB files, diff two 2GB sets of several smaller
files (largest file in set <60MB).

Note: When diff is used to compare several smaller files, it uses much less
memory (<10%).

Note: This has also been seen with ldiskfs, but is easier to repro with zfs.

Issue Links

• is related to: LU-9601 recovery-mds-scale test_failover_mds:...

Comments

[James Casper]

My last successful clean/zfs upgrade was from 2.9.0 to 2.10.0 RC1.

[Peter Jones]

Nathaniel

Could you please advise

Thanks

Peter

[James Casper]

Just tried my diff again on an imported 6.5.9 zpool. This time it was with 2.10.51 b3624, which also uses zfs 6.5.9 (and has an el7.3 kernel). It killed the diff process again. So this is not a zfs 7.1 issue.

[Andreas Dilger]

Could you include the stack traces from the console when the OOM was hit? That should be recorded by conman.

Also, for slabtop (and to a lesser extent top), the output should be sorted by total memory usage rather than number of objects (and CPU, respectively). It looks like "too" is being charged for most of the memory usage (it shows in top as 89%), but it isn't clear what that memory is.

It is definitely strange that ZFS has anything to do with this, because that is running on the OSS, and the memory used is on the client.

[James Casper]

Stack and new slabtop output attached.

FYI: This is not zfs related. I finally saw the same results with ldiskfs.

[James Casper]

I was able to repro and get a crash dump.

[James Casper]

I verified the output from top. The memory usage during diff is dramatically different between reads of a couple 2GB files and reads of several smaller files (60MB or less, total read still ~4GB):

two large files: ~85.0%
many small files: ~00.5%

[Peter Jones]

Bobijam

Could you please advise on this? It seems to be a CLIO issue that has been introduced since 2.10

Peter

[James Casper]

Still seeing 80-90% of memory used during a diff of large files. This is also happening on the latest b2_10 (2.10.2 RC1, b50). The message sent to the console when the memory killer kicks in is "diff: memory exhausted".

[Andreas Dilger]

Looking at the slabtop output it doesn’t seem like any slabs are taking up much space, the largest is tens of MB. If all the memory is accounts by diff, then either there is a significant memory leak in diff itself, or the file pages are being accounted against diff but cannot be released while the files are in use?

[James Casper]

I ran the diff again with the two directories on / rather than /mnt/lustre. The same behavior was seen (diff: memory exhausted), so I believe this is not a lustre issue.