[LU-2613] opening and closing file can generate 'unreclaimable slab' space

• Type: Bug
• Resolution: Fixed
• Priority: Major
• Fix Version/s: Lustre 2.6.0, Lustre 2.5.1
• Affects Version/s: Lustre 2.1.3, Lustre 2.1.4
• Labels: JL mn4
• Severity: 3
• Assignee: Hongchao Zhang
• Reporter: Alexandre Louvet (Inactive)

Description

We have a lot of nodes with a large amount of unreclaimable memory (over 4GB). Whatever we try to do (manually shrinking the cache, clearing lru locks, ...) the memory can't be recovered. The only way to get the memory back is to umount the lustre filesystem.

After some troubleshooting, I was able to wrote a small reproducer where I just open(2) then close(2) files in O_RDWR (my reproducer use to open thousand of files to emphasize the issue).

Attached 2 programs :

• gentree.c (cc -o gentree gentree.c -lpthread) to generate a tree of known files (no need to use readdir in reproducer.c)
• reproducer.c (cc -o reproducer reproduver.c -lpthread) to reproduce the issue.
  The macro BASE_DIR has to be adjust according the local cluster configuration (you should provide the name of a directory located on a lustre filesystem).

There is no link between the 2 phases as rebooting the client between gentree & reproducer does't avoid the problem. Running gentree (which open as much files as reproducer) doesn't show the issue.

Issue Links

• is duplicated by: LU-3399 MDT don't update client last commited correctly so produce OOM on client
• is duplicated by: LU-3381 lock enqueue fails for open(".lustre/fid", ...)
• is related to: LU-4272 lu_device_fini()) ASSERTION( cfs_atomic_read(&d->ld_ref) == 0 ) failed from lovsub_device_free
• is related to: LU-4270 Interop 2.5.0<->2.6 failure on test suite sanity test_209: open/close requests are not freed

Comments

[Peter Jones]

Landed for 2.5.1 and 2.6

[Bob Glossman (Inactive)]

backport to b2_4
http://review.whamcloud.com/8277

[Peter Jones]

Pushing to 2.5.1 because it seems that the patch needs more work

[Niu Yawei (Inactive)]

Seems the server code has to be changed. Anyway, I introduced a new DISP bit (DISP_OPEN_STRIPE) to identify the open which creates stripe, with this manner, the server/protocol changes are less than the former patch (server returning on-disk transno). Mike could you take a look at the patch? Thanks

[Niu Yawei (Inactive)]

Mike, I realized that not only the open creates object (with DISP_OPEN_CREATE) needs be replayed, the open which creates stripe data needs be replayed as well (see mdt_create_data()), and I don't see how to identify such open on client, any good idea?

[Mikhail Pershin]

Niu, I am not so sure it will be easy to implement, this is just possible way to go but if that will work that would be good.

[Niu Yawei (Inactive)]

Mike, your solution looks fine to me, I'll update the patch in this way soon. Thanks.