[LU-7085] Toward smaller memory allocations on wide-stripe file systems

• Type: Improvement
• Resolution: Fixed
• Priority: Minor
• Fix Version/s: Lustre 2.8.0
• Affects Version/s: Lustre 2.8.0
• Environment: Test nodes with 2.7.57-gea38322
• Severity: 3
• Assignee: Yang Sheng
• Reporter: Matt Ezell

Description

I'm testing on a fairly recent master build that includes the patch from LU-6587 (refactor OBD_ALLOC_LARGE to always do kmalloc first). That band-aid has been great a improving performance on our wide-stripe file systems, but in the face of memory pressure/fragmentation, it will still rely on vmalloc to satisfy memory requests. Since users tend to use RAM, I'd like to see if there are any opportunities to reduce allocation sizes.

Anywhere we need to allocate sizeof(something) * num_stripes, we should check to see if there's any way to avoid per-stripe information or at least reduce sizeof(something).

Comments

[Joseph Gmitter (Inactive)]

Landed for 2.8.0

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch http://review.whamcloud.com/17476/
Subject: LU-7085 lov: trying smaller memory allocations
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: e1e56300cac30fe8d9db296107905f5936648c3c

[Gerrit Updater]

Yang Sheng ([e-mail]) uploaded a new patch: http://review.whamcloud.com/17476
Subject: LU-7085 lov: trying smaller memory allocations
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 8c7f009755ef03080c599347cf0452a9bd7cf5f9

[Yang Sheng]

I'll trying second way and then would doing it in first way if we still not satisfy the effect.

Thanks,
YangSheng

[Jian Yu]

Thank you, Yang Sheng. Which way would you prefer to implement?

[Yang Sheng]

After a few test and discuss with clio expert. Looks like lis_subs buffer allocation is not avoid entirely. One way is allocate every osc separately. So we can skip some osc needn't sync data. But it really bring some complexity. Other way just trying to reduce the struct size.

Thanks,
YangSheng

[Jian Yu]

I think we can skip invoke cl_sync_file_range while unlink file.

Hi Yang Sheng, are you going to create a patch for this?

[Yang Sheng]

The 'lio->lis_subs'big buffer allocated from cl_io_init invoked by cl_sync_file_range. The code path as below:
iput=> iput_final => drop_inode => ll_delete_inode => cl_sync_file_range

I think we can skip invoke cl_sync_file_range while unlink file.

[Jian Yu]

There should not be a large buffer allocation for unlinks, that would be a bug I think.

Hi Yang Sheng,

Could you please look into the above issue? Thank you.

[Andreas Dilger]

There should not be a large buffer allocation for unlinks, that would be a bug I think.

As for the max reply size decay, that was something I proposed during patch review but was never implemented. I agree that if access to wide striped files is rare that it may make sense to reduce the allocations again, but the hard part is to figure out what "rarely" means do that there are not continual resends.