report.tgz
|
Related |
|||
| is related to |
|
automate NFSv3/v4 over Lustre Testing | Resolved |
[LU-907] ior test on nfs over lustre causes OOM in nfs server (lustre client) Created: 09/Dec/11 Updated: 29/May/17 Resolved: 29/May/17 |
|
| Status: | Resolved |
| Project: | Lustre |
| Component/s: | None |
| Affects Version/s: | Lustre 2.2.0 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Minor |
| Reporter: | Minh Diep | Assignee: | Lai Siyao |
| Resolution: | Cannot Reproduce | Votes: | 0 |
| Labels: | None | ||
| Attachments: |
report.tgz
|
||||||||
| Issue Links: |
|
||||||||
| Severity: | 3 | ||||||||
| Rank (Obsolete): | 4233 | ||||||||
| Description |
|
When we run ior test on nfs over lustre, we consistently hit OOM on the nfs server/lustre client 10:04:52:Lustre: DEBUG MARKER: == sanity test iorssf: iorssf ======================================================================== 10:04:49 (1323367489) Attached is the maloo's log. I can't get the report because the test hung due to OOM |
| Comments |
| Comment by Peter Jones [ 09/Dec/11 ] |
|
Lai Could you please look into this one? Thanks Peter |
| Comment by Lai Siyao [ 26/Dec/11 ] |
|
Let me clarify one thing first: this OOM happened on MDS, not Lustre client. I'll try to reproduce and collect memory usage on MDS first. |
| Comment by Minh Diep [ 29/Dec/11 ] |
|
actually, we mount lustre client on the mds and make it nfs server. The OOM actually relates to lustre client more. |
| Comment by Lai Siyao [ 29/Dec/11 ] |
|
Yes, I also observed this. In my test environment I installed client and mds on different node, and found OOM on client. |
| Comment by Lai Siyao [ 30/Dec/11 ] |
|
I output /proc/slabinfo before OOM: ... size-2048 229724 229724 2048 2 1 : tunables 24 12 8 : slabdata 114862 114862 84 size-1024(DMA) 0 0 1024 4 1 : tunables 54 27 8 : slabdata 0 0 0 size-1024 914900 914900 1024 4 1 : tunables 54 27 8 : slabdata 228725 228725 0 size-512(DMA) 0 0 512 8 1 : tunables 54 27 8 : slabdata 0 0 0 size-512 229328 229328 512 8 1 : tunables 54 27 8 : slabdata 28666 28666 0 size-256(DMA) 0 0 256 15 1 : tunables 120 60 8 : slabdata 0 0 0 size-256 1890 1920 256 15 1 : tunables 120 60 8 : slabdata 128 128 0 size-128(DMA) 0 0 128 30 1 : tunables 120 60 8 : slabdata 0 0 0 size-64(DMA) 0 0 64 59 1 : tunables 120 60 8 : slabdata 0 0 0 size-64 345858 345858 64 59 1 : tunables 120 60 8 : slabdata 5862 5862 0 size-32(DMA) 0 0 32 112 1 : tunables 120 60 8 : slabdata 0 0 0 size-128 32820 32820 128 30 1 : tunables 120 60 8 : slabdata 1094 1094 0 size-32 233968 233968 32 112 1 : tunables 120 60 8 : slabdata 2089 2089 0 It shows common slab consumes a lot of memory (1.5G against 2G total), and if I stopped test and umount lustre, these slab memory will be freed right after. I'll try to collect some memory debug log to find where such memory is allocated and not freed. |
| Comment by Lai Siyao [ 05/Jan/12 ] |
|
Debug log shows the memory allocated above is uncommitted open request; and after further looking into the code, I found open only fetches LOOKUP|OPEN lock, and a later getattr will take LOOKUP|UPDATE lock, as will conflict with the cached open lock, thus the open lock cache for NFS export doesn't take effect at all (1.8 code will take UPDATE lock for non-create case). I've made a fix for this, and will commit to review after test later. |
| Comment by Lai Siyao [ 05/Jan/12 ] |
|
review is on http://review.whamcloud.com/#change,1921 Minh, could you help verify the fix? |
| Comment by Lai Siyao [ 08/Jan/12 ] |
|
sanityn.sh test 13 failed, I found two more issues for nfs support:
The patch is updated. |
| Comment by Oleg Drokin [ 16/Feb/12 ] |
|
If the problem is a lot of uncommitted file opens, then we should attack the real problem and truncate the open requests in uncommitted queue so that they only include enough data for the current request, and not include all the empty space in the request buffer "just in case", since we know the striping already at that point. |
| Comment by Lai Siyao [ 24/Feb/12 ] |
|
Oleg, I was busy on other bugs, sorry to reply so late. For NFS export, the call trace of lustre client for a normal write is like this: .permission -> take LOOKUP|UPDATE dlm lock .open -> revoke previous lock, take LOOKUP|OPEN lock .getattr -> revoke previous lock, take LOOKUP|UPDATE lock This happens for each nfs write, and it shows the open lock doesn't take any effect, but worsen the situation: update lock is not cached at all. So IMO the real problem is open lock cache doesn't work. Besides, for open(O_READONLY) a LOOKUP|OPEN with mode LCK_CR lock is fetched, if another client calls setattr(LA_MODE), this LOOKUP lock with LCK_CR mode won't be revoked, because setattr takes lock with mode LCK_PW, which is compatible with LCK_CR. This is wrong! So a LOOKUP lock with LCK_CR mode doesn't make sense. If OPEN lock is split from LOOKUP lock, this problem can be solved. Finally if OPEN lock is fetched only, the dentry won't be found by others. But as I said in the beginning, for NFS .permission is always called before .open, so generally a valid dentry is already there. In brief, if OPEN lock is mainly for NFS export usage, it's best to split OPEN lock from LOOKUP lock. |
| Comment by Andreas Dilger [ 29/May/17 ] |
|
Close old ticket. |