[LU-11798] cur_grant goes to 0 and never increases with 2.8 client and 2.10 server

• Type: Bug
• Resolution: Fixed
• Priority: Major
• Fix Version/s: Lustre 2.13.0, Lustre 2.12.1
• Affects Version/s: Lustre 2.8.0, Lustre 2.10.5
• Labels: llnl
• Environment: client: ; lustre-2.8.2_5.chaos-1.ch6.x86_64 ; ; servers: ; zfs-0.7.11-4llnl.ch6.x86_64 ; lustre-2.10.5_2.chaos-3.ch6.x86_64
• Severity: 3
• Assignee: Alex Zhuravlev
• Reporter: Olaf Faaland

Description

The Lustre 2.10.5 / zfs 0.7.11 OST is not allocating grant to the clients.

I set +cache debug flag on both client and server.

On the client, under /proc/fs/lustre/osc/lustre3-OST....:
cur_dirty_bytes == 0
cur_grant_bytes starts out as 2097152

I then issue
dd if=/dev/urandom of=/p/lustre3/foo/toss-4377/ost0/test.10M bs=5K count=2000

cur_grant_bytes goes down, and never up, as writes to the OST occur. Eventually it reaches 0 and then stays == 0.

There is no connection issue - no log messages indicating connections lost/reconnects, and osc/lustre3-OSTXXXX.*/state has "current_state: FULL" for all OSTs.

Issue Links

• is related to: LU-11844 IO pattern causing writes to hang to OST
• is related to: LU-11663 corrupt data after page-unaligned write with zfs backend lustre 2.10

Comments

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/34561/
Subject: LU-11798 grant: prevent overflow of o_undirty
Project: fs/lustre-release
Branch: b2_12
Current Patch Set:
Commit: e13cc954a776f6dec06f01299a55a55018ae6235

[Gerrit Updater]

Minh Diep ([e-mail]) uploaded a new patch: https://review.whamcloud.com/34561
Subject: LU-11798 grant: prevent overflow of o_undirty
Project: fs/lustre-release
Branch: b2_12
Current Patch Set: 1
Commit: 9786a2a7bf1d27539ea5aaa27200a5f651d9b430

[Peter Jones]

Landed for 2.13

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/33948/
Subject: LU-11798 grant: prevent overflow of o_undirty
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: d6f5219162116778e49865d83724276b17b9ce3a

[Gerrit Updater]

Alex Zhuravlev ([e-mail]) uploaded a new patch: https://review.whamcloud.com/33957
Subject: LU-11798 grant: prevent overflow of o_undirty
Project: fs/lustre-release
Branch: b2_10
Current Patch Set: 1
Commit: a8d73df2469a4966f93d61fc4668f581c1dd7422

[Olaf Faaland]

Great, thank you both.

[Peter Jones]

Olaf

Thanks for pushing the patch to master. Alex is out for the holidays but I chatted briefly with him earlier and he confirmed that we could land this fix to master (and then to b2_10 and b2_12 afterwards)

Peter

[Olaf Faaland]

I've pushed p0 as a patch against Master to gerrit, which I will cherry-pick from for our local build.

[Gerrit Updater]

Olaf Faaland-LLNL ([e-mail]) uploaded a new patch: https://review.whamcloud.com/33948
Subject: LU-11798 grant: prevent overflow of o_undirty
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 0b13ea79c3477ed835f40c12a03e38d92e7b77d4

[Olaf Faaland]

So this needs to be fixed, or at least made less bad, in master. And since you're about to stop maintaining 2.10, it seems a shame to leave it this broken.

Happy New Year!   What say you?  Or is there a patch in gerrit I overlooked? Thanks.