[LU-12687] Fast ENOSPC on direct I/O

• Type: Bug
• Resolution: Fixed
• Priority: Minor
• Fix Version/s: Lustre 2.14.0, Lustre 2.12.6
• Severity: 3
• Assignee: Vladimir Saveliev
• Reporter: Vladimir Saveliev

Description

On direct I/O a client gets substantial amount of grants without consuming them so that direct I/O writes face with ENOSPC long before disk space is over.
The below example shows how dd oflag=direct fails to write 400kb to ~70mb ost.

[root@sl75master tests]# OSTSIZE=100000 sh llmount.sh
...
Updated after 6s: wanted 'procname_uid' got 'procname_uid'
disable quota as required
[root@sl75master tests]# lfs df -h
UUID                       bytes        Used   Available Use% Mounted on
lustre-MDT0000_UUID       122.4M        1.9M      109.5M   2% /mnt/lustre[MDT:0]
lustre-OST0000_UUID        69.4M        1.2M       61.4M   2% /mnt/lustre[OST:0]
lustre-OST0001_UUID        69.4M        1.2M       61.4M   2% /mnt/lustre[OST:1]

filesystem_summary:       138.9M        2.5M      122.7M   2% /mnt/lustre

[root@sl75master tests]# dd if=/dev/zero of=/mnt/lustre/file bs=4k count=100 oflag=direct
dd: error writing ‘/mnt/lustre/file’: No space left on device
54+0 records in
53+0 records out
217088 bytes (217 kB) copied, 0.138233 s, 1.6 MB/s
[root@sl75master tests]# 

Issue Links

• is related to: LU-12757 sanity-lfsck test 36a fails with '(N) Fail to resync /mnt/lustre/d36a.sanity-lfsck/f2'
• is related to: LU-13766 tgt_grant_check() lsrza-OST000a: cli dfdf1aff-07d9-53b3-5632-c18a78027eb2 claims 1703936 GRANT, real grant 0
• is related to: LU-13847 sanity test_64f: grants mismatch
• is related to: LU-4198 Improve IO performance when using DIRECT IO using libaio
• is related to: LU-4664 sync write should consume grant on client

Comments

[Mikhail Pershin]

yes, I understand that, meanwhile we don't just consume grants but also increase cl_dirty_pages and obd_dirty_pages counters during that and do that without regards to limits, so technically these values can become greater than their cap and prevent cache entering in osc_enter_cache_try() for a non-DIO writers. So actually the question is - should we increase cl_dirty_pages and obd_dirty_pages at all? Or better keep them untouched with DIO writes. Probably that is not big deal because as you said - it returns back quite quickly

[Andreas Dilger]

Hi Mike, I don't think that the max_dirty_pages limit applies in the case of O_DIRECT writes, because the client is not caching those pages locally. It is immediately sending the pages for write to the OST, so it doesn't actually need the grant or to check the dirty limit. The grant usage is only to prevent clients from holding grant that cannot be used by the O_DIRECT writes. In most cases, the client will get grant back immediately in the reply, unless the filesystem is low on free space.

[Mikhail Pershin]

Vladimir, Andreas, I reopened this due to question I have about already landed patch, please check my previous comment.

[Mikhail Pershin]

Regarding landed patch https://review.whamcloud.com/35896/ - I've noted that added code does pretty the same as osc_enter_cache_try() but skips the following checks:

	if (cli->cl_dirty_pages < cli->cl_dirty_max_pages) {
		if (atomic_long_add_return(1, &obd_dirty_pages) <=
		    obd_max_dirty_pages) {
                    ...

for cl_dirty_max_pages limit. Instead of that we just consuming write grants always. Was that intentional or the same limits check should be added for DIO case too?

[Gerrit Updater]

Olaf Faaland-LLNL ([e-mail]) uploaded a new patch: https://review.whamcloud.com/39517
Subject: LU-12687 canary: comment-only change
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: bcec79eb40dda0c26bffb9ec950a142034866df6

[Gerrit Updater]

Mike Pershin ([e-mail]) uploaded a new patch: https://review.whamcloud.com/39386
Subject: LU-12687 osc: consume grants for direct I/O
Project: fs/lustre-release
Branch: b2_12
Current Patch Set: 1
Commit: c89c8423cf730df97ddc19a8981f978c79fabdfa

[Peter Jones]

Landed for 2.14

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/35896/
Subject: LU-12687 osc: consume grants for direct I/O
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 05f326a7988a7a0d6954d1b0d318315526209ae6

[Andreas Dilger]

Alex, if you are able to reproduce this easily, could you please verify patch https://review.whamcloud.com/35896 "LU-12687 osc: consume grants for direct I/O" fixes it for you? That is about to land.

[Alex Zhuravlev]

I'm hitting this almost every run of sanity.sh in 248b

[Andreas Dilger]

I agree with Patrick here that the LU-4664 patch is the right way to go. If we don't consume grants on O_DIRECT writes, then it is possible for those writes to run out of space when there is still grant available on the client.

Conversely, if O_DIRECT consumes all of the grant, but doesn't request more (as with the patch here) then if there are cached writes after this it will not have any grant and will start with small (PAGE_SIZE) synchronous writes until it has some grant.

It would be great if that patch could be refreshed and landed.