[LU-4139] Significant perforamce issue when user over soft quota limit

• Status: Resolved
• Project: Lustre
• Affects Version/s: Lustre 2.4.1
• Fix Version/s: Lustre 2.6.0, Lustre 2.4.2, Lustre 2.5.1
• Type: Bug
• Priority: Critical
• Reporter: Mahmoud Hanafi
• Assignee: Niu Yawei (Inactive)
• Resolution: Fixed
• Severity: 4

Description

When a user goes over their softlimit there is a major performace hit.

Testing showed a file copied in 3sec when under softlimit and 7 Min when over softlimit.

Can be reproduced by just testing below and over softlimit.

see trace for when the copy was slow.

Issue Links

• is related to: LU-7795 least_qunit should be tunnable

Comments

[Peter Jones]

Niu

Could you please comment on this one?

Thanks

Peter

[Niu Yawei (Inactive)]

Hi, Mahmoud

This is by design but not a bug.

To acheive an accurate grace time management, we have to shrink the qunit size to the least qunit size (1K), then quota slave can only acquire 1K quota limit from master each time, that definitely hurts performance. For good performance, large qunit has to be kept when approaching softlimit, however, grace timer would not be triggered (or stopped) exactly on user's real disk usage, I think that would be really bad user experience.

We think the performance when over softlimit (or approaching hardlimit) is less important comparing with an accurate grace timer (or an accurate -EDQUOT), so we choose to sacrifice the performance (in certain special case) for the accuracy. This is the reason of dynamic qunit introduced actually.

[Andreas Dilger]

Niu, when you write "1k qunit" is that really 1kB of data, or 1024 blocks? It doesn't make sense to only have 1kB of quota if the blocks are allocated in chunks of 4kB.

[Mahmoud Hanafi]

3sec to 7 min is not a very good trade off. This behaviour makes softlimit useless for us. Is there a way we can tune how much min qunit is allocated if a user is over their softlimit.

[Jay Lan (Inactive)]

The qsd_internal.h showed the meaning of (qunit == 1024) as:
1MB or 1K inodes.

[Niu Yawei (Inactive)]

The qsd_internal.h showed the meaning of (qunit == 1024) as:
1MB or 1K inodes.

Right, it is 1K blocks (1M).

3sec to 7 min is not a very good trade off. This behaviour makes softlimit useless for us. Is there a way we can tune how much min qunit is allocated if a user is over their softlimit.

After over softlimit, client turns to sync write instead of writing to cache, and quota slave only acquires minimum quota limit which can just satisfy the incoming write each time, that means two RPC round-trip delay would be added to each write operation, but I don't think the performance gap will be so big (3sec to 7min), could you verify that all data was really flushed back (for the 3sec copy)? Or could you retry the test with direct io to see the difference? Thanks.

[Mahmoud Hanafi]

Here is what my testing showed:

Using IOR with directio 3 threads writing to single ost:
Over Softlimit: 316MB/sec
Under Softlimit: 356MB/sec

Using IOR with buffered IO 3 threads writing to singe ost:
Over Softlimit: 3.33MB/sec
Under Softlimit: 2.91MB/sec

using 'cp' to copy a 10G file
Under Softlimit: 7.8sec
Over softlimit: >10Min. I didn't want to wait for it to finish

What I noticed is when I was over my softlimit using cp all the I/O was 4KB RPCs. I was able to see this happen in the middle of my test as I would go over my softlimit the RPCs would drop from 1MB to 4KB. Also using IOR with buffered I/O all the RPCs where 4K. It seems that the smaller I/O sizes are the main issue.

MY IOR COMMANDS
Buffered IO
mpiexec -n 3 /u/mhanafi/bin/IOR -a POSIX -C -Y -e -i 1 -w -t 1m -b 1g -v -O useExistingTestFile=1,keepFile=1,filePerProc=1,intraTestBarriers=1,testFile=ior,outlierThreshold=1

Direct IO
mpiexec -n 3 /u/mhanafi/bin/IOR -a POSIX -C -B -i 1 -w -t 1m -b 1g -v -O useExistingTestFile=1,keepFile=1,filePerProc=1,intraTestBarriers=1,testFile=ior,outlierThreshold=1

[Andreas Dilger]

Niu,
I think it is unintentional that over softlimit IO is done in 4kB chunks, even if the qunit is getting 1MB chunks. Is it possible to avoid throttling the clients if there is a large gap between the soft and hard quota limit (i.e. treating over softlimit the same as under softlimit if there is still a large margin before the hardlimit)?

[Niu Yawei (Inactive)]

What I noticed is when I was over my softlimit using cp all the I/O was 4KB RPCs. I was able to see this happen in the middle of my test as I would go over my softlimit the RPCs would drop from 1MB to 4KB. Also using IOR with buffered I/O all the RPCs where 4K. It seems that the smaller I/O sizes are the main issue.

For quota accuracy, when approaching (or over) quota hardlimit (or softlimit), client turns to sync write (see bug16642), and in consequence the PRC size will be page size, 4K (page can't be cached on client, it has to be synced out on write).

I think it is unintentional that over softlimit IO is done in 4kB chunks, even if the qunit is getting 1MB chunks. Is it possible to avoid throttling the clients if there is a large gap between the soft and hard quota limit (i.e. treating over softlimit the same as under softlimit if there is still a large margin before the hardlimit)?

As I described above, page size (4KB) io is because of sync write on client. To avoid sync write on client after over softlimit, I think probably we can tweak the qunit size differently when over softlimit. I'll try to cooke a patch.

[Jay Lan (Inactive)]

While the servers run 2.4.1 the clients are 2.1.5. The client code has no knowledge of the new quota rules. Which variable/field enforce sync write, and how the server tells clients to start using sync write? I found where the qunit is adjusted, but I have not figured out how the sync write was enforced.

[Niu Yawei (Inactive)]

While the servers run 2.4.1 the clients are 2.1.5. The client code has no knowledge of the new quota rules. Which variable/field enforce sync write, and how the server tells clients to start using sync write? I found where the qunit is adjusted, but I have not figured out how the sync write was enforced.

The new quota didn't change client protocol, so triggering sync write when approaching limit is same as before, please check the server code qsd_op_begin0():

                        __u64   usage;

                        lqe_read_lock(lqe);
                        usage  = lqe->lqe_usage;
                        usage += lqe->lqe_pending_write;
                        usage += lqe->lqe_waiting_write;
                        usage += qqi->qqi_qsd->qsd_sync_threshold;

                        /* if we should notify client to start sync write */
                        if (usage >= lqe->lqe_granted - lqe->lqe_pending_rel)
                                *flags |= LQUOTA_OVER_FL(qqi->qqi_qtype);
                        else
                                *flags &= ~LQUOTA_OVER_FL(qqi->qqi_qtype);
                        lqe_read_unlock(lqe);

And the client code osc_queue_async_io() -> osc_quota_chkdq().

[Niu Yawei (Inactive)]

Lose some grace time accuracy to improve write performance when over softlimit: http://review.whamcloud.com/8078

[Mahmoud Hanafi]

How does this patch help with the 4k io sizes? I think that is the real issue with the performance.

[Niu Yawei (Inactive)]

How does this patch help with the 4k io sizes? I think that is the real issue with the performance.

4K io size is caused by over quota flag on client, with this patch, slave can acquire/pre-acquire little bit more spare limit each time when over softlimit, then over quota flag won't be set on client anymore.

[Mahmoud Hanafi]

New benchmark number with the patch

Direct I/O
UnderSoftlimit: 383MB/sec
OverSoftlimit: 359MB/sec

Buffered I/O
UnderSoftlimit:316MB.sec
OverSoftlimit: 304MB/sec

So it looks good!

[Jian Yu]

Patch http://review.whamcloud.com/8078 landed on master branch and was cherry-picked to Lustre b2_4 branch.

[Mahmoud Hanafi]

Please close this one.

[Peter Jones]

ok Mahmoud

[Niu Yawei (Inactive)]

backported to b2_5: http://review.whamcloud.com/8603