[LU-4139] Significant perforamce issue when user over soft quota limit

• Type: Bug
• Resolution: Fixed
• Priority: Critical
• Fix Version/s: Lustre 2.6.0, Lustre 2.4.2, Lustre 2.5.1
• Affects Version/s: Lustre 2.4.1
• Severity: 4
• Assignee: Niu Yawei (Inactive)
• Reporter: Mahmoud Hanafi

Description

When a user goes over their softlimit there is a major performace hit.

Testing showed a file copied in 3sec when under softlimit and 7 Min when over softlimit.

Can be reproduced by just testing below and over softlimit.

see trace for when the copy was slow.

Issue Links

• is related to: LU-7795 least_qunit should be tunnable

Comments

[Niu Yawei (Inactive)]

backported to b2_5: http://review.whamcloud.com/8603

[Peter Jones]

ok Mahmoud

[Mahmoud Hanafi]

Please close this one.

[Jian Yu]

Patch http://review.whamcloud.com/8078 landed on master branch and was cherry-picked to Lustre b2_4 branch.

[Mahmoud Hanafi]

New benchmark number with the patch

Direct I/O
UnderSoftlimit: 383MB/sec
OverSoftlimit: 359MB/sec

Buffered I/O
UnderSoftlimit:316MB.sec
OverSoftlimit: 304MB/sec

So it looks good!

[Niu Yawei (Inactive)]

How does this patch help with the 4k io sizes? I think that is the real issue with the performance.

4K io size is caused by over quota flag on client, with this patch, slave can acquire/pre-acquire little bit more spare limit each time when over softlimit, then over quota flag won't be set on client anymore.

[Mahmoud Hanafi]

How does this patch help with the 4k io sizes? I think that is the real issue with the performance.

[Niu Yawei (Inactive)]

Lose some grace time accuracy to improve write performance when over softlimit: http://review.whamcloud.com/8078

[Niu Yawei (Inactive)]

While the servers run 2.4.1 the clients are 2.1.5. The client code has no knowledge of the new quota rules. Which variable/field enforce sync write, and how the server tells clients to start using sync write? I found where the qunit is adjusted, but I have not figured out how the sync write was enforced.

The new quota didn't change client protocol, so triggering sync write when approaching limit is same as before, please check the server code qsd_op_begin0():

                        __u64   usage;

                        lqe_read_lock(lqe);
                        usage  = lqe->lqe_usage;
                        usage += lqe->lqe_pending_write;
                        usage += lqe->lqe_waiting_write;
                        usage += qqi->qqi_qsd->qsd_sync_threshold;

                        /* if we should notify client to start sync write */
                        if (usage >= lqe->lqe_granted - lqe->lqe_pending_rel)
                                *flags |= LQUOTA_OVER_FL(qqi->qqi_qtype);
                        else
                                *flags &= ~LQUOTA_OVER_FL(qqi->qqi_qtype);
                        lqe_read_unlock(lqe);

And the client code osc_queue_async_io() -> osc_quota_chkdq().

[Jay Lan (Inactive)]

While the servers run 2.4.1 the clients are 2.1.5. The client code has no knowledge of the new quota rules. Which variable/field enforce sync write, and how the server tells clients to start using sync write? I found where the qunit is adjusted, but I have not figured out how the sync write was enforced.