[LU-11390] sanity-quota test_61: test timeout

• Status: Resolved
• Project: Lustre
• Affects Version/s: Lustre 2.12.0
• Fix Version/s: Lustre 2.12.0
• Type: Bug
• Priority: Minor
• Reporter: Maloo
• Assignee: Hongchao Zhang
• Resolution: Fixed
• Severity: 3

Description

This issue was created by maloo for sarah <[e-mail]>

This issue relates to the following test suite run: https://testing.whamcloud.com/test_sets/4b207cce-b97d-11e8-9df3-52540065bddc

test_61 failed with the following error:

Timeout occurred after 1369 mins, last suite running was sanity-quota, restarting cluster to continue tests

cannot find error msg

test log

set to use default quota
set default quota
get default quota
Disk default grp quota:
     Filesystem   bquota  blimit  bgrace   iquota  ilimit  igrace
    /mnt/lustre  20480   20480       0      0       0  604800
Test not out of quota
running as uid/gid/euid/egid 60000/60000/60000/60000, groups:
 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d61.sanity-quota/f61.sanity-quota-0] [count=10] [oflag=sync]
10+0 records in
10+0 records out
10485760 bytes (10 MB) copied, 343.958 s, 30.5 kB/s
Test out of quota
CMD: trevis-38vm4 lctl set_param -n os[cd]*.*MDT*.force_sync=1
CMD: trevis-38vm3 lctl set_param -n osd*.*OS*.force_sync=1
running as uid/gid/euid/egid 60000/60000/60000/60000, groups:
 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d61.sanity-quota/f61.sanity-quota-0] [count=40] [oflag=sync]
dd: error writing '/mnt/lustre/d61.sanity-quota/f61.sanity-quota-0': Disk quota exceeded
19+0 records in
18+0 records out
18874368 bytes (19 MB) copied, 636.705 s, 29.6 kB/s
Increase default quota
CMD: trevis-38vm4 lctl set_param -n os[cd]*.*MDT*.force_sync=1
CMD: trevis-38vm3 lctl set_param -n osd*.*OS*.force_sync=1
running as uid/gid/euid/egid 60000/60000/60000/60000, groups:
 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d61.sanity-quota/f61.sanity-quota-0] [count=40] [oflag=sync]
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 124.211 s, 338 kB/s
Set quota to override default quota
CMD: trevis-38vm4 lctl set_param -n os[cd]*.*MDT*.force_sync=1
CMD: trevis-38vm3 lctl set_param -n osd*.*OS*.force_sync=1
running as uid/gid/euid/egid 60000/60000/60000/60000, groups:
 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d61.sanity-quota/f61.sanity-quota-0] [count=40] [oflag=sync]

VVVVVVV DO NOT REMOVE LINES BELOW, Added by Maloo for auto-association VVVVVVV
sanity-quota test_61 - Timeout occurred after 1369 mins, last suite running was sanity-quota, restarting cluster to continue tests

Issue Links

• is related to: LU-11678 sanity-quota test 1 fails with 'user ...

Comments

[Peter Jones]

Hongchao

Can you please investigate?

Thanks

Peter

[Hongchao Zhang]

this issue is caused by the wrong over-quota flag sent back to OSC, which cause the IO to be synchronized for each page.
1, the block size of the file is 1024K(1M) (dnode->dn_datablksz), the qunit is also 1024K(1M)
2, if the client write some data within the existing blocks
for example, the file has 2 blocks (2M) but its data only 1M+4K, and the client write 4K ~ 1023K to OST and the file
will have still 2 blocks (2M) but its data will also be 2M. in this case, the write will reserve 1M quota space and the QSD
acquire 1M (the qunit) quota space from QMT, but it will trigger the over-quota flag in qsd_op_begin0

    static int qsd_op_begin0(const struct lu_env *env, struct qsd_qtype_info *qqi,
                         struct lquota_id_info *qid, long long space,
                         int *flags)
    {
        ...
        if (flags != NULL) {
    out_flags:
                LASSERT(qid->lqi_is_blk);
                if (rc != 0) {
                        *flags |= lquota_over_fl(qqi->qqi_qtype);
                } else {
                        __u64   usage;

                        lqe_read_lock(lqe);
                        usage  = lqe->lqe_usage;
                        usage += lqe->lqe_pending_write;
                        usage += lqe->lqe_waiting_write;
                        usage += qqi->qqi_qsd->qsd_sync_threshold;

                        qtype_flag = lquota_over_fl(qqi->qqi_qtype);
                        /* if we should notify client to start sync write */
                        if (usage >= lqe->lqe_granted - lqe->lqe_pending_rel)
                                *flags |= qtype_flag;
                        else
                                *flags &= ~qtype_flag;
                        lqe_read_unlock(lqe);
                }
        }
    ...
    

After this, the following write at OSC will be synchronized for each page, which in turn cause the over-quota flag
to be marked at OST, which cause the write performance to degrade drastically.

[Gerrit Updater]

Hongchao Zhang ([e-mail]) uploaded a new patch: https://review.whamcloud.com/33238
Subject: LU-11390 quota: mark over-quota flag correctly
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: bfb4285b53f6ab465d379277a6d553b5cbc52236

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/33238/
Subject: LU-11390 quota: mark over-quota flag correctly
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 1ada2c947b894f97ac4e7e9c6435a8b4e42ca1fa

[Peter Jones]

Landed for 2.12