[LU-8999] sanity-quota test_38: skipped id entries

• Type: Bug
• Resolution: Fixed
• Priority: Major
• Fix Version/s: Lustre 2.12.0, Lustre 2.10.7
• Affects Version/s: Lustre 2.10.0, Lustre 2.10.1, Lustre 2.11.0, Lustre 2.10.2
• Environment: RHEL 7.3 Server/Client; master, build# 3486
• Severity: 3
• Assignee: Hongchao Zhang
• Reporter: Maloo

Description

This issue was created by maloo for Saurabh Tandan <[e-mail]>

This issue relates to the following test suite run: https://testing.hpdd.intel.com/test_sets/d6ff8e38-d464-11e6-ace4-5254006e85c2.

The sub-test test_38 failed with the following error:

skipped id entries

test_logs:

== sanity-quota test 38: Quota accounting iterator doesn't skip id entries =========================== 05:03:51 (1483707831)
Waiting for local destroys to complete
Creating test directory
CMD: onyx-35vm3,onyx-35vm4 lctl set_param fail_val=0 fail_loc=0
fail_val=0
fail_loc=0
fail_val=0
fail_loc=0
Create 10000 files...
CMD: onyx-35vm3 lctl set_param -n osd*.*MDT*.force_sync=1
CMD: onyx-35vm4 lctl set_param -n osd*.*OS*.force_sync=1
CMD: onyx-35vm3 /usr/sbin/lctl get_param osd-ldiskfs.lustre-MDT0000.quota_slave.acct_user
Found 10010 id entries
 sanity-quota test_38: @@@@@@ FAIL: skipped id entries

Comments

[Hongchao Zhang]

I have managed to reproduce this issue in my local VMs, and there are two problems to cause it.

1. the problem caused by multiple calls of the "lprocfs_quota_seq_start" of the seq_operations
During the reading of the sequence file, "seq_read" will be called multiple times to read the content,
and start/stop operation will be called in each call of "seq_read", but the start operation will cause
the following problem,
Assume the last stop operation save the current read position at 2000 (qid), the saved block and index is
[1, 2, 3, 100, 200] and [0, 0, 0, 0, 20, 50] (the level of the quota file is 4), then in the next start operation,
"osd_it_acct_load" is called to resume the iteration, but it will change the block and index to
[1, 2, 3, 150, 200] and [0, 0, 0, 30, 50], that is because the last block storing the quota information will be used
sequentially (will look for the first empty slot when a new quota needs to be inserted).

2. there is a problem in "walk_tree_dqentry"

int walk_tree_dqentry(const struct lu_env *env, struct osd_object *obj,
                      int type, uint blk, int depth, uint index,
                      struct osd_it_quota *it)
{       
        dqbuf_t  buf = getdqbuf();
        loff_t   ret;
        u32     *ref = (u32 *) buf;
        
        ENTRY;

        if (!buf)
                RETURN(-ENOMEM);
        ret = quota_read_blk(env, obj, type, blk, buf);
        if (ret < 0) {
                CERROR("Can't read quota tree block %u.\n", blk);
                goto out_buf;
        }
        ret = 1;
                
        for (; index <= 0xff && ret > 0; index++) {
                blk = le32_to_cpu(ref[index]);
                if (!blk)       /* No reference */
                        continue;

                if (depth < LUSTRE_DQTREEDEPTH - 1)
                        ret = walk_tree_dqentry(env, obj, type, blk,
                                                depth + 1, 0, it);
                else
                        ret = walk_block_dqentry(env, obj, type, blk, 0, it);   <--- here, if ret == 0, the "index++" in the "for" clause
                                                                                                          <--- will still be called and cause one entry is skipped.
        }
        it->oiq_blk[depth + 1] = blk;
        it->oiq_index[depth] = index;

out_buf:
        freedqbuf(buf);
        RETURN(ret);
}

[Gerrit Updater]

John L. Hammond ([e-mail]) merged in patch https://review.whamcloud.com/30896/
Subject: LU-8999 test: ignore unrelated quota id
Project: fs/lustre-release
Branch: b2_10
Current Patch Set:
Commit: b5073eb0234e02892d22c5dead4a3a8a3da065bd

[Gerrit Updater]

Hongchao Zhang ([e-mail]) uploaded a new patch: https://review.whamcloud.com/31098
Subject: LU-8999 test: debug patch
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 30f1ac0972d068350cb1cd10146f596bf14814df

[Saurabh Tandan (Inactive)]

Reopening the issue as it still occurs for 2.10.57 "SLES 12 SP3 Server/DNE/ldiskfs SLES 12 SP3 Client" config.
https://testing.hpdd.intel.com/test_sets/acaba252-fd4e-11e7-a7cd-52540065bddc
Looks like it didn't found enough id entries

[Gerrit Updater]

Minh Diep ([e-mail]) uploaded a new patch: https://review.whamcloud.com/30896
Subject: LU-8999 test: ignore unrelated quota id
Project: fs/lustre-release
Branch: b2_10
Current Patch Set: 1
Commit: 01a8b2e0d8afa058e62cac1d24c8b520b81f9671

[Peter Jones]

Landed for 2.11

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/30730/
Subject: LU-8999 test: ignore unrelated quota id
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 8a0a9190cd0a68a88e42210e9c3dac324319fa6d

[Gerrit Updater]

Hongchao Zhang ([e-mail]) uploaded a new patch: https://review.whamcloud.com/30730
Subject: LU-8999 test: ignore unrelated quota id
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: e0d615954ba8eb1303e20296c164c30fca5e3ebe

[Hongchao Zhang]

I have checked the log, there are 13 extra quota id, which could be left by other tests, but the quota id 0 ~ 9999 required by
this test was there and no problem.

[root@zhanghc ~]#grep "id:" quota_ids |awk '{print $3}'|sort -n
0
1
...
9998
9999
10273
14421
18394
19745
20974
23221
23332
25594
27020
29911
30457
30979
31362

- id:      10273
  usage:   { inodes:                    1, kbytes:                    4 }
- id:      14421
  usage:   { inodes:                    1, kbytes:                    4 }
- id:      18394
  usage:   { inodes:                    1, kbytes:                    0 }
- id:      19745
  usage:   { inodes:                    1, kbytes:                    4 }
- id:      20974
  usage:   { inodes:                    1, kbytes:                    4 }
- id:      23221
  usage:   { inodes:                    1, kbytes:                    0 }
- id:      23332
  usage:   { inodes:                    1, kbytes:                    0 }
- id:      25594
  usage:   { inodes:                    1, kbytes:                    4 }
- id:      27020
  usage:   { inodes:                    1, kbytes:                    0 }
- id:      29911
  usage:   { inodes:                    1, kbytes:                    0 }
- id:      30457
  usage:   { inodes:                    1, kbytes:                    0 }
- id:      30979
  usage:   { inodes:                    1, kbytes:                    4 }
- id:      31362
  usage:   { inodes:                    1, kbytes:                    0 }

[Minh Diep]

doesn't look like this is fixed
https://testing.hpdd.intel.com/test_sets/8ac15c2e-f1a3-11e7-8c23-52540065bddc

[Gerrit Updater]

John L. Hammond ([e-mail]) merged in patch https://review.whamcloud.com/30631/
Subject: LU-8999 test: check chown result
Project: fs/lustre-release
Branch: b2_10
Current Patch Set:
Commit: 3a0fc1a8d65d4796a3da895006dcc5012e9e1b3f