Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-18293

sanity-quota test_1a, 1b, 1c: FAIL: user write success, but expect EDQUOT

Details

    • 3
    • 9223372036854775807

    Description

      sanity-quota test_1a, 1b, and 1c failed on 2.16.0 RC1 full-zfs-part-2 test session:
      https://testing.whamcloud.com/test_sets/0d7fdd5a-b404-4b82-bd32-aaf538cee475

      Disk quotas for grp quota_usr (gid 60000):
            Filesystem  kbytes   quota   limit   grace   files   quota   limit   grace
           /mnt/lustre   10245*      0   10240       -       1       0       0       -
        lustre-MDT0000       2       -       0       -       1       -       0       -
        lustre-OST0000       0       -       0       -       -       -       -       -
        lustre-OST0001       0       -       0       -       -       -       -       -
        lustre-OST0002       0       -       0       -       -       -       -       -
        lustre-OST0003   10244       -       0       -       -       -       -       -
        lustre-OST0004       0       -       0       -       -       -       -       -
        lustre-OST0005       0       -       0       -       -       -       -       -
        lustre-OST0006       0       -       0       -       -       -       -       -
      Total allocated inode limit: 0, total allocated block limit: 0
      Files for group (quota_usr), count=1:
        File: /mnt/lustre/d1a.sanity-quota/f1a.sanity-quota-1
        Size: 11534336  	Blocks: 20487      IO Block: 4194304 regular file
      Device: 2c54f966h/743766374d	Inode: 144117285664063501  Links: 1
      Access: (0644/-rw-r--r--)  Uid: (60000/quota_usr)   Gid: (60000/quota_usr)
      Access: 2024-10-01 10:01:26.000000000 +0000
      Modify: 2024-10-01 10:01:36.000000000 +0000
      Change: 2024-10-01 10:01:36.000000000 +0000
       Birth: 2024-10-01 10:01:26.000000000 +0000
       sanity-quota test_1a: @@@@@@ FAIL: user write success, but expect EDQUOT 
      

      Attachments

        Issue Links

          Activity

            [LU-18293] sanity-quota test_1a, 1b, 1c: FAIL: user write success, but expect EDQUOT

            "Andreas Dilger <adilger@whamcloud.com>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/58790/
            Subject: LU-18293 tests: Revert "LU-18293 test: use direct IO for"
            Project: fs/lustre-release
            Branch: master
            Current Patch Set:
            Commit: 3c98e7edd8c94f936b4ec749d1790e0819824f2a

            gerrit Gerrit Updater added a comment - "Andreas Dilger <adilger@whamcloud.com>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/58790/ Subject: LU-18293 tests: Revert " LU-18293 test: use direct IO for" Project: fs/lustre-release Branch: master Current Patch Set: Commit: 3c98e7edd8c94f936b4ec749d1790e0819824f2a

            "Alex Zhuravlev <bzzz@whamcloud.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/58790
            Subject: LU-18293 tests: Revert "LU-18293 test: use direct IO for"
            Project: fs/lustre-release
            Branch: master
            Current Patch Set: 1
            Commit: 149315a89e588d25eddcd90523a501fd594fb5c5

            gerrit Gerrit Updater added a comment - "Alex Zhuravlev <bzzz@whamcloud.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/58790 Subject: LU-18293 tests: Revert " LU-18293 test: use direct IO for" Project: fs/lustre-release Branch: master Current Patch Set: 1 Commit: 149315a89e588d25eddcd90523a501fd594fb5c5

            Note that there is a new test failure rate report that can be used to see the change in subtest failure rates over time, even if a subtest is already failing intermittently:

            https://testing.whamcloud.com/reports?test_set_script_id=61149410-4a46-11e0-a7f6-52540025f9af&sub_test_script_id=91a1b9eb-e7d9-491e-9cdb-b6ccf3a4a53a&source=fail_rate_trend#redirect

            This link is showing sanity-quota test_73 going from 0% failure rate to over 75% failure rate in the past few days. The previous failure spikes in March and January were caused by patches that were failing almost all of the subtests.

            adilger Andreas Dilger added a comment - Note that there is a new test failure rate report that can be used to see the change in subtest failure rates over time, even if a subtest is already failing intermittently: https://testing.whamcloud.com/reports?test_set_script_id=61149410-4a46-11e0-a7f6-52540025f9af&sub_test_script_id=91a1b9eb-e7d9-491e-9cdb-b6ccf3a4a53a&source=fail_rate_trend#redirect This link is showing sanity-quota test_73 going from 0% failure rate to over 75% failure rate in the past few days. The previous failure spikes in March and January were caused by patches that were failing almost all of the subtests.

            hongchao.zhang, as Alex pointed out, there are now quite a lot (155) sanity-quota test_71a failures on master:
            https://testing.whamcloud.com/search?status%5B%5D=FAIL&test_set_script_id=61149410-4a46-11e0-a7f6-52540025f9af&sub_test_script_id=91a1b9eb-e7d9-491e-9cdb-b6ccf3a4a53a&start_date=2025-04-08&end_date=2025-04-14&source=sub_tests#redirect

            These look like they mostly started on 2025-04-10 after this patch landed to master, except for 21 more failures on 2025-04-04 when the patch was included in master-next.

            adilger Andreas Dilger added a comment - hongchao.zhang , as Alex pointed out, there are now quite a lot (155) sanity-quota test_71a failures on master: https://testing.whamcloud.com/search?status%5B%5D=FAIL&test_set_script_id=61149410-4a46-11e0-a7f6-52540025f9af&sub_test_script_id=91a1b9eb-e7d9-491e-9cdb-b6ccf3a4a53a&start_date=2025-04-08&end_date=2025-04-14&source=sub_tests#redirect These look like they mostly started on 2025-04-10 after this patch landed to master, except for 21 more failures on 2025-04-04 when the patch was included in master-next.
            sebastien Sebastien Buisson added a comment - +1 on master: https://testing.whamcloud.com/test_sets/e7eca048-edf9-4715-9470-480bc8e9a551
            bzzz Alex Zhuravlev added a comment - https://testing.whamcloud.com/test_sets/231346ad-6272-4119-87fd-50507a31af7e
            # MDSCOUNT=2 ONLY=71a bash sanity-quota.sh
            ...
            == sanity-quota test 71a: Check PFL with quota pools ===== 18:44:23 (1744483463)
            Waiting for MDT destroys to complete
            Creating test directory
            fail_val=0
            fail_loc=0
            User quota (block hardlimit:100 MB)
            Creating new pool qpool1
            Pool lustre.qpool1 created
            Adding targets to pool
            OST lustre-OST0000_UUID added to pool lustre.qpool1
            OST lustre-OST0001_UUID added to pool lustre.qpool1
            Creating new pool qpool2
            Pool lustre.qpool2 created
            Adding targets to pool
            OST lustre-OST0001_UUID added to pool lustre.qpool2
            -u quota_usr  space:0
            used 0
            Write...
            running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000
             [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d71a.sanity-quota/f71a.sanity-quota-0] [count=10] [oflag=direct]
            dd: error writing '/mnt/lustre/d71a.sanity-quota/f71a.sanity-quota-0': Disk quota exceeded
            10+0 records in
            9+0 records out
            9437184 bytes (9.4 MB, 9.0 MiB) copied, 0.148691 s, 63.5 MB/s
            Disk quotas for usr quota_usr (uid 60000):
                 Filesystem  kbytes  bquota  blimit  bgrace   files  iquota  ilimit  igrace 
                /mnt/lustre    9216       0  102400       -       1       0       0       - 
             lustre-MDT0000       0       -       0       -       1       -       0       - 
             lustre-MDT0001       0       -       0       -       0       -       0       - 
             lustre-OST0000    9216       -   10240       -       1       -       0       - 
             lustre-OST0001       0       -       0       -       0       -       0       - 
            Total allocated inode limit: 0, total allocated block limit: 10240
            Files for user (quota_usr), count=1:
              File: /mnt/lustre/d71a.sanity-quota/f71a.sanity-quota-0
              Size: 9437184   	Blocks: 18432      IO Block: 4194304 regular file
            Device: 2c54f966h/743766374d	Inode: 144115272381366533  Links: 1
            Access: (0644/-rw-r--r--)  Uid: (60000/quota_usr)   Gid: (60000/quota_usr)
            Access: 2025-04-12 18:44:31.000000000 +0000
            Modify: 2025-04-12 18:44:31.000000000 +0000
            Change: 2025-04-12 18:44:31.000000000 +0000
             Birth: -
             sanity-quota test_71a: @@@@@@ FAIL: user write failure, but expect success 
            
            bzzz Alex Zhuravlev added a comment - # MDSCOUNT=2 ONLY=71a bash sanity-quota.sh ... == sanity-quota test 71a: Check PFL with quota pools ===== 18:44:23 (1744483463) Waiting for MDT destroys to complete Creating test directory fail_val=0 fail_loc=0 User quota (block hardlimit:100 MB) Creating new pool qpool1 Pool lustre.qpool1 created Adding targets to pool OST lustre-OST0000_UUID added to pool lustre.qpool1 OST lustre-OST0001_UUID added to pool lustre.qpool1 Creating new pool qpool2 Pool lustre.qpool2 created Adding targets to pool OST lustre-OST0001_UUID added to pool lustre.qpool2 -u quota_usr space:0 used 0 Write... running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [ if =/dev/zero] [bs=1M] [of=/mnt/lustre/d71a.sanity-quota/f71a.sanity-quota-0] [count=10] [oflag=direct] dd: error writing '/mnt/lustre/d71a.sanity-quota/f71a.sanity-quota-0' : Disk quota exceeded 10+0 records in 9+0 records out 9437184 bytes (9.4 MB, 9.0 MiB) copied, 0.148691 s, 63.5 MB/s Disk quotas for usr quota_usr (uid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 9216 0 102400 - 1 0 0 - lustre-MDT0000 0 - 0 - 1 - 0 - lustre-MDT0001 0 - 0 - 0 - 0 - lustre-OST0000 9216 - 10240 - 1 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 10240 Files for user (quota_usr), count=1: File: /mnt/lustre/d71a.sanity-quota/f71a.sanity-quota-0 Size: 9437184 Blocks: 18432 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115272381366533 Links: 1 Access: (0644/-rw-r--r--) Uid: (60000/quota_usr) Gid: (60000/quota_usr) Access: 2025-04-12 18:44:31.000000000 +0000 Modify: 2025-04-12 18:44:31.000000000 +0000 Change: 2025-04-12 18:44:31.000000000 +0000 Birth: - sanity-quota test_71a: @@@@@@ FAIL: user write failure, but expect success

            with this patch landed sanity-quota/71a doesn't pass locally, I did a bisection:

            COMMIT          TESTED  PASSED  FAILED          COMMIT DESCRIPTION
            2f96622a52      5       0       5       BAD     LU-18293 test: use direct IO for quota test
            1811e77562      5       5       0       GOOD    LU-16134 utils: un-deprecate 'lctl set_param -F'
            
            bzzz Alex Zhuravlev added a comment - with this patch landed sanity-quota/71a doesn't pass locally, I did a bisection: COMMIT TESTED PASSED FAILED COMMIT DESCRIPTION 2f96622a52 5 0 5 BAD LU-18293 test: use direct IO for quota test 1811e77562 5 5 0 GOOD LU-16134 utils: un-deprecate 'lctl set_param -F'

            "Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/58533/
            Subject: LU-18293 test: use direct IO for quota test
            Project: fs/lustre-release
            Branch: master
            Current Patch Set:
            Commit: 2f96622a522924c7b9e26fd2dfdf34eb6227b7a6

            gerrit Gerrit Updater added a comment - "Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/58533/ Subject: LU-18293 test: use direct IO for quota test Project: fs/lustre-release Branch: master Current Patch Set: Commit: 2f96622a522924c7b9e26fd2dfdf34eb6227b7a6

            "Hongchao Zhang <hongchao@whamcloud.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/58533
            Subject: LU-18293 test: use direct IO for quota test
            Project: fs/lustre-release
            Branch: master
            Current Patch Set: 1
            Commit: 43184902d29ed8ed053a5da38e70781398b65f19

            gerrit Gerrit Updater added a comment - "Hongchao Zhang <hongchao@whamcloud.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/58533 Subject: LU-18293 test: use direct IO for quota test Project: fs/lustre-release Branch: master Current Patch Set: 1 Commit: 43184902d29ed8ed053a5da38e70781398b65f19

            People

              hongchao.zhang Hongchao Zhang
              yujian Jian Yu
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated: