https://testing.whamcloud.com/test_sets/fd33d7dc-5327-11e9-9720-52540065bddc
This same error made test_33 fail as well...
Same error seen in sanity-quota test_3 for 2.12.1 and 2.13.0 with logs at
https://testing.whamcloud.com/test_sets/81a82ef4-54e6-11e9-b98a-52540065bddc
https://testing.whamcloud.com/test_sets/edda232e-4a56-11e9-a256-52540065bddc
Another instance failed on test_2:
https://testing.whamcloud.com/test_sets/71c71570-71ad-11e9-bd0e-52540065bddc
This appears to be failing only on ZFS. The first similar failure that I could find was on 2018-06-12 with patch https://review.whamcloud.com/31976 "LU-11310 tests: for test.", though this is a test patch for SLES15 ldiskfs that never landed, so it is possible there was something wrong with that kernel/patch?
https://testing.whamcloud.com/test_sets/772bbf04-6dec-11e8-a522-52540065bddc
The next failure is on 2018-08-09 with patch https://review.whamcloud.com/31504 v19 "LU-4684 lmv: support accessing migrating directory", which eventually landed on 2018-10-01, which is after the next failure on 2018-08-23, so that could not have been the patch that introduced the problem.
I'd be inclined to look for a patch that landed just before 2018-08-09 (there was a big batch of landings on 2018-08-06), which has patch https://review.whamcloud.com/32827 "LU-11153 quota: initialize ver for default quota" among others, and more on 2018-08-09.
stancheff, you comment was correct, and looks to be useful! I'm going to push a patch. 
Patrick Farrell (pfarrell@whamcloud.com) uploaded a new patch: https://review.whamcloud.com/35577
Subject: LU-12100 tests: Fix typo in project quota support
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 647cb8165588d8171a0cb1d3095bc673a53f8efd
Nathaniel Clark (nclark@whamcloud.com) uploaded a new patch: https://review.whamcloud.com/35667
Subject: LU-12100 tests: Use minimum soft qunit limit
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 06f63234459f92d8ff5186252ac61457fb6b7e02
ZFS 0.8.0 failures:
https://testing.whamcloud.com/test_sets/b4d07d78-b4d5-11e9-9f36-52540065bddc
https://testing.whamcloud.com/test_sets/09cf7628-b28b-11e9-bcf0-52540065bddc
The issue is that the second dd ("Write to exceed soft limit") seems to run much slower than first dd, and exceeds the grace period.
First dd is between 65 and 200MB/s
Write up to soft limit running as uid/gid/euid/egid 60000/60000/60000/60000, groups: [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d3.sanity-quota/f3.sanity-quota-0] [count=4] 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.0603464 s, 69.5 MB/s
The second is > 0.5MB/s
Write to exceed soft limit running as uid/gid/euid/egid 60000/60000/60000/60000, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d3.sanity-quota/f3.sanity-quota-0] [bs=1K] [count=10] [seek=4096] 10+0 records in 10+0 records out 10240 bytes (10 kB) copied, 33.0001 s, 0.3 kB/s
When the third ("Write before timer goes off") attempts to write the grace timer has in fact already expired.
Disk quotas for usr quota_usr (uid 60000):
Filesystem kbytes quota limit grace files quota limit grace
/mnt/lustre 5125* 4096 0 none 1 0 0 -
Write before timer goes off running as uid/gid/euid/egid 60000/60000/60000/60000, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d3.sanity-quota/f3.sanity-quota-0] [bs=1K] [count=10] [seek=5120] dd: error writing '/mnt/lustre/d3.sanity-quota/f3.sanity-quota-0': Disk quota exceeded
Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/35667/
Subject: LU-12100 tests: Use minimum soft qunit limits
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 37e28b7e05a5b1f77fe663f9407436aea312b3b2
Landed for 2.13
Minh Diep (mdiep@whamcloud.com) uploaded a new patch: https://review.whamcloud.com/36346
Subject: LU-12100 tests: Use minimum soft qunit limits
Project: fs/lustre-release
Branch: b2_12
Current Patch Set: 1
Commit: a42726bd0bc203163644de63705dc3d148a54a0e
Still seeing this on latest master (2.13.50):
https://testing.whamcloud.com/test_sets/8fa3c85c-07e1-11ea-8e77-52540065bddc
Nathaniel Clark (nclark@whamcloud.com) uploaded a new patch: https://review.whamcloud.com/36797
Subject: LU-12100 tests: Use least qunit to set limit
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 38306a6354314cc0b59ae180f311069f8a67796e
Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/36346/
Subject: LU-12100 tests: Use minimum soft qunit limits
Project: fs/lustre-release
Branch: b2_12
Current Patch Set:
Commit: 17802fc3336b0a9546a03dd10efa496f6a7f6f42
https://testing.whamcloud.com/test_sets/efbb05c6-3b34-11ea-971c-52540065bddc
+1 on master: https://testing.whamcloud.com/test_sets/df3f8624-4f73-11ea-a90e-52540065bddc
Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/36797/
Subject: LU-12100 tests: Use least qunit to set limit
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 33e500cfb33406b8dddac46e1dfb5a3d59ff01c5
Landed for 2.14
James Nunez (jnunez@whamcloud.com) uploaded a new patch: https://review.whamcloud.com/38769
Subject: LU-12100 tests: Use least qunit to set limit
Project: fs/lustre-release
Branch: b2_12
Current Patch Set: 1
Commit: c6985624f377888ba134883b742c07e36bd3bae2
Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/38769/
Subject: LU-12100 tests: Use least qunit to set limit
Project: fs/lustre-release
Branch: b2_12
Current Patch Set:
Commit: abb6af1ab8f74df8d0aa1e728c63fe67b7b2d3e1