[LU-19018] fallocate by-passes quota limits initially, subsequent fallocate Fails correctly with EDQUOT

• Type: Bug
• Resolution: Unresolved
• Priority: Blocker
• Affects Version/s: Lustre 2.16.1
• Environment: Lustre 2.15 / EL7 or Lustre 2.16 / EL9
• Severity: 2
• Assignee: Arshad Hussain
• Reporter: Stephane Thiell

Description

Hello! I just realized that posix_fallocate() allows users to bypass Lustre quota enforcement. Tested against Lustre 2.15.5 and 2.16.1.

I'm attaching a simple fallocate helper fallocate.c that I used below.

We assume quota enforcement is enabled here (osd-ldiskfs.lustre-OST*.quota_slave_dt.enabled=up):

User quota enforcement:

$ lfs setquota -u sthiell -B 1T /elm

$ ./fallocate bigfalloc-user $(( 2 * 1024 * 1024 * 1024 * 1024 )) 

$ stat bigfalloc-user
  File: bigfalloc-user
  Size: 2199023255552	Blocks: 4294968080 IO Block: 4194304 regular file
Device: 899af214h/2308633108d	Inode: 144115441093058211  Links: 1
Access: (0644/-rw-r--r--)  Uid: (282232/ sthiell)   Gid: (  100/   users)
Access: 2025-05-14 15:39:29.000000000 -0700
Modify: 2025-05-14 15:39:29.000000000 -0700
Change: 2025-05-14 15:39:29.000000000 -0700
 Birth: 2025-05-14 15:39:29.000000000 -0700

$ lfs quota -u sthiell -h  /elm/
Disk quotas for usr sthiell (uid 282232):
      Filesystem    used   quota   limit   grace   files   quota   limit   grace
           /elm/      2T*     0k      1T       -       5       0       0       -

On another system, I was able to bypass project quotas in the same way.

$ ./fallocate /scratch/users/sthiell/bigfalloc $(( 10 * 1024 * 1024 * 1024 * 1024 ))
Successfully allocated 10995116277760 bytes for file /scratch/users/sthiell/bigfalloc


$ lfs quota -p 282232 -h /scratch/
Disk quotas for prj 282232 (pid 282232):
     Filesystem    used   quota   limit   grace   files   quota   limit   grace
      /scratch/    112T*     0k    100T       -    4849       0 104857600       -
pid 282232 is using default block quota setting

Originally discovered with dtar which makes use of posix_fallocate() to preallocate the resulting tar file. It seems pretty important to fix this ASAP.

Comments

[Sergey Cheremencev]

The problem here is that fallocate doesn't take into account quota overflow flags at the client side. It only fails if you allocate new parts of the file one by one: fallocate -f10M f1, fallocate -f10M -i10M f1, fallocate -f 10M -i 20M f1 ... Besides that, it is still possible to fallocate new file when the client is overquota:

[root@vm3 tests]# ./runas -u quota_usr fallocate -l30M /mnt/lustre/f1
running as uid/gid/euid/egid 1000/1000/1000/1000, groups: 1000
 [fallocate] [-l30M] [/mnt/lustre/f1]
[root@vm3 tests]# lfs quota -u quota_usr /mnt/lustre
Disk quotas for usr quota_usr (uid 1000):
      Filesystem  kbytes   bquota  blimit  bgrace   files   iquota  ilimit  igrace
     /mnt/lustre   30720        0   51200       -       1        0       0       -
[root@vm3 tests]# ./runas -u quota_usr fallocate -l30M /mnt/lustre/f2
running as uid/gid/euid/egid 1000/1000/1000/1000, groups: 1000
 [fallocate] [-l30M] [/mnt/lustre/f2]
[root@vm3 tests]# lfs quota -u quota_usr /mnt/lustre
Disk quotas for usr quota_usr (uid 1000):
      Filesystem  kbytes   bquota  blimit  bgrace   files   iquota  ilimit  igrace
     /mnt/lustre   61440*       0   51200       -       2*       0       0       -
[root@vm3 tests]# ./runas -u quota_usr fallocate -l30M /mnt/lustre/f3
running as uid/gid/euid/egid 1000/1000/1000/1000, groups: 1000
 [fallocate] [-l30M] [/mnt/lustre/f3]
[root@vm3 tests]# lfs quota -u quota_usr /mnt/lustre
Disk quotas for usr quota_usr (uid 1000):
      Filesystem  kbytes   bquota  blimit  bgrace   files   iquota  ilimit  igrace
     /mnt/lustre   92160*       0   51200       -       3*       0       0       -
[root@vm3 tests]# ./runas -u quota_usr fallocate -l30M /mnt/lustre/f4
running as uid/gid/euid/egid 1000/1000/1000/1000, groups: 1000
 [fallocate] [-l30M] [/mnt/lustre/f4]
[root@vm3 tests]# lfs quota -u quota_usr /mnt/lustre
Disk quotas for usr quota_usr (uid 1000):
      Filesystem  kbytes   bquota  blimit  bgrace   files   iquota  ilimit  igrace
     /mnt/lustre  122880*       0   51200       -       4*       0       0       -
[root@vm3 tests]# ./runas -u quota_usr dd if=/dev/urandom of=/mnt/lustre/ffff3 bs=1M count=40
running as uid/gid/euid/egid 1000/1000/1000/1000, groups: 1000
 [dd] [if=/dev/urandom] [of=/mnt/lustre/ffff3] [bs=1M] [count=40]
dd: error writing '/mnt/lustre/ffff3': Disk quota exceeded
5+0 records in
4+0 records out
4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.218391 s, 19.2 MB/s
[root@vm3 tests]# ./runas -u quota_usr fallocate -l30M /mnt/lustre/f55
running as uid/gid/euid/egid 1000/1000/1000/1000, groups: 1000
 [fallocate] [-l30M] [/mnt/lustre/f55]
[root@vm3 tests]# lfs quota -u quota_usr /mnt/lustre
Disk quotas for usr quota_usr (uid 1000):
      Filesystem  kbytes   bquota  blimit  bgrace   files   iquota  ilimit  igrace
     /mnt/lustre  157696*       0   51200       -       6*       0       0       - 

That said, 2 things should fixed here:

1. It shouldn't be possible to fallocate a new file when the user(group or project) hits it's quota limits. The inode at the MDS is still admitted to be created at the MDS side if the QID doesn't hit inode limits. At the same time fallocate at the OSTs should fail with -EDQUOT.
2. fallocate should take into account remaining quotas. I.e. if the QID still has quotas and not hitting any limits, fallocate should not be able to allocate the space above the limit.

Im not sure how this would be implemented. The 1st issue could be fixed pretty easy - something like osc_quota_chkdq should be called to check that we are not over quota yet. Another approach that can be probably used with the 1st one is to return -EDQUOT from the server side.

The 2nd problem looks a bit complicated to fix. Probably we should retrieve quotas(like we do in a regular lfs quota) in fallocate handler to estimate does the requested fallocate size fit the remaining quota. Or we can check that at the server side and return -EDQUOT if the OST cannot supply requested space size.

I haven't looked at the code, so there could be better solutions.

[Arshad Hussain]

If I understand correctly, only the first fallocate beyond the quota limit will succeed, but further calls to fallocate will not. I also tried to write to the fallocated file afterwards and got EDQUOT. So it might not be as bad as I originally thought! Still the initial fallocate allows users to get past their quota limits, at least logically.

Thanks for your report. Yes, I also understand this is the current behaviour. I also think this a known limitation in Lustre (looking at how sanity-quota/1h test-case is done). Please note that if fallocate is called divided in smaller incremental several parts it honores the EDQUOT as demonestrated by lfallocate.sh (attached). I have also updated the title to reflect the bug clearly.  Please feel free to correct the title.

[Stephane Thiell]

Thanks Arshad and Sergey for looking into this! If I understand correctly, only the first fallocate beyond the quota limit will succeed, but further calls to fallocate will not. I also tried to write to the fallocated file afterwards and got EDQUOT. So it might not be as bad as I originally thought! Still the initial fallocate allows users to get past their quota limits, at least logically.

I wanted to note that this is also impacting the DDN ExaScaler stack, it is easily reproduced with the standard fallocate command (-x or not actually). This is on our NVIDIA SuperPod:

sthiell@login-01:/scratch/m000001/sthiell/tmp$ lfs setstripe -c 10 teststripe
sthiell@login-01:/scratch/m000001/sthiell/tmp$ fallocate -x -l 50T teststripe 

sthiell@login-01:/scratch/m000001/sthiell/tmp$ stat teststripe 
  File: teststripe
  Size: 54975581388800	Blocks: 107374201760 IO Block: 4194304 regular file
Device: 2015dafeh/538303230d	Inode: 360288091639906923  Links: 1
Access: (0644/-rw-r--r--)  Uid: (1695273714/ sthiell)   Gid: (851715518/marlowe-m000001)
Access: 2025-06-06 12:11:59.000000000 -0700
Modify: 2025-06-06 12:11:53.000000000 -0700
Change: 2025-06-06 12:11:53.000000000 -0700
 Birth: 2025-06-06 12:11:53.000000000 -0700

 sthiell@login-01:/scratch/m000001/sthiell/tmp$ lfs quota -h -p 851715518 /scratch/
Disk quotas for prj 851715518 (pid 851715518):
     Filesystem    used   quota   limit   grace   files   quota   limit   grace
      /scratch/  50.13T*     0k     15T       -  641726       0  831488       -

Then, a write to this file quickly generates EDQUOT, and the overall blocks allocated is somehow reduced:

sthiell@login-01:/scratch/m000001/sthiell/tmp$ dd if=/dev/zero of=teststripe bs=16M count=1000 seek=1M
dd: error writing 'teststripe': Disk quota exceeded
5+0 records in
4+0 records out
67108864 bytes (67 MB, 64 MiB) copied, 0.0529419 s, 1.3 GB/s

sthiell@login-01:/scratch/m000001/sthiell/tmp$ stat teststripe 
  File: teststripe
  Size: 17592253153280	Blocks: 34359826608 IO Block: 4194304 regular file
Device: 2015dafeh/538303230d	Inode: 360288091639907030  Links: 1
Access: (0644/-rw-r--r--)  Uid: (1695273714/ sthiell)   Gid: (851715518/marlowe-m000001)
Access: 2025-06-06 13:23:23.000000000 -0700
Modify: 2025-06-06 13:23:23.000000000 -0700
Change: 2025-06-06 13:23:23.000000000 -0700
 Birth: 2025-06-06 13:20:36.000000000 -0700

It's not great if a successful fallocated file cannot be written to afterwards. The point of using fallocate is to make sure the blocks are reserved. It would be better if the initial fallocate could fail with EDQUOT, I think. Thanks again.

[Arshad Hussain]

Hi Stephane,

My first reaction was this was quota. Which let me down different path of analysis. Quota internals is very less known to me and I was wrong. sanity-quota/1h test-case does do the checking of EDQUOT. However, it does that dividing into several parts . I could make your fallocate.c work by calling fallocate in steps. (Please see Sergey's answer/comments in this ticket). My max_dirty_mb is was set to 100M and I was testing with files less than 100M. I could
still see qsd_op_begin0() correctly hit EDQUOT. 

fallocate_mod.c (gcc -o posix_fallocate fallocate_mod.c) is your fallocate.c which is slightly modified to accept start offset instead of 0. lfallocate.sh
is adapted from sanity-quota/1h to write call fallocate_mod.c(binary) in steps.

This is what I am getting:

Quota set to 14M

$ lfs quota -u quota_usr -h /mnt/lustre/dir/
Disk quotas for usr quota_usr (uid 1002):
     Filesystem    used  bquota  blimit  bgrace   files  iquota  ilimit  igrace 
/mnt/lustre/dir/     4k      0k     14M       -       1       0       0       - 

 

Write 15M.

$ ./lfallocate.sh /mnt/lustre/dir/f1 15728640
Successfully allocated 7864320 bytes for file /mnt/lustre/dir/f1
posix_fallocate: Disk quota exceeded
./lfallocate.sh: failed for /mnt/lustre/dir/f1

 

Write 12M

./lfallocate.sh /mnt/lustre/dir/f1 12582912
Successfully allocated 6291456 bytes for file /mnt/lustre/dir/f1
Successfully allocated 6291456 bytes for file /mnt/lustre/dir/f1

Verify

lfs quota -u quota_usr -h /mnt/lustre/dir/
Disk quotas for usr quota_usr (uid 1002):
     Filesystem    used  bquota  blimit  bgrace   files  iquota  ilimit  igrace 
/mnt/lustre/dir/     12M      0k     14M       -       2       0       0       - 

[Arshad Hussain]

Thanks scherementsev.  for you inputs.

. It does it in manner of reporting, i.e. it says to QMT: the usage has been increased, but doesn't ask "can I increase the usage?"

This implies that, it appears to allocate first and then report the usage afterward?  What I understand. fallocate declares quota usage through osd_declare_inode_qid() it may not query the QMT(Master) before proceeding?

I am tracing through osd_declare_inode_qid() and it did seem to call qsd_op_begin().
osd_declare_inode_qid
->qsd_op_begin()

Small snip of call graph (just tracing lquota.ko)

# tracer: function_graph                                                        
CPU  DURATION                  FUNCTION CALLS                                 
# |     |   |                     |   |   |   |                                 
 1)   0.235 us    |  qsd_op_begin [lquota]();                                   
 1)   0.114 us    |  qsd_op_begin [lquota]();                                   
 1)   0.124 us    |  qsd_op_begin [lquota]();                                   
 1)   0.111 us    |  qsd_op_begin [lquota]();                                   
 1)   0.146 us    |  qsd_op_end [lquota]();     
...
...   

 

Note, that it is possible to hit quota limits with fallocate only when allocate the file. At the 2nd time, if you try to fallocate larger size it fails:

I can confirm I have hit this too.  This means it is querying QMT and reporting error. What conditions are met here?  Any hints.

[Sergey Cheremencev]

Hi arshad512 ,

Regarding your example with DD. I think it is a well known behaviour when the data is firstly stored in a cache on the client. If the "cache size" is big enough, it may cause to hit the limit. You could try the same but ater setting max_dirty_mb to 10M. Or just try to write the same amount of date divided to several parts: 40M + 20M + 10M - I expect you will get -EDQUOT trying to write 20M. I think this doesn't relate to fallocate problem from the description.

Im pretty busy right now and can't spend looking to the details. But at first look the problem with fallocate is that QSD doesn't acquire the quota during ofd_object_fallocate. It does it in manner of reporting, i.e. it says to QMT: the usage has been increased, but doesn't ask "can I increase the usage?" before allocating. This behaviour might have some strong reasons in that I'm not aware of. But at first look it can be fixed if required.

Note, that it is possible to hit quota limits with fallocate only when allocate the file. At the 2nd time, if you try to fallocate larger size it fails:

[root@vm1 tests]# lfs quota -u quota_usr /mnt/lustre
Disk quotas for usr quota_usr (uid 1000):
     Filesystem  kbytes   quota   limit   grace   files   quota   limit   grace
    /mnt/lustre       0       0   20480       -       0       0       0       -
[root@vm1 tests]# ./runas -u quota_usr fallocate -l100M /mnt/lustre/ff
running as uid/gid/euid/egid 1000/1000/1000/1000, groups: 1000
 [fallocate] [-l100M] [/mnt/lustre/ff]
[root@vm1 tests]# ./runas -u quota_usr fallocate -l110M /mnt/lustre/ff
running as uid/gid/euid/egid 1000/1000/1000/1000, groups: 1000
 [fallocate] [-l110M] [/mnt/lustre/ff]
fallocate: fallocate failed: Disk quota exceeded