It seems that if fallocate() is called with a large enough range, that the ext4_fallocate() code will restart the transaction internally. That makes sense, since the number of blocks needed for the operation may be very large.
I don't think there is any atomicity requirement for fallocate() of the blocks themselves, so long as the size (if needed) is only changed at the end when the blocks are actually allocated.
It seems like the right thing to do is extend the transaction after >fallocate() is called to avoid running out of credits at the end when the other Lustre updates are done? We could probably compute the number of credits reserved by Lustre in other declares by looking at the handle before >fallocate is called and subtracting the amount that osd_declare_fallocate() added? A bit of a hack, but avoids hard-coding some amount into this function.
Arshad, would you be able to look into this?
Shuichi, you could work around this for now by temporarily changing the client and/or OFD to return -EOPNOTSUPP for all fallocate calls.
Hi Andreas, - Yes, I will look into this.
Arshad Hussain (arshad.super@gmail.com) uploaded a new patch: https://review.whamcloud.com/39342
Subject: LU-13765 osd-ldiskfs: Increase credit for large fallocate range
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 1dad03a65179a23a3fc41beb2aac17eacfee63d3
I think the problem is in ext4_fallocate() we will start_journal and end_journal ourselves, however,Lustre start journal handle already, we have nested journal here, JBD2 logic will try to return previous start journal in this case.
Since we might have several loops to finish allocation, that means journal credits could not be enough here..
And it looks pretty hard to predict how much credits we might use for fallocation operation, for example, in fresh filesystem, we could think 128M per extent we could get, but in a most worst case, it could be 4K blocks per extent...
And if we always blindly reserve a big credits for fallocate, that could be pretty hurting for performance, osd_punch have similar problem, but it is a bit different, we could just add orphan inode and move it out of main transaction.
I haven't got some better idea for this, maybe bzzz have some better ideas for how to solve this?
Shilong, I agree we can't predict the total number of credits needed in advance. However, we can extend the journal handle after the ->fallocate() method is called if there are not enough credits left at that point. This is the same as with ->truncate().
The patch https://review.whamcloud.com/39342 is not quite correct, but is going in the right direction.
Hi Wang Shilong,
>And it looks pretty hard to predict how much credits we might use for fallocation operation, for example, in fresh filesystem, we could think 128M per extent we could get, but in a most worst case, it could be 4K blocks per extent...
Yes i can see that now. I misunderstood Andreas's direction/hints on this. I am creating local test-bed for this now.
but in a most worst case, it could be 4K blocks per extent...
how this is possible?
I think Andreas's idea to restart transaction is correct approach - this is what truncate does. plus Lustre doesn't require whole thing to be atomic.
Hi Andreas,
I noticed the bug priority moved to "critical" (I did not wanted that...
). I did commit but really caught up other stuff and could not focus. Unfortunately, I still could not set time for this until mid-next week as I see it now. I still want to finish it, can this wait until start of next week (IE, week ending 09/06/2020) ? Sorry for the delay on this.
Thanks
Arshad
Any updates and progress of patch https://review.whamcloud.com/39342 ? fallocate is enabled in fio by default. it's very annoying all OSTs are remounted as readonly when client calls fallocate().
Hi Shuichi - Apologies for the delay. Now, at least on my local testing it is withstanding 300MB with fallocate. Unfortunately I could not have any larger OST to test out. I should be pushing the patch for Andreas/Want Shilong/Alex for review in next 30 min or so. Will try to get this fix very soon.
# fio --name=write --directory=/mnt/lustre/fio --rw=write --ioengine=libaio --bs=128K --numjobs=1 --filesize=300M --filename=test2 --fallocate=1
write: (g=0): rw=write, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=1
fio-3.7
Starting 1 process
write: Laying out IO file (1 file / 300MiB)
Jobs: 1 (f=1): [W(1)][100.0%][r=0KiB/s,w=24.2MiB/s][r=0,w=193 IOPS][eta 00m:00s]
write: (groupid=0, jobs=1): err= 0: pid=24383: Wed Sep 9 15:01:59 2020
write: IOPS=248, BW=31.1MiB/s (32.6MB/s)(300MiB/9651msec)
slat (usec): min=529, max=67978, avg=4008.99, stdev=3956.89
clat (nsec): min=2114, max=66329, avg=4229.04, stdev=1932.25
lat (usec): min=531, max=67989, avg=4015.17, stdev=3957.63
clat percentiles (nsec):
| 1.00th=[ 2352], 5.00th=[ 2960], 10.00th=[ 3216], 20.00th=[ 3504],
| 30.00th=[ 3664], 40.00th=[ 3824], 50.00th=[ 3984], 60.00th=[ 4128],
| 70.00th=[ 4384], 80.00th=[ 4704], 90.00th=[ 5408], 95.00th=[ 6176],
| 99.00th=[ 7520], 99.50th=[ 8640], 99.90th=[32640], 99.95th=[33536],
| 99.99th=[66048]
bw ( KiB/s): min=19968, max=63744, per=100.00%, avg=31847.79, stdev=10501.34, samples=19
iops : min= 156, max= 498, avg=248.74, stdev=82.05, samples=19
lat (usec) : 4=51.71%, 10=47.96%, 20=0.12%, 50=0.17%, 100=0.04%
cpu : usr=0.09%, sys=14.57%, ctx=2274, majf=1, minf=31
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,2400,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1Run status group 0 (all jobs):
WRITE: bw=31.1MiB/s (32.6MB/s), 31.1MiB/s-31.1MiB/s (32.6MB/s-32.6MB/s), io=300MiB (315MB), run=9651-9651msec
#
Test 2:
# fio --name=write --directory=/mnt/lustre/fio --rw=write --ioengine=libaio --bs=128K --numjobs=3 --filesize=100M --filename=test2 --fallocate=1
write: (g=0): rw=write, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=1
...
fio-3.7
Starting 3 processes
<SNIP>
Run status group 0 (all jobs):
WRITE: bw=17.8MiB/s (18.7MB/s), 9113KiB/s-9637KiB/s (9331kB/s-9868kB/s), io=200MiB (210MB), run=10626-11237msec
#
Thanks Arshad! let me test your updated patch.
Hi Ihara,
I have updated the patch, test it locally and succeeded.
Please try it.
Regards,
Qian
Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/39342/
Subject: LU-13765 osd-ldiskfs: Extend credit correctly for fallocate
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 93f700ca241a98630fc5ff19a041e35fbdbf0385
Landed for 2.14
Arshad Hussain (arshad.super@gmail.com) uploaded a new patch: https://review.whamcloud.com/40509
Subject: LU-13765 osd-ldiskfs: Rename dt_declare_falloc to dt_declare_fallocate
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 6f1d2085d2ee8330484091ada5748eb62f7a10b5
Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/40509/
Subject: LU-13765 osd-ldiskfs: Rename dt_declare_falloc to dt_declare_fallocate
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 6a2b9ef84274e319bce9a367cb0261aff3ae636b