Gu Zheng (gzheng@ddn.com) uploaded a new patch: http://review.whamcloud.com/19366
Subject: LU-7990 rpc: increase bulk size
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 17aaa0120997e1b6428c449454e45be9c11ef62f
Gu Zheng (gzheng@ddn.com) uploaded a new patch: http://review.whamcloud.com/19367
Subject: LU-7990 ofd: add a parameter to adjust preferred brw size
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: bf1ebfadacd9f77c0e46103c1c76b8da9e1cb1b0
Gu Zheng (gzheng@ddn.com) uploaded a new patch: http://review.whamcloud.com/19368
Subject: LU-7990 clio: revise readahead to support 16MB IO
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: ed70b211cbed6d74487707746f098be98787b777
Oleg Drokin (oleg.drokin@intel.com) merged in patch http://review.whamcloud.com/19366/
Subject: LU-7990 rpc: increase bulk size
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 51b32ac2b9b86a600e92ab03f387717e526153d3
I like to see DDN work to decrease a overall Lustre performance. Last merged patch replace a kmalloc allocations with vmalloc so dramatically increase vmalloc lock contention for kernels without per CPU vmalloc code.
Ouch that needs to be fixed.
I guess you are referring to the memory allocation in ptlrpc_new_bulk(). With 16M RPC support, the maximum value of nfrags is 4096, and this will lead to a memory allocation of 64KB in maximum. This is less than the maximum memory size kmalloc() can allocate in modern systems. If the memory allocation can;t be fulfilled by kmalloc(), it will then fall down to vmalloc(). I didn't see any problem with it.
Is it? What is that maximum size? I thought it was page size times two?
Hi Alexey, did you actually see a performance degression? Performance is really important to us, so please open a ticket if so. 
I do agree with Jinshan on the change of ptlrpc_new_bulk().
Oleg Drokin (oleg.drokin@intel.com) merged in patch http://review.whamcloud.com/19367/
Subject: LU-7990 ofd: add a parameter to adjust preferred brw size
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 60032c840c3764755861a61a5ffd6d2ccd5446c9
Li Xi,
we have lots problems with vmalloc on RHEL6 kernels. That kernel have a single spinlock to protect a vmalloc allocations.
So it's very high loaded spinlock on server side when we need an allocate a bulk for each BRW request.
Typical performance down is 20% in that case. Probably you run own tests on less loaded system or at system with kernels 3.8 and up, where some vmalloc improvements merged into kernel, in that case you may don't hit it problem.
Hi Alexey, thanks for sharing this information. We will run benchmarks to check this too.
From my point view, you should look to very different side if you want to increase a size of transfer.
Currently we use a single page per element in transfer, it increase overhead on each parts. Size of array, number WR / SGE requests on IB transfer, ... but two main points exist.
1) one object <= page size limitation exist only for socklnd, due bug / feauture to use an tcp_sendpage, but if we kill it and return to use kernel_sockmsg or tcp_sendpages directly we able to send more data via single call. IB lnd, GNI lnd, OmniPath .. all able to send up 2Gb per single SGE / WR entry.
2) OSC always send a data via continues region. Yes, region is continues in virtual memory, but we need anyway remap pages to DMA area when we need a prepare a SGE list, from other side it have 16 continues regions.
So from my point of view - bulk code should be changed to access an OSC view as input and have a region to SGE page conversion inside of LNet as SGE list don't have a limitation to number of segments and may better controlled by low level drivers.
It's decrease a memory usage on OST side also.
You make valid points Alexey. I think I like to bring up is that it has been recommend that we move from kernel_sock** to using the netlink api. I currently don't have the cycles for that but its something we should look into.
In a related question - does the large BRW RPC size also increase the initial ocd_grant request, so that the client can at least form one full-size RPC to the OST after a connect? The default is currently 2MB (i.e. 2x 1MB RPCs) but this initial grant should probably be increased to 8MB or more when the RPC size is increased.
Alexey, would you please give us speicfic IO pattern you think performance drops? we had a lot of benchmark and didn't see any perforamnce regresions so far and singinicant improved performance, instead.
The ptlrpc_new_bulk() code is changed from calling OBD_ALLOC() to use OBD_ALLOC_LARGE(), which in Lustre 2.7.55 and later is:
#define OBD_ALLOC_LARGE(ptr, size) \ do { \ OBD_ALLOC_GFP(ptr, size, GFP_NOFS | __GFP_NOWARN); \ if (ptr == NULL) \ OBD_VMALLOC(ptr, size); \ } while (0)
so it will try kmalloc() first at whatever size is requested, and only if this fails will it fall back to vmalloc() if there isn't a large-order allocation available to fulfill this request. For a fragmented 16MB RPC this will be 64KB at most, so it is likely that kmalloc() will succeed to get an order-4 allocation in most cases. Also, the size of this allocation is driven by the actual RPC size, so it won't do a larger allocation if large RPCs are not enabled. I don't see any significant problem in the current implementation.
Alexey, there is work being done on the upstream kernel to allow higher-order page allocations for IO, which would reduce the number of fragments in the IB SG list and could optimize both Lustre RPCs and IB-connected storage like SRP. It would be welcome if you would investigate your proposal to use higher-order allocations or virtual mappings to reduce the SG list and to update the socklnd code to be more efficient. Patches should probably be submitted under a different LU ticket.
It sounds like there will be significant changes from VFS level for the support of large page order allocation. Right now kernel dirties pages one by one therefore the order of page allocation is always 1.
Alexey, It's good to see this problem from a higher level. Please document your proposal in detail so that we can revisit this later after we have sufficient support from kernel.
Andreas - I'm reluctant to increase the initial grant because I would like to be conservative before 16M RPC is widely used. Without this change, the first RPC would be small in size, but after that the RPC size should be 16MB, not a big deal.
Andreas,
64k allocations failed easly after some time. If you look into several LNet OOM tickets you may see it. So any aged system including servers will switch to use vmalloc for this code.
As about SG lists - server side R/O cache kills an IB OFED cache as pool code can't find older map. I worked on other possibilities time to time. when looked to rework an o2ib LND.
Over time these larger allocations are going to fail due to the increasing memory fragmentation so the vmalloc penalty will show up.
Oleg Drokin (oleg.drokin@intel.com) merged in patch http://review.whamcloud.com/19368/
Subject: LU-7990 clio: revise readahead to support 16MB IO
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: d8467ab8a2ca15fbbd5be3429c9cf9ceb0fa78b8
Patches have landed to master for 2.9.0
Add LUDOC-356 into lustre manual tracking for Large bulk IO feature.
Andreas Dilger (andreas.dilger@intel.com) uploaded a new patch: https://review.whamcloud.com/26955
Subject: LU-7990 llite: increase whole-file readahead to RPC size
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 25f80fc94b7988e9d356de200573027bc5178a32
Oleg Drokin (oleg.drokin@intel.com) merged in patch https://review.whamcloud.com/26955/
Subject: LU-7990 llite: increase whole-file readahead to RPC size
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 627d0133d9d7bef908544537a8a2b2093719fc72
Minh Diep (minh.diep@intel.com) uploaded a new patch: https://review.whamcloud.com/29738
Subject: LU-7990 llite: increase whole-file readahead to RPC size
Project: fs/lustre-release
Branch: b2_10
Current Patch Set: 1
Commit: 9482cd464b354cd1a3edddb158d42eea2d733419
John L. Hammond (john.hammond@intel.com) merged in patch https://review.whamcloud.com/29738/
Subject: LU-7990 llite: increase whole-file readahead to RPC size
Project: fs/lustre-release
Branch: b2_10
Current Patch Set:
Commit: 5c1877c4143bddcd3c5c4f2397db5622c2116fce