[LU-1431] Support for larger than 1MB sequential I/O RPCs

• Type: New Feature
• Resolution: Fixed
• Priority: Major
• Fix Version/s: Lustre 2.4.0
• Affects Version/s: Lustre 2.4.0
• Labels: performance
• Bugzilla ID: 16,900

Epic

lnet

• Assignee: Peter Jones
• Reporter: James A Simmons

Description

Currently Lustre maximum buffer size for a RPC sending I/O is 1MB. This work looks to change the amount of data transfer to allow the data sent to be a size to achieve peak performance with large I/O transfers to the back end disk. Also an additional benefit is the reduction in the round trip time to send the data.

Issue Links

• is related to: LU-4533 rpc_stats histogram does not support max_rpcs_in_flight greater than 31
• is related to: LU-3308 large readdir chunk size slows unlink/"rm -r" performance
• is related to: LU-2791 Stuck client on server OOM/lost message
• is related to: LU-2816 sanity-benchmark test_bonnie slow after 4MB BRW RPC patch
• is related to: LU-2598 obdfilter-survey LBUG ASSERTION( iobuf->dr_npages < iobuf->dr_max_pages ) failed
• is related to: LU-2790 Failure to allocated osd keys leads to ofd_intent_policy()) ASSERTION( res_lvb != ((void *)0) ) failed
• is related to: LU-3438 replay-ost-single test_5 failed with error int check_write_rcs() "Unexpected # bytes transferred"
• is related to: LU-2748 OSD uses kmalloc with high order to allocate a keys
• is related to: LUDOC-80 4MB RPC Doc Changes
• is related to: LUDOC-80 4MB RPC Doc Changes
• is related to: LU-2424 add memory limits for ptlrpc service
• Trackbacks: Lustre Community Development in Progress Features are being developed for future Lustre releases both at Whamcloud and by other organizations in the Lustre community. These will be eligible for inclusion in future Lustre releases as per our processes

Comments

[Artem Blagodarenko (Inactive)]

Xyratex-bug-id: MRP-687
Xyratex-bug-id: MRP-319

[Peter Jones]

Thanks Artem!

[Artem Blagodarenko (Inactive)]

LU-3438 is created.

[Peter Jones]

Artem

Could you please open a new ticket for this failure so we can track it?

Thanks

Peter

[Artem Blagodarenko (Inactive)]

Our testing system shows, that there is failed test eplay-ost-single.test_5

Lustre: DEBUG MARKER: == replay-ost-single test 5: Fail OST during iozone == 21:21:13 (1369851673)
Lustre: Failing over lustre-OST0000
LustreError: 11-0: an error occurred while communicating with 0@lo. The ost_write operation failed with -19
LustreError: Skipped 1 previous similar message
Lustre: lustre-OST0000-osc-ffff8800514d3400: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: Skipped 1 previous similar message
Lustre: lustre-OST0000: shutting down for failover; client state will be preserved.
Lustre: OST lustre-OST0000 has stopped.
Lustre: server umount lustre-OST0000 complete
LustreError: 137-5: UUID 'lustre-OST0000_UUID' is not available for connect (no target)
LustreError: Skipped 1 previous similar message
LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: 
LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: 
Lustre: 16962:0:(ldlm_lib.c:2195:target_recovery_init()) RECOVERY: service lustre-OST0000, 2 recoverable clients, last_transno 1322
Lustre: lustre-OST0000: Now serving lustre-OST0000 on /dev/loop1 with recovery enabled
Lustre: 2398:0:(ldlm_lib.c:1021:target_handle_connect()) lustre-OST0000: connection from lustre-MDT0000-mdtlov_UUID@0@lo recovering/t0 exp ffff88005ca19c00 cur 1369851700 last 1369851697
Lustre: 2398:0:(ldlm_lib.c:1021:target_handle_connect()) Skipped 3 previous similar messages
Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect
Lustre: lustre-OST0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted.
Lustre: lustre-OST0000-osc-MDT0000: Connection restored to lustre-OST0000 (at 0@lo)
Lustre: Skipped 1 previous similar message
LustreError: 1716:0:(osc_request.c:1232:check_write_rcs()) Unexpected # bytes transferred: 65536 (requested 32768)
LustreError: 1716:0:(osc_request.c:1232:check_write_rcs()) Unexpected # bytes transferred: 2097152 (requested 1048576)
Lustre: lustre-OST0000: received MDS connection from 0@lo
Lustre: MDS mdd_obd-lustre-MDT0000: lustre-OST0000_UUID now active, resetting orphans
Lustre: DEBUG MARKER: iozone rc=1
Lustre: DEBUG MARKER: replay-ost-single test_5: @@@@@@ FAIL: iozone failed

This messages looks related to 4mb IO patch

LustreError: 1716:0:(osc_request.c:1232:check_write_rcs()) Unexpected # bytes transferred: 65536 (requested 32768)
LustreError: 1716:0:(osc_request.c:1232:check_write_rcs()) Unexpected # bytes transferred: 2097152 (requested 1048576)

I believe, that this test is failed in intel's master branch, but they skip it as SLOW during testing
https://maloo.whamcloud.com/test_sets/dd033a98-7264-11e2-aad1-52540035b04c

test_5	SKIP	0	0	skipping SLOW test 5

Could you, please, start this test (it marked as SLOW) and check if it failed?

[Andreas Dilger]

LUDOC-80 landed, closing bug.