[#LU-13004] Use a single style of IO buf descriptor

[ 2135.474369] LustreError: 9149:0:(client.c:264:__ptlrpc_prep_bulk_page()) ASSERTION( desc->bd_iov_count < desc->bd_max_iov ) failed:  8 vs 8
[ 2135.474726] LustreError: 9149:0:(client.c:264:__ptlrpc_prep_bulk_page()) LBUG
[ 2135.474896] Pid: 9149, comm: mdt_out00_000 3.10.0 #1 SMP Tue Jun 11 05:40:47 EDT 2019
[ 2135.474897] Call Trace:
[ 2135.474901]  [<ffffffffa00889dd>] libcfs_call_trace+0x7d/0xa0 [libcfs]
[ 2135.474911]  [<ffffffffa0088a7c>] lbug_with_loc+0x4c/0xa0 [libcfs]
[ 2135.474917]  [<ffffffffa04c9e8a>] __ptlrpc_prep_bulk_page+0x1ca/0x200 [ptlrpc]
[ 2135.474951]  [<ffffffffa04caded>] ptlrpc_prep_bulk_frag_pages+0x7d/0x130 [ptlrpc]
[ 2135.474978]  [<ffffffffa0544f67>] tgt_send_buffer+0xd7/0x200 [ptlrpc]
[ 2135.475013] LustreError: 13486:0:(lmv_obd.c:1259:lmv_statfs()) lustre-MDT0001-mdc-ffff8800b7579bf8: can't stat MDS #0: rc = -11
[ 2135.476026]  [<ffffffffa0550641>] out_read+0x751/0xa70 [ptlrpc]
[ 2135.476059]  [<ffffffffa0553d47>] out_handle+0x14f7/0x1b80 [ptlrpc]
[ 2135.476088]  [<ffffffffa054ac5a>] tgt_request_handle+0x94a/0x15f0 [ptlrpc]
[ 2135.476116]  [<ffffffffa04ee386>] ptlrpc_server_handle_request+0x256/0xb20 [ptlrpc]
[ 2135.476144]  [<ffffffffa04f2808>] ptlrpc_main+0xc48/0x1c70 [ptlrpc]
[ 2135.476170]  [<ffffffff8110aad4>] kthread+0xd4/0xe0
[ 2135.476174]  [<ffffffff81839777>] ret_from_fork_nospec_end+0x0/0x39
[ 2135.476177]  [<ffffffffffffffff>] 0xffffffffffffffff

I've added small debug in assert and it shows that bd_max_iov is set to lower value than is used later. It looks like in tgt_send_buffer the pages are calculated wrongly and is smaller value than amount of __ptlrpc_prep_bulk_page() calls in later cycle through rdbuf->rb_nbufs

I've check that once more and it seems it can be possible if offset of first chunk is not 0, though that is also not likely, because its size is 4K in case of 8 chunks as above. I am using SLUB allocator in kernel, just in case.

00000020:00000400:2.0:1581670655.078022:0:8712:0:(tgt_handler.c:1095:tgt_send_buffer()) buf[0]: offset 2024, len 4096 00000020:00000400:2.0:1581670655.078032:0:8712:0:(tgt_handler.c:1095:tgt_send_buffer()) buf[1]: offset 8, len 4096 00000020:00000400:2.0:1581670655.078033:0:8712:0:(tgt_handler.c:1095:tgt_send_buffer()) buf[2]: offset 8, len 4096 00000020:00000400:2.0:1581670655.078034:0:8712:0:(tgt_handler.c:1095:tgt_send_buffer()) buf[3]: offset 680, len 4096 00000020:00000400:2.0:1581670655.078035:0:8712:0:(tgt_handler.c:1095:tgt_send_buffer()) buf[4]: offset 1352, len 4096 00000020:00000400:2.0:1581670655.078035:0:8712:0:(tgt_handler.c:1095:tgt_send_buffer()) buf[5]: offset 8, len 4096 00000020:00000400:2.0:1581670655.078036:0:8712:0:(tgt_handler.c:1095:tgt_send_buffer()) buf[6]: offset 2024, len 4096 00000020:00000400:2.0:1581670655.078037:0:8712:0:(tgt_handler.c:1095:tgt_send_buffer()) buf[7]: offset 344, len 4096 00000020:00020000:2.0:1581670655.078038:0:8712:0:(tgt_handler.c:1096:tgt_send_buffer()) on step #4 pages: 8 vs 8

for (i = 0; i < rdbuf->rb_nbufs; i++) /* There is only one caller (out_read) and we *know* that * bufs are at most 4K, and 4K aligned, so a simple DIV_ROUND_UP * is always sufficient. */ pages += DIV_ROUND_UP(rdbuf->rb_bufs[i].lb_len, PAGE_SIZE);

[LU-13004] Use a single style of IO buf descriptor - iovec Created: 24/Nov/19 Updated: 31/Jan/21 Resolved: 27/May/20
Status:	Resolved
Project:	Lustre
Component/s:	None
Affects Version/s:	None
Fix Version/s:	Lustre 2.14.0

[LU-13004] Use a single style of IO buf descriptor - iovec Created: 24/Nov/19 Updated: 31/Jan/21 Resolved: 27/May/20