[LU-11093] clients hang when over quota

Related — Mon, 13 Aug 2018 13:50:07 +0000

Hi,

clients nodes hang and loop forever when codes go over group quota. Lustre is very verbose when this happens. the below is pretty typical. john50 is a client. arkles are OSS's.

Jun 23 13:22:42 john50 kernel: LNetError: 895:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.32@o2ib44, match 1603955949771616 length 104
8576 too big: 1048208 left, 1048208 allowed
Jun 23 13:22:42 arkle2 kernel: LustreError: 297785:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff8815b4044a00
Jun 23 13:22:42 arkle2 kernel: LustreError: 297785:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff8815b4044a00
Jun 23 13:22:42 arkle2 kernel: LustreError: 272882:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE  req@ffff8816bf52cc50 x1603955949771616/t0(0) o4->fe53a66a-b8c6-e1de-1353-a3b91bd42058@192.168.44.150@o2ib44:548/0 lens 608/448 e 0 to 0 dl 1529724168 ref 1 fl Interpret:/0/0 rc 0/0
Jun 23 13:22:42 arkle2 kernel: LustreError: 272882:0:(ldlm_lib.c:3253:target_bulk_io()) Skipped 73 previous similar messages
Jun 23 13:22:42 arkle2 kernel: Lustre: dagg-OST0002: Bulk IO write error with fe53a66a-b8c6-e1de-1353-a3b91bd42058 (at 192.168.44.150@o2ib44), client will retry: rc = -110
Jun 23 13:22:42 arkle2 kernel: Lustre: Skipped 73 previous similar messages
Jun 23 13:22:49 john50 kernel: Lustre: 906:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1529724162/real 1529724162]  req@ffff8817b7ab4b00 x1603955949771616/t0(0) o4->dagg-OST0002-osc-ffff882fcafb0800@192.168.44.32@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1529724169 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Jun 23 13:22:49 john50 kernel: Lustre: 906:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
Jun 23 13:22:49 john50 kernel: Lustre: dagg-OST0002-osc-ffff882fcafb0800: Connection to dagg-OST0002 (at 192.168.44.32@o2ib44) was lost; in progress operations using this service will wait for recovery to complete
Jun 23 13:22:49 arkle2 kernel: Lustre: dagg-OST0002: Client fe53a66a-b8c6-e1de-1353-a3b91bd42058 (at 192.168.44.150@o2ib44) reconnecting
Jun 23 13:22:49 arkle2 kernel: Lustre: Skipped 72 previous similar messages
Jun 23 13:22:49 arkle2 kernel: Lustre: dagg-OST0002: Connection restored to  (at 192.168.44.150@o2ib44)
Jun 23 13:22:49 arkle2 kernel: Lustre: Skipped 59 previous similar messages
Jun 23 13:22:49 john50 kernel: Lustre: dagg-OST0002-osc-ffff882fcafb0800: Connection restored to 192.168.44.32@o2ib44 (at 192.168.44.32@o2ib44)
Jun 23 13:22:49 john50 kernel: LNetError: 893:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.32@o2ib44, match 1603955949773296 length 1048576 too big: 1048208 left, 1048208 allowed
Jun 23 13:22:49 arkle2 kernel: LustreError: 297783:0:(events.c:449:server_bulk_callback()) event type 5, status -61, desc ffff881055ca0e00
Jun 23 13:22:49 arkle2 kernel: LustreError: 297783:0:(events.c:449:server_bulk_callback()) event type 3, status -61, desc ffff881055ca0e00
Jun 23 13:22:56 john50 kernel: Lustre: 906:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1529724169/real 1529724169]  req@ffff8817b7ab4b00 x1603955949771616/t0(0) o4->dagg-OST0002-osc-ffff882fcafb0800@192.168.44.32@o2ib44:6/4 lens 608/448 e 0 to 1 dl 1529724176 ref 2 fl Rpc:X/2/ffffffff rc 0/-1
Jun 23 13:22:56 john50 kernel: Lustre: dagg-OST0002-osc-ffff882fcafb0800: Connection to dagg-OST0002 (at 192.168.44.32@o2ib44) was lost; in progress operations using this service will wait for recovery to complete
Jun 23 13:22:56 john50 kernel: Lustre: dagg-OST0002-osc-ffff882fcafb0800: Connection restored to 192.168.44.32@o2ib44 (at 192.168.44.32@o2ib44)
Jun 23 13:22:56 john50 kernel: LNetError: 895:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-192.168.44.32@o2ib44, match 1603955949773424 length 1048576 too big: 1048208 left, 1048208 allowed

the messages don't go away when the code exits. the node stays (somewhat) broken afterwards. only rebooting the client seems to fix it.

reports from the users seem to indicate that it's not 100% repeatable, but is reasonably close.

our OSTs are pretty plain and simple z3 12+3's with 4 of those making up one OST pool. 2M recordsize and compression on.

[arkle2]root: zfs get all | grep /OST | egrep 'compression|record'
arkle2-dagg-OST2-pool/OST2  recordsize            2M                                         local
arkle2-dagg-OST2-pool/OST2  compression           lz4                                        inherited from arkle2-dagg-OST2-pool
arkle2-dagg-OST3-pool/OST3  recordsize            2M                                         local
arkle2-dagg-OST3-pool/OST3  compression           lz4                                        inherited from arkle2-dagg-OST3-pool

cheers,
robin

Whamcloud Community JIRA

[LU-11093] clients hang when over quota