[LU-14447] LustreError: 13814:0:(ldlm_lib.c:3273:target_bulk_io()) @@@ Eviction on bulk READ

Fri, 19 Feb 2021 13:49:00 +0000

Feb 18 21:18:08 oss25 crmd[13943]: notice: State transition S_IDLE -> S_POLICY_ENGINE
Feb 18 21:18:09 oss25 kernel: Lustre: ost_io: This server is not able to keep up with request traffic (cpu-bound).
Feb 18 21:18:09 oss25 kernel: Lustre: 34958:0:(service.c:1541:ptlrpc_at_check_timed()) earlyQ=19 reqQ=0 recA=122, svcEst=123, delay=6545
Feb 18 21:18:18 oss25 kernel: LustreError: 32605:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 99s: evicting client at 10.10.21.210@o2ib ns: filter-public3-OST0018_UUID lock: ffff98cd8c56d440/0xd09e6e7eca1fc51e lrc: 4/0,0 mode: PR/PR res: [0x6ac35f4:0x0:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x60000400010020 nid: 10.10.21.210@o2ib remote: 0x7557ceb9931fe797 expref: 336 pid: 34891 timeout: 11157804 lvb_type: 1
Feb 18 21:18:18 oss25 kernel: LustreError: 32605:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 3 previous similar messages
Feb 18 21:18:19 oss25 kernel: LustreError: 32605:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 10.10.21.210@o2ib ns: filter-public3-OST0014_UUID lock: ffff98dc01de6240/0xd09e6e7eca1ee16d lrc: 4/0,0 mode: PR/PR res: [0x6a2cc22:0x0:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x60000400010020 nid: 10.10.21.210@o2ib remote: 0x7557ceb9931fa63d expref: 399 pid: 35938 timeout: 11157805 lvb_type: 1
Feb 18 21:18:19 oss25 kernel: LustreError: 32605:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 7 previous similar messages
Feb 18 21:18:19 oss25 kernel: LustreError: 35885:0:(ldlm_lib.c:3273:target_bulk_io()) @@@ Eviction on bulk READ req@ffff98dd5260c050 x1689836064090368/t0(0) o3->35aa455e-9908-3b48-c2aa-0fed07b5d743@10.10.21.210@o2ib:306/0 lens 504/440 e 2 to 0 dl 1613654256 ref 1 fl Interpret:/0/0 rc 0/0
Feb 18 21:18:19 oss25 kernel: LustreError: 35885:0:(ldlm_lib.c:3273:target_bulk_io()) Skipped 11 previous similar messages
Feb 18 21:18:19 oss25 kernel: Lustre: 13762:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (98:10s); client may timeout. req@ffff98e8f6acd050 x1678524515830400/t0(0) o4->051b5f5c-84c1-dce0-650e-bc6ad2cee64c@10.10.30.89@o2ib:339/0 lens 488/0 e 0 to 0 dl 1613654289 ref 1 fl Interpret:/0/ffffffff rc 0/-1
Feb 18 21:18:19 oss25 kernel: Lustre: 13762:0:(service.c:2165:ptlrpc_server_handle_request()) Skipped 35 previous similar messages
Feb 18 21:18:19 oss25 kernel: LustreError: 13814:0:(ldlm_lib.c:3273:target_bulk_io()) @@@ Eviction on bulk READ req@ffff98e4038f2850 x1689836064107456/t0(0) o3->35aa455e-9908-3b48-c2aa-0fed07b5d743@10.10.21.210@o2ib:264/0 lens 488/440 e 0 to 0 dl 1613654214 ref 1 fl Interpret:/0/0 rc 0/0
Feb 18 21:18:20 oss25 kernel: LustreError: 32605:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 99s: evicting client at 10.10.21.210@o2ib ns: filter-public3-OST0010_UUID lock: ffff98dbfa50cd80/0xd09e6e7eca1f13cb lrc: 4/0,0 mode: PR/PR res: [0x69442c6:0x0:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x60000400010020 nid: 10.10.21.210@o2ib remote: 0x7557ceb9931fa92a expref: 430 pid: 11902 timeout: 11157806 lvb_type: 1
Feb 18 21:18:20 oss25 kernel: LustreError: 32605:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 9 previous similar messages
Feb 18 21:18:23 oss25 kernel: LNetError: 40010:0:(o2iblnd_cb.c:3308:kiblnd_cm_callback()) 10.10.51.91@o2ib DISCONNECTED
Feb 18 21:18:23 oss25 kernel: LNetError: 40010:0:(o2iblnd_cb.c:3308:kiblnd_cm_callback()) Skipped 4 previous similar messages
Feb 18 21:18:23 oss25 kernel: LustreError: 34982:0:(ldlm_lib.c:3269:target_bulk_io()) @@@ timeout on bulk WRITE after 100+1602496494s req@ffff98cd85d1b050 x1691273802424576/t0(0) o4->63eb8e43-2df7-40db-ac89-7cec4059f1dc@10.10.24.165@o2ib:306/0 lens 488/448 e 2 to 0 dl 1613654256 ref 1 fl Interpret:/0/0 rc 0/0
Feb 18 21:18:24 oss25 kernel: LustreError: 29813:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff98cd87afec00
Feb 18 21:18:24 oss25 kernel: LustreError: 29815:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff98de2a576400
Feb 18 21:18:24 oss25 kernel: LustreError: 29814:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff98de2a576400
Feb 18 21:18:24 oss25 kernel: LustreError: 29813:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff98de2a576400
Feb 18 21:18:24 oss25 kernel: LustreError: 29812:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff98e7f6046800

Whamcloud Community JIRA

[LU-14447] LustreError: 13814:0:(ldlm_lib.c:3273:target_bulk_io()) @@@ Eviction on bulk READ