[LU-10647] Lustre Errors reported by MDS server under load

Thu, 8 Feb 2018 19:59:33 +0000

Hi All,
we are seeing lustre errors reported by the MDS/MGS servers when the system is under load, below are the captured logs from MDS/MGS servers..

snapshot log:

Ex. from cngb-mds-a20-1 ( Hardware, SBB)
[Thu Jan 25 17:37:30 2018] LustreError: 5800:0:(ldlm_lib.c:2376:target_queue_recovery_request()) @@@ dropping resent queued req req@ffff881fdfa11200 x1589920133725824/t0(189028088566) o101->0b81aa25-deec-835e-aae5-bcecb47fcbc7@10.53.0.42@tcp:119/0 lens 576/0 e 0 to 0 dl 1516873099 ref 2 fl Interpret:/6/ffffffff rc 0/-1
[Thu Jan 25 17:37:31 2018] LustreError: 5289:0:(ldlm_lib.c:2376:target_queue_recovery_request()) @@@ dropping resent queued req req@ffff881011138300 x1581593857853392/t0(189048295667) o35->5ed295b4-d432-14cd-5df6-5c09bc7c23d7@10.53.6.42@tcp:120/0 lens 392/0 e 0 to 0 dl 1516873100 ref 2 fl Interpret:/6/ffffffff rc 0/-1
[Thu Jan 25 17:37:31 2018] LustreError: 5289:0:(ldlm_lib.c:2376:target_queue_recovery_request()) Skipped 2 previous similar messages
[Thu Jan 25 17:37:35 2018] LustreError: 5793:0:(ldlm_lib.c:2376:target_queue_recovery_request()) @@@ dropping resent queued req req@ffff880073f5f800 x1589919936542592/t0(189046881163) o101->7bd43f2c-1419-ec9b-c622-378cadbda13a@10.53.2.213@tcp:125/0 lens 584/0 e 0 to 0 dl 1516873105 ref 2 fl Interpret:/6/ffffffff rc 0/-1
[Thu Jan 25 17:37:35 2018] LustreError: 5793:0:(ldlm_lib.c:2376:target_queue_recovery_request()) Skipped 1 previous similar message
[Thu Jan 25 17:37:40 2018] LustreError: 5276:0:(ldlm_lib.c:2376:target_queue_recovery_request()) @@@ dropping resent queued req req@ffff8810101f4800 x1590452589095504/t0(189047967098) o101->4fcea924-dd73-062e-1f6a-4fd90aa35aae@10.53.3.4@tcp:129/0 lens 608/0 e 0 to 0 dl 1516873109 ref 2 fl Interpret:/6/ffffffff rc 0/-1
[Thu Jan 25 17:37:40 2018] LustreError: 5276:0:(ldlm_lib.c:2376:target_queue_recovery_request()) Skipped 6 previous similar messages
[Thu Jan 25 17:37:42 2018] Lustre: zfsz3-MDT0000: Client 996c31e5-38a9-678c-d539-9697767f6546 (at 10.53.3.14@tcp) reconnecting, waiting for 402 clients in recovery for 13:16
[Thu Jan 25 17:37:42 2018] Lustre: Skipped 97 previous similar messages
[Thu Jan 25 17:37:48 2018] LustreError: 5584:0:(ldlm_lib.c:2376:target_queue_recovery_request()) @@@ dropping resent queued req req@ffff882013b53000 x1584144704015776/t0(189027939916) o101->f62d4978-e6d7-f79d-f8eb-ef7a7d41058e@10.53.3.60@tcp:137/0 lens 584/0 e 0 to 0 dl 1516873117 ref 2 fl Interpret:/6/ffffffff rc 0/-1
[Thu Jan 25 17:37:48 2018] LustreError: 5584:0:(ldlm_lib.c:2376:target_queue_recovery_request()) Skipped 6 previous similar messages
[Thu Jan 25 17:38:04 2018] LustreError: 5614:0:(ldlm_lib.c:2376:target_queue_recovery_request()) @@@ dropping resent queued req req@ffff88100e191500 x1589921628468096/t0(189028085455) o101->e27e0512-2434-d18a-ad66-6aeffe3928da@10.53.0.49@tcp:153/0 lens 576/0 e 0 to 0 dl 1516873133 ref 2 fl Interpret:/6/ffffffff rc 0/-1
[Thu Jan 25 17:38:04 2018] LustreError: 5614:0:(ldlm_lib.c:2376:target_queue_recovery_request()) Skipped 26 previous similar messages
[Thu Jan 25 17:38:14 2018] Lustre: zfsz3-MDT0000: Client 5d6d19f1-8bce-de13-d9f7-ca520a17fb02 (at 10.53.6.66@tcp) reconnecting, waiting for 402 clients in recovery for 12:44
[Thu Jan 25 17:38:14 2018] Lustre: Skipped 223 previous similar messages
[Thu Jan 25 17:38:17 2018] Lustre: MGS: Client eda1ebf3-9799-6b47-ce64-1d6fd3484f8f (at 10.53.128.124@tcp) reconnecting
[Thu Jan 25 17:38:18 2018] LustreError: 5617:0:(tgt_handler.c:509:tgt_filter_recovery_request()) @@@ not permitted during recovery req@ffff88100d37d700 x1589552690075280/t0(0) o601->zfsz3-MDT0000-lwp-OST0028_UUID@10.53.19.60@tcp:167/0 lens 336/0 e 0 to 0 dl 1516873147 ref 1 fl Interpret:/0/ffffffff rc 0/-1
[Thu Jan 25 17:38:36 2018] LustreError: 5885:0:(ldlm_lib.c:2376:target_queue_recovery_request()) @@@ dropping resent queued req req@ffff88200bfe1500 x1575583787563824/t0(189046963265) o101->ef255176-549b-e004-82d9-e8c67323fdd5@10.0.10.93@tcp:185/0 lens 640/0 e 0 to 0 dl 1516873165 ref 2 fl Interpret:/6/ffffffff rc 0/-1
[Thu Jan 25 17:38:36 2018] LustreError: 5885:0:(ldlm_lib.c:2376:target_queue_recovery_request()) Skipped 83 previous similar messages
[Thu Jan 25 17:39:10 2018] Lustre: MGS: Client eda1ebf3-9799-6b47-ce64-1d6fd3484f8f (at 10.53.128.124@tcp) reconnecting
[Thu Jan 25 17:39:18 2018] Lustre: zfsz3-MDT0000: Client bb7ed2ef-9e72-ae69-f1b6-d0226a1b3623 (at 10.53.20.171@tcp) reconnecting, waiting for 402 clients in recovery for 11:40
[Thu Jan 25 17:39:18 2018] Lustre: Skipped 578 previous similar messages
[Thu Jan 25 17:39:40 2018] LustreError: 5278:0:(ldlm_lib.c:2376:target_queue_recovery_request()) @@@ dropping resent queued req req@ffff8810084d9500 x1590370043828320/t0(189048105760) o101->590855c8-9819-edd0-0383-e7833c33670f@10.53.14.172@tcp:250/0 lens 608/0 e 0 to 0 dl 1516873230 ref 2 fl Interpret:/6/ffffffff rc 0/-1
[Thu Jan 25 17:39:40 2018] LustreError: 5278:0:(ldlm_lib.c:2376:target_queue_recovery_request()) Skipped 154 previous similar messages
[Thu Jan 25 17:39:53 2018] LNetError: 5209:0:(acceptor.c:412:lnet_acceptor()) Refusing connection from 10.53.2.224: insecure port 48622
[Thu Jan 25 17:39:53 2018] Lustre: zfsz3-MDT0000: Denying connection for new client 364d3384-1168-1bc1-d6d0-1aa89761591c (at 10.53.2.224@tcp), waiting for all 402 known clients (284 recovered, 100 in progress, and 0 evicted) to recover in 11:04
[Thu Jan 25 17:39:54 2018] LNetError: 5209:0:(acceptor.c:412:lnet_acceptor()) Refusing connection from 10.53.2.234: insecure port 51422
[Thu Jan 25 17:39:54 2018] LNetError: 5209:0:(acceptor.c:412:lnet_acceptor()) Skipped 10 previous similar messages
[Thu Jan 25 17:39:54 2018] Lustre: zfsz3-MDT0000: Denying connection for new client 2e3c13d6-1b09-2972-5356-053252a1cf79 (at 10.53.2.234@tcp), waiting for all 402 known clients (284 recovered, 100 in progress, and 0 evicted) to recover in 11:03
[Thu Jan 25 17:39:54 2018] Lustre: Skipped 10 previous similar messages
[Thu Jan 25 17:39:56 2018] LNetError: 5209:0:(acceptor.c:412:lnet_acceptor()) Refusing connection from 10.54.24.152: insecure port 38508
[Thu Jan 25 17:39:56 2018] LNetError: 5209:0:(acceptor.c:412:lnet_acceptor()) Skipped 4 previous similar messages
[Thu Jan 25 17:39:57 2018] Lustre: MGS: Client e2e7ab79-33eb-7f09-8d23-dc7f7aec31fb (at 10.53.128.123@tcp) reconnecting
[Thu Jan 25 17:39:58 2018] LNetError: 5209:0:(acceptor.c:412:lnet_acceptor()) Refusing connection from 10.54.24.151: insecure port 33020
[Thu Jan 25 17:39:58 2018] LNetError: 5209:0:(acceptor.c:412:lnet_acceptor()) Skipped 3 previous similar messages
[Thu Jan 25 17:40:40 2018] Lustre: MGS: Client e2e7ab79-33eb-7f09-8d23-dc7f7aec31fb (at 10.53.128.123@tcp) reconnecting
[Thu Jan 25 17:40:40 2018] Lustre: Skipped 1 previous similar message
[Thu Jan 25 17:40:48 2018] Lustre: MGS: Client eda1ebf3-9799-6b47-ce64-1d6fd3484f8f (at 10.53.128.124@tcp) reconnecting
[Thu Jan 25 17:41:08 2018] Lustre: zfsz3-MDT0000: Denying connection for new client 364d3384-1168-1bc1-d6d0-1aa89761591c (at 10.53.2.224@tcp), waiting for all 402 known clients (286 recovered, 100 in progress, and 0 evicted) to recover in 9:49
[Thu Jan 25 17:41:08 2018] Lustre: Skipped 4 previous similar messages
[Thu Jan 25 17:41:26 2018] Lustre: zfsz3-MDT0000: Client 04e14b1e-0ec5-6fa4-c20a-8197a0a209c2 (at 10.53.14.181@tcp) reconnecting, waiting for 402 clients in recovery for 9:32
[Thu Jan 25 17:41:26 2018] Lustre: Skipped 1153 previous similar messages
[Thu Jan 25 17:41:32 2018] Lustre: MGS: Client e2e7ab79-33eb-7f09-8d23-dc7f7aec31fb (at 10.53.128.123@tcp) reconnecting
[Thu Jan 25 17:41:49 2018] LustreError: 5285:0:(ldlm_lib.c:2376:target_queue_recovery_request()) @@@ dropping resent queued req req@ffff882009e86000 x1588671876880608/t0(189048016558) o101->058ad300-211e-adeb-fccf-2bef7a0cbdf3@10.53.14.197@tcp:378/0 lens 608/0 e 0 to 0 dl 1516873358 ref 2 fl Interpret:/6/ffffffff rc 0/-1
[Thu Jan 25 17:41:49 2018] LustreError: 5285:0:(ldlm_lib.c:2376:target_queue_recovery_request()) Skipped 302 previous similar messages

Thanks,

Abe

Whamcloud Community JIRA

[LU-10647] Lustre Errors reported by MDS server under load