Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-10494

clients in locked state: spirit-29 kernel: LustreError: 11-0: zfstest-OST0000-osc-ffff881010be4000: operation ldlm_enqueue to node 192.168.2.1@o2ib failed: rc = -107

    XMLWordPrintable

Details

    • Bug
    • Resolution: Duplicate
    • Critical
    • None
    • Lustre 2.11.0, Lustre 2.10.3
    • None
    • Performance Spirit Cluster.
      Servers and Clients 2.10.3_RC1
    • 3
    • 9223372036854775807

    Description

      Clients in locked state:
      OSS: spirit-aeon-1.log

      spirit-aeon-1.log:Jan 11 18:25:19 spirit-aeon-1 kernel: LustreError: 0:0:(ldlm_lockd.c:334:waiting_locks_callback()) ### lock callback timer expired after 101s: evicting client at 192.168.1.21@o2ib  ns: filter-zfstest-OST0002_UUID lock: ffff880085f45000/0x78dafdfb659c8831 lrc: 3/0,0 mode: PR/PR res: [0x2f948:0x0:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x60000400010020 nid: 192.168.1.21@o2ib remote: 0x444cce0aa8c1b51c expref: 93939 pid: 10432 timeout: 4295871382 lvb_type: 1
spirit-aeon-1.log:Jan 11 18:25:19 spirit-aeon-1 kernel: LustreError: 10447:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED   req@ffff88081d26bc00 x1589320376498320/t0(0) o104->zfstest-OST0002@192.168.1.18@o2ib:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1
spirit-aeon-1.log:Jan 11 18:25:19 spirit-aeon-1 kernel: LustreError: 10447:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 1 previous similar message
spirit-aeon-1.log:Jan 11 18:25:21 spirit-aeon-1 kernel: LustreError: 10481:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED   req@ffff880820cdbc00 x1589320376498464/t0(0) o104->zfstest-OST0002@192.168.1.21@o2ib:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1
spirit-aeon-1.log:Jan 11 18:25:21 spirit-aeon-1 kernel: LustreError: 10481:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 1 previous similar message
spirit-aeon-1.log:Jan 11 18:25:21 spirit-aeon-1 kernel: LustreError: 0:0:(ldlm_lockd.c:334:waiting_locks_callback()) ### lock callback timer expired after 101s: evicting client at 192.168.1.20@o2ib  ns: filter-zfstest-OST0000_UUID lock: ffff8808130f5800/0x78dafdfb6598887f lrc: 3/0,0 mode: PR/PR res: [0x2b7d8:0x0:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x60000400010020 nid: 192.168.1.20@o2ib remote: 0x1f6e0be3f64f847c expref: 91031 pid: 10470 timeout: 4295873298 lvb_type: 1
spirit-aeon-1.log:Jan 11 18:25:21 spirit-aeon-1 kernel: LustreError: 0:0:(ldlm_lockd.c:334:waiting_locks_callback()) Skipped 4 previous similar messages
spirit-aeon-1.log:Jan 11 18:25:32 spirit-aeon-1 kernel: LustreError: 22938:0:(ldlm_lockd.c:334:waiting_locks_callback()) ### lock callback timer expired after 101s: evicting client at 192.168.1.16@o2ib  ns: filter-zfstest-OST0000_UUID lock: ffff8807bf296c00/0x78dafdfb6597d9a2 lrc: 3/0,0 mode: PR/PR res: [0x2b6b5:0x0:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x60000400010020 nid: 192.168.1.16@o2ib remote: 0x270a88b21111f96b expref: 85199 pid: 10554 timeout: 4295884213 lvb_type: 1
spirit-aeon-1.log:Jan 11 18:25:32 spirit-aeon-1 kernel: LustreError: 22938:0:(ldlm_lockd.c:334:waiting_locks_callback()) Skipped 2 previous similar messages
spirit-aeon-1.log:Jan 11 18:25:32 spirit-aeon-1 kernel: LustreError: 10550:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED   req@ffff88081d268600 x1589320376498832/t0(0) o104->zfstest-OST0000@192.168.1.16@o2ib:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1
spirit-aeon-1.log:Jan 11 18:25:32 spirit-aeon-1 kernel: LustreError: 10550:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 4 previous similar messages
spirit-aeon-1.log:Jan 11 18:25:34 spirit-aeon-1 kernel: LustreError: 10431:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED   req@ffff88081d26b600 x1589320376499072/t0(0) o104->zfstest-OST0002@192.168.1.21@o2ib:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1
spirit-aeon-1.log:Jan 11 18:25:34 spirit-aeon-1 kernel: LustreError: 10431:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 6 previous similar messages
spirit-aeon-1.log:Jan 11 18:25:38 spirit-aeon-1 kernel: LustreError: 10550:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED   req@ffff88003516f000 x1589320376499360/t0(0) o104->zfstest-OST0000@192.168.1.16@o2ib:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1
spirit-aeon-1.log:Jan 11 18:25:38 spirit-aeon-1 kernel: LustreError: 10550:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 6 previous similar messages
spirit-aeon-1.log:Jan 11 18:25:47 spirit-aeon-1 kernel: LustreError: 10299:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED   req@ffff880820cda100 x1589320376499840/t0(0) o104->zfstest-OST0000@192.168.1.16@o2ib:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1
spirit-aeon-1.log:Jan 11 18:25:47 spirit-aeon-1 kernel: LustreError: 10299:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 6 previous similar messages
spirit-aeon-1.log:Jan 11 18:25:59 spirit-aeon-1 kernel: LustreError: 16964:0:(ldlm_lockd.c:334:waiting_locks_callback()) ### lock callback timer expired after 101s: evicting client at 192.168.1.27@o2ib  ns: filter-zfstest-OST0002_UUID lock: ffff880793e21e00/0x78dafdfb659be2d8 lrc: 3/0,0 mode: PR/PR res: [0x2f1f8:0x0:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x60000400010020 nid: 192.168.1.27@o2ib remote: 0x4d4a0c820ccb8af9 expref: 54505 pid: 10542 timeout: 4295911095 lvb_type: 1
spirit-aeon-1.log:Jan 11 18:26:03 spirit-aeon-1 kernel: LustreError: 10247:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED   req@ffff880846aaf300 x1589320376504784/t0(0) o104->zfstest-OST0002@192.168.1.27@o2ib:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1
spirit-aeon-1.log:Jan 11 18:26:03 spirit-aeon-1 kernel: LustreError: 10247:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 27 previous similar messages

      Clients:

      spirit-29.log:Jan 11 18:25:24 spirit-29 kernel: LustreError: 11-0: zfstest-OST0000-osc-ffff881010be4000: operation ldlm_enqueue to node 192.168.2.1@o2ib failed: rc = -107
spirit-29.log:Jan 11 18:25:24 spirit-29 kernel: LustreError: 167-0: zfstest-OST0000-osc-ffff881010be4000: This client was evicted by zfstest-OST0000; in progress operations using this service will fail.
spirit-29.log:Jan 11 18:25:37 spirit-29 kernel: LustreError: 3048:0:(ldlm_resource.c:1100:ldlm_resource_complain()) zfstest-OST0000-osc-ffff881010be4000: namespace resource [0x145010:0x0:0x0].0x0 (ffff881005cf2f00) refcount nonzero (1) after lock cleanup; forcing cleanup.
spirit-29.log:Jan 11 18:25:37 spirit-29 kernel: LustreError: 3048:0:(ldlm_resource.c:1682:ldlm_resource_dump()) --- Resource: [0x145010:0x0:0x0].0x0 (ffff881005cf2f00) refcount = 1
spirit-29.log:Jan 11 18:25:37 spirit-29 kernel: LustreError: 3048:0:(ldlm_resource.c:1682:ldlm_resource_dump()) --- Resource: [0x138010:0x0:0x0].0x0 (ffff881004979140) refcount = 1

      Attachments

        1. lustre_log_dump.txt
          59 kB
          Saurabh Tandan
        2. spirit-11.log
          2.34 MB
          Saurabh Tandan
        3. vmcore-dmesg.txt
          115 kB
          Saurabh Tandan

        Issue Links

          duplicates

          Bug - A problem which impairs or prevents the functions of the product. LU-9230 soft lockup on v2.9 Lustre clients (ldlm?)

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Activity

            People

              utopiabound Nathaniel Clark
              standan Saurabh Tandan (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: