Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-4593

Lustre OSTs on one of the login nodes are evicted but never reconnect.

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Major
    • None
    • Lustre 2.4.1
    • 2
    • 12546

    Description

      AWE has reported the following problem about Lustre.
      Occasionally some of the Lustre OSTs on sprig2 (one of the login nodes) are evicted but never reconnect and they see the following error in the syslog:

      Jan 14 08:54:13 sprig2 kernel: [680476.616831] LustreError: 809:0:(cl_lock.c:1420:cl_unuse_try()) result = -108, this is unlikely!

      extract of syslog
      Jan 14 08:55:19 sprig2 kernel: [680478.577953] LustreError: 829:0:(cl_lock.c:1435:cl_unuse_locked()) } lock@ffff882d00691eb8
      Jan 14 08:55:19 sprig2 kernel: [680478.577958] LustreError: 829:0:(cl_lock.c:1435:cl_unuse_locked()) 3 0: —
      Jan 14 08:55:19 sprig2 kernel: [680478.577963] LustreError: 829:0:(cl_lock.c:1435:cl_unuse_locked()) 4 0: —
      Jan 14 08:55:19 sprig2 kernel: [680478.577972] LustreError: 829:0:(cl_lock.c:1435:cl_unuse_locked()) 5 0: lock@ffff8838357b34d8[0 5 0 0 0 00000000] R(1):[0, 18446744073709551615]@[0x100070000:0x41dd7e:0x0] {
      Jan 14 08:55:19 sprig2 kernel: [680478.577982] LustreError: 829:0:(cl_lock.c:1435:cl_unuse_locked()) lovsub@ffff882cb4efc760: [5 ffff883bbf9a88e8 P(0):[0, 1844674
      4073709551615]@[0x5000013f2:0x2872:0x0]]
      Jan 14 08:55:19 sprig2 kernel: [680478.577992] LustreError: 829:0:(cl_lock.c:1435:cl_unuse_locked()) osc@ffff883cb5239d80: ffff883c470b7b40 0x20000041001 0x54d9d8518dc4ea31

      Looks similar to issue reported on https://jira.hpdd.intel.com/browse/LU-3889.

      Attachments

        1. dmesg-20140304
          212 kB
        2. lctl_ping.txt
          13 kB
        3. LU-4593-lustre-log.1393924031.9587.gz
          234 kB
        4. lustre-evict-7mar14-mds1.tar
          150 kB
        5. lustre-evict-7mar-client.tar
          1.91 MB
        6. lustre-log.1393924031.9592
          185 kB
        7. mds1-messages
          0.9 kB
        8. mds2-messages
          3 kB
        9. messages-20140304
          118 kB
        10. messages-mds1
          4 kB
        11. messages-mds2
          5 kB
        12. messages-oss1
          7 kB
        13. messages-oss2
          7 kB
        14. messages-sprig2
          28 kB
        15. messages-sprig3
          40 kB
        16. oss1-messages
          6 kB
        17. oss2-messages
          5 kB
        18. sprig2-lfs.txt
          2 kB
        19. sprig2-messages-20140130.txt
          27 kB
        20. sprig5-dmesg-20140226.txt
          20 kB
        21. sprig5-lustre-log.1393411844.12537
          270 kB
        22. sprig5-messages-20140226.txt
          20 kB
        23. sprig-mds1-messages.txt
          0.7 kB
        24. sprig-mds2-messages.txt
          0.4 kB
        25. sprig-oss1-messages.txt
          4 kB
        26. sprig-oss2-messages.txt
          5 kB

        Activity

          People

            bobijam Zhenyu Xu
            orentas Oz Rentas (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            12 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: