Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-4593

Lustre OSTs on one of the login nodes are evicted but never reconnect.

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Major
    • None
    • Lustre 2.4.1
    • 2
    • 12546

    Description

      AWE has reported the following problem about Lustre.
      Occasionally some of the Lustre OSTs on sprig2 (one of the login nodes) are evicted but never reconnect and they see the following error in the syslog:

      Jan 14 08:54:13 sprig2 kernel: [680476.616831] LustreError: 809:0:(cl_lock.c:1420:cl_unuse_try()) result = -108, this is unlikely!

      extract of syslog
      Jan 14 08:55:19 sprig2 kernel: [680478.577953] LustreError: 829:0:(cl_lock.c:1435:cl_unuse_locked()) } lock@ffff882d00691eb8
      Jan 14 08:55:19 sprig2 kernel: [680478.577958] LustreError: 829:0:(cl_lock.c:1435:cl_unuse_locked()) 3 0: —
      Jan 14 08:55:19 sprig2 kernel: [680478.577963] LustreError: 829:0:(cl_lock.c:1435:cl_unuse_locked()) 4 0: —
      Jan 14 08:55:19 sprig2 kernel: [680478.577972] LustreError: 829:0:(cl_lock.c:1435:cl_unuse_locked()) 5 0: lock@ffff8838357b34d8[0 5 0 0 0 00000000] R(1):[0, 18446744073709551615]@[0x100070000:0x41dd7e:0x0] {
      Jan 14 08:55:19 sprig2 kernel: [680478.577982] LustreError: 829:0:(cl_lock.c:1435:cl_unuse_locked()) lovsub@ffff882cb4efc760: [5 ffff883bbf9a88e8 P(0):[0, 1844674
      4073709551615]@[0x5000013f2:0x2872:0x0]]
      Jan 14 08:55:19 sprig2 kernel: [680478.577992] LustreError: 829:0:(cl_lock.c:1435:cl_unuse_locked()) osc@ffff883cb5239d80: ffff883c470b7b40 0x20000041001 0x54d9d8518dc4ea31

      Looks similar to issue reported on https://jira.hpdd.intel.com/browse/LU-3889.

      Attachments

        1. dmesg-20140304
          212 kB
          Rajeshwaran Ganesan
        2. lctl_ping.txt
          13 kB
          Oz Rentas
        3. LU-4593-lustre-log.1393924031.9587.gz
          234 kB
          Rajeshwaran Ganesan
        4. lustre-evict-7mar14-mds1.tar
          150 kB
          Rajeshwaran Ganesan
        5. lustre-evict-7mar-client.tar
          1.91 MB
          Rajeshwaran Ganesan
        6. lustre-log.1393924031.9592
          185 kB
          Rajeshwaran Ganesan
        7. mds1-messages
          0.9 kB
          Oz Rentas
        8. mds2-messages
          3 kB
          Oz Rentas
        9. messages-20140304
          118 kB
          Rajeshwaran Ganesan
        10. messages-mds1
          4 kB
          Oz Rentas
        11. messages-mds2
          5 kB
          Oz Rentas
        12. messages-oss1
          7 kB
          Oz Rentas
        13. messages-oss2
          7 kB
          Oz Rentas
        14. messages-sprig2
          28 kB
          Oz Rentas
        15. messages-sprig3
          40 kB
          Oz Rentas
        16. oss1-messages
          6 kB
          Oz Rentas
        17. oss2-messages
          5 kB
          Oz Rentas
        18. sprig2-lfs.txt
          2 kB
          Oz Rentas
        19. sprig2-messages-20140130.txt
          27 kB
          Oz Rentas
        20. sprig5-dmesg-20140226.txt
          20 kB
          Oz Rentas
        21. sprig5-lustre-log.1393411844.12537
          270 kB
          Oz Rentas
        22. sprig5-messages-20140226.txt
          20 kB
          Oz Rentas
        23. sprig-mds1-messages.txt
          0.7 kB
          Oz Rentas
        24. sprig-mds2-messages.txt
          0.4 kB
          Oz Rentas
        25. sprig-oss1-messages.txt
          4 kB
          Oz Rentas
        26. sprig-oss2-messages.txt
          5 kB
          Oz Rentas

        Activity

          People

            bobijam Zhenyu Xu
            orentas Oz Rentas (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            12 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: