Details

    • Technical task
    • Resolution: Unresolved
    • Critical
    • Upstream
    • Lustre 2.10.0, Upstream
    • 9223372036854775807

    Description

      The new done callback is implemented in three ways:

      1- Direct (no polling)
      2- softirq (our callback gets called by the IRQ_POLL_SOFTIRQ mechanism)
      3- WorkQueues

      It is very tempting to replace the kiblnd_scheduler() and its use of wait queues with the WorkQueue approach. However, this has two major problems:

      1- There is no way to bind the WorkQueue to a specific CPT without submitting a change to the RDMA code base. I'm not interested in doing this.
      2- It is unclear how the kernel threads for WorkQueues are created/destroyed. If not done efficiently, this will cause a performance degradation to LNet.

      So, my recommendation is to bind our current kiblnd_cq_completion() to the softirq callback (with necessary semantic changes). The main loop for the scheduler, kiblnd_scheduler(), will need to be updated to not do any polling of the cq as that will be done for us by the new callback mechanism. All of o2iblnd needs to be scanned for any cq polling and that needs to be turned off.

      Attachments

        Issue Links

          Activity

            [LU-8875] Change to new RDMA done callback mechanism
            simmonsja James A Simmons made changes -
            Assignee Original: Doug Oucharek [ dougo ] New: James A Simmons [ simmonsja ]
            ashehata Amir Shehata (Inactive) made changes -
            Remote Link New: This issue links to "Page (HPDD Community Wiki)" [ 22713 ]
            jgmitter Joseph Gmitter (Inactive) made changes -
            Fix Version/s Original: Lustre 2.11.0 [ 13091 ]
            pjones Peter Jones made changes -
            Assignee Original: Amir Shehata [ ashehata ] New: Doug Oucharek [ dougo ]
            mdiep Minh Diep made changes -
            Assignee Original: WC Triage [ wc-triage ] New: Amir Shehata [ ashehata ]
            mdiep Minh Diep made changes -
            Assignee Original: Amir Shehata [ ashehata ] New: WC Triage [ wc-triage ]
            doug Doug Oucharek (Inactive) made changes -
            Assignee Original: Doug Oucharek [ doug ] New: Amir Shehata [ ashehata ]
            simmonsja James A Simmons made changes -
            Link Original: This issue is related to LU-6215 [ LU-6215 ]
            doug Doug Oucharek (Inactive) made changes -
            Status Original: Open [ 1 ] New: In Progress [ 3 ]
            doug Doug Oucharek (Inactive) made changes -
            Status Original: In Progress [ 3 ] New: Open [ 1 ]

            People

              simmonsja James A Simmons
              doug Doug Oucharek (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated: