Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-21272

o2iblnd: map bulk DMA in IOVA-cacheable chunks

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Medium Medium
    • None
    • None
    • None
    • 3
    • 9223372036854775807

      Behind a translating IOMMU, every dma_map_sg() allocates one IOVA
      range for the whole scatterlist. The IOMMU layer only caches ranges
      up to dma_opt_mapping_size() (128 KiB on current kernels) per CPU.
      o2iblnd maps a whole LNet bulk (up to 1 MiB, 256 pages) in one call,
      so every bulk map and unmap goes to the rbtree allocator under the
      IOMMU domain's iova_rbtree_lock, first trying the 32-bit window. On
      RoCE clients with the IOMMU in translated mode, Lustre bulk throughput
      drops to 1/2 or 1/3 of line rate on about a third of boots. With
      iommu=pt, 0 of 21 boots degraded, versus 8 of 21 without it.

            cbordage Cyril Bordage
            cbordage Cyril Bordage
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: