[LU-7475] lnetctl import not honoring nbuffers, credits, or mincredits

• Status: Resolved
• Project: Lustre
• Affects Version/s: Lustre 2.7.0
• Fix Version/s: Lustre 2.8.0
• Type: Bug
• Priority: Minor
• Reporter: Jesse Hanley
• Assignee: Amir Shehata (Inactive)
• Resolution: Fixed

Environment

RHEL 6.6
lustre-client-2.7.1-2.6.32_504.30.3.el6.x86_64_g43f9fd9.x86_64

• Severity: 3

Description

Using lnetctl import produces different results than using lnet module parameters.

Entry in lnet.conf modprobe file:

options lnet check_routers_before_use=1 \
             router_ping_timeout=130 \
             dead_router_check_interval=300 \
             live_router_check_interval=600\
             tiny_router_buffers=8192 \
             small_router_buffers=131072 \
             large_router_buffers=4096 \
             peer_buffer_credits=64

options lnet forwarding="enabled"

Will create the following setup:

# lnetctl routing show
routing:
    - cpt[0]:
          tiny:
              npages: 0
              nbuffers: 2048
              credits: 2048
              mincredits: 2048
          small:
              npages: 1
              nbuffers: 32768
              credits: 32768
              mincredits: 32768
          large:
              npages: 256
              nbuffers: 1024
              credits: 1024
              mincredits: 1024
......

But, specifying these parameters in /etc/sysconfig/lnet.conf does not. Entry in /etc/sysconfig/lnet.conf:

routing:
    - cpt[0]:
          tiny:
              npages: 0
              nbuffers: 2048
              credits: 2048
              mincredits: 2048
          small:
              npages: 1
              nbuffers: 32768
              credits: 32768
              mincredits: 32768
          large:
              npages: 256
              nbuffers: 1024
              credits: 1024
              mincredits: 1024

Result:

# lnetctl routing show
routing:
    - cpt[0]:
          tiny:
              npages: 0
              nbuffers: 512
              credits: 512
              mincredits: 512
          small:
              npages: 1
              nbuffers: 4096
              credits: 4096
              mincredits: 4096
          large:
              npages: 256
              nbuffers: 256
              credits: 256
              mincredits: 256
......

Are these settings not supported by lnetctl, or is using the module parameters still the preferred method?

Thanks!
–
Jesse

Comments

[Amir Shehata (Inactive)]

The syntax to configure buffers from YAML is:

routing:
        enable: 1

buffers:
        tiny: 8192
        small: 131072
        large: 4096

[Andreas Dilger]

Amir, it should be possible to take the output of lnetctl show and use that as the input (i.e. the input and output should describe the same parameters in the same way).

[Jesse Hanley]

Thanks for the info Amir. I just tested it and it appears to work as expected. Does this just mean that the buffers section should be added to lnetctl export and to the man page?

[Amir Shehata (Inactive)]

I think the best way to do it is to sum up the output of the routing YAML block for each of the tiny/small/large buffers and add it in the show block as the buffers section. I think we need to do that because if you have multiple CPTs in the system, then when you show routing, each CPT will show the number of buffers assigned to it. But if you want to use the output to configure and get the exact same result, the total number of buffers should be fed back to LNet, and the only way to do that is to sum up the buffers for all the CPTs. Technically, you can do that in the LNet module, but I would rather keep these type of calculations in user space.

I'll create a patch for this.

[Gerrit Updater]

Amir Shehata ([e-mail]) uploaded a new patch: http://review.whamcloud.com/17370
Subject: LU-7475 lnet: ensure buffer config symmetry
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 5f7939976652e16bdc9eb4b5dcbe023fc75f80ea

[Jesse Hanley]

Thanks Amir.

Will this also allow us to set the lnet params for check_routers_before_use, avoid_asym_router_failure, live_router_check_interval, dead_router_check_interval, and router_ping_timeout, or will we still need to use a modprofile file for this?

Thanks again,
–
Jesse

[James A Simmons]

Currently there is no way to set those values outside of the modprobe config file

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch http://review.whamcloud.com/17370/
Subject: LU-7475 lnet: ensure buffer config symmetry
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: e48f5a556f05e8dd6f2ffc57ad7c1220c24c9be1

[Peter Jones]

Landed for .2.8