[LU-8507] DLC: Enable peer_credit management

• Status: Resolved
• Project: Lustre
• Affects Version/s: Lustre 2.9.0
• Fix Version/s: Lustre 2.9.0
• Type: Bug
• Priority: Major
• Reporter: Doug Oucharek (Inactive)
• Assignee: Doug Oucharek (Inactive)
• Resolution: Fixed
• Severity: 3

Description

The code to allow peer_credits to be set per NI was originally "left inactive" because there were concerns about peer_credits interfering with the ability for IB nodes to connect to each other when peer_credits are not the same (peer_credits controls the queue depth for IB).

With LU-3322, the values do not have to match so it is now safe to enable this code so peer_credits can be set per NI.

Comments

[Gerrit Updater]

Doug Oucharek ([e-mail]) uploaded a new patch: http://review.whamcloud.com/21948
Subject: LU-8507 lnet: Enable setting per NI peer_credits
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 54959518d88190ebccf3a1737344c6579a998842

[James A Simmons]

Will this be meant mostly for back porting since multi-rail fixes this.

[Doug Oucharek (Inactive)]

I don't think multi-rail will fix this. Customers may want to adjust the peer_credit value per NI using DLC even with multi-rail.

[James A Simmons]

With this patch I will no longer need to have a ko2ilnd modprobe config file which is awesome. So I tried your patch but I'm seeing some changes in what values are being set. With a modprobe config file with peer_credits=63 I see the following with lnetctl export > settings.conf.
This is the desired settings and what I placed in my setting.conf for lnetctl import.

BTW the command I used to setup my configure for both cases is:

modprobe lnet;lnetctl lnet configure;lnetctl import < ~/settings.conf

• net: o2ib1
  nid: 10.37.202.11@o2ib1
  status: up
  interfaces:
  0: ib0
  lnd tunables:
  peercredits_hiw: 62
  map_on_demand: 16
  concurrent_sends: 31
  fmr_pool_size: 1280
  fmr_flush_trigger: 1024
  fmr_cache: 1
  tunables:
  peer_timeout: 180
  peer_credits: 63
  peer_buffer_credits: 0
  credits: 2560
  CPT: "[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]"

Here is the default settings when I remove my modprobe conf file with lnetctl export > setttings.conf

• net: o2ib1
  nid: 10.37.202.11@o2ib1
  status: up
  interfaces:
  0: ib0
  lnd tunables:
  peercredits_hiw: 7
  map_on_demand: 16
  concurrent_sends: 16
  fmr_pool_size: 1280
  fmr_flush_trigger: 1024
  fmr_cache: 1

As you can see peercredits_hiw and concurrent_sends is set to what I want. Any ideas why?

[James A Simmons]

Dough I figured out the problem. Its a old bug so this need to be back ported to several branches. Basically the data the user passed in using
struct lnet_ioctl_config_data wasn't being used to set struct lnet_ni fields like ni->ni_peertxcredits until after lnd->lnd_startup() was called. This meant the o2iblnd driver was never seeing the ni->ni_peerXXX fields changed by the lnet_ioctl_config_data. So I moved the setting of the ni->ni_peerXXX fields with the lnet_ioctl_config_data before the call to lnd->lnd_startup(). Give it a try now.

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch http://review.whamcloud.com/21948/
Subject: LU-8507 lnet: Enable setting per NI peer_credits
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: ebac11d264b306b694684f1ee91080a5e31684d4

[Peter Jones]

Landed for 2.9