[LU-6099] DLC: YAML output must adhere to parsing rules

• Type: Bug
• Resolution: Fixed
• Priority: Critical
• Fix Version/s: Lustre 2.7.0
• Labels: HB
• Severity: 3
• Project: Dynamic LNET Configuration
• Assignee: Amir Shehata (Inactive)
• Reporter: Amir Shehata (Inactive)

Description

when CPT parameter is included in YAML output, the value should be enclosed in double quotes (").

Currently

net:
    - net: lo
      nid: 0@lo
      status: up
      tunables:
          peer_timeout: 0
          peer_credits: 0
          peer_buffer_credits: 0
          credits: 0
          CPT: [0,0]

should be

net:
    - net: lo
      nid: 0@lo
      status: up
      tunables:
          peer_timeout: 0
          peer_credits: 0
          peer_buffer_credits: 0
          credits: 0
          CPT: "[0,0]"

Issue Links

• is related to: LU-6043 DLC: lnetctl import does not configure any networks

Comments

[Frank Zago (Inactive)]

I see. Sorry, it appeared to me that CPT was a list/array (and valid YAML). If it's not, then yes, using quotes is fine. The application just has to parse the string.

[Amir Shehata (Inactive)]

That means if you have over 10 CPTs, which I guess is entirely possible on multicore systems, you'll have something like:

CPT:
  - 1
  - 2
  - 3
  - 4
  - 5
  - 6
  - 7
  - 8
  - 9
  - 10

With larger number of CPTs, the output would be fairly large. It does give you the advantage of being converted into a python list, but the disadvantage is that you'll have fairly large output. And there will be no way to express a CPU range. For example "CPT: "[0-4]"

Due to these reasons, I believe that the current solution advantages, outweigh the disadvantages.

[Frank Zago (Inactive)]

What about this?

  CPT: 
    - 0
    - 1
    - 2

[Amir Shehata (Inactive)]

There is no existing external tools that parse that output. So there is no backwards compatibility issue.

the problem as I have explained in the patch is that the yaml 3rd party library that we use does not parse input such as [0,1...]. It has to be enclosed in double quotes. This causes a problem since the output can be without the double quotes, but if you take that output and try to pipe it into lnetctl again (ex: lnetctl import < config.yaml), it'll have a parse error.

The requirement that the lnetclt export output has to be usable with lnetctl import, is higher priority than having the CPT parameter as parsable from python. Therefore the above solution was implemented.

Another possible solution is to simply have the output as:

CPT: 0,1,2

However, that would parse in python as a string as well. So, you'll end up with the same issue.

An easy way to deal with this is simply to parse out the CPT output into a list as below

YAML output:

net:
    - net: tcp
      nid: 192.168.205.158@tcp
      status: up
      interfaces:
          0: eth4
      tunables:
          peer_timeout: 180
          peer_credits: 8
          peer_buffer_credits: 0
          clnetredits: 256
      CPT: "[1,0]"

Python Interpetation:

{'net': [{'CPT': '[1,0]',
          'interfaces': {0: 'eth4'},
          'net': 'tcp',
          'nid': '192.168.205.158@tcp',
          'status': 'up',
          'tunables': {'clnetredits': 256,
                       'peer_buffer_credits': 0,
                       'peer_credits': 8,
                       'peer_timeout': 180}}]}

if we assign the above python expression to a variable x, then you can do the following
cpt_list = x['net'][0]['CPT'].replace('[', '').replace(']', '').split(',')

then you can print items in the list by:

int(cpt_list[0])

You can also create a loop
>>> for item in cpt_list:
...     print int(item)

If you have a better solution please share it.

[Christopher Morrone (Inactive)]

I agree with Frank, that seems like an odd choice to me.

[Frank Zago (Inactive)]

I'm not affected by this, but that fix looks strange. It changes CPT from being an array to being a string. Is there any external tool parsing these files too? Because they're likely to break on that new input.

Here's for instance how python decodes the test file before and after:

{'net': [{'status': 'up', 'tunables': {'peer_timeout': 0, 'credits': 0, 'CPT': [0, 0], ...

{'net': [{'status': 'up', 'tunables': {'peer_timeout': 0, 'credits': 0, 'CPT': '[0,0]', ...

[Jodi Levi (Inactive)]

Patch landed to Master.

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch http://review.whamcloud.com/13304/
Subject: LU-6099 lnet: correct YAML output
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 53cc2a1fe6befcd51826a9b42e8dff9cc33ee8a3

[James A Simmons]

Patch fixes the issue.

[Gerrit Updater]

Amir Shehata ([e-mail]) uploaded a new patch: http://review.whamcloud.com/13304
Subject: LU-6099 lnet: correct YAML output
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 71abe659e67129280c8ec58c2782dcb6b1fb86d5