[LU-9680] Improve the user land to kernel space interface for lustre

• Status: In Progress
• Project: Lustre
• Affects Version/s: Lustre 2.11.0
• Fix Version/s: Upstream
• Type: Improvement
• Priority: Major
• Reporter: James A Simmons
• Assignee: James A Simmons
• Resolution: Unresolved

Description

Lustre currently has a complex assortment of ioctls for both the file system component and the LNet layer. Many of the those ioctls are no longer used or can be replaced with accessing files in the sysfs or debugfs file systems. With the initial review of lustre in the upstream kernel two topics on how ioctls are managed to brought to our attention. One is the dislike of the ioctl redirection that is done in lustre. The seconds is the request that we use the netlink userland API, in particular with the LNet layer. The netlink API could also be used for lustre.

Issue Links

• is related to: LU-6202 clean up ioctl handling
• is related to: LU-8834 Don't hide new functionality behind i...
• is related to: LU-9667 LNet Kernel/Userspace Interface
• is related to: LU-7659 Replace KUC by more standard mechanisms
• is related to: LU-10968 add coordinator bypass upcalls for HS...
• is related to: LU-16715 Resolve genl_lock might_sleep bug
• is related to: LU-14391 Large network routes
• is related to: LU-11850 Relocating /proc/fs/lustre/ost to /sy...
• is related to: LU-8915 Don't use linux list structure as use...
• is related to: LU-10003 lnetctl error "cannot add network: in...
• is related to: LU-16460 fix lnetctl command line option bugs
• is related to: LU-12198 lnetctl peer show hangs for ~2600 cli...
• is related to: LU-14732 struct hsm_user_request extent fields...
• is related to: LU-16241 Kernel 3.10 Build Fail on Master (3ff...
• is related to: LU-16346 lctl stack smashing on aarch64 platform
• is related to: LU-16462 conf-sanity sles12.5 test_43a: lctl: ...
• is related to: LU-16552 LNet: crash on deleting an NI using l...
• is related to: LU-12511 Prepare lustre for adoption into the ...
• is related to: LU-14680 remove strict dependency between user...

Comments

[Amir Shehata (Inactive)]

James, we already started working on a project to access module parameters and stats from sysfs. So we should coordinate that effort so we don't over write each other. I attached a requirements document I wrote that outlines what we plan to do for sysfs. Please take a look at it and provide feedback. My opinion is to use that as a starting point to plan out what needs to be done. thoughts?

[James A Simmons]

The first candidate for netlink usage has shown up from ticket LU-7659. First it would be good to go over why I'm pushing Lustre to move toward netlink to fill in people like Yohn. The major reason is the data size limitation with ioctls. Some of the data reported by lustre overflows the size limitation such as stats. In the past the way around this was using proc which by using kernel seq_files allowing virtual unlimited size. With the move away from proc we end up in debugfs which only root is suppose to access. Second benefit is for I/O forwarding system which tend not to like ioctls.

So the areas I was hoping to tackle with netlink are as follows:

1) LNet seftest (LU-8915) which currently pass in kernel link list which is bad in many ways.

2) lustre stats like md_stats so non-root users can collect stats.

3) Replace the generic lustre ioctl interface we have. (LU-6202)

4) Use it to replace the HSM communication infrastructure. (LU-7659)

---------------------------------------------------------------------------------------------------------

In the linux kernel their is only a limited number of core netlink services available. 32 to be exact and 22 have been taken. So lustre would need 2 since LNet is its own separate entity. It would be really hard press to get 2. What the kernel has done to get around this limitation is create a 'generic' netlink service. I would suggest we move to the interface. This way we don't need one netlink server in obdclass as well. 

[James A Simmons]

Adding LU-7659 since the first implementation of netlink is being done.

[Gerrit Updater]

James Simmons ([e-mail]) uploaded a new patch: https://review.whamcloud.com/34230
Subject: LU-9680 utils: add netlink infrastructure to liblustreapi
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 30b2f9ee687f0521ac09278e56cf36b0e94b1a27

[James A Simmons]

Discussing with Amir and Olaf Weber the decision was to present all network collected data in YAML format.  With this approach we can actually piece together complex trees. One of things to come out of this is listing obd devices. With YAML output I have currently:

devices:

• index: 0
    status: UP
    type: mgc
    name: MGC10.37.248.114@o2ib1
    uuid: 1a74da83-ca50-4
    refcount: 4

Would anyone want a different format? For me I think replacing index as the first field with the actual device name.

[Olaf Weber]

Including the device name makes sense to me. Whether index as a separate field is worth retaining at that point is a different question. It depends on what consumers of this data will want to use it for.

[James A Simmons]

That is the beauty of netlink. You can transmit only what you want. This leads me to the idea of creating templates. I was planning on using this concept for stats and you gave me the idea that the template the user provide could take the ordering literally. So the ordering of the template fields that the user asked for can be used to create an YAML output tree in the same exact order. The other idea with this template approach is to also added filters so only under certain conditions will the data set be returned. For example we have the ioctl OBD_IOC_NAME2DEV. Here the filter would be the obd device name and the template would only request the obd index. 

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/34230/
Subject: LU-9680 utils: add netlink infrastructure
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 3c39dac19aaf7f3f4fdee104ce6da92dd1962776

[James A Simmons]

Their is another patch outstanding.

[Steve Crusan]

> Would anyone want a different format? For me I think replacing index as the first field with the actual device name.
 
This will be amazing!

One comment I do have, is that with with YAML (this is a problem with JSON too), is that it would be great if the actual key names were fixed, i.e. 'name', 'index', 'target', etc. Just nothing dynamic!

good

devices:
  - name: osc.testfs2-OST0000
    status: UP
    uuid: dog 
  - name: osc.testfs2-OST0001
    status: UP
    uuid: cat 

bad: 

devices:
  - osc.testfs2-OST0000:
    status: UP
    uuid: dog 
  - osc.testfs2-OST0001:
    status: UP
    uuid: cat  

bad2:

devices:
  osc.testfs2-OST0000:
    status: UP
    uuid: dog 
  osc.testfs2-OST0001:
    status: UP
    uuid: cat  

it makes it much, much easier to unmarshal this type of data with fixed keys, i.e: https://zhwt.github.io/yaml-to-go/

I've had to deal with this with REST APIs before that had dynamic keys (w/ JSON), and it's sadness. 

[Gerrit Updater]

James Simmons ([e-mail]) uploaded a new patch: https://review.whamcloud.com/44358
Subject: LU-9680 net: Netlink improvements
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 16d33295fd1480cdd37384abb6e4a3635b5b47d9

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/44358/
Subject: LU-9680 net: Netlink improvements
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 82835a1952dcb37e8aac8c9d0666016557ea58e7

[James A Simmons]

More work is left.

[Gerrit Updater]

"James Simmons <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/47802
Subject: LU-9680 utils: fix Netlink / YAML library handling
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 565c30a5878b96a471d88b4cb04f74b6f2944bf2

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/47802/
Subject: LU-9680 utils: fix Netlink / YAML library handling
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 70149f4ea89aa9f5c002b63f56d0c98afcafbad9

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/31618/
Subject: LU-9680 obdclass: user netlink to collect devices information
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 86ba46c24430f67bbe0d2768f67ed0c894b2ece1

[James A Simmons]

much more coming

[Alex Zhuravlev]

sanity/104d fails constantly on a local setup:

== sanity test 104d: runas -u 500 -g 500 lctl dl test ==== 03:25:28 (1665631528)
running as uid/gid/euid/egid 500/500/500/500, groups:
 [/mnt/build/lustre/tests/../utils/lctl] [dl]
running as uid/gid/euid/egid 500/500/500/500, groups:
 [/mnt/build/lustre/tests/../utils/lctl] [dl]
 sanity test_104d: @@@@@@ FAIL: lctl dl reports wrong number of OST devices 

possible fix:

@@ -11990,11 +11990,12 @@ test_104d() {
        [ "$($RUNAS $LCTL dl | wc -l)" -ge 3 ] ||
                error "lctl dl doesn't work for non root"
 
-       ost_count="$($RUNAS $LCTL dl | grep $FSNAME-OST* | wc -l)"
+       ost_count="$($RUNAS $LCTL dl | grep "obdfilter $FSNAME-OST*" | wc -l)"
+       $RUNAS $LCTL dl | grep obdfilter*$FSNAME-OST*
        [ "$ost_count" -eq $OSTCOUNT ]  ||
                error "lctl dl reports wrong number of OST devices"
 
-       mdt_count="$($RUNAS $LCTL dl | grep $FSNAME-MDT* | wc -l)"
+       mdt_count="$($RUNAS $LCTL dl | grep "mdt $FSNAME-MDT*" | wc -l)"
        [ "$mdt_count" -eq $MDSCOUNT ]  ||
                error "lctl dl reports wrong number of MDT devices"
 }

[Gerrit Updater]

"Jian Yu <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/49091
Subject: LU-9680 utils: fix Netlink / YAML library handling
Project: fs/lustre-release
Branch: b2_15
Current Patch Set: 1
Commit: 87672c666a8b4990043ec1751dd9eed74ff58e57

[Gerrit Updater]

"James Simmons <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/49361
Subject: LU-9680 lnet: replace IOC_LIBCS_PING_PEER with Netlink
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: cf4b2a1589ea5cd12c8086d631c92981558ba059

[Gerrit Updater]

"James Simmons <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/49491
Subject: LU-9680 utils: new llapi_param_display_value().
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: afef4c07a724d32015fc54187032d7af6bb4ca0a

[Gerrit Updater]

"James Simmons <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/49516
Subject: LU-9680 lnet: collect data about peer_ni by using Netlink
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 4ccc999280b9e1221b715e8b8f76d358d6325e4f

[Gerrit Updater]

"Jian Yu <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/49775
Subject: LU-9680 obdclass: user netlink to collect devices information
Project: fs/lustre-release
Branch: b2_15
Current Patch Set: 1
Commit: 946036ace0e81e92b85a6e138bcad4c3bea0c84e

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/49491/
Subject: LU-9680 utils: new llapi_param_display_value().
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: ba0d5ffc1c0eaa0ac0b59cacc992928e80a15bf2

[Alex Zhuravlev]

with LU-9680 utils: new llapi_param_display_value() landed I'm hitting number of failures, especially in sanity-quota. and it seems the patch breaks lctl dl's output:

 31 UP mdc lustre-MDT0000-mdc-ffff8d0a1a124000 43e60cbe-0f7e-4f8d-8b45-020bcdd4b297 4
 32 UP mdc lustre-MDT0001-mdc-ffff8d0a1a124000 43e60cbe-0f7e-4f8d-8b45-020bcdd4b297 4
 33 UP osc lustre
 34 UP osc lustre-OST0001-osc-ffff8d0a1a124000 43e60cbe-0f7e-4f8d-8b45-020bcdd4b297 4

notice line 33 UP ...

[Gerrit Updater]

"jsimmons <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/49839
Subject: LU-9680 lnet: collect data about routes by using Netlink
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 00bd32ea0902abeb75dcc9fc4f50a4e4d73b123b

[Gerrit Updater]

"jsimmons <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/50026
Subject: LU-9680 lnet:
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 6fb935b1c3f0cc1891dd7670cb1d758e4c2f41e5

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/49091/
Subject: LU-9680 utils: fix Netlink / YAML library handling
Project: fs/lustre-release
Branch: b2_15
Current Patch Set:
Commit: 4ebc91a7b8caea4a99529fa00f9713c9e45033be

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/50026/
Subject: LU-9680 lnet: handle multi-rail setups
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 6fab1fe4a5c5615d4d8f74832c9c5984625b8668

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/49839/
Subject: LU-9680 lnet: collect data about routes by using Netlink
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 4ccac8297d0a600275f1e04deaad5ff9e1148e4d

[Gerrit Updater]

"James Simmons <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/51508
Subject: LU-9680 contrib: share libyaml C code generator
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 3746701cc76529b5b34dd9727eb9b9ab3f7c6d8f

[Gerrit Updater]

"James Simmons <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/51715
Subject: LU-9680 utils: add updating the key table for Netlink.
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 19ae0f22817bda193adf917a8b63bc7082bdb58d

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/51508/
Subject: LU-9680 contrib: share libyaml C code generator
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: ebcfe25232387c894d8284e76f8fe6011364f25e

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/51715/
Subject: LU-9680 utils: add updating the key table for Netlink.
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: dff466c2bacd2db0d28087bba8c6250cdd5eeb8e

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/49516/
Subject: LU-9680 lnet: collect data about peer_ni by using Netlink
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: f0be00678cc8005ec122fa1982d3aa432f6e3cbd

[Gerrit Updater]

"James Simmons <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/53004
Subject: LU-9680 lnet: add NLM_F_DUMP_FILTERED support
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: c12bab0ded1af72b4de1160207206e36b9629d14

[Gerrit Updater]

"Chris Horn <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/53366
Subject: LU-9680 lnet: Empty route/peer table is not an error
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 33b298ea5b35ebbcb08cccecf3e091e9f23fb529

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/53366/
Subject: LU-9680 lnet: Empty route/peer table is not an error
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 0031ab371ddd0ffda157629fb64c101203a3b1e2

[Gerrit Updater]

"Chris Horn <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/53731
Subject: LU-9680 lnet: Convert net_fault.c to work with large NIDs
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: d51f88e49bdbf94cdb10479e53b1cdc99fe66165

[Gerrit Updater]

"Chris Horn <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/53732
Subject: LU-9680 lnet: Fault injection add/del ioctls to netlink
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 85559cb1eacc03eff5c41301ab54f2418b70b44e

[Gerrit Updater]

"Chris Horn <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/53733
Subject: LU-9680 lnet: Fault injection list ioctls to netlink
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 94b2c1948673f9c4702799301219bd86322db257

[Gerrit Updater]

"Chris Horn <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/53734
Subject: LU-9680 lnet: Convert lnetctl debug recovery to netlink
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: f6e4fecc98f25dcc57aa6b6e905ab11ff5d1d9b4

[Gerrit Updater]

"James Simmons <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/53889
Subject: LU-9680 utils: fix nested attribute handling in liblnetconfig
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: b86ab2cb34936524987fd510a6185d3ceab5a942