[LU-5969] Create an LGPL version of liblustreapi

• Type: Improvement
• Resolution: Unresolved
• Priority: Major
• Affects Version/s: Lustre 2.7.0
• Assignee: Hongchao Zhang
• Reporter: Frank Zago (Inactive)

Description

Cray would like to have an LGPL version of the liblustreapi
library. We believe that there is also a desire in the
community to do that move also.

A starting point would be to create that new library with the existing
code already licensed under the LGPL (such as HSM and layout), and
start from there.

We should take this opportunity to do some cleanup in the library, and
start a new one, independent of the older code. Some of the things
that we propose are:

• prefix all API functions with lustre_, as opposed to llapi_

• have a better API to work with the copytool and changelog
  readers. As these programs know which filesystem they are dealing
  with, there could be an lustre_open_fs / lustre_close_fs type of
  API, which would prevent calls to function such as get_root_path for
  every request. The HSM API already has something similar, and should
  be reused.

• every function must have a test. Currently the liblustreapi is only
  tested indirectly with program like lfs, and that's not good enough.

• every function must have a comment describing what it does, its
  parameters and return code. This is done for some but not all
  functions the existing library.

As a test of the viability for this plan, we'd like to first port the
posix tool to this new API. Currently, besides the API already in some
LGPLed source files, the following functions and their dependencies are needed:

       get_param
       get_root_path
       libcfs_ukuc_get_rfd
       libcfs_ukuc_msg_get
       libcfs_ukuc_start
       libcfs_ukuc_stop
       llapi_chomp_string
       llapi_create_volatile_idx
       llapi_error
       llapi_error_callback_set
       llapi_fd2fid
       llapi_fid2path
       llapi_file_open_pool
       llapi_get_agent_uuid
       llapi_get_mdt_index_by_fid
       llapi_msg_set_level
       llapi_open_by_fid
       llapi_parse_size
       llapi_search_fsname
       llapi_search_mounts

Cray would like to do the preliminary implementation of this new library, and would like to know if the copyright holders of the above mentioned functions agree to relicense their code under the LGPL. Otherwise they will have to be recoded.

The second phase would be to also port Robinhood to it, since the most
of the API should be there already, except for the changelog
processing.

Issue Links

• is duplicated by: LU-5712 Derive LGPL library from libcfs
• is related to: LU-6202 clean up ioctl handling
• is related to: LU-8282 missing trailing newline when getting Lustre version param
• is related to: LU-9897 Tracking of lustre library improvements.
• is related to: LU-1707 Add llapi_* function to return the OST/MDT count for a path.
• is related to: LU-3602 make utility functions availalbe to user space
• is related to: LU-6613 lib-dlc.h is under LGPLv3
• is related to: LU-6142 Enforce Linux kernel coding style in all code

Comments

[Gerrit Updater]

Andreas Dilger ([e-mail]) uploaded a new patch: https://review.whamcloud.com/26440
Subject: LU-5969 obdclass: deprecate OBD_GET_VERSION ioctl
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: e87f4290bdf794955a8ded44c5aee325970c0820

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/26047/
Subject: LU-5969 tests: allow "version" without "lustre:" prefix
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: d73cf7722417c78ad07d88189421cfc8ef3fb8a7

[Gerrit Updater]

Andreas Dilger ([e-mail]) uploaded a new patch: https://review.whamcloud.com/26047
Subject: LU-5969 tests: allow "version" without "lustre:" prefix
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 34a97c24c4326712e072fd40d7b03225a8c90e45

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/25324/
Subject: LU-5969 lustreapi: allow "version" without "lustre:"
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: d89191163324200d1f57a095faef1253c7d9fe11

[Gerrit Updater]

Andreas Dilger ([e-mail]) uploaded a new patch: https://review.whamcloud.com/25324
Subject: LU-5969 lustreapi: allow "version" without "lustre:"
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: a832cfdbdcf21e113aa42419f5e874b6f3eb29eb

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch http://review.whamcloud.com/20809/
Subject: LU-5969 procfs: restore missing newline for version param
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 78fa16a2031b0b242e4d7bd441aba0956f9353d2

[Gerrit Updater]

John L. Hammond ([e-mail]) uploaded a new patch: http://review.whamcloud.com/20809
Subject: LU-5969 procfs: restore missing newline for version param
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 5e55f75b18b9e2b9ac701e3e5930ad36ec20570f

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch http://review.whamcloud.com/16721/
Subject: LU-5969 lustreapi: replace llapi_get_version()
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 0c5fbd80f1ba60a1c6e523864d67a4d9dcf468e6

[Andreas Dilger]

Unfortunately, it seems that while there is some interest in a major refactoring of both the kernel interfaces and library, there is little motivation from that camp to actually work on such a large project. In the meantime, it doesn't make sense to leave Cray's library gathering dust when it could get us out of the situation we are in with the GPL userspace library that cannot be used by non-GPL code such as HSM copytools. What I don't want is continued inaction for months and years discussing a perfect solution when we have something usable today that solves a real problem and can always be improved in the future.

I would be in support of submitting the Cray code to the tree, and the proponents of reworking the whole library and kernel interfaces can still pursue that avenue as their schedules permit (this should also take into account the upstream kernel's request to refactor the Lustre ioctls in general). The existing liblustreapi code can be adapted to work with the newer interfaces as they arrive, so that user tools using this library continue to function across releases while the lower level interfaces are reworked, and existing applications can move over to the new API after it is well established.

[Christopher Morrone (Inactive)]

I certainly like the general approach suggested by Robert. I have not yet formed an opinion on whether Frank's library is too high level or not, because most of it currently concerns HSM and that is an area of Lustre about which I know very little.

I think there are probably lots of great side effects of separating the kernel and user space code. The build and packaging systems could potentially be cleaned up and simplified quite a bit, for instance.

[Robert Read]

It seems to me we have a good opportunity here to refactor our user space utilities out of the Lustre tree. If we define a clean kernel-user interface, then our user space tools won't be dependent on Lustre internals, and we can move all the utilities into their own repository. This will make tool development easier since a tool-only patch would not require a full build and test of the entire Lustre stack, as it does today. The faster development cycle may encourage experimentation and new tool development. It would also provide an easier entry path for new developers interested in contributing to Lustre, but intimidated with the level of effort require to contribute even small changes to the core tree. (A level of effort which is certainly understandable for kernel modules, but perhaps not as much for more fungible utilities.)

If this is not a desirable goal we, as a community, want to work towards, then I agree that it makes sense to provide both the base primitives and abstract interface in a single library in the tree, and I'd be OK with that. On the other hand, if this is desirable then perhaps this ticket should be about providing the right LGLP library so Frank's library can continue to to develop externally instead of being incorporated.