[LU-5969] Create an LGPL version of liblustreapi

• Type: Improvement
• Resolution: Fixed
• Priority: Major
• Affects Version/s: Lustre 2.7.0
• Assignee: Andreas Dilger
• Reporter: Frank Zago (Inactive)

Description

Cray would like to have an LGPL version of the liblustreapi
library. We believe that there is also a desire in the
community to do that move also.

A starting point would be to create that new library with the existing
code already licensed under the LGPL (such as HSM and layout), and
start from there.

We should take this opportunity to do some cleanup in the library, and
start a new one, independent of the older code. Some of the things
that we propose are:

• prefix all API functions with lustre_, as opposed to llapi_

• have a better API to work with the copytool and changelog
  readers. As these programs know which filesystem they are dealing
  with, there could be an lustre_open_fs / lustre_close_fs type of
  API, which would prevent calls to function such as get_root_path for
  every request. The HSM API already has something similar, and should
  be reused.

• every function must have a test. Currently the liblustreapi is only
  tested indirectly with program like lfs, and that's not good enough.

• every function must have a comment describing what it does, its
  parameters and return code. This is done for some but not all
  functions the existing library.

As a test of the viability for this plan, we'd like to first port the
posix tool to this new API. Currently, besides the API already in some
LGPLed source files, the following functions and their dependencies are needed:

       get_param
       get_root_path
       libcfs_ukuc_get_rfd
       libcfs_ukuc_msg_get
       libcfs_ukuc_start
       libcfs_ukuc_stop
       llapi_chomp_string
       llapi_create_volatile_idx
       llapi_error
       llapi_error_callback_set
       llapi_fd2fid
       llapi_fid2path
       llapi_file_open_pool
       llapi_get_agent_uuid
       llapi_get_mdt_index_by_fid
       llapi_msg_set_level
       llapi_open_by_fid
       llapi_parse_size
       llapi_search_fsname
       llapi_search_mounts

Cray would like to do the preliminary implementation of this new library, and would like to know if the copyright holders of the above mentioned functions agree to relicense their code under the LGPL. Otherwise they will have to be recoded.

The second phase would be to also port Robinhood to it, since the most
of the API should be there already, except for the changelog
processing.

Issue Links

• is duplicated by: LU-5712 Derive LGPL library from libcfs
• is related to: LU-6202 clean up ioctl handling
• is related to: LU-8282 missing trailing newline when getting Lustre version param
• is related to: LU-9897 Tracking of lustre library improvements.
• is related to: LU-1707 Add llapi_* function to return the OST/MDT count for a path.
• is related to: LU-3602 make utility functions availalbe to user space
• is related to: LU-6613 lib-dlc.h is under LGPLv3
• is related to: LU-6142 Enforce Linux kernel coding style in all code

Comments

[Cory Spitz]

Robert, if we are going to continue to refactor things in order to develop base primitives would it be ok to have two libraries in the Lustre tree? Or, would it be so bad to keep a single LGPL library that contains the complexity you don't want. You don't have to use the complex routines in your applications as long as the lib contains all the base primitives you want. If things are more cleanly implemented it shouldn't be a problem.

[Cory Spitz]

An argument for going with liblustre, of course, is that users can link via -llustre.

[Robert Read]

If the complexity is actually needed, then maybe, but the existing ball of mud in liblustreapi is a great example of pushing too much complexity into a library. I'd prefer for liblustreapi to stay as low-level as possible, and allow application developers build the higher level abstractions that make sense for their applications. Libraries like Frank's are still very useful, but they should be external to the Lustre tree and not be the interface Lustre provides.

[Andreas Dilger]

Robert, isn't that an argument for keeping the complexity inside liblustreapi? That avoids the need to develop such an abstraction for every user/interface to this code (e.g. Python, Perl, C applications, etc).

[Robert Read]

We're using this library in a different language (Go), and we've already built this style of abstraction layer in our target language. I'd rather implement this on top of the simpler primitives instead of adding another layer of abstraction.

[Frank Zago (Inactive)]

What are your reasons?
If you don't let the library perform some complex operations, then you'll have to do them in your applications.

[Robert Read]

I would like to see a new library, too, but I would prefer to see a simpler library with just the base primitives exposed by Lustre, and not the higher level abstract interface provided by Frank's library.

[Andreas Dilger]

The easiest way to do that would be to add it as liblustreapi2, as is done with many other libraries. I don't think it makes sense to do a new .so version, since the API (AFAIK, haven't looked at it) will be different for many functions. It should be built with proper .so versioning itself from the beginning. This way liblustreapi and liblustreapi2 can coexist in the tree for some time as needed, though they could both link in the existing LGPL code.

[Cory Spitz]

Should we plan to add Frank's LGPL liblustre in the tree alongside the GPL liblustreapi? Then, when he pushes to Gerrit you all can discuss the functions and error handling in code review?

[Christopher Morrone (Inactive)]

Cray would like to have this library eventually adopted by the community and not stay a Cray only project. At this stage it's still a proposal.

Ah, when you said that Cray's copytool is using it, I got the impression that you are using this on customers' systems. If it is still in proposal mode and not in the wild, then no harm, no foul.

If you can't get over your dislike of the Lustre/POSIX way of handling errors and make it consistent with the existing LGPL parts of the API, that could be a deal breaker. It would be a shame to let all of the good work go to waste over something like that. Not only that, right now your API seems strangely internally inconsistent in its error approach, at least judging by what the man pages say. One function works the POSIX way using errno, others don't...I don't get it.

[Frank Zago (Inactive)]

Christopher, Cray would like to have this library eventually adopted by the community and not stay a Cray only project. At this stage it's still a proposal.

James, that can't happen. When you mix GPL+LGPL code, the result is GPL, which is exactly what we don't want. Also, liblustreapi is never going to be LGPL, which is why we started this effort.