[LU-12821] Replace the posix copytool with robinhood's generic copytool

Wed, 5 Aug 2020 13:27:55 +0000

As discussed on LAD (with Andreas/Nathan), replacing the posix copytool by robinhood's generic copytool might be interesting - the robinhood isn't a good place for it to start with, and it looks like the posix copytool hasn't had much love.

Robinhood's generic copytool can be found here:

https://github.com/cea-hpc/robinhood/blob/master/src/tools/lhsmtool_cmd.c

Unfortunately it makes extensive use of glib, I'd rather not pull that for lustre tools, so we're talking about a full rewrite... I had forgotten that point last week

the most problematic might be the g_shell_parse_argv function, that could probably be replaced by wordexp(3) though; even if I'm not too keen of how overpowerful it is.

the regexp replaces for {fd} etc are simple replaces we could do manually with strstr and friends, but if we go with wordexp we we could just say the syntax changed and define some $FD $FID and $CTDATA variables for wordexp to expand by itself - as long as people quote variables properly it does a good job at expanding as one would expect.

the ini config file parsing... Could switch to yaml or something else we already have some of, I guess.

g_thread/async queue: I'd say single thread depopping events and forking as required is going to be just as efficient, and allow much simpler code... Do hsm-related actions (getting fid, ctdata etc) before fork to be safe, and slower stuff in child process (open & execve basically), and ct_fini upon receiving a sigchld (some eventfd for this, and polling on the kuc->lk_rfd and that, perhaps? would rather not do more in signal handler, even if the single thread model assumes ct_fini will be fast...)

All in all that seems to be a much bigger work than I thought it'd be last week, but it's not that much code either (<1kloc) so we might be able to make it happen somehow... I'd still say new code would be faster than trying to adapt the current posix copytool so I wouldn't go that way, but happy to take opinions here, and this LU can be used as reminder for myself

Whamcloud Community JIRA

[LU-12821] Replace the posix copytool with robinhood's generic copytool