[LU-6246] Add a JSON decoder

• Status: Resolved
• Project: Lustre
• Affects Version/s: Lustre 2.8.0
• Type: New Feature
• Priority: Minor
• Reporter: Frank Zago (Inactive)
• Assignee: WC Triage
• Resolution: Won't Do

Description

For the HSM migration enhancement (LU-6081), it would be nice to be able to transfer data (stripe informaton and mdt_index) between lfs and the copytool in an exchange format, namely JSON.

The current liblustreapi_json.c can only encode JSON. Use the CCAN (Comprehensive C Archive Network) JSON which it relatively small, can encode/decode, and has a decent testsuite.

Replace liblustreapi_json.c with the new implementation.

Issue Links

• is blocking: LU-6081 hsm: add file migrate support
• is related to: LU-9897 Tracking of lustre library improvements.

Comments

[Gerrit Updater]

frank zago ([e-mail]) uploaded a new patch: http://review.whamcloud.com/13770
Subject: LU-6246 utils: add CCAN JSON parser / emitter from
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: fb7008342fa71cc919a1c743f304fa053fdaac65

[Gerrit Updater]

frank zago ([e-mail]) uploaded a new patch: http://review.whamcloud.com/13771
Subject: LU-6246 tests: add CCAN Test Framework from CCAN
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 0db83eecf2101326b1e7c7ea23a13f8e3f94646f

[Gerrit Updater]

frank zago ([e-mail]) uploaded a new patch: http://review.whamcloud.com/13772
Subject: LU-6246 tests: integrate CCAN JSON
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: a46eae5c5712bfd6be1071f999a42a1464a59638

[Gerrit Updater]

frank zago ([e-mail]) uploaded a new patch: http://review.whamcloud.com/13773
Subject: LU-6246 tests: split JSON_NUMBER into int and float
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 13a952f777981a83de7b0dc1068a9c48d813004b

[Gerrit Updater]

frank zago ([e-mail]) uploaded a new patch: http://review.whamcloud.com/13774
Subject: LU-6246 user: replace liblustreapi JSON
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 910e31fe91844a4308e839736c07798c3ff8f15d

[Frank Zago (Inactive)]

The licenses used by CCAN JSON are:

• Test anything protocol - tap.c: FreeBSD license (aka 2 clauses BSD)
• Test anything protocol - compiler.h dependency: CC0 (Creative Commons 0)
• JSON: MIT

All 3 licenses are GPLv2 and v3 compatible.

[James A Simmons]

Hi Frank. With LUG coming up I like to discuss on developer's day about using JSON in replacement of YAML. Especially since json-c seems to be the standard now.

[Frank Zago (Inactive)]

That one: https://github.com/json-c/json-c ?

[James A Simmons]

Yes.

[Frank Zago (Inactive)]

I did a quick test to compare the CCAN JSON and the JSON-C (0.11-3 as provided by Ubuntu 12.04).

The test decodes a 1290 bytes strings 1000000 times in a loop.

CCAN JSON does it in 8.58s and JSON-C does it in 20.70s, so it's about 2.4 times slower.

On the other hand, that's still fast enough for the usage being done here, and JSON-C is present in CentOS 6 & 7 and Debian/Ubuntu. The API looks also similar.

So I wouldn't mind dropping these patches if JSON-C has better chance of being used. If JSON-C turns out to be a bottleneck, we can replace it.

[Frank Zago (Inactive)]

Hi James,
Was there any discussion/decision on that subject at LUG?

[James A Simmons]

Once LU-9324 lands liblustreapi.so will be supporting YAML. Since this is the case I don't think we need to look at adding a JSON decoder. Instead we need to look at using libYAML to handle JSON. Should we close this ticket or look to changing the direction?

[James A Simmons]

I see the latest LU-6081 work has moved from using JSON to YAML. Since this is the case should we close this ticket and abandon the patches related to this work?

[Nikitas Angelinas]

Hi James. Yes, I think so as well, unfortunately. This would probably have been useful to land, but as you had suggested, since libyaml is now a dependency of Lustre it makes sense to use that instead.

[James A Simmons]

Will use YAML instead.