Make cli output json

[benthecarman]

Previously we would output the debug format of each of our return times which worked, but was not very usable. You could not easily pipe results with cli tools or anything like that. This changes the results to be output to json to make it easier to use and more similiar to what people expect.

This had one annoyance because since our structs are generated using prost, we cannot just simply add the serde traits to the structs, so instead we recreate the types we are going to output in the cli and covert them. There are a couple places where I just created "empty" default versions of types because we had to handle the None case because of how proto handles structs, not totally sure if this is the best move or if we should do something like an unreachable instead

[ldk-reviews-bot]

👋 I see @tnull was un-assigned.
If you'd like another reviewer assignment, please click here.

[tnull]

Hmm, could be debated if we want to deviate from the 'protobuf-over-HTTP' approach, but I'm not sure if we should introduce all the boilerplate to support yet another encoding of the same request/response data?

[benthecarman]

to be clear, this dep was unused here, that's why i removed it, we get it from ldk-server-client.

but yeah it does suck to have a separate encoding but its pretty important to have your cli output be human and machine readable

[tnull]

to be clear, this dep was unused here, that's why i removed it, we get it from ldk-server-client.

Ah, gotcha.

but yeah it does suck to have a separate encoding but its pretty important to have your cli output be human and machine readable

Isn't machine readability mostly important if the cli is considered the main API to interact with a service? We however went out of our way to ensure we provide a 'proper' API via protobufs and even provide a ready-to-go client library to interact with it programatically. I think we def. want to discourage parsing cli stdout/stdin to interact with the service?

[benthecarman]

Isn't machine readability mostly important if the cli is considered the main API to interact with a service? We however went out of our way to ensure we provide a 'proper' API via protobufs and even provide a ready-to-go client library to interact with it programatically. I think we def. want to discourage parsing cli stdout/stdin to interact with the service?

Yeah I agree we don't want the cli to be the primary api, but I still think machine readability is important. I often pipe bitcoind and lnd cli commands though jq to more easily read stuff or to do quick computations. This is often useful as well for chaining commands together

[tnull]

Okay, but I have to say I still hate the idea of taking on the maintainance burden of almost 1000 LoC just to be able to print as valid JSON. If that's important we should really find another way to do it rather than just having an agent spit out a lot of boilerplate that nobody ever wants to read/maintain.

Turns out it's really easy to add serde/JSON support on the generated protos by just adding:

diff --git a/ldk-server-protos/Cargo.toml b/ldk-server-protos/Cargo.toml
index 5d16484..65894e2 100644
--- a/ldk-server-protos/Cargo.toml
+++ b/ldk-server-protos/Cargo.toml
@@ -7,6 +7,8 @@ build = "build.rs"

 [dependencies]
 prost = { version = "0.11.6", default-features = false, features = ["std", "prost-derive"] }
+serde = { version = "1.0", features = ["derive"] }
+bytes = { version = "1", features = ["serde"] }

 [target.'cfg(genproto)'.build-dependencies]
 prost-build = { version = "0.11.6" , default-features = false}
diff --git a/ldk-server-protos/build.rs b/ldk-server-protos/build.rs
index 76eb77a..ac5d0ce 100644
--- a/ldk-server-protos/build.rs
+++ b/ldk-server-protos/build.rs
@@ -13,6 +13,8 @@ fn main() {
 #[cfg(genproto)]
 fn generate_protos() {
        prost_build::Config::new()
+               .type_attribute(".", "#[derive(serde::Serialize, serde::Deserialize)]")
+               .type_attribute(".", "#[serde(rename_all = \"snake_case\")]")
                .bytes(&["."])
                .compile_protos(
                        &[

Also see tokio-rs/prost#75 and https://protobuf.dev/programming-guides/proto3/#json

From there we could consider hiding the additional attributes behind a serde or json feature, so that they are only used by/exposed to the cli, avoiding any confusion which serialization/deserialization you want to use in other contexts.

[benthecarman]

Okay good find, much easier

[elnosh]

maybe for separate PR but I think it would be good for some fields of certain types (Bytes) to have custom serializers. Otherwise the json would be something like:

{
  "payments": [
    {
      "id": "16e03992d1bd35760d7bb03fea0770a2f57d9800dcdbb0d0672512cb4ef5ef6d",
      "kind": {
        "kind": {
          "bolt11": {
            "hash": "16e03992d1bd35760d7bb03fea0770a2f57d9800dcdbb0d0672512cb4ef5ef6d",
            "preimage": "0db215671266a7f5659d18625935c95b9a14c7a39be61372c8a307e0da677ea0",
            "secret": [
              90,
              76,
              7,
              206,
              138,
              238,
              94,
              53,
              214,
              194,
              106,
              151,
              173,
              27,
              70,
              57,
              140,
              61,
              202,
              220,
              130,
              111,
              225,
              71,
              79,
              55,
              117,
              96,
              226,
              183,
              161,
              102
            ]
          }
        }
      },
      "amount_msat": 50000000,
      "fee_paid_msat": 0,
      "direction": 1,
      "status": 1,

Also others like status:1 since that is not very friendly

[benthecarman]

Yeah I think in a follow up i'll go through them all and find ones like this to pretty up. Was planning similiar for #84 to make the page token a single field as well