Support standard data format for config files (ex: JSON)

[ThinkChaos]

Hi,

As part of migrating my mail server over to NixOS, I'm now driving chasquid's configuration files (chasquid.conf, $domain/aliases, hooks/*, etc.) from Nix code that builds-up the data in Nix native structures, like maps and arrays, and then serializes it to chasquid's expected formats.
I had to write a bit of code to generate chasquid.conf's pbtxt format, and the custom aliases format. It's nothing especially complex, but as always, less code implies less bugs, so I'd prefer to avoid it if possible!

Hence I'm wondering if you would be open to supporting a more common & standard formats in addition to the current ones?
I saw the protoc generated Config struct already includes json tags, so it should be easy to parse that, but I would understand if you'd rather not support multiple data formats.
I've already got the serialization working, so I'm totally fine if the answer is no, but I figured I might as well ask.

For the pbtxt format, I could not find any tool online that generates it as chasquid expects, and even tried writing my own in Go, but I always ended up with output like message Config { <key: value\n ... > } instead of just <key: value \n ...> at the top level.
(I expect it would work if I had used the Protobuf schema, but I wanted to avoid that so the solution would work for any input and generate any expected schema, which is how JSON/YAML/etc. serialization is handled in NixOS).

(None of the Nix stuff is public ATM, but I'm considering upstreaming the package and config module in NixOS once I'm happy with it)

[albertito]

Thank you for filing this!

Supporting multiple configs formats is doable, but I'd want to be very careful because it's easy to end up with a big mess. But maybe json isn't too crazy since (as you point out) the generated protobuf already includes some struct tags for it.

If I were to start over, I agree there are much better options for the main config file than textpb, but we're now stuck with it for the foreseeable future :S

I wonder if we could use Cue for at least some of this.

It doesn't support exporting to pbtxt natively yet, but maybe there are ways to ingest json/yaml and export textproto using the experimental proto APIs as part of the schema definition.

Same for the aliases file, since the format is pretty simple and can likely be expressed in Cue.

Would having that type of conversion tool help with the Nix use case? I am not familiar enough with NixOS but from your message, it seems conversion tools can be used if needed?

I'm also open to un-internalizing the config package or proto definition if it would help!

FYI, I'm not going to be able to work on this for a few weeks, but I will post another update once I've had a chance to explore a few options and look a bit more into it.

Thank you!

[albertito]

Curiosity got the best of me and I spent a few minutes playing around.

For the main config, you can convert from json/yaml into textproto with cue directly:

$ cat -p chasquid.json 
{
  "hostname" : "test 123"
}

$ cue export internal/config/config.proto chasquid.json -d '#Config' -o textproto:chasquid.conf

$ cat -p chasquid.conf
hostname: "test 123"

The export is schema-aware, it will read the proto definition and make sure the fields are valid, and error if not.

If you just do cue export chasquid.json -o textproto:chasquid.conf, it will do the conversion, just without any schema checking.

I know this doesn't fully solve all your questions, there's also the aliases file, but maybe it helps in the meantime.

[ThinkChaos]

Would having that type of conversion tool help with the Nix use case? I am not familiar enough with NixOS but from your message, it seems conversion tools can be used if needed?

Yes. Basically Nix has native support for generating JSON, and it's a generic build system, so it can run tools, so combining those we're good.

I'll play with Cue, thanks for the hint!

FYI, I'm not going to be able to work on this for a few weeks, but I will post another update once I've had a chance to explore a few options and look a bit more into it.

No trouble, take your time!

[albertito]

Would having that type of conversion tool help with the Nix use case? I am not familiar enough with NixOS but from your message, it seems conversion tools can be used if needed?

Yes. Basically Nix has native support for generating JSON, and it's a generic build system, so it can run tools, so combining those we're good.

I'll play with Cue, thanks for the hint!

How did this go? :)

[ThinkChaos]

How did this go? :)

I got "sidetracked" by my main-track :D
Basically focused on migrating everything and didn't remember to come back to this. Thanks for the reminder!

I tried it out with my real setup, and the snag I hit was cue converts the .proto field names from snake_case to camelCase. I had to change my Nix config to match that, and the chasquid docs are no longer 1:1 applicable.
Also kind of minor, but it means the Cue binary is downloaded to build the config, but not used in the system's "closure", so it'll get cleaned up by GC and be downloaded again next time you change things and need to regenerate the .conf.

Not big issues, but a couple papercuts/non-ideal UX, so I'll keep the pure Nix string conversion compared to this option :)

For posterity, here's how I generated the .conf using the Cue CLI:

configFile =
  pkgs.runCommandLocal "chasquid.conf"
    {
      passAsFile = [ "json" ];
      json = builtins.toJSON cfg.settings;
    }
    ''
      ln -s "$jsonPath" config.json # cue needs a .json extension to detect the format
      ${lib.getExe pkgs.cue} export ${cfg.package.src}/internal/config/config.proto --schema '#Config' config.json -o textproto:"$out"
    '';