grpc/xds: Add feature-gated xDS proto codegen and the generated .rs code

[gu0keno0]

Add xds protos, codegen and generated .rs files

Motivation

The usual places for the generated code of xDS protobuf is https://github.com/envoyproxy, e.g. https://github.com/envoyproxy/go-control-plane. However, currently there's no official https://github.com/envoyproxy/rust-control-plane.

For gRPC, we need to use the Google protobuf codegen to generate the xDS structs, therefore created this PR to do the job.

Solution

1. Add import script to download relevant protobuf sources from the SOT locations. This is mirroring grpc-java
2. Add codegen logic to the existing codegen crate to generate .rs files for the downloaded protobuf sources. By default, the Rust protoc codegen library assumes all types are in a flattened namespace in a single crate. This does not work for us because it will cause name collisions in multiple xDS protobuf source files. Therefore, the codegen generates a aggregation .rs file to re-export the protobuf types as a module, which also provides namespace isolations and organizes the generated protobuf in the same hierarchy as the original protobuf sources.
3. Gate xds feature in both codegen and grpc crates.

Test Plan

1. cargo build + CI
2. I also created a local test to verify that the generated types work as expected

#[cfg(test)]
mod tests {
    //! Demonstrates building the four core xDS resource messages
    //! (LDS/RDS/CDS/EDS) from the generated types and round-tripping them
    //! through the protobuf wire format.

    // `.serialize()` and `T::parse(&bytes)` come from the protobuf message
    // traits; the prelude brings them into scope under `_` aliases.
    use protobuf::prelude::*;

    use super::generated::envoy::config::{
        cluster::v3::cluster::{cluster::DiscoveryType, Cluster},
        endpoint::v3::endpoint::ClusterLoadAssignment,
        listener::v3::listener::Listener,
        route::v3::route::RouteConfiguration,
    };

    /// Encodes then decodes a message, exercising the protobuf wire round-trip.
    fn round_trip<M: protobuf::Message>(msg: &M) -> M {
        M::parse(&msg.serialize().expect("serialize")).expect("parse")
    }

    #[test]
    fn lds_listener_round_trip() {
        let mut listener = Listener::new();
        listener.set_name("grpc-listener");

        let decoded = round_trip(&listener);
        assert_eq!(decoded.name().as_bytes(), b"grpc-listener");
    }

    #[test]
    fn rds_route_configuration_round_trip() {
        let mut route_config = RouteConfiguration::new();
        route_config.set_name("grpc-routes");

        let decoded = round_trip(&route_config);
        assert_eq!(decoded.name().as_bytes(), b"grpc-routes");
    }

    #[test]
    fn cds_cluster_round_trip() {
        let mut cluster = Cluster::new();
        cluster.set_name("grpc-cluster");
        // A CDS cluster whose endpoints are resolved via EDS.
        cluster.set_type(DiscoveryType::Eds);

        let decoded = round_trip(&cluster);
        assert_eq!(decoded.name().as_bytes(), b"grpc-cluster");
        assert_eq!(decoded.r#type(), DiscoveryType::Eds);
    }

    #[test]
    fn eds_cluster_load_assignment_round_trip() {
        let mut endpoints = ClusterLoadAssignment::new();
        // EDS keys its endpoints by the owning cluster's name (the CDS -> EDS link).
        endpoints.set_cluster_name("grpc-cluster");

        let decoded = round_trip(&endpoints);
        assert_eq!(decoded.cluster_name().as_bytes(), b"grpc-cluster");
    }
}

3. Soon we'll add meaningful handlings of these generated structs when we implement the xDS dependency manager.

[ejona86]

I've been poking at this today on my computer, trying to understand what protobuf is doing and trying to understand our options.

Some notes that don't impact the heart of the PR too much:

1. I don't think we want to be using codegen/ for this. I think we'd want to use a build.rs instead. We could use GRPC_RUST_REGENERATE_PROTO=1 so it isn't done on every build, but... that's invalidated by the next point
2. I don't think we want to check in this much code. I think we want to regenerate it for right now. Since xds is not generally enabled that is not a problem for users today. Once we get further then we'll want a separate crate in a separate repo, so it could be committed then, but not to this repo

I had thought (2) is what we said would be best during our meeting. @dfawley can confirm. (I'll be on vacation tomorrow.)

At this point I think I see why you've made the various choices you did. I think I agree that is at least close to optimal right now. But I think we'd also all agree this is ridiculous. I want to think more about what we can suggest to protobuf to make the situation better. I feel we'll be doing everyone a disservice if we don't figure out a better approach before protobuf stabilizes.

(I don't yet see a reason protoc is run per-file instead of per-directory/proto-package. I'm right now assuming both ways would work and you just chose something and went with it.)

[gu0keno0]

Thanks for looking. Answer the questions inline.

I've been poking at this today on my computer, trying to understand what protobuf is doing and trying to understand our options.

Some notes that don't impact the heart of the PR too much:

1. I don't think we want to be using codegen/ for this. I think we'd want to use a build.rs instead. We could use GRPC_RUST_REGENERATE_PROTO=1 so it isn't done on every build, but... that's invalidated by the next point

2. I don't think we want to check in this much code. I think we want to regenerate it for right now. Since xds is not generally enabled that is not a problem for users today. Once we get further then we'll want a separate crate in a separate repo, so it could be committed then, but not to this repo

I had thought (2) is what we said would be best during our meeting. @dfawley can confirm. (I'll be on vacation tomorrow.)

In #2728 @dfawley is suggesting to put the generated code into a separate crate grpc-xds. I think build.rs makes sense. For Envoy go-control-plaine and java-control-plane, they have to check in generated code because they don't have build.rs . For Rust it's not a problem.

At this point I think I see why you've made the various choices you did. I think I agree that is at least close to optimal right now. But I think we'd also all agree this is ridiculous. I want to think more about what we can suggest to protobuf to make the situation better. I feel we'll be doing everyone a disservice if we don't figure out a better approach before protobuf stabilizes.

(I don't yet see a reason protoc is run per-file instead of per-directory/proto-package. I'm right now assuming both ways would work and you just chose something and went with it.)

The per-file approach is because there are naming conflicts. If we feed all of them directly into protoc, the generated code will have naming conflicts under the same flattened namespace (5000+ errors), because in generated code, the proto types are re-exported by a statement like "pub use internal_do_not_use_:: re-export." and there are the name conflicts such as "Policy" and "Filter". The per-file approach creates a module wrapper for each file, and therefore eliminates the name collisions.

I think this shenanigan will go away if Protobuf natively support organizing the generated code by protobuf package modules (similar to this PR but natively supported by protoc).