Throughput benchmarks

[snowp]

Add benchmarks providing some baseline for performance. An example of this would be a producer that produces new snapshots at a rapid pace and seeing how long it takes for these changes to get sent to the client.

Should at the very least cover the new delta code (as it's the more performant protocol), but could also target sotw.

This would help us better understand the impact of larger changes (like #413) and the cost of per resource computation in delta.

@alecholmez

[github-actions]

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or "no stalebot" or other activity occurs. Thank you for your contributions.

[alecholmez]

@snowp We definitely want this, once the delta code lands I think it'd be worth while spending some time on really figuring out the performance gains of delta.

Just an FYI, I introduced this a while ago: #362 but it never got merged. Might be worth revisiting at some point.

[alecholmez]

I think I'm going to take this next, we need to test the performance of what we have now. Utilizing the integration test with something like pprof to scrape for statistics seems appealing.

[snowp]

Running on top the integration tests seems like a good start!

One thing to think about beyond that is how performance would differ under different load patterns: lots of streams against the same type, lots of streams against an opaque type, lots of streams with different types, etc. and how the types of updates that are happening affect the stream (what % of streams are getting updates, the ADS case of multiple resources being sent over the same stream etc.)

[alecholmez]

Excellent, myself and @dougfort might iterate on a design doc to see what we can come up with. We'll see what we can capture, theres a gold mine of data here

[alecholmez]

@snowp if we intend to utilize the integration tests for what we want, is it safe to assume we don't intend on doing any long term storage of this throughput data? I was iterating design with a few co-workers and we were curious as to what you think is a good fit for this situation:

1. A one-time executable benchmark that simply outputs formatted data for users of this repo to consume
2. Or a complex system that holds data over time

We initially sketched out a mix of 1 and 2 but we don't want to creep the scope here to something that is unnecessary for the data we want to collect.

Here's a quick proposal we drafted up. It just sketches out the concepts of each component of the benchmark. We can go back and add technical artifacts to that later.

With a code example of the producer we talk about.

I think we want a system that will allow us to at least run pprof so we can see what's going on at runtime. The go benchmarking framework is limited as we've come to learn. If we do end up going with a system that complex it might even be worth separating from the integration test as a whole and having some sort of standalone test under pkg/test/benchmark.

[alecholmez]

Want to note that eventually we should use this benchmark to test code change too: #451