Envoy WASM extensions in the present and its future (Proxy-Wasm)

[marc-barry]

Title: Envoy WASM extensions in the present and its future (Proxy-Wasm)

Description:

Envoy current supports WASM extensions via the WASM filter. I am aware of the following warning:

The Wasm filter is experimental and is currently under active development. Capabilities will be expanded over time and the configuration structures are likely to change.

The documentation for the feature is fairly terse and I largely used articles like https://tetrate.io/blog/wasm-modules-and-envoy-extensibility-explained-part-1/ and there was Google Document proposal that I had read a while back but I'm now unable to find the document. We have hit a number of issues in terms of the documentation, extension development process and clear understanding of the adhered ABI spec and plans for that.

• The ABI that Envoy seems to have chosen is https://github.com/proxy-wasm. But if you look at the spec as it is defined today under https://github.com/proxy-wasm/spec/tree/main/abi-versions it is in-fact empty at this time. Envoy appears to use v0.1.0 of the spec, although that isn't clear documented. Spec documentation for v0.1.0? proxy-wasm/spec#46 is a request for information on the spec and where it might be defined. Document Proxy-Wasm ABI v0.2.1. proxy-wasm/spec#42 appears to attempt to document v0.2.1 of the spec but it is coming up on a year without that being merged.
• The organization https://github.com/orgs/proxy-wasm/people only has one member (https://github.com/lizan) at this time. It does appear that the organization and hence the spec has been abandoned at this time.
• https://github.com/tetratelabs/proxy-wasm-go-sdk was what I used to get it all working. This is maintained outside the aforementioned organization and with https://github.com/tetratelabs. It also references the ABI spec for https://github.com/proxy-wasm which we have established isn't up-to-date or really documented.
• Kong even seems to have adopted Proxy-Wasm as they have https://github.com/Kong/ngx_wasm_module/. They actually had some good examples of writing filters which could largely be used with Envoy. Also see https://konghq.com/blog/engineering/proxy-wasm.
• Even Istio has a similar documentation for using WASM under https://istio.io/latest/docs/concepts/wasm/ but has the same documentation issues as Envoy but does refer to a decent set of examples under https://github.com/istio-ecosystem/wasm-extensions. There are lots of articles from the Istio community with announcements like https://opensource.googleblog.com/2020/03/webassembly-brings-extensibility-to.html and https://www.solo.io/blog/an-extended-and-improved-webassembly-hub-to-helps-bring-the-power-of-webassembly-to-envoy-and-istio/. Unfortunately, I get blocked from using their announced https://webassemblyhub.io/ and I'm not even sure if it is alive and used today.
• I noticed that Envoy Gateway 1.0 was announced (https://gateway.envoyproxy.io/news/blogs/1.0-release/1.0-release/). WASM extensions are part of its release and the documentation exists at https://gateway.envoyproxy.io/docs/tasks/extensibility/wasm/ but once again it really doesn't cover how to build an extension and doesn't even mention Proxy-Wasm.

There are many more references across the Internet with reference to WASM extensions for things that use Envoy under the hood or have decided to also adopt it as its ABI. But as I mentioned above the spec isn't really even defined in the public space and all work on it appears abandoned or stalled.

What I'm trying to determine is the following:

• Does Envoy in-fact use Proxy-Wasm and v0.1.0 of the spec?
• Since the spec doesn't exist in consumable form under https://github.com/proxy-wasm/spec/tree/main/abi-versions how can I determine the ABI that Envoy is using?
• Does Envoy intend to move to later versions of the spec, assuming they are documented and agreed upon at some point in the future?
• When products like https://gateway.envoyproxy.io refer to WASM extensibility I assume they are using Envoy under the hood and that in fact Envoy's WASM extensions are used. Is this a correct assumption?
• Does anyone have a good understanding of the future of WASM extensions in Envoy and what the support and evolution might look like?

Relevant Links:

• proposal: design Wasm ABI for tracing providers #10563
• https://github.com/proxy-wasm
• https://github.com/proxy-wasm/spec
• https://github.com/tetratelabs/proxy-wasm-go-sdk
• https://istio.io/latest/docs/concepts/wasm/
• https://github.com/istio-ecosystem/wasm-extensions
• https://webassemblyhub.io/
• https://gateway.envoyproxy.io/docs/tasks/extensibility/wasm/

[phlax]

@marc-barry i would be happy to review any improvements to the docs - sounds like we have some gaps in current docs

cc @mpwarres wrt substantive questions

[marc-barry]

@phlax thanks for answering. I also don't mind helping improve this in the community and contributing. Perhaps we can start with documenting the ABI interface details that Envoy uses so that developers can reference this when developing extensions. I couldn't find any examples for Envoy and perhaps I could contribute a simple example that we could include.

https://www.envoyproxy.io/docs/envoy/v1.31.0/api-v3/extensions/wasm/v3/wasm.proto#envoy-v3-api-msg-extensions-wasm-v3-pluginconfig is documented quite well but the gap is, for example, from say Go code to a WASM plugin that can be loaded. If you navigate down to allowed_capabilities you'll see a reference to The capability names are given in the [Proxy-Wasm ABI](https://github.com/proxy-wasm/spec/tree/master/abi-versions/vNEXT). but if you follow that link it no longer exists. I think we could focus on cleaning this up which would make it more clear. We should probably figure out how to improve the https://github.com/proxy-wasm contribution situation as this is then something that the Envoy docs could just reference.

[phlax]

yeah, makes sense

we have this example https://www.envoyproxy.io/docs/envoy/latest/start/sandboxes/wasm-cc, i tried previously to add something similar for rust but didnt get too far

@lizan is not as active as before but might have ideas about who to speak to - i know there is quite a bit of commitment on the google side to develop/maintain the wasm filter

[mathetake]

@mpwarres @martijneken are the current owners of Proxy-Wasm and Envoy codebase, so I am truly hoping they share what direction this goes and how the evolution looks like - as a former maintainer there, I wanted to express my sorry here for the mess and I actually failed to bring it to the healthy state. I completely share the frustration that I sense you have. I hope the folks mentioned ^^ clarify their stance and how this will be resolved, especially Google's point of view on this matter. cc @alyssawilk

[mathetake]

I think one idea is to eliminate the Proxy-Wasm organization dependency completely, and make the Envoy codebase self-contained. After that, host the spec and complete user-level documentation here in envoyproxy.io. Plus, host the SDKs here as well (we are more than happy donating the Go SDK). Given that there was no inter-proxy collaboration per-se since the release, there's absolutely zero benefit to Envoy in having the spec separated from Envoy at this point. pure technical debt is from my view the consensus from the conversation with other community members in the last few weeks. That's my take. That's just a random idea, but having the central place to look at is as a user myself is less confusing as well as better IMHO.

[mathetake]

^^ if this sounds good to other maintainers, I am more than happy to help and maintain again - I really feel the obligation to fix this once and for all

[kyessenov]

I am happy to review any doc improvements since I'm familiar with the extensions as well. A lot of the base code is used in production, for real products, so there's definitely a valid use case, but it is difficult to figure out where the stability ends and the rough edges start without looking at the code. The larger problem is that it is simply hard to start writing Wasm no matter which language one chooses (not everything will learn Rust for changing headers), so there's a barrier of entry that the improved docs may not fully address.

I think the current Wasm implementation is no longer experimental and reached some stability (for core parts), but it also failed to reach 1.0. It doesn't really matter what the version number says, since in practice, there's just one ABI used by various Proxy-Wasm efforts.

[mathetake]

yeah I completely agree with @kyessenov, and in order to improve the situation (not saying I am sure how the end stability means here), I think as @marc-barry hinted (and I believe everyone is aware), the unnecessary dependency on the Proxy-Wasm org makes the situation worse or standstill. What I am suggesting is to document and host everything in envoyproxy.io so there is just one single source of truth. This is not only about the documentation, but also how the ecosystem around it works like who's responsible for what, how's the issue handled, where's the place to report issues, what's the support policy etc. Currently all of them is a mess. But open to suggestions, and curious how others think about the coupling with Proxy-Wasm and leaving everything there as-is benefits Envoy just by looking at the history

[thenewwazoo]

I'm an onlooker, but thought I might weigh in. My employer is interested in leveraging existing logic written in Rust (or in porting Java code to Rust) in order to embed it in multiple places, one of which is Envoy filters. We benchmarked C++, Lua, and Rust WASM (using the v8 runtime) and found the overhead of WASM to be a show-stopper. I have been experimenting for the last ~week with building NullVM filters to compile into v1.30 but ran into this problem after some significant struggles bringing everything up-to-date, and have stopped trying.

I'm not opposed per se to proxy-wasm, but I'm doubtful as to its value given that adoption has been very poor over the last ~4 years since it was introduced. The overhead of a "real" WASM runtime is too high for us, and NullVM has (as far as I can tell) never quite been finished (and its overhead is still to be measured). As such, I'll be exploring the recent work on loadable modules. IMO that's where development effort should be directed.

[kyessenov]

@thenewwazoo The criticism is shared but it's an ecosystem problem with Wasm, and not something Envoy can fix. IMO, a stable ABI is all that matters, and Wasm in Envoy has kept the stability promise despite many internal changes inside Envoy. This is no worse than other extension points, e.g. Lua, and it's more flexible since you can also use dynload "nullVM" and not be blocked by runtime performance.