feat: add ssz to engine api#764
Conversation
…eflect optionality removal Removes `Optional[T]` mapping as it is replaced by specific zero/empty value encoding in container definitions. Updates `PayloadStatusV1`, `ForkchoiceUpdatedResponseV1`, `ExecutionPayloadBodyV1`, and `ExecutionPayloadBodyV2` to use non-optional types where JSON mapping implies a zero/empty value instead of true optionality. Updates SSZ mappings for `engine_getPayloadBodiesByHashV1/V2` and `engine_getBlobsV1/V2/V3` to use nested lists (`List[List[T, 1]]`) instead of `Optional[T]` to represent the presence or absence of data, consistent with non-null SSZ encoding for absent data. Adds notes explaining the zero/empty encoding for absent fields.
…update examples The documentation for how `T or null` maps to SSZ encoding is being clarified. Replaced the generic statement about `Optional[T]` being encoded as `List[T, 1]` with a more direct explanation that it is represented as `List[T, 1]`. Also updated the description for `payload_attributes` in `ForkchoiceUpdated` requests to explicitly state that presence is indicated by a list with 1 element, matching the SSZ type definition. Additionally, added missing vocabulary words to `wordlist.txt` to improve future documentation generation tools.
…ation to reflect capability exchange The transport negotiation mechanism has been updated to exclusively use `engine_exchangeCapabilities` over JSON-RPC for determining support of SSZ REST endpoints. This change clarifies the required steps for clients to discover and utilize the binary SSZ transport.
Update documentation to better explain how JSON-RPC remains the default for negotiation and fallback, explicitly stating when binary SSZ is used. This clarifies the steps involved for CL and EL during initialization.
| - The CL uses SSZ natively, forcing a round-trip conversion (SSZ to JSON, then JSON to internal types) at the Engine API boundary. | ||
|
|
||
| Binary SSZ eliminates all of this. The CL sends raw SSZ bytes over HTTP; the EL deserializes directly. No hex encoding, no JSON parsing, no intermediate representations. Payload sizes are reduced by 50% or more compared to JSON-RPC, and serialization is no longer a bottleneck in the critical path between CL and EL. |
There was a problem hiding this comment.
The EL uses RLP, why not RLP? The EL does not currently support SSZ while the CL does support RLP for various reasons. This would reduce the number of libraries in the EL but be net zero for the CL.
What unique utility does SSZ provide?
There was a problem hiding this comment.
The CL does not support RLP, some clients do, but it is unnecessary, it is an optimization.
|
|
||
| ### Binary SSZ transport | ||
|
|
||
| Clients **MAY** support a binary SSZ transport as an alternative to JSON-RPC. The binary transport uses resource-oriented REST endpoints with raw SSZ request and response bodies (`application/octet-stream`), eliminating JSON and hex-encoding overhead for fast CL-EL communication. Endpoints follow Beacon API conventions with path-based versioning (e.g., `POST /engine/v5/payloads`). |
There was a problem hiding this comment.
Current specs define the capabilities as JSON-RPC methods names like:
- engine_newPayloadV2
- engine_getPayloadV5
This suggestion change the capabilities vocabulary to strings like POST /engine/v5/payloads.
This is protocol change, not just a transport addition.
|
|
||
| --- | ||
|
|
||
| #### `POST /engine/v1/capabilities` — Exchange capabilities |
There was a problem hiding this comment.
Related to earlier point, If we want to add this endpoint then we should remove engine_exchangeCapabilities reference from the documents and that will be a protocol change.
Or we keep both but must keep the return format of both to one canonical format, and that will existing response format of the engine_exchangeCapabilities to avoid a breaking change to protocol.
| ### Client errors | ||
|
|
||
| | Status | Meaning | Usage | | ||
| | - | - | - | | ||
| | `400` | Bad Request | Malformed SSZ encoding | | ||
| | `401` | Unauthorized | Missing or invalid JWT token | | ||
| | `404` | Not Found | Unknown payload ID | | ||
| | `409` | Conflict | Invalid forkchoice state | | ||
| | `413` | Request Too Large | Request exceeds maximum element count | | ||
| | `422` | Unprocessable Entity | Invalid payload attributes | | ||
|
|
||
| ### Server errors | ||
|
|
||
| | Status | Meaning | Usage | | ||
| | - | - | - | | ||
| | `500` | Internal Server Error | Unexpected server error | | ||
|
|
||
| Error responses use `Content-Type: text/plain` with a human-readable error message body. |
There was a problem hiding this comment.
The existing Engine API has meaningful machine-readable error codes:
- unknown payload
- invalid forkchoice state
- invalid attributes
- too large request
- unsupported fork
The SSZ HTTP version partly maps these, but not completely:
- unsupported fork is missing
- invalid params vs malformed SSZ are not fully separated
- text/plain error bodies are not machine-stable
Would suggest a normative mapping from JSON-RPC error codes to HTTP status + a small structured error body, even if the success path stays raw SSZ.
|
|
||
| When a new fork introduces a new method version, a new versioned endpoint is added. Older versioned endpoints **MAY** be deprecated but **SHOULD** remain available for backwards compatibility. | ||
|
|
||
| ### Negotiation and fallback |
There was a problem hiding this comment.
Negotiation says “if both advertise, use SSZ”, but what if:
- EL advertises endpoint support but returns 404/415/500
- one endpoint version is implemented incorrectly while others work
Do we:
- permanently downgrade the whole transport?
- downgrade only that method version?
- retry JSON-RPC immediately?
Without clear rules for negotiation and fallback behavior for each client may diverge.
…llable types
- Drop misleading "idempotent" claim on GET /payloads/{payload_id} and
require Cache-Control: no-store; the payload mutates until the slot
deadline so caches/intermediaries must not store or revalidate it.
- Expand security considerations with explicit DoS guidance: pre-read
Content-Length rejection, length/offset validation before allocation,
and operationally enforced per-endpoint body caps. The protocol-level
maxima bound on-chain validity, not per-request resource use.
- Encode truly-nullable fields per the documented List[T, 1] rule:
PayloadStatusV1.latest_valid_hash, ForkchoiceUpdatedResponseV1.payload_id,
and ExecutionPayloadBodyV2.block_access_list. Restores parity with the
JSON spec (each is non-required / oneOf null) and removes the
zero-sentinel ambiguity. Example response length updated 37 -> 41.
| | `Content-Type` (request) | `application/octet-stream` | SSZ-encoded request container | | ||
| | `Content-Type` (response) | `application/octet-stream` | SSZ-encoded response (success) | | ||
| | `Content-Type` (response) | `text/plain` | Human-readable error message | | ||
| | `Accept` (request) | `application/octet-stream` | Client accepts SSZ-encoded responses | |
There was a problem hiding this comment.
I would also like that we support both application/json and application/octet-stream. The application/json would just the existing JSON-RPC with layer with REST interface. That will be very easy for everyone to implement. Later we can add application/octet-stream support for individual endpoint.
|
The motivation for binary SSZ is clear and the performance gains it promises, especially with blobs, are significant and necessary for the protocol's evolution. However, I have some concerns about coupling the transport layer migration (JSON-RPC to REST) with the encoding semantics change (JSON/hex to SSZ). I propose a staged approach that separates these concerns, allowing for a more robust and predictable transition for CL and EL clients. Core Recommendation: Implement REST Transport with Current JSON Semantics First, Then Introduce SSZ Gradually. This approach involves two main phases: Phase 1: Introduce REST Transport with Existing JSON Semantics
Phase 2: Introduce Optional Binary SSZ Encoding Gradually, Endpoint-by-Endpoint
This phased approach embodies the principle of "slow is smooth, smooth is fast" for protocol evolution. It allows implementers to digest one architectural change at a time, leading to a more stable and predictable ecosystem. |
…ConfigurationV1 - Remove INVALID_BLOCK_HASH from the PayloadStatusV1 status enum. The value only applied to engine_newPayloadV1 (Paris) and was supplanted by INVALID starting Shanghai (V2+); all SSZ endpoints in this spec are V2+, so it was dead weight. Addresses @LukaszRozmej review. - Remove the TransitionConfigurationV1 container, the /engine/v1/transition-configuration endpoint section, the endpoint-summary row, and both TOC entries. The corresponding engine_exchangeTransitionConfigurationV1 method was deprecated in Cancun (clients MAY remove support).
|
Alternative: #793 |
I would change transport first and then later we can refactor. Not a fan of doing 2 things at once. This way we can reuse handler code and focus on transport layer. We already implemented this one but we can also support yours if it gains traction |
I will post this here but also copy this message over to the PR:
here is how versioning should be done in REST: https://restfulapi.net/versioning/, we will still need different versions for the containers. getting rid of the versioning just makes everything harder and overall worse for no benefit (you still need different containers) |
7 participants
Core change: Full binary SSZ over REST. No JSON, no hex encoding - raw SSZ bytes over HTTP.
Rationale: