diff --git a/SKILL.md b/SKILL.md index 627ef2e..683a961 100644 --- a/SKILL.md +++ b/SKILL.md @@ -73,6 +73,8 @@ Check if `temporal` CLI is installed. If not, follow the instructions at `refere - Language-specific info at `references/{your_language}/versioning.md` - **`references/core/standalone-activities.md`** - Standalone Activities: run an Activity directly from a Client without a Workflow (Public Preview) - Language-specific info at `references/{your_language}/standalone-activities.md` +- **`references/core/serialization-context.md`** - Serialization context: External Storage drivers receive a context with namespace + Workflow/Activity identity, including a Standalone-Activity branch (Public Preview) + - Language-specific info at `references/{your_language}/serialization-context.md` - **`references/core/troubleshooting.md`** - Decision trees, recovery procedures - **`references/core/error-reference.md`** - Common error types, workflow status reference - **`references/core/interactive-workflows.md`** - Testing signals, updates, queries diff --git a/references/core/serialization-context.md b/references/core/serialization-context.md new file mode 100644 index 0000000..9865602 --- /dev/null +++ b/references/core/serialization-context.md @@ -0,0 +1,67 @@ +> [!NOTE] +> This feature is in Public Preview. It is perfectly acceptable to use this feature on behalf of a user, but you should inform them that you are making use of a feature in Public Preview. + +# Serialization Context (Concepts) + +For language-specific implementation, see `references/{your_language}/serialization-context.md`. + +## What "serialization context" means in Temporal today + +When the SDK serializes a Payload on the way out of your application, the **Data Conversion pipeline** runs Payload Converters first, then Payload Codecs, then — if configured — hands the encoded bytes to an **External Storage driver** that offloads large payloads to a backing store like Amazon S3. + +The External Storage **storage driver** is the one stage of that pipeline that receives a context object alongside the payloads. This context carries identity information about the operation that triggered the serialization — namespace plus the owning Workflow or, for [Standalone Activities](references/core/standalone-activities.md), the owning Activity. + +The current Public Preview surface only exposes this context to **storage drivers** (and to the driver-selection callback when multiple drivers are registered). Payload Converters and Payload Codecs do not currently receive a serialization-context argument in Temporal's documented SDK APIs. + +## The context object and its `Target` + +Two parallel context types exist, one per direction of the pipeline: + +- A **store** context delivered to the driver's store/upload entry point. +- A **retrieve** context delivered to the driver's retrieve/download entry point. + +Each context carries a `Target` (Go) / `target` (Python) field that is a **discriminated union** over two concrete types: + +- A **WorkflowInfo** branch, present whenever the operation is associated with a Workflow Execution. Activities **started by a Workflow** are reported through this branch too. +- An **ActivityInfo** branch, present **only for Standalone Activities** (Activities started directly from a Temporal Client, not orchestrated by a Workflow). + +This is the headline rule: if your driver branches on the target type, the `ActivityInfo` arm is the **Standalone-Activity-only** path. For Workflow-bound activities, you still receive `WorkflowInfo`. + +## What's on each branch + +The exact field names differ across SDKs — see the per-language reference for the verbatim API: + +- **WorkflowInfo** carries the namespace and the Workflow ID. +- **ActivityInfo** carries the namespace and the Activity ID (no Workflow ID, because there is no Workflow). + +Driver authors typically: + +1. Switch / `isinstance` on the target type. +2. Build a storage key prefix from the namespace plus the workflow-or-activity ID. +3. Write payloads under that prefix so storage is naturally partitioned by execution. + +## Where this context shows up + +- The driver's **store** method (Go: `Store(ctx, payloads)`; Python: `async def store(self, context, payloads)`). +- The driver's **retrieve** method (Go: `Retrieve(ctx, claims)`; Python: `async def retrieve(self, context, claims)`). +- The **driver-selector** callback used when more than one storage driver is registered. The selector receives the store context plus the payload and returns the driver to use (or `nil` / `None` to keep the payload inline). + +## SDK coverage + +Documented SDK guides for External Storage and its context object exist for: + +- **Go SDK** — see `references/go/serialization-context.md`. +- **Python SDK** — see `references/python/serialization-context.md`. + +The encyclopedia External Storage page lists only those two SDKs. The Temporal **.NET SDK** documentation is currently silent on a serialization-context object; treat `references/dotnet/serialization-context.md` as a placeholder until .NET docs add coverage. + +## Common mistakes + +- **Assuming `ActivityInfo` covers Workflow-bound activities.** It does not. An Activity scheduled by a Workflow surfaces as `WorkflowInfo` (Workflow ID is the orchestrator's ID). Only [Standalone Activities](references/core/standalone-activities.md) produce `ActivityInfo`. +- **Reaching for workflow info from inside a custom Payload Converter or Payload Codec.** No documented SDK API exposes a context to those stages today; only the StorageDriver does. If you need the owning Workflow ID at serialization time, implement a custom storage driver (or driver selector) rather than smuggling state into a converter. +- **Treating `Target` as a struct with a `Kind` enum.** It is a discriminated union — a Go interface satisfied by either concrete type, or a Python target whose runtime type you check with `isinstance`. Use a type switch / `isinstance`. + +## Resources + +- External Storage encyclopedia page: +- Standalone Activities concept page: see `references/core/standalone-activities.md`. diff --git a/references/dotnet/data-handling.md b/references/dotnet/data-handling.md index 8d0bb23..faf7348 100644 --- a/references/dotnet/data-handling.md +++ b/references/dotnet/data-handling.md @@ -215,3 +215,5 @@ public class MyWorkflow 3. Encrypt sensitive data with `IPayloadCodec` 4. Use `Workflow.NewGuid()` and `Workflow.Random` for deterministic values 5. Use camelCase converter if interoperating with other SDKs + +For the serialization-context object that other SDKs pass to External Storage drivers — and the current .NET coverage gap — see `references/dotnet/serialization-context.md`. diff --git a/references/dotnet/dotnet.md b/references/dotnet/dotnet.md index 8ed4a53..2ec9eda 100644 --- a/references/dotnet/dotnet.md +++ b/references/dotnet/dotnet.md @@ -201,3 +201,4 @@ See `references/dotnet/testing.md` for info on writing tests. - **`references/dotnet/versioning.md`** — Patching API, workflow type versioning, Worker Versioning - **`references/dotnet/standalone-activities.md`** — Standalone Activities: run an Activity directly from a Client without a Workflow (Public Preview). Concept overview at `references/core/standalone-activities.md`. - **`references/dotnet/determinism-protection.md`** — Runtime task detection, .NET Task determinism rules +- **`references/dotnet/serialization-context.md`** — Serialization context for External Storage drivers. .NET docs are currently silent on this surface; the file is a placeholder with `` markers. Concept overview at `references/core/serialization-context.md`. diff --git a/references/dotnet/serialization-context.md b/references/dotnet/serialization-context.md new file mode 100644 index 0000000..3e28994 --- /dev/null +++ b/references/dotnet/serialization-context.md @@ -0,0 +1,31 @@ +> [!NOTE] +> External Storage and its serialization-context object are in Public Preview in the SDKs that ship it. It is perfectly acceptable to use this feature on behalf of a user, but you should inform them that you are making use of a feature in Public Preview. + +## Overview + +[Cross-SDK concept file](references/core/serialization-context.md). + +The Temporal **.NET SDK documentation currently has no published surface for a serialization-context object** delivered to Data Converter components. The encyclopedia External Storage page lists SDK-specific guides for the Go SDK and Python SDK only; there is no .NET SDK guide. + +What the .NET docs **do** describe is the existing Data Converter and Codec surface in `Temporalio.Converters`. `IPayloadCodec` exposes `EncodeAsync(IReadOnlyCollection)` and `DecodeAsync(IReadOnlyCollection)` — neither overload receives a context parameter. + +Treat this page as a placeholder. If a user asks how to access Workflow/Activity identity inside a .NET custom converter or codec today, the supported answer is "the .NET SDK does not document one." + +## Prerequisites + +## Storage driver context + +## Reading workflow vs activity metadata + +## Driver selection + +## Common mistakes + +- **Assuming the .NET SDK has the same `SerializationContext` / `StorageDriver` API as Go or Python.** It does not, as of the current published documentation. Don't generate .NET code that imports `Temporalio.Converters.StorageDriverStoreContext` or similar based on the Go/Python names. +- **Reaching for a context parameter on `IPayloadCodec.EncodeAsync` / `DecodeAsync`.** Neither method receives a context argument in the documented .NET API. + +## Resources + +- .NET Data Converters and encryption: (see "Converters and encryption"). +- External Storage encyclopedia page (Go and Python only, today): +- Cross-SDK concept file: `references/core/serialization-context.md`. diff --git a/references/go/data-handling.md b/references/go/data-handling.md index 18ccf57..5e4c467 100644 --- a/references/go/data-handling.md +++ b/references/go/data-handling.md @@ -262,3 +262,5 @@ err := workflow.UpsertMemo(ctx, map[string]interface{}{ 3. Keep payloads small -- see `references/core/gotchas.md` for limits 4. Use `PayloadCodec` for encryption; never store sensitive data unencrypted 5. Configure the same data converter on both client and worker + +For External Storage drivers that need the owning Workflow or Standalone-Activity identity, see `references/go/serialization-context.md`. diff --git a/references/go/go.md b/references/go/go.md index 6c42bed..18269e7 100644 --- a/references/go/go.md +++ b/references/go/go.md @@ -252,3 +252,4 @@ See `references/go/testing.md` for info on writing tests. - **`references/go/data-handling.md`** - Data converters, payload codecs, encryption - **`references/go/versioning.md`** - Patching API (`workflow.GetVersion`), Worker Versioning - **`references/go/determinism-protection.md`** - Information on **`workflowcheck`** tool to help statically check for determinism issues. +- **`references/go/serialization-context.md`** - Serialization context delivered to External Storage drivers (`converter.StorageDriverStoreContext` / `converter.StorageDriverRetrieveContext`), including the Standalone-Activity branch. Concept overview at `references/core/serialization-context.md`. diff --git a/references/go/serialization-context.md b/references/go/serialization-context.md new file mode 100644 index 0000000..2d34e81 --- /dev/null +++ b/references/go/serialization-context.md @@ -0,0 +1,89 @@ +> [!NOTE] +> This feature is in Public Preview. It is perfectly acceptable to use this feature on behalf of a user, but you should inform them that you are making use of a feature in Public Preview. + +## Overview + +[Cross-SDK concept file](references/core/serialization-context.md). + +The Go SDK delivers a **serialization context** — `converter.StorageDriverStoreContext` on the way in and `converter.StorageDriverRetrieveContext` on the way out — to every custom External Storage driver. The context's `Target` field is a discriminated union (`converter.StorageDriverWorkflowInfo` or `converter.StorageDriverActivityInfo`) that carries namespace and execution-identity metadata, including the Standalone-Activity case. + +## Prerequisites + +- A Temporal Go SDK version that supports External Storage and the `converter.StorageDriver` interface. External Storage is in Public Preview; consult the Go SDK release notes for the minimum supported version. +- An existing custom storage driver, or willingness to write one — see the `references/go/data-handling.md` file for the broader Data Converter setup that External Storage plugs into. + +## Storage driver context + +A `converter.StorageDriver` exposes `Store` and `Retrieve` methods. The first argument to each is the context object: + +```go +func (d *MyDriver) Store( + ctx converter.StorageDriverStoreContext, + payloads []*commonpb.Payload, +) ([]converter.StorageDriverClaim, error) { ... } + +func (d *MyDriver) Retrieve( + ctx converter.StorageDriverRetrieveContext, + claims []converter.StorageDriverClaim, +) ([]*commonpb.Payload, error) { ... } +``` + +Both context types carry a `Target` field that identifies the owning execution. + +## Reading workflow vs activity metadata + +`ctx.Target` is an interface satisfied by one of two concrete types. Use a Go type switch to extract identity fields: + +```go +switch info := ctx.Target.(type) { +case converter.StorageDriverWorkflowInfo: + // Workflow-scoped operation, including Activities started by a Workflow. + // Fields: info.Namespace, info.WorkflowID + if info.WorkflowID != "" { + // build key from info.Namespace + info.WorkflowID + } +case converter.StorageDriverActivityInfo: + // StorageDriverActivityInfo is only used for standalone (non-workflow-bound) + // activities. Activities started by a workflow use StorageDriverWorkflowInfo. + // Fields: info.Namespace, info.ActivityID + if info.ActivityID != "" { + // build key from info.Namespace + info.ActivityID + } +} +``` + +Key facts: + +- The `StorageDriverActivityInfo` arm fires **only for [Standalone Activities](references/core/standalone-activities.md)** — Activities started directly from a Client without a Workflow. +- An Activity scheduled by a Workflow is reported on the `StorageDriverWorkflowInfo` arm, with the orchestrating Workflow's ID. +- The docs describe the identity fields as "namespace, Workflow ID" depending on the operation; use the type switch to access the concrete values. + +## Driver selection + +When more than one driver is registered, a `StorageDriverSelector` decides which driver stores a given payload. Its `SelectDriver` method receives the same `StorageDriverStoreContext`, so the selection logic can branch on namespace, Workflow ID, or whether the operation is a Standalone Activity: + +```go +func (s *PreferredSelector) SelectDriver( + ctx converter.StorageDriverStoreContext, + payload *commonpb.Payload, +) (converter.StorageDriver, error) { + // Inspect ctx.Target the same way Store does. + return s.preferred, nil +} +``` + +Return `nil` for the driver to keep a payload inline in Event History instead of offloading. + +## Common mistakes + +- **Switching on `Target` with a default-case fallthrough that assumes Workflow identity.** A driver that runs against Standalone Activities will land in `StorageDriverActivityInfo` and have no `WorkflowID`. Branch explicitly. +- **Reading `info.WorkflowID` on `StorageDriverActivityInfo`.** The activity branch carries `info.Namespace` and `info.ActivityID` only. +- **Expecting a context on `PayloadConverter` or `PayloadCodec`.** The Go SDK's serialization-context object is only passed to the `StorageDriver` interface (and the `StorageDriverSelector`). To key behavior off Workflow/Activity identity, do it in the storage driver, not in a custom Payload Converter. +- **Changing a driver's `Name()` after payloads have been written.** The SDK records the name on each claim; renaming breaks retrieval. + +## Resources + +- External Storage encyclopedia page: +- Go SDK External Storage guide: +- Cross-SDK concept file: `references/core/serialization-context.md`. +- Standalone Activities (Go): see the language-specific reference under `references/{your_language}/standalone-activities.md`. diff --git a/references/python/advanced-features.md b/references/python/advanced-features.md index 6ad8ae8..4cb4ad6 100644 --- a/references/python/advanced-features.md +++ b/references/python/advanced-features.md @@ -134,7 +134,7 @@ client = await Client.connect( - The only field is `resolution_interval_millis: int = 30000` — how often to re-resolve DNS, in milliseconds. - `DnsLoadBalancingConfig.default` is a pre-built instance with the default 30-second interval. -- `dns_load_balancing_config` defaults to 30 seconds if you don't pass anything explicitly. +- `dns_load_balancing_config` defaults to 30 seconds if you don't pass anything explicitly. - Pass `dns_load_balancing_config=None` to disable DNS load balancing entirely. ### Mutual exclusion with HTTP CONNECT proxy diff --git a/references/python/data-handling.md b/references/python/data-handling.md index 65f4a99..f7b184f 100644 --- a/references/python/data-handling.md +++ b/references/python/data-handling.md @@ -230,3 +230,5 @@ class MyWorkflow: 3. Encrypt sensitive data with PayloadCodec 4. Use dataclasses for simple data structures 5. Use `workflow.uuid4()` and `workflow.random()` for deterministic values + +For External Storage drivers that need the owning Workflow or Standalone-Activity identity, see `references/python/serialization-context.md`. diff --git a/references/python/integrations/google-adk.md b/references/python/integrations/google-adk.md index 4d59f4d..3e0e015 100644 --- a/references/python/integrations/google-adk.md +++ b/references/python/integrations/google-adk.md @@ -9,7 +9,6 @@ The integration is built on the Python SDK [Plugin system](https://docs.temporal > [!NOTE] > This feature is in Public Preview. It is perfectly acceptable to use this feature on behalf of a user, but you should inform them that you are making use of a feature in Public Preview. - For general Temporal AI/LLM patterns (retries, rate limits, multi-agent orchestration) see `references/core/ai-patterns.md` and `references/python/ai-patterns.md`. ## Prerequisites diff --git a/references/python/integrations/langgraph.md b/references/python/integrations/langgraph.md index 2675672..4237ca7 100644 --- a/references/python/integrations/langgraph.md +++ b/references/python/integrations/langgraph.md @@ -214,4 +214,4 @@ For LangSmith tracing of LangGraph nodes and Temporal Activities together, use t - `references/python/ai-patterns.md` — Python AI/LLM patterns (Pydantic data converter, LLM Activity design, retry/error classification). - `references/core/ai-patterns.md` — language-agnostic AI/LLM patterns. -- `references/python/integrations/langsmith.md` - Companion LangSmith plugin. +- `references/python/integrations/langsmith.md` - Companion LangSmith plugin. diff --git a/references/python/integrations/langsmith.md b/references/python/integrations/langsmith.md index 98db967..a0ab26a 100644 --- a/references/python/integrations/langsmith.md +++ b/references/python/integrations/langsmith.md @@ -100,7 +100,6 @@ The plugin makes `@traceable` replay-safe in the Workflow sandbox. You do not ne - The plugin injects metadata using `workflow.now()` for timestamps and `workflow.random()` for UUIDs instead of `datetime.now()` and `uuid4()`. - LangSmith HTTP calls run on a background thread pool that does not interfere with deterministic Workflow execution. - ## Context propagation Trace context flows automatically across Client → Workflow → Activity → Child Workflow → Nexus via Temporal headers. Do not pass context manually. diff --git a/references/python/integrations/openai-agents-sdk.md b/references/python/integrations/openai-agents-sdk.md index 3807eb7..8ddf36a 100644 --- a/references/python/integrations/openai-agents-sdk.md +++ b/references/python/integrations/openai-agents-sdk.md @@ -164,7 +164,6 @@ Note that the initial run context comes from the `context=...` argument you pass In addition, since a `@function_tool` runs in the workflow, they can also call Temporal activities or other durable primitives themselves. - **Don't put I/O, system clock, or sources of randomness inside a `@function_tool` body.** Make it an `@activity.defn` and wrap with `activity_as_tool` instead. ### Picking between the two diff --git a/references/python/python.md b/references/python/python.md index 2528e97..0abaa8e 100644 --- a/references/python/python.md +++ b/references/python/python.md @@ -181,6 +181,7 @@ See `references/python/testing.md` for info on writing tests. - **`references/python/data-handling.md`** - Data converters, Pydantic, payload encryption - **`references/python/versioning.md`** - Patching API, workflow type versioning, Worker Versioning - **`references/python/standalone-activities.md`** - Standalone Activities: run an Activity directly from a Client without a Workflow (Public Preview). Concept overview at `references/core/standalone-activities.md`. +- **`references/python/serialization-context.md`** - Serialization context delivered to External Storage drivers (`StorageDriverStoreContext` / `StorageDriverRetrieveContext`), including the Standalone-Activity branch. Concept overview at `references/core/serialization-context.md`. - **`references/python/determinism-protection.md`** - Python sandbox specifics, forbidden operations, pass-through imports - **`references/python/ai-patterns.md`** - LLM integration, Pydantic data converter, AI workflow patterns - **`references/python/workflow-streams.md`** - Public-Preview `temporalio.contrib.workflow_streams` library: durable, offset-addressed event channel for streaming progress to subscribers. diff --git a/references/python/serialization-context.md b/references/python/serialization-context.md new file mode 100644 index 0000000..e6b2b8f --- /dev/null +++ b/references/python/serialization-context.md @@ -0,0 +1,85 @@ +> [!NOTE] +> This feature is in Public Preview. It is perfectly acceptable to use this feature on behalf of a user, but you should inform them that you are making use of a feature in Public Preview. + +## Overview + +[Cross-SDK concept file](references/core/serialization-context.md). + +The Python SDK delivers a **serialization context** — `StorageDriverStoreContext` on the way in and `StorageDriverRetrieveContext` on the way out — to every custom External Storage driver. The context's `target` attribute is a discriminated union (`StorageDriverWorkflowInfo` or `StorageDriverActivityInfo`) that carries namespace and execution-identity metadata, including the Standalone-Activity case. + +## Prerequisites + +- A Temporal Python SDK version that exposes the External Storage `StorageDriver` abstract class. External Storage is in Public Preview; consult the Python SDK release notes for the minimum supported version. +- An existing custom storage driver, or willingness to write one — see `references/python/data-handling.md` for the broader Data Converter setup that External Storage plugs into. + +## Storage driver context + +A custom Python storage driver subclasses `StorageDriver` and implements async `store` and `retrieve` methods. The second argument to each is the context object: + +```python +class MyDriver(StorageDriver): + async def store( + self, + context: StorageDriverStoreContext, + payloads: Sequence[Payload], + ) -> list[StorageDriverClaim]: + ... + + async def retrieve( + self, + context: StorageDriverRetrieveContext, + claims: Sequence[StorageDriverClaim], + ) -> list[Payload]: + ... +``` + +Both context types expose a `target` attribute that identifies the owning execution. + +## Reading workflow vs activity metadata + +`context.target` is one of two concrete classes. Use `isinstance` to extract identity fields: + +```python +target = context.target +if isinstance(target, StorageDriverWorkflowInfo) and target.id: + # Workflow-scoped operation, including Activities started by a Workflow. + # Attributes: target.namespace, target.id (Workflow ID) + prefix = os.path.join(base_dir, target.namespace, target.id) +elif isinstance(target, StorageDriverActivityInfo): + # StorageDriverActivityInfo only fires for standalone (non-workflow-bound) + # activities. Activities started by a workflow use StorageDriverWorkflowInfo. + ... +``` + +Key facts: + +- The `StorageDriverActivityInfo` branch fires **only for [Standalone Activities](references/core/standalone-activities.md)** — Activities started directly from a Client without a Workflow. +- An Activity scheduled by a Workflow is reported on `StorageDriverWorkflowInfo`, with the orchestrating Workflow's ID. +- The docs describe the identity attributes as "namespace, Workflow ID, or Activity ID" depending on the operation. +- The transcribed snippet uses `target.id` for the Workflow ID and `target.namespace` for the namespace on the `StorageDriverWorkflowInfo` branch. + +## Driver selection + +When more than one driver is registered, you must pass a `driver_selector` callable to `ExternalStorage`. The selector receives the same `StorageDriverStoreContext` plus the payload and returns the driver to use — or `None` to keep the payload inline in Event History: + +```python +ExternalStorage( + drivers=[preferred_driver, legacy_driver], + driver_selector=lambda context, payload: preferred_driver, +) +``` + +Branch inside the selector on `context.target` the same way the driver does. + +## Common mistakes + +- **Falling through to a Workflow-only code path when `target` is `StorageDriverActivityInfo`.** Standalone Activities have no Workflow ID; check the branch explicitly. +- **Expecting a context on `EncodingPayloadConverter` or `PayloadCodec`.** The Python SDK's serialization-context object is only passed to the `StorageDriver` (and the `driver_selector` callable). To key behavior off Workflow/Activity identity, do it in the storage driver, not in a custom Payload Converter. +- **Changing a driver's `name()` after payloads have been written.** The SDK records the name on each claim; renaming breaks retrieval. + +## Resources + +- External Storage encyclopedia page: +- Python SDK External Storage guide: +- Cross-SDK concept file: `references/core/serialization-context.md`. +- Standalone Activities (Python): `references/python/standalone-activities.md`. diff --git a/references/python/workflow-streams.md b/references/python/workflow-streams.md index 84ecbd7..b53915b 100644 --- a/references/python/workflow-streams.md +++ b/references/python/workflow-streams.md @@ -11,7 +11,6 @@ Use it for modest fan-out progress streaming: AI-agent runs, order pipelines, mu Only available in the Python SDK today; cross-language is on the roadmap. - ## When to use / not to use - Use it for: updating a UI as an AI agent works; surfacing status from a payment or order pipeline; reporting intermediate results from a data job. diff --git a/references/ruby/gotchas.md b/references/ruby/gotchas.md index c2e962c..e56391d 100644 --- a/references/ruby/gotchas.md +++ b/references/ruby/gotchas.md @@ -59,7 +59,6 @@ require_relative 'activities/my_activity' Transient network errors should be retried. Authentication errors should not be. See `references/ruby/error-handling.md` to understand how to classify errors with `non_retryable: true` and `non_retryable_error_types`. - ## Heartbeating ### Forgetting to Heartbeat Long Activities