Pure-Go SDK vs. cgo Go SDK: customer-facing differences & should we make them source-compatible?

[zlata-stefanovic-db]

We now have two Go SDKs in the repo — the original cgo SDK (go/, imports the Rust core over cgo + FFI) and the new pure-Go SDK (purego/zerobus, native gRPC, no cgo). They talk to the same Zerobus service and share the same performance model (ingest in a loop, Flush() once), but the public APIs are not source-compatible: every type and most method signatures differ.

I wrote up the customer-facing differences and want to open a discussion on one question in particular: should we make the two source-compatible, and if so, how? Summary below — feedback welcome.

Where they differ (what a user actually notices)

Build & runtime

• Pure-Go: no cgo, no C compiler, no Rust, no prebuilt native libs; cross-compiles like any Go program. Requires Go 1.25.
• cgo: needs a C compiler (and Rust when built from source), limited to platforms with prebuilt libs. Builds on Go 1.21.

Naming / entry points (nothing is a drop-in swap)

Concept	Pure-Go	cgo
SDK type / ctor / teardown	SDK / New / Close() error	ZerobusSdk / NewZerobusSdk / Free()
Stream type	Stream	ZerobusStream
Error type	Error	ZerobusError
Options	functional (WithJSON(), WithRecovery(...))	StreamConfigurationOptions struct
User-agent	zerobus-sdk-go-purego/<ver>	zerobus-sdk-go/<ver>

API shape

• Payloads: pure-Go is uniformly []byte / [][]byte; cgo is interface{} (string for JSON, []byte for proto).
• Context: pure-Go takes context.Context on CreateStream and offers ...Context variants on ingest/flush/wait; cgo has none.
• Config units: pure-Go functional options with time.Duration; cgo struct fields in millisecond uint64.
• Errors: pure-Go *Error with Op + Unwrap + package-level Retryable(err); cgo *ZerobusError{Message, IsRetryable} via type assertion.
• Custom auth: HeadersProvider differs — pure-Go GetHeaders(ctx, tableName) + Invalidate(...); cgo GetHeaders().

Feature gaps (either direction)

• Pure-Go only: ack callback (WithAckCallback), async open + WithWaitForReady, custom TLS (WithTLSConfig), byte-based backpressure caps, Stream.ID()/ServerID()/IsClosed(), NewStaticHeadersProvider, and dynamic-proto with UC schema fetch (CreateDynamicProtoStream / IngestJSONStringOffset).
• cgo only: Arrow Flight ingestion (Beta), plus the deprecated IngestRecord/RecordAck/*Nowait surface.

Should we make them source-compatible?

My take: don't rename the pure-Go API to match cgo. The pure-Go surface is deliberately cleaner (context, byte payloads, functional options, errors.Is/As, no deprecated methods). Matching cgo would drag its legacy warts in permanently.

Instead, if we want to cut migration cost, add an opt-in compat subpackage (e.g. .../purego/zerobus/compat) that re-exposes the cgo names and delegates to the pure-Go core. Existing users change one import path; new users get the idiomatic API.

What a shim would translate

Mismatch	Shim translation
Type names	Wrapper types (or aliases where shapes match)
Free()	Calls Close(), discards the error
CreateStream(TableProperties, ...) (no ctx)	Call ctx variant with context.Background(); DescriptorProto → WithProto(...)
StreamConfigurationOptions	Field-by-field → WithX options; *Ms uint64 → time.Duration
RecordType field	→ WithJSON() / WithProto(desc)
interface{} / string payloads	Type-switch to []byte
*ZerobusError	Wrap pure-Go error: Message: err.Error(), IsRetryable: zerobus.Retryable(err)
HeadersProvider.GetHeaders()	Adapt to GetHeaders(ctx, tableName); no-op Invalidate
IngestRecord/RecordAck/*Nowait	Reimplement on IngestRecordOffset/IngestRecordsOffset

Hard blockers a shim can't fix

• Arrow Flight — no pure-Go equivalent yet, so cgo Arrow users can't migrate.
• Go 1.25 minimum — a project pinned below 1.25 can't switch regardless of API.
• Behavioral, not signature, differences — async-by-default open (failures surface later on Flush/WaitForOffset) and MaxInflight counting ingest calls vs requests mean identical-looking code can behave differently.
• Error identity — wrapping *Error as *ZerobusError preserves Retryable() but changes the concrete type for errors.As.

Questions for discussion

1. Is a compat subpackage worth maintaining, or should cgo→pure-Go just be a documented one-time migration?
2. If we do it, how long should compat live — permanent, or a deprecation bridge?
3. Are the behavioral differences (async open, inflight semantics) acceptable under a "compatible" banner, or disqualifying?

[zlata-stefanovic-db]

@teodordelibasic-db Discussion about Pure-Go vs. cgo Go SDK