Boiler blate code to enable getLedgerEntries, simulateTransaction, sendTransaction on v2 (#884). Wiring comes in (#889) - #903
Merged
karthikiyer56 merged 8 commits intoJul 31, 2026
Conversation
) These three endpoints read CURRENT ledger state, which lives in captive core's bucket list rather than in v2's stores. The existing implementations (internal/preflight, internal/ledgerentries) are already written against internal/host's interfaces, so v2 needs one concrete Daemon over its own captive core — not reimplemented handlers. host.Daemon.GetCore() handed out the whole SDK core object for the sake of one version string, which v2's stream deliberately hides (it runs a fresh core per supervised run). Narrowing it to CoreVersion() lets both daemons answer from the configured binary path alone. Core's HTTP ports go on the LIVE core's toml only. A no-lake backfill starts one bounded-replay core per chunk in parallel, and offline mode drops HTTP_PORT but not HTTP_QUERY_PORT — so a shared toml would have those cores contending on the query port.
Setting HTTPQueryServerParams for the first time made an SDK path reachable that PANICS instead of reporting a conflict: its query-server mismatch branches format the error with params.PeerPort, which this daemon never sets. Peek the four core-HTTP keys out of the captive-core file and reject a disagreement here, with a message naming both sides, before the SDK ever compares them. The backfill toml was only portless because no test fixture declared ports. The SDK keeps a file-declared HTTP_QUERY_PORT, and the bounded-replay config clears only HTTP_PORT — so parallel replays would have contended on the query port after all. Clear the fields outright rather than relying on absent params. core_url gains form validation: a missing scheme otherwise started fine and then failed on every submission while the daemon still reported healthy.
The design doc's config tables had drifted into implementation notes — flag machinery internals, an SDK panic, validation rules — the opposite of what that doc is for. It now lists the keys and the one design decision behind them: core's HTTP ports go on the live core only. Code comments say the same things in fewer, plainer words. Config struct fields keep their explanations, since the schema is what needs describing; test functions lose docstrings their names already carry.
v2 serves nothing today, and #889 alone does not change that: the handlers it registers need the router-backed store readers, so serving arrives across the epic's sub-tasks rather than at one ticket.
The captive-core file's HTTP_PORT, HTTP_QUERY_PORT, QUERY_THREAD_POOL_SIZE and QUERY_SNAPSHOT_LEDGERS are now overridden with a warning rather than refused. A conflict check compared post-default values, so a core file with QUERY_THREAD_POOL_SIZE set and no [ingestion] value started on one host and failed on another purely by CPU count. The four settings are written onto each generated toml instead of passed as CaptiveCoreTomlParams, which also keeps the SDK's nil-PeerPort panic branches unreachable. Also: - reject a loopback core_url whose port is not core_http_port; it can only reach some other process, often another daemon's captive core - bound the stellar-core version read with a context and a 5s timeout - turn enable_debug off in the sample config
There was a problem hiding this comment.
Pull request overview
Adds v2 plumbing for captive-core-backed RPC methods without exposing the handlers yet.
Changes:
- Adds v2 core clients, preflight workers, and live/backfill core configuration.
- Extends configuration, validation, flags, documentation, and tests.
- Narrows the host daemon interface and enables Rust preflight linkage.
Reviewed changes
Copilot reviewed 22 out of 22 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
Makefile |
Links v2 with Rust libraries. |
design-docs/full-history-streaming-workflow.md |
Documents new configuration. |
cmd/stellar-rpc/rpcv2/sample_config_test.go |
Tests sample defaults. |
cmd/stellar-rpc/rpcv2/rpc-v2-sample-config.toml |
Adds example settings. |
cmd/stellar-rpc/internal/rpcv2/startup.go |
Updates server references. |
cmd/stellar-rpc/internal/rpcv2/helpers_test.go |
Adds log capture helper. |
cmd/stellar-rpc/internal/rpcv2/health.go |
Updates server references. |
cmd/stellar-rpc/internal/rpcv2/daemon.go |
Wires core state and preflight. |
cmd/stellar-rpc/internal/rpcv2/daemon_test.go |
Tests core opener behavior. |
cmd/stellar-rpc/internal/rpcv2/corestate/corestate.go |
Implements v2 host daemon. |
cmd/stellar-rpc/internal/rpcv2/corestate/corestate_test.go |
Tests core-state integration. |
cmd/stellar-rpc/internal/rpcv2/config/flags.go |
Adds Boolean flags. |
cmd/stellar-rpc/internal/rpcv2/config/flags_test.go |
Tests new flags. |
cmd/stellar-rpc/internal/rpcv2/config/config.go |
Defines new configuration. |
cmd/stellar-rpc/internal/rpcv2/config/config_test.go |
Tests defaults and cascading. |
cmd/stellar-rpc/internal/rpcv2/config_validate.go |
Validates core and preflight settings. |
cmd/stellar-rpc/internal/rpcv2/config_validate_test.go |
Tests validation rules. |
cmd/stellar-rpc/internal/rpcv1/daemon/metrics.go |
Adapts v1 core version access. |
cmd/stellar-rpc/internal/methods/get_version_info.go |
Uses narrowed daemon API. |
cmd/stellar-rpc/internal/host/noOpDaemon.go |
Updates no-op implementation. |
cmd/stellar-rpc/internal/host/host.go |
Narrows the daemon interface. |
cmd/stellar-rpc/docker/Dockerfile.rpcv2 |
Documents required Rust tooling. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+163
to
+164
| if host == "localhost" { | ||
| return true |
| ctx, cancel := context.WithTimeout(ctx, coreVersionTimeout) | ||
| defer cancel() | ||
|
|
||
| out, err := exec.CommandContext(ctx, binaryPath, "version").Output() |
Comment on lines
+146
to
+156
| // This daemon starts the core it submits to, and that core binds | ||
| // core_http_port, so a local core_url on any other port reaches something | ||
| // else — often a second daemon's captive core, which would accept the | ||
| // transaction — with neither startup nor getHealth reporting a problem. A | ||
| // non-local core_url is left alone: that is the reason to set it at all. | ||
| if isLoopbackHost(u.Hostname()) && u.Port() != strconv.FormatUint(uint64(*ing.CoreHTTPPort), 10) { | ||
| return fmt.Errorf("[ingestion].core_url %q points at this host but not at %s (%d) — "+ | ||
| "the captive core this daemon starts listens on that port. Leave core_url unset to "+ | ||
| "derive it, or give it a non-local address if core's admin server really is elsewhere", | ||
| ing.CoreURL, keyCoreHTTPPort, *ing.CoreHTTPPort) | ||
| } |
Comment on lines
+136
to
+139
| ctx, cancel := context.WithCancel(context.Background()) | ||
| cancel() | ||
|
|
||
| d, err := New(ctx, cfg) |
tamirms
approved these changes
Jul 30, 2026
tamirms
left a comment
Contributor
There was a problem hiding this comment.
I think the copilot findings are worth reviewing. But apart from that the PR looks good!
Contributor
Author
done. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TL;DR
serveReadsis a no-op stub. This PR builds the plumbing the three captive-core endpoints need. Registering them is Wire the v2 binary's JSON-RPC handlers #889, and their handlers also need the router-backedstore.LedgerReader(LedgerReader + LedgerReaderTx adapter #885–EventReader adapter #887).host.Daemonover its own captive core (newinternal/rpcv2/corestate), so Wire the v2 binary's JSON-RPC handlers #889 can run v1's existinginternal/preflightandinternal/ledgerentriescode unmodified.host.Daemon.GetCore() *ledgerbackend.CaptiveStellarCorenarrows toCoreVersion() string— the whole SDK core object was being handed out for one version string.[ingestion]core-HTTP keys, and[service.preflight]— including the flag machinery's first bool.v1 and v2 both take these four settings from RPC config and write them into the captive-core toml. They differ in what happens when the operator's captive-core file also names them.
NewCaptiveCoreTomlFromFiletoml.go'svalidate:HTTP_PORT in captive core config file: %d does not match passed configuration (%d). It is never a silent override.core_http_query_thread_pool_sizedefaults toNumCPU(). Under a conflict check, a core file withQUERY_THREAD_POOL_SIZE = 8and no[ingestion]value starts on an 8-CPU host and refuses to start on a 16-CPU host — the operator set one value, and a default disagreed with them. Only this key is machine-dependent; the other three compare against fixed defaults.params.PeerPort, which this daemon never sets, so a disagreeing file panics rather than reporting the conflict. v2 passes no HTTP params at all, which makes those branches unreachable.HTTP_PORT = 11625moves core to11626, so firewall rules or monitoring scrapes pointed at the old port need updating. The warning names both values:NETWORK_PASSPHRASEis unchanged and still owned by the captive-core file — the ownership split is deliberate, not blanket.What a running daemon actually does differently
[ingestion]overrides the captive-core fileHTTP_PORT,HTTP_QUERY_PORT,QUERY_THREAD_POOL_SIZEorQUERY_SNAPSHOT_LEDGERSis overridden, with a warning naming both values. Startup is not blocked.core_request_timeout, a malformedcore_url, a localcore_urlthat doesn't matchcore_http_port(or useshttps— the local core speaks plain HTTP), acore_urlport past 65535, or a zero preflight worker count all fail startup.corestate.Newstellar-core versiononce, under a 5s timeout tied to the daemon's context (plus a 1sWaitDelay, so a wrapper's child holding stdout can't stall past the kill).worker_countidle goroutines and registers 5 Prometheus collectors.Inert until #889 — the request paths. Nothing calls
CoreClient(),FastCoreClient(),LedgerEntryGetter()orpool.GetPreflight(), so the clients never issue a request and the pool never gets a job.Why a Daemon and not three handlers
sendTransactionCoreClient→ admin portgetLedgerEntriesFastCoreClient→ query portsimulateTransactionlibpreflightin-processFastCoreClientAll three already sit behind
internal/host's interfaces. This PR supplies the missing implementation; #889 assembles the method table over #885's reader.New config keys (15 keys, all with v1 counterparts)
[service.methods.<method>]— three new tables, sameMethodConfigshape:sendTransaction.queue_limit500request-backlog-send-transaction-queue-limitsendTransaction.max_execution_duration"15s"max-send-transaction-execution-durationsimulateTransaction.queue_limit100request-backlog-simulate-transaction-queue-limitsimulateTransaction.max_execution_duration"15s"max-simulate-transaction-execution-durationgetLedgerEntries.queue_limit1000request-backlog-get-ledger-entries-queue-limitgetLedgerEntries.max_execution_duration"5s"max-get_ledger-entries-execution-duration[ingestion]— core's HTTP surface, placed with the process it configures:core_http_port11626stellar-captive-core-http-portcore_urlhttp://localhost:{core_http_port}stellar-core-urlcore_request_timeout"2s"stellar-core-timeoutcore_http_query_port11628stellar-captive-core-http-query-portcore_http_query_thread_pool_sizeNumCPU…-http-query-thread-pool-sizecore_http_query_snapshot_ledgers4…-http-query-snapshot-ledgers[service.preflight]— serving policy, like[service.fee_stats]:worker_countNumCPUpreflight-worker-countworker_queue_sizeNumCPUpreflight-worker-queue-sizeenable_debugtruepreflight-enable-debug--dotted.pathCLI flag, free from the reflection-derived flag set.core_request_timeoutjoins the ≥1ms nanosecond-typo guard next to[backfill.bsb].retry_wait.core_urlis derived after flags are overlaid, so--ingestion.core_http_port=31626alone moves the submission URL with it.enable_debug = false— a deliberate departure from thetruedefault, since the sample is a production starting point.What changed where
internal/host/host.go,noOpDaemon.goGetCore()→CoreVersion() string; theledgerbackenddependency leaves the interfaceinternal/methods/get_version_info.godaemon.CoreVersion()inside the handler closureinternal/rpcv1/daemon/metrics.god.core.GetCoreVersion()internal/rpcv2/corestate/(new)Daemon: two*stellarcore.Clients at the two ports, registry/namespace, version string.LedgerEntryGetter(store.LedgerReader)is the seam #889 calls, pinning entry reads to the ledger v2 last committedinternal/rpcv2/config/config.go[service.preflight], six[ingestion]core keys, sharedNumCPU()helperinternal/rpcv2/config/flags.gokindBoolinleafKind/BindFlags/setLeaf,GetBoolonFlagOverridesinternal/rpcv2/config_validate.govalidateIngestion+validatePreflight; the three methods joinvalidateServiceinternal/rpcv2/daemon.goresolveCorereturnsresolvedCore{live, backfill, networkPassphrase, binaryPath}; both openers come from ONE read of the captive-core file; corestate + preflight pool built once, outside the supervised loopMakefile,docker/Dockerfile.rpcv2build-rpc-v2: build-libs; the rust stage's now-wrong "not needed yet" comments are gonefriendbot_urlWhy core's HTTP ports land on the live core only
runDaemonWithhands oneCoreOpenerto two consumers: the live ingestion loop, and (with no lake)captiveSource, which starts a fresh bounded-replay core per chunk, in parallel.HTTP_PORTbut notHTTP_QUERY_PORT, so the collision is real.applyHTTPServerssets all four settings on the live toml,disableHTTPServersclears them on the backfill toml.CaptiveCoreTomlParams. The SDK treats params as a cross-check rather than an override, and its three query-server mismatch branches format the error withparams.PeerPort— always nil here — so they panic instead of reporting. Passing no HTTP params leaves those branches unreachable.Design decisions
[ingestion]overrides the captive-core file's four HTTP keys, with a warning, rather than rejecting the config•
core_http_query_thread_pool_sizedefaults toNumCPU, so a conflict check would refuse to start on a host with a different CPU count than the one the config was written on• the warning keeps the override visible: a v1 file with
HTTP_PORT = 11625may have firewall rules pointed at 116251..65535and must differ; v1's "0disables core's HTTP server" is rejected• a disabled server is a broken deployment, not a mode
core_urlmust namecore_http_port• non-local
core_urls are left alone; that's the reason to set onegetVersionInfocallsdaemon.CoreVersion()per requeststellarcore.Clients; v1'sCoreClientWithMetricstxsub summaries not reproducedinternal/rpcv1/daemon, not a shared package•
TODO(#889)at the client, since #889 is what wires sendTransactionstellar-core versionlookup logs and yields""setCoreVersion• bounded by
exec.CommandContext+ 5s, so a wrapper script or stalled mount can't freeze startup before anything is loggedresolveCorereturns two openers instead of wideningCoreOpenerrunDaemonWithwithdefer Close()• built outside
supervise, since re-registering the pool's collectors panicshttp.Clients, one per core serverhttp.DefaultTransport, whose pool already keys connections byhost:port• kept separate so #889 can wrap the submission client alone for v1's txsub metrics
enable_debugis an ordinary pflag bool--service.preflight.enable_debug=falseturns it off; a bare flag sets true• the flag's own
falsedefault never leaks in, sinceApplyFlagsreads only flags the user setFixes from the branch code review
HTTPQueryServerParamsmakes an SDK path reachable that panics on a nil*params.PeerPortSIGSEGVatledgerbackend/toml.go:728with a core file declaringHTTP_QUERY_PORT = 11111HTTP_QUERY_PORT, andCatchupToml()clearsHTTP_PORTbut never the query portHTTP_QUERY_PORTdisableHTTPServersclears all four fields, the same mechanismCatchupTomluses. Test asserts on the marshalled toml, direct and catchupcore_urlhad no form validation, socore.internal:11626started fine and failed per request while health looked greenvalidateIngestionrequires anhttp/httpsscheme and a host, and rejects a loopbackcore_urlwhose port isn'tcore_http_portQUERY_THREAD_POOL_SIZEin a core file could start on one host and fail on another purely by CPU countreadCoreVersionranstellar-core versionwith no context or timeout, on the startup path before any log lineexec.CommandContextwith a 5s cap, bound to the daemon's contexthttp.Clients so a slow submission cannot tie up a connection the query path needs" — both sharehttp.DefaultTransport, so that isn't what the split buysTODO(#889)— needs v1's wrapper lifted out ofinternal/rpcv1/daemonfirstFixes from the Copilot review
CommandContextkills only the direct process; a wrapper's child holding stdout keepsOutputwaiting past the 5s timeoutcmd.WaitDelay = time.Secondforces the pipes closed after the killNew, so the script never started and the timeout was never exercisedisLoopbackHostmissedLOCALHOSTandlocalhost.https://localhost:11626passed validation but the local core speaks plain HTTP, so every submission would fail its TLS handshakehttpsto a loopbackcore_urlis a startup errorhttp://core.internal:70000) parsed fine and failed only at dial timecore_urlports are range-checked 1–65535Testing
go build ./...go test ./cmd/stellar-rpc/...(incl. the ~3min goleak-guarded rpcv2 suite)golangci-lint run ./cmd/stellar-rpc/...int → uint, so nolintlint calls the//nolint:goseconuint(runtime.NumCPU())unused. CI's pinned 2.11.3 needs the directive; v1 carries the identical line atrpcv1/config/options.go:40with the same skewmake build-rpc-v2nm stellar-rpc-v2shows_preflight_invoke_hf_op, so the rust bindings are really in the binaryNew tests:
corestate— client URLs/timeouts per port, registry identity, incomplete-config rejection, version read from a fakestellar-core versionbinary, empty on a missing binary, a live deadline proving a hanging binary is killed rather than waited out, and a wrapper whose backgrounded child holds stdout open (theWaitDelaycase). Anhttptestserver provesLedgerEntryGetterhits the query port asking for the daemon's own latest ledger.rpcv2— the live-vs-backfill port split (including when the core file declares ports itself), four override cases asserting the live toml's value plus the warning, no warning when the file already agrees, injected-opener-serves-both-roles, and the[ingestion]/[service.preflight]/core_urlrejection cases.config— defaults and cascade participation for the three new tables, preflight defaults incl. an explicitfalse, core-HTTP defaults,core_urlderivation, and the bool flag override.