Skip to content

feat: launch ingesters per enabled network in the bootstrap (#117) - #155

Open
gideonpius7-design wants to merge 1 commit into
Miracle656:mainfrom
gideonpius7-design:feat/issue-117-per-network-ingesters
Open

feat: launch ingesters per enabled network in the bootstrap (#117)#155
gideonpius7-design wants to merge 1 commit into
Miracle656:mainfrom
gideonpius7-design:feat/issue-117-per-network-ingesters

Conversation

@gideonpius7-design

Copy link
Copy Markdown
Contributor

Summary

src/index.ts previously started exactly one instance of each of the five venue ingesters (SDEX, AMM, Soroswap, Snapshot, Aquarius), always bound to whatever network was active at startup. To index more than one network in the same process, each ingester now runs once per enabled network.

The per-network client factories (#139) and per-network venue config (#138) this builds on are already on main, so this diff is just the orchestration change.

Changes

  • src/network/enabledNetworks.ts (new): getEnabledNetworks() reads the comma-separated ENABLED_NETWORKS env var (e.g. testnet,mainnet), trims, lowercases, dedupes, and filters to known network names. When unset — or when nothing valid remains after filtering — it falls back to just the currently active network (STELLAR_NETWORK), so a single-network deployment starts exactly one instance of each ingester rather than one per network.
  • src/index.ts: loops over the enabled networks and starts SDEX, AMM, Soroswap, Snapshot, and Aquarius once per network. restartIngester's retry closure now captures the specific (venue, network) pair, so a crash in, say, Soroswap on mainnet only restarts that one instance and cannot affect the ingesters running on testnet.
  • src/ingesters/snapshot.ts: appendSnapshots() / startSnapshotIngester() now take a network argument. price_snapshots rows are network-scoped via the (network, pair, ts) primary key, so a single shared instance could only ever snapshot the active network — each enabled network now gets its own loop.
  • src/ingest/venues/aquarius.ts: startAquariusIngester() / ingestAquariusPair() take a network argument and resolve the per-network Aquarius block via getNetworkConfig(network) (enabled flag + API URL).
  • .env.example: documents ENABLED_NETWORKS.
  • src/__tests__/enabledNetworks.test.ts (new): covers the default fallback, explicit multi-network lists, whitespace/casing normalization, deduplication, and filtering out unrecognized network names.

Acceptance criteria (#117)

  • Each ingester runs once per enabled network
  • Restart logic isolates per-network instances (restartIngester keyed on (venue, network))
  • Single-network mode still starts five loops, not ten (ENABLED_NETWORKS unset → [activeNetwork])

Testing

  • npx tsc --noEmit passes
  • npx vitest run passes — 312 passed, 1 skipped (7 new tests in enabledNetworks.test.ts)

closes #117

🤖 Generated with Claude Code

…56#117)

src/index.ts previously started exactly one instance of each venue ingester
(SDEX, AMM, Soroswap, Snapshot, Aquarius), always bound to whatever network
was active at startup. To index more than one network in the same process,
each ingester now runs once per enabled network.

- src/network/enabledNetworks.ts (new): getEnabledNetworks() reads the
  comma-separated ENABLED_NETWORKS env var (e.g. "testnet,mainnet"), trims,
  lowercases, dedupes and filters to known network names. When unset — or
  nothing valid remains — it falls back to [activeNetwork], so a
  single-network deployment starts exactly one instance of each ingester.
- src/index.ts: loops over the enabled networks and starts each ingester
  with its network context. restartIngester now captures the specific
  (venue, network) pair, so a crash in one instance restarts only that
  instance and cannot affect the others.
- src/ingesters/snapshot.ts: appendSnapshots() / startSnapshotIngester()
  take a network arg. price_snapshots rows are network-scoped via the
  (network, pair, ts) PK, so a single instance could only ever snapshot the
  active network — each enabled network now gets its own loop.
- src/ingest/venues/aquarius.ts: startAquariusIngester() / ingestAquariusPair()
  take a network arg and resolve the per-network Aquarius block via
  getNetworkConfig(network).
- .env.example: documents ENABLED_NETWORKS.
- src/__tests__/enabledNetworks.test.ts (new): covers the fallback, explicit
  multi-network lists, whitespace/casing normalization, dedupe and filtering
  of unrecognized names.

closes Miracle656#117

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019ky4A59maw6dB2CptXyhdW
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Launch ingesters per network in the bootstrap

1 participant