Spike: Persistent, multi-instance-safe storage strategy to replace in-memory service state

[meshackyaro]

Description

Every domain service in the backend — EscrowService, GigService, UserProfileService, IpfsPinningService, and the auth NonceStoreService — stores its state in a process-local in-memory Map. Only rate-limiting (src/common/rate-limit/) actually uses the Redis client that's wired up in src/common/redis/redis.module.ts.

This means:

• All application state (open gigs, escrows, user profiles, pinned CIDs) is lost on every restart/deploy.
• Running more than one instance behind a load balancer causes state to diverge — a request hitting instance A won't see a gig created on instance B.
• Background sweep workers (GigExpiryWorkerService, RepinWorkerService) each run their own setInterval per process, so N instances means N-way duplicate sweeps and N-way duplicate webhook dispatches for the same expiring/degraded record.

The migration framework (src/migration/) already models schema migrations against a stand-in in-memory "gigs" table, suggesting persistence is planned but not yet decided.

Component

Backend

Difficulty

🟣 Spike — investigation required, scope not yet defined

Tasks

• Inventory every in-memory store and its access patterns (read/write frequency, query shapes)
• Evaluate persistence options (Postgres, DynamoDB, etc.) against the existing migration framework's expectations
• Design a leader-election or distributed-lock strategy so sweep workers run exactly once cluster-wide (e.g. Redis-backed lock, given Redis is already a hard dependency)
• Prototype migrating one service (recommend GigService, the newest/smallest) to the chosen persistence layer
• Document the migration path for the remaining services

Acceptance Criteria

• A written recommendation for the persistence layer and the singleton-worker strategy
• A working prototype of at least one service migrated off in-memory storage
• Follow-up implementation issue(s) filed per remaining service
• Any blocking unknowns (infra cost, ops burden) explicitly flagged

Estimated Time

2-3 days (time-boxed spike)

[balisdev]

Hi maintainer, I’d like to claim this spike issue.
This is a critical architectural transition for horizontally scaling the backend and preventing state loss. Here is my planned breakdown to execute this spike:

1. Persistence Layer Evaluation & Strategy
   State Audit: Map out query shapes, indexing requirements, and read/write load for EscrowService, GigService, UserProfileService, IpfsPinningService, and NonceStoreService.

Storage Options: Evaluate Postgres (relational consistency for financial/escrow states and complex queries) vs. Redis/NoSQL alternatives against the existing src/migration/ framework expectations, weighing setup/ops friction vs. consistency guarantees.

2. Cluster-Wide Singletons & Locking

Utilize the existing Redis client (src/common/redis/) to prototype a distributed lock (e.g., Redlock pattern or Redis-backed leader election) for GigExpiryWorkerService and RepinWorkerService.

Ensure background jobs execute strictly once across $N$ replica nodes to prevent duplicate webhook dispatches and race conditions.

3. Execution & Deliverables
   Prototype: Migrate GigService as the pilot service, replacing its process-local Map with the recommended persistent layer + unit/integration tests to ensure state survives process bounces.

Documentation & Follow-ups: Deliver a clean RFC/Spike document outlining infra/ops trade-offs, alongside structured follow-up implementation issues for each remaining service.

Could you please assign this to me? I'll open a draft PR/RFC with the initial storage matrix and lock prototype once ready.

[grantfox-oss]

🦊 GrantFox — @balisdev has been assigned to this issue as part of the Third Campaign campaign!

Next steps:

1. Open a Pull Request referencing this issue (e.g., Closes #181)
2. Your PR will be reviewed by the trustflow-protocol maintainers

Good luck! Track your progress on GrantFox.