[Bug]:Account/user created on one instance is invisible to other instances in multi-instance deployment

[xialuodba-bit]

Bug Description

When OpenViking is deployed with multiple instances behind a load balancer, account and user information created through the admin API on one instance is not propagated to other instances. Subsequent requests routed to a different instance fail authentication because
the in-memory cache of that instance does not contain the newly created account/user.

Steps to Reproduce

1. Deploy two OpenViking instances (Instance A and Instance B) behind a round-robin load balancer sharing the same AGFS/VikingFS storage
2. Call POST /api/v1/admin/accounts or POST /api/v1/admin/accounts/{account_id}/users — the request lands on Instance A
3. Instance A writes the new account/user to AGFS and updates its own in-memory APIKeyManager._accounts
4. Send an authenticated request using the new API key — the request is routed to Instance B
5. Instance B has no knowledge of the new account/user (its cache was loaded at startup and never refreshed)

Expected Behavior

All instances share a consistent view of accounts and users. A newly created account/user should be immediately usable regardless of which instance handles the next request.

Actual Behavior

Instance B returns an authentication error (401) for the new API key because APIKeyManager._accounts on that instance is stale. The account/user only exists in AGFS (persisted) and in Instance A's memory.

Minimal Reproducible Example

# Instance A: create a new user
  import requests
  resp = requests.post("http://instance-a/api/v1/admin/accounts/acct1/users", ...)
  api_key = resp.json()["api_key"]

  # Next request hits Instance B via load balancer
  resp = requests.get("http://load-balancer/api/v1/...", headers={"Authorization": f"Bearer {api_key}"})
  # → 401 Unauthorized

Error Logs

# Instance B logs
  AuthenticationError: API key not found

OpenViking Version

v0.3.22

Python Version

3.12

Operating System

Linux

Model Backend

None

Additional Context

Three failure modes exist:

┌─────────────────────┬──────────────────────────────────────────────────┐
│ Operation │ Effect on other instances │
├─────────────────────┼──────────────────────────────────────────────────┤
│ Create account/user │ New credentials rejected (401) │
├─────────────────────┼──────────────────────────────────────────────────┤
│ Regenerate API key │ Old key still accepted (security risk) │
├─────────────────────┼──────────────────────────────────────────────────┤
│ Delete user │ Deleted user still authenticated (security risk) │
└─────────────────────┴──────────────────────────────────────────────────┘

Workaround: Route all admin API requests to a single designated instance, and restart other instances after any account change.

[MaojiaSheng]

you should have a destributed filesystem like minio/rustfs

[xialuodba-bit]

you should have a destributed filesystem like minio/rustfs

The suggestion to use a distributed filesystem like minio/rustfs addresses the wrong layer of the problem.

The root cause is in-memory cache staleness, not storage persistence. Each instance loads account data into memory at startup (APIKeyManager._accounts) and never refreshes it. When Instance A creates a new account via the admin API, Instance B's in-memory cache remains
stale — it has no knowledge of the change regardless of what filesystem is used.

Even with minio/rustfs:

1. The write would land on shared storage ✓
2. But Instance B's in-memory cache is still the old data until it restarts ✗

So the distributed filesystem only solves persistence sharing, not cache synchronization — which is the actual problem here.

Better fixes:

• Short-term / simple: Skip the cache on every auth check and read directly from storage. Slight performance cost, but correct.
• Proper fix: Add a cache invalidation mechanism — after any write, broadcast an invalidation signal to other instances (e.g. Redis pub/sub, a message queue).
• Cleanest fix: Store account data in a shared backend (Redis, MySQL, etc.) so all instances read from the same source of truth instead of relying on startup-time local snapshots.

The filesystem is orthogonal to this problem. The real issue is that there's no synchronization mechanism between instances when account data changes.

[zhoujh01]

Thank you for your feedback. We will schedule a resolution for this issue in the near future. We also welcome community members to share their code.