Add position health updated event

[illia-malachyn]

Motivation

The FCM observer currently fetches positions’ health by running scripts (get_position_by_id.cdc). This approach doesn’t scale well.

For example, with ~10,000 positions, we need to execute 10,000 script calls. Access nodes are rate-limited (typically ~1–5 requests/sec), so processing all positions takes a significant amount of time.

Change

• Introduce an event-based approach for tracking position health.
• Instead of polling via scripts, we subscribe to events over a WebSocket (TCP) stream. This allows us to process a high volume of updates in real time, without being constrained by request rate limits.

Result

• Eliminates the need for per-position script calls
• Avoids access node rate limits
• Enables near real-time updates
• Scales much better with the number of positions

[holyfuchs]

The event is only emitted inside _queuePositionForUpdateIfNecessary, which is triggered on deposit, withdrawal, or async update loop processing. It won't fire when health drifts passively due to interest accrual or price oracle changes, which are arguably the most critical cases for the FCM observer to catch.
As-is, this doesn't fully replace polling for positions that aren't being actively touched.

Could get_position_by_id.cdc be extended to accept a list of position IDs and return health for all of them in a single script call? That would collapse 10k requests into far fewer batched calls and might be a simpler fix than the event approach.

Another option would be to calculate position health off-chain entirely.
To do this, the observer would need:

• the scaledBalance per token per position (only changes on withdraw, deposit)
• the pool's current interest indices (creditIndex / debitIndex)
• and the current oracle prices.
  With those three inputs, health can be computed as effectiveCollateral / effectiveDebt for any position without any script calls. The interest indices and oracle prices could be fetched periodically in a single read rather than per-position, making this approach scale to any number of positions.

[illia-malachyn]

@holyfuchs Thanks for starting this discussion!

"It won't fire when health drifts passively due to interest accrual or price oracle changes"**

• Price oracle changes: From what I can see in code, when the oracle is updated via setPriceOracle(), we explicitly queue all positions for update: self.state.setPositionsNeedingUpdates(self.positions.keys). These queued positions are then processed through asyncUpdate() -> asyncUpdatePosition() -> _queuePositionForUpdateIfNecessary().

• Interest accrual: If you're referring to a scenario where interest accrues but no transaction touches the position - how would the script-based approach catch this? Scripts read on-chain state, and if the health factor isn't recalculated on-chain, the script will return stale data as well.

"Could get_position_by_id.cdc be extended to accept a list of position IDs?"**

Yes, we could write a batch script, but it only mitigates the problem rather than solving it:

• Scale ceiling is unknown: I don't know the upper bound of how much data a single script execution can return. With 100k positions, if the script can only handle 500-1000 at a time, we're still making hundreds of calls and falling behind. For our liquidation monitoring (currently manual with Grafana alerts), being slow to detect unhealthy positions could be critical.
• Polling vs streaming: Batch scripts are still polling - we'd be trading many small requests for fewer large requests, but we're still bound by polling intervals and access node rate limits. Events over WebSocket give us a real-time push-based stream with no rate limit concerns.

off-chain health calculation

I think it adds significant complexity - we'd need to replicate the health calculation logic off-chain, keep it in sync with any contract changes, and independently fetch and combine interest indices + oracle prices + per-position balances. I think events are a much simpler and more maintainable approach, and I'd prefer to go with it and only fall back to scripts if events turn out not to be doable for some reason.

[holyfuchs]

Price oracle changes: From what I can see in code, when the oracle is updated via setPriceOracle(), we explicitly queue all positions for update

Yes, but this only queues positions when the PriceOracle contract is swapped out, not when prices change. That will extremely rarely happen. The much more common case — prices moving continuously — is not covered.

Interest accrual: If you're referring to a scenario where interest accrues but no transaction touches the position - how would the script-based approach catch this?

Scripts can't persist state changes, but they do reflect them during execution. getPositionDetails calls _borrowUpdatedTokenState which calls state.updateForTimeChange() — so the interest accrual is applied transiently within the script call and the returned health value is accurate, even though nothing is written on-chain. So a batch script would actually catch interest-driven health drift correctly.

Events

Position health changes constantly — every second due to interest accrual, and on every price tick. Events are only useful if you emit one regularly for every position. You could add a function that iterates all positions and emits health events on a schedule, but unlike a script this isn't free — each event costs gas, so at any meaningful scale this gets expensive fast. Scripts are cheaper here precisely because they don't write to chain.

Off-chain health calculation

I agree it definitely adds a significant amount of complexity but I don't think there is a nice solution here.

[illia-malachyn]

@holyfuchs thanks for the reply!

After syncing with @vishalchangrani, it looks like tracking health factors for all positions is only a temporary requirement. Given that, we probably don’t need to worry about scaling here and can stick with a script-based approach for now.

That said, there’s a separate issue: fcm-observer is stateless, so after a restart/redeploy we lose the IDs of previously opened positions. I see a couple of ways to recover them:

1. Script to return all open position IDs. (sth like Add function and script that return IDs of open positions #289)
   Drawback: scripts have a 100k gas/computation limit, so not sure this would fit even if the number of positions is not large.
   I don't even know how to estimate the script computation, actually. I tried calling Access Node to get the info about that, but all of them returned 0/100000 computation used for a simple script like get_position_by_id.cdc
2. Script to return the latest position ID, then iterate from 0 to latest. (sth like Add Pool.getPositionCount() and get_position_count script #270)
   We’d register scripts for each ID and filter out closed positions along the way.
   Drawbacks: more complex, and startup could be slow since we’d scan through a lot of already closed positions.

Curious if you have thoughts on this, or if there’s a better pattern I'm missing here.

[holyfuchs]

Ran a small test and for bigger resource in the map it fails at > 150k keys. So adding that function should be fine.
For smaller resources it works until around 400k keys.

[zhangchiqing]

I wonder how do we ensure that we've captured all the cases where the position health is updated. Note, this function has a few returns that skips this emit.

And is it possible the position health will be updated outside of this function?

This function only queues the update, asyncUpdatePosition is actually the function updating the position. Should we emit event there instead?

And this is when the position is updated from internally. What about externally? For instance, since position health depends on the price oracle, so the position health is changed as soon as the price oracle updates the price.

[illia-malachyn]

Your concerns are valid and discussed in this PR. This PR is to be closed. We're going to gather position health factors via a script.

[illia-malachyn]

We decided to move on with a script-based approach.