Skip to content

[fal.ai/livepeer-staging] Trickle subscriber GET fails mid-session — sub cannot complete GET on segment /10 #915

@livepeer-tessa

Description

@livepeer-tessa

Summary

The livepeer_gateway.trickle_subscriber is throwing an error during an active session while trying to GET a trickle segment from the orchestrator. This is distinct from the publisher errors (#912, #846) — the subscriber side (reading processed output back from the orch) is failing mid-session.

cc @mjh1 @emranemran

Error Logs (Grafana/Loki — 2026-04-11 ~14:06 UTC)

Session manifest_id: 2153e39d
fal_app_name: github_f1lhgmk5v76a0ev1w0u378by-scope-livepeer

2026-04-11 14:06:55,287 - livepeer_gateway.trickle_subscriber - ERROR - Trickle sub failed to complete GET https://orch-staging-1.daydream.monster:8935/ai/trickle/2153e39d-1-in/10

  File "/app/.venv/lib/python3.12/site-packages/livepeer_gateway/trickle_subscriber.py", line 150, in _preconnect

Context

  • Session started at 14:00:57 UTC (subscribing to control channel 2153e39d-control)
  • Error occurs ~6 minutes in at 14:06:55 UTC, on segment /10 of the 2153e39d-1-in stream
  • The session was actively publishing successfully before this point (seen from the MediaPublishStats logs on the companion scope-livepeer job running at the same time: 300+ segments completed, 0 failed)

Analysis

The subscriber is attempting to GET the orchestrator's input segment /10 but the GET fails. This could be:

  1. A network blip between the fal.ai worker and orch-staging-1.daydream.monster:8935
  2. The orchestrator dropped the trickle stream / stream not found on that endpoint
  3. A timeout in _preconnect before the segment is available

The failure occurs on the input stream (-1-in) rather than the output, suggesting the subscriber is feeding processed video back into the session pipeline and the orch-side endpoint has gone away.

Affected Components

  • livepeer_gateway.trickle_subscriber._preconnect (line 150)
  • Orchestrator: orch-staging-1.daydream.monster:8935

Impact

  • Subscriber cannot receive processed frames from orchestrator
  • Session may continue running but output will stall/degrade
  • No automatic recovery observed in logs

Related Issues

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions