Delegate RunLogsService.TailLogs to DataProxyService to remove duplicate implementation

[pingsutw]

What

RunLogsService.TailLogs (runs/service/run_logs_service.go:44) and dataproxy.Service.TailLogs (dataproxy/service/dataproxy_service.go:503) are essentially doing the same thing:

1. Resolve the LogContext for the given action attempt.
2. Hand it to a LogStreamer which streams pod logs back to the client.

The canonical home for this is now the DataProxy service. The RunLogsService.TailLogs RPC is only kept for backward compatibility with existing clients.

Why

Today both services:

• Depend on a LogStreamer implementation (runs/service/k8s_log_streamer.go and dataproxy/logs/k8s_log_streamer.go — two copies).
• Implement their own concurrency limit / context resolution code path.
• Must be kept in sync whenever we evolve log streaming behavior (filtering, pagination, new backends, auth, etc.).

This is duplicate surface area with no good reason now that DataProxy owns logs.

Proposed change

Refactor RunLogsService.TailLogs into a thin shim that forwards to DataProxyServiceClient.TailLogs:

• Inject a dataproxyconnect.DataProxyServiceClient into RunLogsService instead of (or in addition to) the LogStreamer.
• In TailLogs:
  1. Validate action_id / acquire the concurrency semaphore as today.
  2. Open a client stream to DataProxyService.TailLogs with the same action_id + attempt.
  3. Forward each dataproxy.TailLogsResponse chunk to the outbound workflow.TailLogsResponse stream (converting the proto message — the schemas need to be checked for 1:1 field compatibility).
• Delete runs/service/k8s_log_streamer.go and its test once the delegation is in place; DataProxy's logs/k8s_log_streamer.go becomes the single implementation.
• Keep the runs RPC registered so existing clients keep working.

Open questions

• Response proto shape: confirm workflow.TailLogsResponse and dataproxy.TailLogsResponse are structurally compatible, or define a small converter.
• Wiring: the run service will need to be given a DataProxy client at startup (likely the in-process one in the all-in-one binary).
• Concurrency limit: decide whether the defaultMaxConcurrentStreams semaphore stays on the run-service shim, moves entirely to DataProxy, or both.

References

• runs/service/run_logs_service.go:44-61 — current run-service TailLogs
• dataproxy/service/dataproxy_service.go:503-520 — DataProxy TailLogs (target delegate)
• runs/service/k8s_log_streamer.go / dataproxy/logs/k8s_log_streamer.go — duplicated streamer impls

[runllm]

It could be possible that the Flyte codebase supports refactoring RunLogsService.TailLogs to delegate log streaming to DataProxyService.TailLogs, as both services appear to resolve log context and stream pod logs using similar logic. The proposal to inject a DataProxyServiceClient into RunLogsService and forward requests aligns with the goal of reducing duplicate code and centralizing log streaming logic, as described in the issue body. However, the data sources do not provide explicit details about the proto message compatibility or the exact wiring for dependency injection, so those aspects may require further investigation.

Though I'm not entirely certain, it might be that concurrency limits and proto conversion will need careful handling, as the data sources mention concurrency semaphores and the need to check for 1:1 field compatibility between response types. The removal of the duplicate streamer implementation and keeping the legacy RPC for backward compatibility also seem reasonable based on the outlined rationale. Would you like a step-by-step breakdown of how to implement this delegation, including code structure and migration steps?

Sources:

• https://github.com/flyteorg/flyte/blob/master/flyteidl/gen/pb-go/gateway/flyteidl/service/dataproxy.pb.gw.go
• https://github.com/flyteorg/flyte-sdk/blob/main/src/flyte/remote/_client/_protocols.py
  Note: I wasn’t able to find highly relevant data sources, but above are a few potentially relevant links.

---

This is an AI-generated response and your feedback is appreciated! Please leave a 👍 if this is helpful and 👎 if it is not.

Mention @RunLLM to ask follow-up questions. See runllm.com for more!

[Carina-TzuHsuan]

#take