Skip to content

ops: Configure Rootly alerts for IISA service #36

Description

@MoonBoi9001

Overview

Configure Rootly alerting for the IISA HTTP service to ensure operational visibility and rapid incident response.

Context

IISA runs as a separate containerized service that dipper depends on for indexer selection. The service has two operational modes:

  • Inference (per-request): Fast selection using pre-computed weights
  • Training (daily): Linear regression job updating weights (~15 min, 32GB+ RAM) (via CronJob, technically no longer managed insid the IISA container)

Failures in IISA will cause dipper to fall back to random indexer selection after 6 hours (per PR #532), but alerts should fire well before that threshold.

Alerts to Configure

  • Inference Failure: IISA service unreachable or returning errors for indexer selection requests
  • Training Failure: Daily regression job failed to complete

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions