Skip to content

[Performance] Bound S3 backup log retention queries to avoid full-table scans #10

Description

@chromonav

Branch reviewed: s3

Problem

Retention logic fetches all successful backup logs and old logs without explicit limits, which can degrade sharply as history grows.

Evidence (file + line)

  • aws_integration/aws_integration/s3/backup.py#L540
  • aws_integration/aws_integration/s3/backup.py#L563
  • Related candidate generation path: aws_integration/aws_integration/s3/backup.py#L104

Risk / impact

  • Full-table scans and large in-memory lists in maintenance jobs.
  • Longer backup windows and higher DB pressure.
  • Increased timeout risk on large installations.

Replication suggestion

  1. Seed S3 Backup Log with high row count (e.g., 100k+ success logs).
  2. Run backup rotation.
  3. Measure query time/memory and job duration.

Resolution suggestion

  • Query only required columns and bounded windows.
  • Use retention-count based slicing via SQL with limits.
  • For retention-days, delete in paginated batches using indexed fields (completed_at, status).
  • Consider server-side delete query strategy to avoid loading all names into Python sets.

Acceptance criteria

  • Rotation performs bounded queries and batch deletes.
  • Runtime remains stable as log table grows.
  • Benchmarks show improved query/memory profile on large datasets.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions