Harden Aurora backups: 35-day PITR, AWS Backup enrollment, and monthly logical dumps

[danielbowne]

Gives the Aurora cluster a three-legged backup strategy, sized to how late an unnoticed bad edit can plausibly surface:

discovery lag	PITR @ 35d	AWS Backup d15_w90	monthly dump
<= 35 days	yes	yes	yes
36-95 days	no	yes	yes
beyond	no	no	yes

The incident that prompted this surfaced 35 days after the edit - exactly the Aurora PITR ceiling - so PITR alone covers the worst case already observed with zero margin. The fix is three legs with different horizons: prod PITR raised to the 35-day maximum, enrollment in the CMS OIT Daily15_Weekly90 AWS Backup plan (tag-selected, no new backup stack), and a monthly pg_dump to a versioned, Object-Locked S3 bucket with no expiry. The dump leg is logical on purpose: recovery for a bad row is "stand up a reference copy and diff", never "roll prod back a month", so readability beats restorability. A daily probe measures the age of the newest dump from the bucket itself and alarms past 40 days - measured, not inferred from the job's own heartbeat, because CloudWatch cannot alarm across a monthly gap.

Riders, each argued inline in the diff: deletion_protection, pinned backup/maintenance windows, copy_tags_to_snapshot, a pinned cluster parameter group, and ignore_changes on engine_version so auto minor upgrades stop producing perpetual downgrade plans.

Testing

terraform fmt, tflint, and terraform validate pass on the rebase against current main (clean rebase, no conflicts). The dump script verifies its own artifact (pg_restore --list) and refuses undersized uploads; runtime behavior is exercised in dev first (db_dump_enabled = true there) before prod depends on it. No application, API, or migration changes - infra and the ops image only.

Deploy notes (read before merging)

1. Ops image ordering. The ops-image workflow rebuilds on backend/ops/** and updates the ztmf_ops_tag SSM parameter, but the terraform apply in the same pipeline can race it. If the apply wins, the dump/probe task definitions pin the previous image until the next apply. Verify the ops-image run finished, then re-apply (or let the next merge do it).
2. Day-one ALARM is expected. The probe reports an empty bucket as maximally stale by design, and the monthly schedule may be weeks out. Clear it by running the dump task once by hand (aws ecs run-task with the schedule's subnets/SG); the next daily probe reads the fresh dump and the alarm returns to OK. The enablement sequence is also documented at the top of backup-dumps.tf.
3. Do not let this sit deployed-to-dev but unmerged. A later main-based apply would try to destroy the dump bucket; once a dump has landed under Object Lock the destroy fails and blocks everyone's dev deploys. Merge promptly after the DEV gate. If dev state does get tangled or reverted mid-window, that is acceptable for dev data - we refresh the dev database from prod routinely, so there is no data-loss concern on the dev side; the sequencing note is about not wedging the deploy lane, not about protecting dev data.

[jsos3-cms]

Thanks for this @danielbowne . Let me start digging in!

[jsos3-cms]

Thanks for the writeup, @danielbowne — this is a lot more considered than "needs-refinement" implies, and several things I went in skeptical of held up under checking (Object Lock vs. lifecycle transitions, arn_without_revision, the write-only S3 IAM shape, the mktemp/readonly/trap ordering, and the "ALARM then clears" claim). I cross-checked the load-bearing claims against the AWS provider docs and the RDS/S3/CloudWatch references, plus an independent review pass. Verdicts on your five, then the changes I'd ask for.

Your five judgment calls

1. Object Lock GOVERNANCE 365d — keep it. The threat model here is accidental/unnoticed edits, not a malicious admin destroying backups; GOVERNANCE covers that and leaves the documented break-glass path. COMPLIANCE would be over-committing on an account we don't solely control. One thing to keep true: no routine or CI role should hold s3:BypassGovernanceRetention — the dump task role is write-only with no delete, which is exactly right.
2. Dev stays enrolled in d15_w90 — keep it, strongly. Two cents a month to prove the CMS-managed selection role can actually back up Aurora via the tag path before prod leans on it is a good trade.
3. Pinned backup/maintenance windows — keep. Non-disruptive. Only nit: the monthly dump fires at 06:30, exactly the end of the 06:00-06:30 backup window — nudge it to 07:00 for a clean gap so a slightly-long backup can't contend.
4. engine_version — keep auto_minor_version_upgrade = true + ignore_changes, but add drift alerting (C2 below). I'd rather keep CVE-patch currency than manage the version by hand, but not silently — since ignore_changes deliberately blinds terraform plan, the signal has to come from outside Terraform.
5. Second task definition — keep two. I confirmed ecs_parameters has no command/env override, so the templated ECS target genuinely can't vary either per-invocation. The universal-target form (arn:aws:scheduler:::aws-sdk:ecs:runTask + input JSON with Overrides) would collapse it to one task def, but it trades the typed, provider-validated ecs_parameters/network_configuration blocks for a hand-authored JSON blob — worse for an audited repo. Two small task defs is the readable choice.

Changes I'd ask for

C1 — blocker: check-dump-age.sh crashes on an empty bucket

check-dump-age.sh's S3-listing JMESPath raises a JMESPathTypeError when the prefix is empty, because list-objects-v2 omits Contents entirely (not an empty array). Under set -euo pipefail that aborts the script before the if [[ -z "${newest}" || "${newest}" == "None" ]] fallback that's supposed to publish age_days=9999 — so on a fresh/empty bucket the daily probe crash-loops and never writes its sentinel or log line (the alarm still ends up correct via treat_missing_data=breaching, but for the wrong reason). Null-coalesce before sorting:

sort_by(Contents[?ends_with(Key, `.dump`)] || `[]`, &LastModified)[-1].LastModified

C2 — engine_version drift alerting

Since plan can't see an AWS-applied patch by design, surface it with a native RDS event subscription to the existing alarms topic. The DB-cluster maintenance category carries RDS-EVENT-0173 (engine version upgraded / patch complete) and RDS-EVENT-0290 (patched X⇒Y):

resource "aws_db_event_subscription" "ztmf_db_engine" {
  name        = "ztmf-db-engine-events-${var.environment}"
  sns_topic   = aws_sns_topic.ztmf_alarms.arn
  source_type = "db-cluster"
  source_ids  = [aws_rds_cluster.ztmf.cluster_identifier]
  # maintenance carries the applied-patch events (RDS-EVENT-0173 / -0290) and
  # the "upgrade available" heads-up (RDS-EVENT-0156).
  event_categories = ["maintenance"]
}

Verify RDS can publish to ztmf_alarms (same-account default is usually enough; add an sns:Publish statement for the RDS service principal if delivery fails). Cluster-level only — don't also subscribe the instance, or you'll get duplicate noise.

C3 — alarm sliding-window jitter

ztmf_db_dump_stale uses period=86400, evaluation_periods=1 on a once-daily metric. The default sliding (non-calendar-aligned) window means routine Fargate start-time jitter can momentarily leave a 24h window with neither yesterday's nor today's datapoint → a ~1-minute false ALARM→OK blip straight to the email SNS. The provider pinned here (~> 5.82) has no evaluation_window/WallClockWindow argument to fix it properly, so widen period to 93600 (26h) to absorb the jitter — still far under the 604800s ceiling and far below the 40-day intent.

C4 — IAM PassRole self-containment

aws_iam_role_policy.ztmf_db_dump_check_scheduler grants iam:PassRole only for module.db_dump_check.role_arn, but the check task's execution_role_arn is module.ops_task_execution.role_arn — authorized only by the sibling dbDumpSchedulerPermissions (both attach to the same scheduler role). It works today, but if the dump leg is ever removed the check schedule breaks with an opaque AccessDenied on PassRole. Add module.ops_task_execution.role_arn to the check-scheduler policy's PassRole Resource so each policy stands on its own.

C5 — fresh-enable alarm

The alarm sits in ALARM from first enable until the first monthly dump lands (and with C1 unfixed, the probe crash-loops meanwhile) — not "clears once the probe has run," since the probe reports maximum age on an empty bucket. Fix that line in the description, and add a runbook step: after enabling, run one dump immediately (aws ecs run-task against the dump task def — the image is preloaded) to seed the bucket and clear the alarm. Do this only post-merge — a seeded object in the shared-dev bucket pre-merge can wedge the dev pipeline, since the next PR's apply-from-main would try to revert the bucket and Object Lock turns that into a failed destroy.

C6 — optional / discretion

• Mirror the cluster's "remove this block for a deliberate version change" comment onto the instance's ignore_changes block. The instance block is redundant (Aurora won't let the instance version diverge from the cluster) but harmless; the comment prevents a future asymmetric edit.
• The 0.0.0.0/0:443 egress matches the existing ztmf_sync_lambda SG, so it's not new posture. If you want to close it repo-wide, adding "monitoring" to the aws_vpc_endpoint.ztmf for_each would drop the NAT path for CloudWatch here and for kion's PutMetricData too — marginal cost given nine endpoints already exist. Separate hygiene item, not a blocker.

Follow-up

The scheduling-convergence work is tracked in CMS-Enterprise/ztmf-misc#169 — migrate the repo's scheduled Lambdas (incl. ztmf-kion-key-rotate) onto aws_scheduler_schedule, for which this PR is the first reference implementation. Related to this but explicitly not a dependency.

[jsos3-cms]

Separately from the design feedback above — the deploy-safety gates to clear before this applies (these are pre-apply checks, not change requests):

• Prod plan. Prod hasn't been planned yet. Run terraform plan against prod and confirm the blast radius: backup_retention_period 1→35 is the only retention change, the db_cluster_parameter_group_name pin shows no change, the backup/maintenance window pins are in-place (not replacements), and deletion_protection is in-place.
• Parameter group. Confirm the current db_cluster_parameter_group in both dev and prod is already default.aurora-postgresql16 (aws rds describe-db-clusters). A cluster on a custom group would be force-reverted to default on apply — disruptive.
• Prod ops image. Confirm the ops image is preloaded to prod ECR with SSM ztmf_ops_tag pointing at it (dev was preloaded; prod wasn't) before the prod task definitions register against an image with no scripts in it.

[jsos3-cms]

@danielbowne checking in on this one. Nothing has moved since the review notes on the 29th, and the PR is still a draft, so I want to make sure it isn't waiting on me.

Where it stands: C1 is the only blocker (the check-dump-age.sh JMESPath crash on an empty bucket), and the three pre-apply gates above are still unchecked, notably the prod plan and the prod ops image. Everything else in that comment was design discussion or optional.

Are you picking the C1 fix up, or would you rather I take it? Happy either way, I just want to know whether to plan around it. If this is parked behind other work, say so and I'll leave it alone.

[danielbowne]

Picking this back up next sprint. I drafted it a while ago out of concern about our data-loss exposure, and the substance still stands - cleaning it up against current main for a re-release.