feat(xtra-prod): single-AZ (us-east-1a), keep intra-zone HA, all t3.medium - #282
Merged
Conversation
Per client direction, drop cross-AZ HA for PROD at this stage and run a minimal 2-VM footprint in us-east-1a (1 system + 1 app node). - Pin both node groups to the us-east-1a private subnet (new local single_az_private_subnet_ids); fixed at 1 node each (min=max=desired=1). - Enable VPC CNI prefix delegation + a launch template raising the system node's kubelet maxPods to 110, so one t3.medium can host all platform pods. - Reduce api and worker to 1 replica each (2x each will not fit one t3.large: ~2.6 vCPU requested vs ~1.93 allocatable). Removes app-tier redundancy. - Trim ingress-nginx and metrics-server to 1 replica. - Ignore RDS engine_version changes (auto_minor_version_upgrade is on) to stop plan drift; DB otherwise untouched. Redis EBS volume already in us-east-1a, so no volume migration. NOTE: not yet applied. Requires a maintenance window (both node groups are replaced; single-replica app has a brief reschedule blip) and stakeholder sign-off on the app-replica reduction.
rohit-joy
reviewed
Sep 7, 2026
Collaborator
Author
|
Thanks Rohit. All-medium works. One heads-up: the worker runs around ~1.2 GB steady and extraction jobs can spike, so on a 4 GB node it will hit memory pressure sooner than it did on the large. That is fine for surfacing breaking points, just flagging it. I will right-size the container requests/limits so it fits medium cleanly, keep 2 app + 2 system nodes (both t3.medium) in us-east-1a, and make sure the api/worker replicas spread across the two nodes so we keep real HA within the zone. |
Per Rohit's PR feedback: keep HA (2 nodes per group, 2 replicas), just drop multi-AZ. Revert the earlier minimal/no-HA design. - Pin both node groups to us-east-1a (single_az_private_subnet_ids); keep both at 2 nodes (min=max=desired=2). All node groups on t3.medium (app was t3.large). - Keep api/worker at 2 replicas; add hostname topologySpreadConstraints so the replicas land on different app nodes (real HA within the zone). - Right-size api/worker requests+limits to fit t3.medium (api 768Mi/1.5Gi, worker 1.5Gi/2.5Gi; actual usage ~0.5Gi and ~1.2Gi). Lower burst ceiling than t3.large, flagged to Rohit. - Drop prefix delegation + launch template (not needed at 2 nodes per group). - Keep RDS engine_version ignore_changes (drift no-op).
arielr-lt
changed the title
arielr-lt
force-pushed
the
feat/xtra-prod-single-az
branch
from
September 7, 2026 21:42
5713f9c to
73a8be7
Compare
rohit-joy
reviewed
Sep 8, 2026
rohit-joy
approved these changes
Sep 8, 2026
added 2 commits
September 8, 2026 08:45
Make the single-AZ node-group cutover near-zero-downtime: - node groups get a -1a name suffix + create_before_destroy, so the new 1a t3.medium nodes come up before the old ones drain. - Add PodDisruptionBudgets (minAvailable 1) for api and worker so drains keep a replica serving (ingress-nginx and coredns already had PDBs).
Per Rohit's review: allow one node of burst headroom per group under load. Applies to both system and app groups (baseline stays min=desired=2).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Per Rohit's review: keep PROD highly available, just not across zones. Move everything into us-east-1a while retaining HA within the zone (2 nodes per group, 2 replicas), and standardize all node groups on t3.medium. Revisit multi-AZ when the app scales up.
Reworked from the earlier minimal/no-HA draft. Not yet applied.
Design
single_az_private_subnet_ids).topologySpreadConstraints(hostname) so the replicas land on different app nodes — real HA across a node failure within the zone.ignore_changes = [engine_version](auto-minor-upgrade drift no-op); DB untouched.Trade-off flagged to Rohit
t3.medium (4 GB) gives the worker a lower memory burst ceiling than t3.large (8 GB) did. Worker runs ~1.2 GB steady; a heavy extraction spike will hit memory pressure sooner on medium. Acceptable per his "find the breaking point" approach, but noted.
terraform plan
DB shows no change.
Storage
Redis EBS volume already in us-east-1a, so no volume migration. EFS regional. RDS stays Multi-AZ.
Rollout (maintenance window)
Node groups get replaced (subnet change is immutable). With 2 nodes per group + soft topology spread, roll one group at a time to keep capacity; brief blips only. Staged: system group, then app group.