AWS Health Issue - Resolve Amazon EKS cluster health issues for AWS Account 996810415034 View details in service console

[jeannekitchens]

[Action required by 2026-08-19] Resolve Amazon EKS cluster health issues for AWS Account 996810415034
View details in service console
Hello,

Amazon EKS detected cluster health issues in your AWS account 996810415034.

The following is a list of affected clusters with their cluster arns, cluster health status and corresponding cluster health issues(s):
arn:aws:eks:us-east-1:996810415034:cluster/ce-registry-eks : IMPAIRED : One or more of the subnets associated with your cluster does not have enough available IP addresses for Amazon EKS to perform cluster management operations. Free up addresses in the subnet(s), or associate different subnets to your cluster using the Amazon EKS update-cluster-config API.

The health of an EKS cluster is a shared responsibility between AWS and customers. You must resolve these issues to maintain operational stability for your EKS cluster(s). Cluster health issues can prevent Amazon EKS from patching your clusters or prevent you from upgrading to newer Kubernetes versions.

Starting on 2026-08-19, Amazon EKS will patch clusters to the latest supported platform version [1]. Clusters that are unstable due to outstanding health issues may experience loss in connectivity between the Kubernetes control plane instances and worker nodes where your workload runs. To avoid this, we recommend that you resolve outstanding cluster health issues [2] before this date.

You can also view your affected clusters in the 'Affected resources' tab in your AWS Health Dashboard or by using the DescribeCluster API [3].

[1] https://docs.aws.amazon.com/eks/latest/userguide/platform-versions.html
[2] https://docs.aws.amazon.com/eks/latest/userguide/troubleshooting.html#cluster-health-status
[3] https://docs.aws.amazon.com/eks/latest/APIReference/API_DescribeCluster.html

Sincerely,
Amazon Web Services.

Message metadata
Affected account

996810415034

Event type code

AWS_EKS_CLUSTER_HEALTH_ISSUES

Event region

us-east-1

End time

Tue, 1 Sep 2026 15:00:42 GMT

Service

EKS

Event type category

accountNotification

Start time

Wed, 12 Aug 2026 15:00:42 GMT

AWS managed notification details
AWS Health: Operations events notifications are generated by AWS and sent to selected account contacts. You can add additional delivery channels for these notifications using AWS managed notifications subscriptions.
Event type

AWS Health Event

Category

Operations

[arielr-lt]

@jeannekitchens @chuang-CE
What AWS flagged: Our production Kubernetes cluster briefly ran low on available network addresses in one zone, so AWS raised a health warning.

What actually caused it: This was a side effect of application instability. Several application components (the main app, Redis, and the logging agent) have been running out of memory and repeatedly crashing/restarting. Every time the system tries to recover from these crashes, it spins up additional servers to keep the app available. More servers means more network addresses in use — and that temporary pile-up is what tripped AWS's warning.

The key point: When the application is healthy, the system runs on far fewer servers and there is plenty of network address capacity. The shortage only appears when the memory crashes force the system to scale up. In other words, fix the crashes and the network warning goes away on its own.

Recommended path (in order):

Tune application memory — right-size the memory settings for the main app, Redis, and logging so they stop running out of memory and crashing. (This is the real fix.)
Let the system scale back down — once the crashing stops, the extra servers retire automatically and network addresses free up.
Network resizing is not required — it would be costly and disruptive, and it treats the symptom, not the cause. We'd only revisit it if the app is fully healthy and capacity is still tight (which the evidence says it won't be).
Bottom line: This is an application resource-tuning issue. Investing in fixing the out-of-memory crashes solves both the stability problem and the AWS warning.

The system looks more stable since the latest deployment — we're seeing fewer restarts and no memory-related crashes so far. I'm happy to keep monitoring to confirm it holds and to continue tuning resources for performance and stability

[jeannekitchens]

@arielr-lt we need dependability every day. Seems to me the memory issue needs to be addressed.

[arielr-lt]

@jeannekitchens agreed — daily dependability is the priority, and the memory issue is the core thing to address. That's where we're focused.

Here's the plan:

Now (immediate stability): We'll right-size the application's memory requests so the system stops overcommitting and the crash-and-restart cycle settles down. This also clears the AWS health alert, since that was a side effect of the crashes forcing extra servers to spin up.

Ongoing: We'll open a PR to tweak the production deployment manifest with the updated resource settings, then keep monitoring to confirm stability holds and continue tuning as needed.

[arielr-lt]

Please let me know when we should be able to proceed @jeannekitchens @chuang-CE

[rohit-joy]

After two of these out of memory incidents, do we know what is the memory leak? Without understanding the reason for the leaky pipe, the plumbing cannot be permanently fixed and are just throwing $ at the problem to buy bigger buckets hoping it contains it. It doesn’t!

Secondly, why is Redis running out of memory? Isn’t that all temporary data? I mean, Registry only uses it for the workflows, and after that the data could be thrown out, right?

Thirdly, I’m curious about this running out of IP address stuff. That would indicate the IPs are still allocated during scaling up even though the pods on VMs have crashed and burned - correct me if I’m wrong. Does that also mean we are holding on to unwanted resources? If so, isn’t there a limit to the auto scaling? No organization can scale to infinity since no one has infinite $.

Same question about logging agent. Why is it not clearing its memory? Fluent-bit is supposed to be the tiniest app.

Please go ahead and send your PR for the manifests, to allow for the EKS patching to take place.

Also, let me know the answers to above questions.

[arielr-lt]

@Rohit good questions, let me go through them.

On the leak. You're right that just giving it more memory doesn't fix anything, we're on the same page there. Here's what we know so far. Main-app memory creeps up steadily the longer a pod runs, roughly 300MB an hour, and it does that regardless of how much traffic it's getting (CPU basically idles the whole time). A fresh pod starts around 350Mi and by about 21 hours it's up near 7.6Gi, hits the ceiling, restarts, and starts over. So we've got a clear picture of the behavior, but not the exact line of code yet. That needs proper memory profiling on the app, which is on us at LT and already in progress. One quick thing we want to try early: Ruby with a lot of threads tends to fragment memory, and capping the malloc arenas (or switching to jemalloc) often knocks 20 to 40% off, and sometimes makes an apparent "leak" flatten out entirely. The manifest PR is really just to keep prod steady while we chase the real thing.

On Redis. You're right, it's transient. It's capped at 500mb with LRU eviction and it's back down to around 10Mi now, so the data isn't piling up. The OOMs weren't the dataset growing, they were short memory spikes. Redis has append-only persistence turned on, and when the app was thrashing and hammering it with writes, the periodic rewrite forks briefly doubled its memory and blew past the 1Gi container cap. Since it's basically cache and queue data, the clean move is to just turn persistence off. The one thing to check first is that Sidekiq keeps its job queue in Redis, so without persistence anything queued wouldn't survive a Redis restart. If you're fine with that we'll switch it off, otherwise we give the fork some headroom instead.

On the IPs. Small correction here: crashed pods don't actually hold onto IPs. When a pod dies its IP gets handed back right away (we checked, there were no orphaned network interfaces). What was holding them was the extra nodes the autoscaler spun up, since each node grabs a pool of spare IPs. So it was too many nodes, not zombie pods. And it sorts itself out. Once the churn stopped, the fleet shrank back to 2 nodes on its own and the free IPs recovered (that subnet went from 8 free back up to 38). On the "scale to infinity" worry, no, everything's capped. The app's node pool maxes at 8, so there's a hard ceiling. What made it worse this time was the autoscaler being told to scale on memory, and since memory was leaking, it kept adding pods and nodes chasing it. The PR takes that memory trigger out so it only scales on real load.

On Fluent-bit. It's normally tiny (uses about 25Mi against a 250Mi cap) and it does have buffer limits set. It fell over because the crash-looping app was throwing off a firehose of logs, and it buffers those in memory while shipping to CloudWatch, which overflowed the cap. It's back-pressure from the log volume, not a leak, and the pods that came up after the incident have zero restarts. There's some optional hardening we can do there but it's low priority.

Basically Redis, Fluent-bit and the IP alert were all knock-on effects of the same thing, main-app churning. That app work is ours at LT and it's already moving, and sorting it out is what makes the rest settle instead of us patching each piece. The PR just keeps prod stable and the cluster patchable while we get there.

[rohit-joy]

That you for details!

Let’s cap the VMs at 2, especially until the memory leak root cause is found. We don’t have a lot of traffic AFAIK. And the large VMs you have allocated should cover it.

In addition we need to be able to reproduce the memory leak. Not hide it. The VM size was already increased last time. The app pod will restart automatically when the pod crashes due to OOM anyway.

Main-app memory creeps up steadily the longer a pod runs, roughly 300MB an hour, and it does that regardless of how much traffic it's getting (CPU basically idles the whole time).

What file has this code that’s adding 300MB per hour? Can you point me to the lines?

Once the churn stopped, the fleet shrank back to 2 nodes on its own and the free IPs recovered (that subnet went from 8 free back up to 38).

Are you saying there were 38 VMs allocated at one point? That’s $ down the drain.

Can you share the logs in Slack as ZIP files?

[arielr-lt]

@rohit-joy Re: "Are you saying there were 38 VMs allocated at one point? That’s $ down the drain." that sounds like a misread of my wording and worth correcting cleanly: 38 was free IP addresses in the subnet, not VMs. The app node pool is capped at 8, crept up during the thrash. Nowhere near 38 machines.

[arielr-lt]

@rohit-joy Re: "What file has this code that's adding 300MB.." The "300MB/hour" isn't a code measurement, it's the average slope I read off the data. A fresh pod sits around 350Mi, and by about 21 hours it's near 7.6Gi, which works out to roughly 345MB an hour. So it's the rate the pod's memory grows over its lifetime, not a single function allocating 300MB on a timer.

[arielr-lt]

@rohit-joy Re: "Let's cap to 2 VM..".

before I pin it, one honest flag. For context, the app is running on 3 machines right now, and 3 is about the sweet spot for keeping it dependable at peak. So the real question isn't adding anything, it's whether we push it down to 2.
We can, but 2 leaves very little headroom under load, so there's a real chance it won't hold during a busy stretch and we'd end up back at 3 anyway. I'd rather raise that now than cap it and have it bite us.

Your call:

1. Push to 2 as you said, and bump back to 3 if it proves tight. Saves one machine, small risk of a rough patch under load.
2. Hold at 3, which is where it sits today, and keep it dependable with no change in cost.

Happy either way, just want your go-ahead before I lock it in

[rohit-joy]

@rohit-joy Re: "Are you saying there were 38 VMs allocated at one point? That’s $ down the drain." that sounds like a misread of my wording and worth correcting cleanly: 38 was free IP addresses in the subnet, not VMs. The app node pool is capped at 8, crept up during the thrash. Nowhere near 38 machines.

If there were 38 IPs left, then we are not running out of IPs. Correct? So what happened that we ran out of these internal IPs in the subnet? What are the total number of IPs available in the subnet?

Hold at 3, which is where it sits today, and keep it dependable with no change in cost.

Thanks for the numbers! That’s what I needed. Keep 3 VMs as max in the node pool for auto scaling limit. Minimum 2 VMs.