Skip to content

Balancer: point balancerproject.org at Envoy LB by 2026-07-27 #160

Closed
Closed
Balancer: point balancerproject.org at Envoy LB by 2026-07-27#160

Description

@themightychris
themightychris
opened on Jul 14, 2026

For the Balancer team. Deadline: 2026-07-27 — after that the TLS cert expires and the site breaks.

balancerproject.org still points at the old ingress-nginx load balancer, which can no longer renew certificates (see #144). It must move to Envoy.

What to change

In whatever DNS provider hosts balancerproject.org (Google Cloud DNS, project unknown to us — we don't have access):

Record Type Value
balancerproject.org A 45.79.246.168

www.balancerproject.org is a CNAME to the apex, so it follows automatically. Set TTL to 60 first if it isn't already, so rollback is fast.

What to expect

  • Everything is staged on our side. We verified the ACME challenge already answers correctly through Envoy, so the cert issues automatically within ~1-2 minutes of the DNS change. No action needed from us.
  • ~60–90 seconds of downtime at the moment of the flip: Envoy redirects HTTP→HTTPS, and its HTTPS listener doesn't come up until the cert lands.
  • Rollback = set the A record back to 104.237.148.175. This only works until we decommission ingress-nginx, so go before then if you want a bail-out.

Do it whenever you like, but before Jul 27

Two hard limits:

  1. 2026-07-27 — current cert expires.
  2. ingress-nginx decommission (Migrate to Envoy Gateway: ingress-nginx → Gateway API + cert-manager gatewayHTTPRoute #144 phase 5.5) — blocked on this issue.

Please comment here when it's done.

Activity

  1. TineoC commented on Jul 14, 2026

    @TineoC
    TineoC
    on Jul 14, 2026
    Contributor
    Image

    Done 💯

  2. TineoC commented on Jul 14, 2026

    @TineoC
    TineoC
    on Jul 14, 2026
    Contributor

    It might take a few minutes, as TTL is 4 hours

  3. TineoC commented on Jul 14, 2026

    @TineoC
    TineoC
    on Jul 14, 2026
    Contributor
    $ curl -v --max-time 10 http://balancerproject.org/ 2>&1 | head -30
% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Host balancerproject.org:80 was resolved.
* IPv6: (none)
* IPv4: 45.79.246.168
*   Trying 45.79.246.168:80...
* Connected to balancerproject.org (45.79.246.168) port 80
* using HTTP/1.x
> GET / HTTP/1.1
> Host: balancerproject.org
> User-Agent: curl/8.12.1
> Accept: */*
> 
* Request completely sent off
< HTTP/1.1 404 Not Found
< date: Tue, 14 Jul 2026 16:27:40 GMT
< content-length: 0
< 

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
* Connection #0 to host balancerproject.org left intact

    Could you please confirm if the app is ready to use the new ip?

    I don't think we have HTTPRoutes in this environment yet

  4. TineoC commented on Jul 14, 2026

    @TineoC
    TineoC
    on Jul 14, 2026
    Contributor
    Image

  5. TineoC commented on Jul 14, 2026

    @TineoC
    TineoC
    on Jul 14, 2026
    Contributor

    DNS change complete — balancerproject.org → 45.79.246.168 (Envoy LB) ✅

    Root cause found: HTTP returns 404 for the apex domain because the per-app Gateway/ListenerSet claims balancerproject.org for HTTPS only. With mergeGateways: true, the shared main-gateway's catch-all HTTP→HTTPS redirect stops applying to claimed hostnames. The www subdomain works because no Gateway explicitly claims it.

    Fix

    Two PRs opened:

    1. balancer-main#517 — Adds balancer-http-redirect HTTPRoute attached to the http section of the ListenerSet, returning 301→HTTPS. Also includes kustomize hygiene fixes (disableNameSuffixHash, corrected ClusterIssuer annotation).

    2. cfp-live-cluster#166 — Removes the conflicting standalone Gateway in _gateways/balancer.yaml (redundant with the ListenerSet approach).

    Merge order

    1. ✅ Merge balancer-main#517
    2. ➡️ Trigger hologit rebuild (build-k8s-manifests workflow on main)
    3. ➡️ Merge cfp-live-cluster#166

    After deploy: HTTP should return 301→HTTPS, ACME challenge should complete, and the TLS cert should auto-issue within ~1-2 minutes.

  6. TineoC commented on Jul 14, 2026

    @TineoC
    TineoC
    on Jul 14, 2026
    Contributor

    Status update

    Done:

    • ✅ balancer-main#517 — merged (added HTTP redirect route to ListenerSet)
    • ✅ balancer-main#463 — merged (develop→main release, gets fix onto the branch hologit tracks)

    Still needed (requires admin on this repo):

    • ➡️ Trigger build-k8s-manifests workflow (push to main or manual dispatch) — regenerates the holobranch with the fixed ListenerSet + redirect
    • ➡️ Merge cfp-live-cluster#166 — removes the conflicting standalone Gateway (safe only after hologit rebuild)

  7. TineoC commented on Jul 14, 2026

    @TineoC
    TineoC
    on Jul 14, 2026
    Contributor

    @themightychris @dherbst — hologit rebuild on cfp-live-cluster is needed to pick up the balancer-main fix. Can someone trigger the build-k8s-manifests workflow (push to main or manual dispatch) and merge #166?

  8. themightychris commented on Jul 14, 2026

    @themightychris
    themightychris
    on Jul 14, 2026
    MemberAuthor

    merging #166 will do it

  9. TineoC commented on Jul 14, 2026

    @TineoC
    TineoC
    on Jul 14, 2026
    Contributor

    Go ahead

  10. themightychris commented on Jul 14, 2026

    @themightychris
    themightychris
    on Jul 14, 2026
    MemberAuthor

    Done — balancerproject.org is on Envoy and serving a fresh cert.

    The DNS change landed; I cleared a stale ACME authorization that had expired during the 57 days the hostname was stuck on nginx, and the cert issued immediately.

    balancerproject.org   http=301 -> https   https=200
balancer-gw-tls       Ready, valid to 2026-10-12

    The HTTP 404 reported in #166 is gone, and it resolved itself: it was cert-manager's ACME solver HTTPRoute (cm-acme-http-solver-*), which declares an exact hostname and therefore outranks http-redirect's catch-all, while only matching the challenge path. cert-manager deletes that route the moment the challenge succeeds. No Gateway change and no ListenerSet were needed — see #166 for the full trace.

    Two things for the balancer team:

    Your upstream Gateway API manifests aren't consumed by this cluster. Routing for balancerproject.org comes from _gateways/balancer.yaml here. gateway-listeners.yaml declares a ListenerSet, which Envoy Gateway v1.7.3 does not reconcile (XListenerSet CRD not found, skipping XListenerSet watch) — it would apply cleanly and be silently ignored. Revisit at EG v1.8. Adoption of v1.1.6 is tracked in #168.

    www.balancerproject.org has never worked — the legacy cert covered only the apex, the Ingress served only the apex, and www fails TLS on nginx too. It CNAMEs to the apex and now reaches Envoy, but no listener matches its SNI. If you want www, it needs its own listener + cert.

  11. TineoC commented on Jul 14, 2026

    @TineoC
    TineoC
    on Jul 14, 2026
    Contributor

    Are there any rules for www? In the gateway or you prefer to redirect www.balancerproject.org to balancerproject.org

  12. themightychris commented on Jul 15, 2026

    @themightychris
    themightychris
    on Jul 15, 2026
    MemberAuthor

    Done — balancerproject.org is on the Envoy LB with a valid cert (verified serving HTTPS 200), and routing moved to the app-owned HTTPRoute contract via #170. Nothing left here.

  13. themightychris
    closed this as completedon Jul 15, 2026
  14. sahilds1
    added a commit that references this issue on Jul 15, 2026

    fix: add HTTP redirect route to ListenerSet for apex domain

    455d822
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions