Jg glue table crd - #4
Draft
JonathanGraniero wants to merge 4 commits into
Draft
Conversation
michaelhtm
pushed a commit
that referenced
this pull request
Aug 4, 2026
Release artifacts for `glue-controller` version `v1.0.0`. Opened by `ack-workspace release`. --- ## Soak Test Summary Soak testing is a prerequisite for cutting a new major version. This release was validated against a **24h soak run** on the `ack-soak-glue` cluster (controller up 47h, runner Job `Complete`). Criteria and the summary format follow the [soak test checklist](https://github.com/aws-controllers-k8s/test-infra/blob/main/soak/SOAK_TEST_CHECKLIST.md). **Result: PASS** — all required and recommended checks passed. ### Verdict table | # | Check | Required | Result | |---|-------|----------|--------| | 1 | No panics / restarts | ✅ | PASS | | 2 | No reconcile errors | ✅ | PASS | | 3 | No infinite loops | ✅ | PASS | | 4 | No leaks (mem/goroutine/fd) | ✅ | PASS | | 5 | e2e all passing | ✅ | PASS | | 6 | No 5xx / unexpected 4xx | ✅ | PASS | | 7 | Latency stable | ⬜ | PASS | | 8 | No dangling resources | ⬜ | PASS | | 9 | CPU bounded | ⬜ | PASS | | 10 | API volume reasonable | ⬜ | PASS | <details> <summary><b>Evidence log</b> (what we looked at, commands run, observed values, conclusion)</summary> ``` #1 No panics / restarts [PASS] Looked at: reconcile/webhook panic counters; controller pod restarts Command: curl .../query?query=controller_runtime_reconcile_panics_total curl .../query?query=controller_runtime_webhook_panics_total kubectl get pods -n ack-system Observed: reconcile_panics_total=0, webhook_panics_total=0; controller pod RESTARTS=0 (up 47h) Conclusion: all zero -> PASS #2 No reconcile errors [PASS] Looked at: controller_runtime_reconcile_errors_total Command: curl .../query?query=controller_runtime_reconcile_errors_total{controller="job"} Observed: job=0, fieldexport=0 Conclusion: zero errors -> PASS #3 No infinite reconcile loops [PASS] Looked at: reconcile-rate shape; workqueue depth/longest-running Command: curl .../query_range?query=sum(rate(controller_runtime_reconcile_total{controller="job"}[5m]))... curl .../query?query=workqueue_depth{name="job"} Observed: reconcile_total = 6138 success + 6138 requeue_after, 0 error; rate peaked 0.144/s and fell to 0.000 after the runner finished; workqueue_depth{name="job"}=0; longest_running=0s Conclusion: rate decays to 0, queue drained -> PASS (retries_total~=6138 is the expected requeue_after) #4 No leaks (memory / goroutine / fd) [PASS] Looked at: RSS, heap objects, goroutines, threads, fds as trends (job=ack-controller-glue) Command: curl .../query_range?query=process_resident_memory_bytes{job="ack-controller-glue"}&step=3600 (repeat for go_memstats_heap_objects, go_goroutines, go_threads, process_open_fds) Observed: RSS 63.0 -> 64.0 MiB (min 63 / max 67.3 over 48h); heap_objects 44k-95k saw-tooth; goroutines 79 -> 78; os_threads 8 (flat); open_fds 10-11 (flat) Conclusion: no monotonic ramp anywhere -> PASS #5 e2e all passing [PASS] Looked at: pytest summary lines across all iterations Command: kubectl logs -n ack-system glue-soak-test-<pod> > glue-soak.log grep -cE '1 passed' glue-soak.log -> 3061 grep -cE '[1-9][0-9]* failed' glue-soak.log -> 0 grep -cE '[0-9]+ error' glue-soak.log -> 0 Observed: 3061 iterations, all "1 passed"; 0 failed; 0 errors Conclusion: zero failures across 3061 iterations -> PASS aws-controllers-k8s#6 No 5xx / unexpected 4xx AWS faults [PASS] Looked at: ack_outbound_api_requests_error_total by op + fault enum (status_code is a smithy FAULT ENUM: 1=server, 2=client -- NOT an HTTP code) Command: curl .../query?query=ack_outbound_api_requests_error_total curl .../query?query=ack_outbound_api_requests_total Observed: status_code="1" (server): none. status_code="2" (client): only GetJob = 3069, which equals the CreateJob/DeleteJob count (read-before-create not-found). Total calls 33,759: GetJob 12276, ListTagsForResource 9207, Create/Update/Delete/TagJob 3069 each. Conclusion: no server faults; only expected GetJob not-found client faults -> PASS aws-controllers-k8s#7 Reconcile latency stable [PASS] Command: curl .../query?query=max_over_time((histogram_quantile(1.0,sum by(le)(rate(controller_runtime_reconcile_time_seconds_bucket{controller="job"}[5m])))*1000)[2d:5m]) Observed: p100 peak 700 ms; no growing tail Conclusion: bounded -> PASS aws-controllers-k8s#8 No dangling resources [PASS] Command: curl .../query?query=max_over_time(apiserver_storage_objects{resource=~".*services.k8s.aws"}[2d]) Observed: jobs.glue.services.k8s.aws peaked at 1; fieldexports/iamroleselectors = 0 Conclusion: at most 1 resource at a time, no accumulation -> PASS aws-controllers-k8s#9 CPU bounded [PASS] Command: curl .../query?query=max_over_time(sum(rate(container_cpu_usage_seconds_total{namespace="ack-system",pod=~"soak-test-glue.*"}[5m]))[2d:5m]) Observed: peak 0.0024 cores; idled after run Conclusion: nowhere near a limit -> PASS aws-controllers-k8s#10 API volume reasonable [PASS] Command: curl .../query?query=ack_outbound_api_requests_total Observed: 33,759 calls scaling with iterations; GetJob highest (read-heavy reconcile), no anomaly Conclusion: proportional to load -> PASS ``` </details> **AWS API faults note:** for aws-sdk-go-v2 controllers the `status_code` label on `ack_outbound_api_requests_error_total` is a smithy fault enum (`1`=server, `2`=client), not an HTTP status -- so checks query `status_code="1"`/`"2"` rather than the dashboard's `4..`/`5..` panels.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.