feat(gastown): emit reconciler metrics to Analytics Engine and add Grafana dashboard panels#1372
Open
Conversation
…afana dashboard panels - Extend writeEvent() to support double3-double10 fields for reconciler metrics - Emit reconciler_tick event after each alarm tick with all 9 metrics - Add Reconciler row to Grafana dashboard with 6 panels: 1. Events drained per tick (timeseries) 2. Actions emitted per tick by type (stacked bar) 3. Side effects attempted/succeeded/failed (timeseries) 4. Invariant violations (stat with >0 alert threshold) 5. Reconciler wall clock time (timeseries with >500ms threshold) 6. Pending event queue depth (gauge with >50 threshold)
![kilo-code-bot[bot]](https://avatars.githubusercontent.com/in/2193792?s=60&v=4){kind=small}
| "interval": "", | ||
| "intervalFactor": 1, | ||
| "nullifySparse": false, | ||
| "query": "SELECT SUM(double8 * _sample_interval) / SUM(_sample_interval) AS pending_events FROM gastown_events WHERE $timeFilter AND blob1 = 'reconciler_tick' ORDER BY timestamp DESC LIMIT 1", |
Contributor
There was a problem hiding this comment.
WARNING: This gauge is showing a time-window average, not the current queue depth
SUM(double8 * _sample_interval) / SUM(_sample_interval) collapses every reconciler_tick in the selected range into a single weighted average, so the panel will not show the latest backlog value. For a queue-depth gauge we need the most recent double8 sample instead, and rawSql should be updated to match.
Contributor
Code Review SummaryStatus: 1 Issues Found | Recommendation: Address before merge Overview
Fix these issues in Kilo Cloud Issue Details (click to expand)WARNING
Other Observations (not in diff)None. Files Reviewed (3 files)
Reviewed by gpt-5.4-20260305 · 550,264 tokens |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
writeEvent()inanalytics.util.tsto supportdouble3–double10fields, enabling reconciler metrics emission without breaking existing callers (new fields default to 0).reconciler_tickevent emission after each alarm tick inTown.do.ts, carrying all 9ReconcilerMetricsfields (wallClockMs,eventsDrained,actionsEmitted,sideEffectsAttempted/Succeeded/Failed,invariantViolations,pendingEventCount) plusactionsByTypeas JSON inblob10.Verification
pnpm typecheck— passes (all workspace projects clean)Visual Changes
N/A
Reviewer Notes
writeEvent()doubles array grew from 2 to 10 entries. Analytics Engine supports up to 20, so this is well within limits.actionsByTypeis stored as JSON string in blob10 (vialabelfield) and parsed in Grafana usingJSONExtractKeysAndValues(blob10, 'Float64').SUM(doubleN * _sample_interval) / SUM(_sample_interval)for weighted averages andSUM(doubleN * _sample_interval)for counts — appropriate aggregation for sampled AE data.