Repository navigation
docs(performance): [develop] record and verify paired runtime benchmark measurements - #658
Open
somethingwithproof wants to merge 4 commits into
Open
somethingwithproof wants to merge 4 commits into
somethingwithproof wants to merge 4 commits into
Conversation
somethingwithproof
requested review from
cigamit
and removed request for
a team
October 11, 2026 10:26
This was referenced Oct 11, 2026
Open
Contributor
There was a problem hiding this comment.
🔵 Needs a closer look
Historical provenance, dependent revisions, and squash-order restacking require final human verification.
0 open findings
What changed in this PR
Records reproducible benchmark evidence for the recovered runtime changes while preserving historical comparisons and operational boundaries.
Changes:
- Adds raw paired measurements, summaries, and provenance for two runtime comparisons.
- Verifies every checked-in benchmark record and includes all evidence in release archives.
- Documents TLS policy, dependency decisions, and unresolved issue boundaries.
| File | Description |
|---|---|
tests/tools/test_benchmark_record.py |
Verifies every checked-in benchmark record. |
Makefile.am |
Adds benchmark and guidance files to distributions. |
Makefile.in |
Regenerates the distribution manifest. |
docs/operations/issue-boundaries.md |
Records remaining cross-project acceptance work. |
docs/operations/database-tls.md |
Documents explicit database TLS policies. |
docs/adr/0004-formatting-and-json-dependencies.md |
Records formatting and JSON dependency decisions. |
docs/benchmarks/README.md |
Links and scopes recorded comparisons. |
docs/benchmarks/results/2026-10-11-runtime-ownership/README.md |
Interprets the historical runtime comparison. |
docs/benchmarks/results/2026-10-11-runtime-ownership/build-settings.txt |
Records historical build settings. |
docs/benchmarks/results/2026-10-11-runtime-ownership/builds.sha256 |
Records source and binary identities. |
docs/benchmarks/results/2026-10-11-runtime-ownership/compiler.txt |
Records compiler identity. |
docs/benchmarks/results/2026-10-11-runtime-ownership/cpu-quota.txt |
Records the CPU quota. |
docs/benchmarks/results/2026-10-11-runtime-ownership/cycles.csv |
Stores historical raw measurements. |
docs/benchmarks/results/2026-10-11-runtime-ownership/dependencies.txt |
Records dependency versions. |
docs/benchmarks/results/2026-10-11-runtime-ownership/metadata.json |
Records workload and environment metadata. |
docs/benchmarks/results/2026-10-11-runtime-ownership/revisions.txt |
Identifies compared revisions. |
docs/benchmarks/results/2026-10-11-runtime-ownership/summary.json |
Stores the computed summary. |
docs/benchmarks/results/2026-10-11-recovered-runtime/README.md |
Interprets the active recovered-runtime comparison. |
docs/benchmarks/results/2026-10-11-recovered-runtime/baseline-binary.sha256 |
Identifies the baseline binary. |
docs/benchmarks/results/2026-10-11-recovered-runtime/baseline-revision.txt |
Identifies the baseline revision. |
docs/benchmarks/results/2026-10-11-recovered-runtime/build-settings.txt |
Records recovered build settings. |
docs/benchmarks/results/2026-10-11-recovered-runtime/candidate-binary.sha256 |
Identifies the candidate binary. |
docs/benchmarks/results/2026-10-11-recovered-runtime/candidate-revision.txt |
Identifies the candidate revision. |
docs/benchmarks/results/2026-10-11-recovered-runtime/compiler.txt |
Records compiler identity. |
docs/benchmarks/results/2026-10-11-recovered-runtime/cycles.csv |
Stores recovered raw measurements. |
docs/benchmarks/results/2026-10-11-recovered-runtime/dependencies.txt |
Records dependency versions. |
docs/benchmarks/results/2026-10-11-recovered-runtime/metadata.json |
Records workload and environment metadata. |
docs/benchmarks/results/2026-10-11-recovered-runtime/summary.json |
Stores the verified summary. |
🧠 Review effort: Balanced
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
This was referenced Oct 11, 2026
TheWitness
pushed a commit
that referenced
this pull request
Oct 11, 2026
…lity (#656) Polling workers currently return permits before the parent can prove their threads have exited, and database pool slots have no lease identity. Add joined worker ownership and thread-owned, generation-checked database leases before shared-state cleanup. Preserve early ping-only output behavior and keep the tested external-script limits and destination-aware upserts from #655. Add explicit local/remote database TLS modes, protocol/cipher constraints, opt-in checked JSON logging, and the paired-build benchmark harness with its recorded original measurements. Legacy connection behavior and text logging remain defaults. Unsupported TLS policies fail closed. The updated man page documents the implemented contracts and connector limitations. Recovered work is isolated from the original uncommitted checkouts. The CMake fork behavior suite is preserved separately on `somethingwithproof/spine` branch `fix/recovered-cmake-behavior-contracts`; all 27 native behavior tests passed. Its fork-specific interfaces are not imported into Cacti's Autotools source layout. Validation for head `3db06ab315041187680b896c3658db0fbe8c60e3`: - All 34 GitHub checks passed: GCC/Clang with MariaDB/MySQL clients, Fedora 44, Rocky Linux 9, macOS, release distcheck, configure contracts, static analysis, integration regressions, sanitizers, nightly ASan/TSan, Memcheck, Helgrind, and concurrency soak. - GitHub and independently measured Linux coverage: **396/396 changed executable production lines (100%)**. Worker ownership: **71/71 lines and 56/56 branches**; database leases: **65/65 lines and 72/72 branches**. Local production-source hashes exactly match this revision. - Native suite: **20 passed, 2 Linux-only prerequisite skips**. Linux suite: **24 passed, 1 expected prerequisite skip**, followed by real database/SNMP/SNMPv3 fault contracts. - Disposable TLS fixtures passed trusted-certificate, wrong-identity, untrusted-CA, protocol/cipher mismatch, and plaintext rejection contracts for local and remote connections. - JSON encoder: 3 tests passed; benchmark evidence self-tests: 7 passed; original recorded benchmark verified. Mandatory commit checks and man-page lint passed. - New paired Linux ARM benchmark compares exact #655 head `a8bd78d7` with this head using identical GCC/dependencies/`-O2 -g0`, 20 measured pairs plus 2 warmup pairs, 64 devices, 16 items/device, 4 workers, alternating order. All 44 cycles verified all 1,024 expected outputs. Median elapsed time: baseline 0.752536 s, candidate 0.749570 s; median CPU time: baseline 0.023159 s, candidate 0.023171 s. Median paired elapsed delta is -0.002300 s. This noisy loopback fixture establishes no general performance improvement. TLS is disabled for both benchmark builds. **Final-head one-hour memory soak passed:** 5,612 cycles, each exiting successfully and writing all 200 results; 5,609 cycles fitted after 3 warmups. RSS slope -7.6 KB/hour; projected growth over the run -8 KB, within the 1,024 KB limit. [Final soak run](https://github.com/Cacti/spine/actions/runs/38128178635). This PR is ready for review. The older committed benchmark record remains explicitly scoped to its original revisions. Original uncommitted checkouts were preserved, and the recovery archive includes restored-and-verified Git objects for all 1,238 orphan commit heads captured during the audit. Depends on #655 and its earlier stack. No live polling endpoints, installations, database settings, or production privilege configuration were changed. ## Current merge dependencies Current base: `fix/post-c17-execution-hardening`. Immediate dependency: [#655](#655). Next open PR: [#658](#658). Merge order: #641 → #644 → #654 → #646 → #648 → #649 → #650 → #651 → #655 → #656 → #658 → #659. This repository uses squash-only merges and deletes merged branches. Record the old parent SHA, squash one PR at a time, then restack and retarget dependents before merging the next PR. Re-run merge validation and checks on the resulting heads; an ordinary clean merge does not replace squash/restack validation. --------- Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
Base automatically changed from
fix/recovered-ownership-tls-observability
to
fix/post-c17-execution-hardening
October 11, 2026 22:56
Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
somethingwithproof
force-pushed
the
docs/runtime-benchmark-records
branch
from
October 11, 2026 23:53
1bea4b0 to
5d300eb
Compare
somethingwithproof
changed the title
Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
TheWitness
pushed a commit
that referenced
this pull request
Oct 12, 2026
…ence (#641) Classify numeric, undefined, multipart and Cacti octet results through one parser, normalize undefined agent values before persistence and flush every SNMP batch consistently. Validate whole non-negative uptimes, retain unknown reindex baselines and free script results through their owning pointer. The mocked poll_host suite checks session-profile boundaries, spike handling, output_regex input and completion writes. Originally step 4/14 of the C17/native-correctness series; its preceding steps have merged. The current base is develop. Earlier validation, before this review follow-up, in isolated copied snapshots on Linux arm64: - Clean GCC build and make check: 15 passed, 1 existing prerequisite skips, 0 failures. - ASan/UBSan: 14 passed, 2 existing privilege/allocator-interposition prerequisite skips, 0 failures; no sanitizer findings. - Production source/profile integrity verification against disposable MariaDB and loopback SNMP fixtures passed. - Release archive extracted into a separate source tree and built/tested out of tree; fuzz checks passed. Earlier local profiles retain source hashes for the copied validation tree; GitHub checks validate the published commit. No production database/device, host installation or live privilege change is part of validation. Existing prerequisite skips were retained; no test or finding was suppressed to obtain a pass. Review follow-up (2026-10-09): Restacked on the corrected preceding branches. The SNMP status fix is introduced in #638, the reindex/completion fixes in #640, and focused regression coverage in #650. Restacked tip validation: isolated Linux arm64 build and make check passed (18 passed, 1 setuid prerequisite skip, no failures). GitHub checks validate the updated published commits; earlier profile/archive/sanitizer evidence above belongs to the previous revisions. ## Current merge dependencies Current base: `develop`. Immediate dependency: `develop` (earlier steps already merged). Next open PR: [#644](#644). Merge order: #641 → #644 → #654 → #646 → #648 → #649 → #650 → #651 → #655 → #658 → #659. This repository uses squash-only merges and deletes merged branches. Record the old parent SHA, squash one PR at a time, then restack and retarget dependents before merging the next PR. Re-run merge validation and checks on the resulting heads; an ordinary clean merge does not replace squash/restack validation. --------- Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
TheWitness
approved these changes
Oct 14, 2026
TheWitness
pushed a commit
that referenced
this pull request
Oct 15, 2026
…644) Polling workers need larger stacks on platforms such as musl. This step initializes worker attributes with the measured minimum, preserves larger platform defaults, and reports attribute failures. Compile-time bounds reject result-buffer sizes that overflow multiplication, addition, or rounding. The default minimum remains 384 KiB. The diff contains step 7 only: nine files for worker stack sizing and its existing fault/worker contracts. Native helper changes remain in #654. Immediate dependency: #660 (`refactor/c17-series-05-06-runtime-build`). #641 is already merged. Merge order: #660 → #644 → #654 → #646 → #648 → #649 → #650 → #651 → #655 → #658 → #659. The branch includes the current #660 parent. This dependency repair preserves the reviewed source tree. After each squash merge, restack and retarget the next dependent PR before merging it; the repository deletes merged branches. Validation: macOS build and `make check` passed, with 13 passes and two Linux-only prerequisite skips. Previous stack-bound compiler probes and worker fault contracts remain in the branch. GitHub checks must pass on the updated head before merge. --------- Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The paired benchmark job already implemented in #656 retained measurements without checking in the recovered-runtime comparison. Record its original raw cycles, full-precision summary, build revisions, binary identities, compiler/dependency versions and a scoped interpretation. Keep the historical default-logging and separate issue-resolution comparisons explicitly tied to their own revisions.
The active #655-to-#656 comparison used identical GCC 12.2.0 settings and dependencies, the same server/agent, four threads, 64 devices, 16 items per device and disabled TLS. Two warmup pairs preceded 20 measured pairs in alternating AB/BA order. All 44 cycles verified 1,024 outputs. Median wall time was 0.752536 s baseline and 0.749570 s candidate; median process CPU time was 0.023159 s and 0.023171 s. The overlapping ranges and shared-host limitations do not establish a general improvement; no percentage speedup is claimed.
Preserve the dependency decision, TLS operations documentation and explicit remaining issue-acceptance boundaries from the historical branch. These documents do not claim the broader ownership audit or poller-wide fatal subprocess shutdown is complete.
Add every benchmark script and checked-in record to the release distribution and verify all recorded summaries in the evidence tests.
Makefile.inis regenerated with the existing Automake 1.19 version.Validation: all eight benchmark evidence tests passed, including every checked-in record; the recovered inputs' fixture/configuration and binary hashes were independently checked against metadata; workflow actionlint and build-script shellcheck passed. Native
make distincluded all 39 benchmark/guidance inputs, and all eight evidence tests passed again from the extracted release archive. The historical issue-resolution source completed its full CI matrix and live coverage before its measurements were documented. Runtime code from merged #656 is now inherited through #655. The restacked head5d300ebb84c7e840860959459f2aaea7f506fc4bpasses its current required CI, regression, integration, static and Rocky Linux checks; all eight evidence tests and release packaging were rechecked after restacking.Depends on #655 and its earlier stack after #656 merged into #655. This replaces the duplicate integration PR #657; its branch and measured revisions are preserved. Human review and ordered restacking remain necessary under the repository's squash-only merge policy.
Current merge dependencies
Current base:
fix/post-c17-execution-hardening. Immediate dependency: #655. Next open PR: #659.Merge order: #641 → #644 → #654 → #646 → #648 → #649 → #650 → #651 → #655 → #658 → #659.
This repository uses squash-only merges and deletes merged branches. Record the old parent SHA, squash one PR at a time, then restack and retarget dependents before merging the next PR. Re-run merge validation and checks on the resulting heads; an ordinary clean merge does not replace squash/restack validation.