Skip to content

fix(results): [develop] normalize output and unify SNMP batch persistence - #641

Merged
TheWitness merged 26 commits into
developfrom
refactor/c17-series-04-results
Oct 12, 2026
Merged

TheWitness merged 26 commits into
developfrom
refactor/c17-series-04-results

Conversation

@somethingwithproof

@somethingwithproof somethingwithproof commented Oct 9, 2026 •

Copy link
Copy Markdown
Member

Classify numeric, undefined, multipart and Cacti octet results through one parser, normalize undefined agent values before persistence and flush every SNMP batch consistently. Validate whole non-negative uptimes, retain unknown reindex baselines and free script results through their owning pointer. The mocked poll_host suite checks session-profile boundaries, spike handling, output_regex input and completion writes.

Originally step 4/14 of the C17/native-correctness series; its preceding steps have merged. The current base is develop.

Earlier validation, before this review follow-up, in isolated copied snapshots on Linux arm64:

  • Clean GCC build and make check: 15 passed, 1 existing prerequisite skips, 0 failures.
  • ASan/UBSan: 14 passed, 2 existing privilege/allocator-interposition prerequisite skips, 0 failures; no sanitizer findings.
  • Production source/profile integrity verification against disposable MariaDB and loopback SNMP fixtures passed.
  • Release archive extracted into a separate source tree and built/tested out of tree; fuzz checks passed.

Earlier local profiles retain source hashes for the copied validation tree; GitHub checks validate the published commit. No production database/device, host installation or live privilege change is part of validation. Existing prerequisite skips were retained; no test or finding was suppressed to obtain a pass.

Review follow-up (2026-10-09): Restacked on the corrected preceding branches. The SNMP status fix is introduced in #638, the reindex/completion fixes in #640, and focused regression coverage in #650.

Restacked tip validation: isolated Linux arm64 build and make check passed (18 passed, 1 setuid prerequisite skip, no failures).
GitHub checks validate the updated published commits; earlier profile/archive/sanitizer evidence above belongs to the previous revisions.

Current merge dependencies

Current base: develop. Immediate dependency: develop (earlier steps already merged). Next open PR: #644.

Merge order: #641 → #644 → #654 → #646 → #648 → #649 → #650 → #651 → #655 → #658 → #659.

This repository uses squash-only merges and deletes merged branches. Record the old parent SHA, squash one PR at a time, then restack and retarget dependents before merging the next PR. Re-run merge validation and checks on the resulting heads; an ordinary clean merge does not replace squash/restack validation.

Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
@somethingwithproof
somethingwithproof force-pushed the refactor/c17-series-03-worker-failures branch from 07962c3 to e29a4d0 Compare October 9, 2026 23:01
@somethingwithproof
somethingwithproof force-pushed the refactor/c17-series-04-results branch from 49733e7 to 98be364 Compare October 9, 2026 23:01
Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
@somethingwithproof
somethingwithproof force-pushed the refactor/c17-series-03-worker-failures branch from e29a4d0 to 873c982 Compare October 10, 2026 00:17
@somethingwithproof
somethingwithproof force-pushed the refactor/c17-series-04-results branch from 98be364 to 3355a4b Compare October 10, 2026 00:17
@somethingwithproof somethingwithproof self-assigned this Oct 10, 2026
TheWitness pushed a commit that referenced this pull request Oct 11, 2026
…638)

Introduce bounded native interfaces and grouped runtime state, then link
regression and fault contracts against the real poller implementations.
The contracts capture CLI/configuration input, availability
combinations, SNMP request/response ownership, script framing,
reindexing, output persistence, worker completion and shutdown. This
foundational change is larger because the shared interfaces and their
existing production contracts must move together; later changes have
separate review boundaries.

Step 1/14 of the complete C17/native-correctness series. Base: develop.

Earlier validation, before this review follow-up, in isolated copied
snapshots on Linux arm64:

- Clean GCC build and make check: 13 passed, 0 existing prerequisite
skips, 0 failures.
- ASan/UBSan: 12 passed, 1 existing privilege/allocator-interposition
prerequisite skips, 0 failures; no sanitizer findings.
- Production source/profile integrity verification against disposable
MariaDB and loopback SNMP fixtures passed.
- Release archive extracted into a separate source tree and built/tested
out of tree; fuzz checks passed.

Earlier local profiles retain source hashes for the copied validation
tree; GitHub checks validate the published commit. No production
database/device, host installation or live privilege change is part of
validation. Existing prerequisite skips were retained; no test or
finding was suppressed to obtain a pass.


The worker contracts publish owned heap state and release it only after
joining workers and restoring the runtime table. Coverage counters use
atomic updates; cppcheck receives an explicit model of the fatal exit
contract. Fixture images use public mirrors to avoid shared Docker Hub
pull limits.


Series navigation: review and merge in order. Every PR targets the
preceding branch so its diff contains one step.

1. [refactor(core): add bounded native interfaces and polling
contracts](#638)
2. [fix(privileges): drop root before input and retain owned ICMP
sockets](#639)
3. [fix(poller): isolate worker failures and bound database
reconnects](#640)
4. [fix(results): normalize output and unify SNMP batch
persistence](#641)
5. [fix(runtime): bound probe retries and validate CLI and
configuration](#642)
6. [build: select GNU C17 explicitly and probe hardening
support](#643)
7. [fix(threads): size worker stacks and guard size
arithmetic](#644)
8. [refactor(c17): simplify ownership guards and share test
support](#645)
9. [refactor(core): extract configuration polling and runtime
modules](#646)
10. [fix(boundaries): validate script slots and make cleanup
portable](#647)
11. [refactor(layout): organize responsibilities and separate main from
runtime](#648)
12. [docs(license): adopt SPDX notices and retain contributor
credits](#649)
13. [test(runtime): measure all suites and cover protocol
boundaries](#650)
14. [build(ci): validate connectors time width and native C17
modules](#651)

Review follow-up (2026-10-09): Publish the decoded GET/GETNEXT status
back to the host so availability checks observe exception and
missing-response failures, while preserving the legacy transport-success
behavior for agent PDU errors.

Updated step 1 snapshot: isolated Linux arm64 build and make check
passed (13 passed, no skips or failures).
The restacked tip passes all five clang-tidy gate regression tests and
real clang-tidy analysis (26 reviewed baseline findings, no new
findings). Actionlint and matching DCO trailers pass. The setuid test
also passes when run in the separate disposable root container.
GitHub checks validate the updated published commits; earlier
profile/archive/sanitizer evidence above belongs to the previous
revisions.

---------

Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
)

Propagate database and device failures through worker completion rather
than terminating the entire poller. Count reconnect attempts even with
stale EINTR, apply session SQL mode after reconnect, preserve failed
reindex assertions, keep completion writes outside the thread-accounting
lock and redact script command arguments from logs. Linker fault
injection drives the real connection, query, worker and startup paths.

Step 3/14 of the complete C17/native-correctness series. Base: the
preceding series branch; merge/review in order so this PR shows only its
own step.

Earlier validation, before this review follow-up, in isolated copied
snapshots on Linux arm64:

- Clean GCC build and make check: 13 passed, 1 existing prerequisite
skips, 0 failures.
- ASan/UBSan: 12 passed, 2 existing privilege/allocator-interposition
prerequisite skips, 0 failures; no sanitizer findings.
- Production source/profile integrity verification against disposable
MariaDB and loopback SNMP fixtures passed.
- Release archive extracted into a separate source tree and built/tested
out of tree; fuzz checks passed.

Earlier local profiles retain source hashes for the copied validation
tree; GitHub checks validate the published commit. No production
database/device, host installation or live privilege change is part of
validation. Existing prerequisite skips were retained; no test or
finding was suppressed to obtain a pass.


Series navigation: [Series
index](#638); [previous
step](#639). [Next
step](#641).

Review follow-up (2026-10-09): Treat a failed reindex SELECT as a device
polling failure and stop item collection. Finish each partition's
host_errors write before publishing completion; record device polling
time before the final schedule update, and mark the device complete only
after all required writes succeed. Database waits remain outside
LOCK_THDET.

Updated step 3 snapshot: isolated Linux arm64 build and make check
passed (13 passed, 1 setuid prerequisite skip, no failures).
The restacked tip passes all five clang-tidy gate regression tests and
real clang-tidy analysis (26 reviewed baseline findings, no new
findings). Actionlint and matching DCO trailers pass. The setuid test
also passes when run in the separate disposable root container.
GitHub checks validate the updated published commits; earlier
profile/archive/sanitizer evidence above belongs to the previous
revisions.

---------

Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
Base automatically changed from refactor/c17-series-03-worker-failures to refactor/c17-series-02-privileges October 11, 2026 00:03
…642)

Give each ICMP retry its own deadline, accept connected UDP replies and
resolve transport-qualified/bracketed IPv6 names correctly. Reject
non-numeric thread/host options and clamp thread limits. Avoid large
logging buffers on thread stacks and reject truncated configuration
keys/values/lines. Native socket/time interception and
logging/configuration suites check retries, formatting, bounds and
cleanup.

Step 5/14 of the complete C17/native-correctness series. Base: the
preceding series branch; merge/review in order so this PR shows only its
own step.

Earlier validation, before this review follow-up, in isolated copied
snapshots on Linux arm64:

- Clean GCC build and make check: 17 passed, 1 existing prerequisite
skips, 0 failures.
- ASan/UBSan: 16 passed, 2 existing privilege/allocator-interposition
prerequisite skips, 0 failures; no sanitizer findings.
- Production source/profile integrity verification against disposable
MariaDB and loopback SNMP fixtures passed.
- Release archive extracted into a separate source tree and built/tested
out of tree; fuzz checks passed.

Earlier local profiles retain source hashes for the copied validation
tree; GitHub checks validate the published commit. No production
database/device, host installation or live privilege change is part of
validation. Existing prerequisite skips were retained; no test or
finding was suppressed to obtain a pass.


Series navigation: [Series
index](#638); [previous
step](#641). [Next
step](#643).

Review follow-up (2026-10-09): Restacked on the corrected preceding
branches. The SNMP status fix is introduced in #638, the
reindex/completion fixes in #640, and focused regression coverage in
#650.

Restacked tip validation: isolated Linux arm64 build and make check
passed (18 passed, 1 setuid prerequisite skip, no failures).
GitHub checks validate the updated published commits; earlier
profile/archive/sanitizer evidence above belongs to the previous
revisions.

Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
Explicitly select GNU C17 when the caller has not chosen a language
standard, preserving caller CC/CFLAGS precedence and warning about older
explicit modes. Probe compiler/linker hardening before enabling it,
retain preset fortify levels and support --disable-hardening. Autotools
and pthreads remain in place; no Annex K interfaces are introduced.

Step 6/14 of the complete C17/native-correctness series. Base: the
preceding series branch; merge/review in order so this PR shows only its
own step.

Earlier validation, before this review follow-up, in isolated copied
snapshots on Linux arm64:

- Clean GCC build and make check: 17 passed, 1 existing prerequisite
skips, 0 failures.
- ASan/UBSan: 16 passed, 2 existing privilege/allocator-interposition
prerequisite skips, 0 failures; no sanitizer findings.
- Production source/profile integrity verification against disposable
MariaDB and loopback SNMP fixtures passed.
- Release archive extracted into a separate source tree and built/tested
out of tree; fuzz checks passed.

Earlier local profiles retain source hashes for the copied validation
tree; GitHub checks validate the published commit. No production
database/device, host installation or live privilege change is part of
validation. Existing prerequisite skips were retained; no test or
finding was suppressed to obtain a pass.


Series navigation: [Series
index](#638); [previous
step](#642). [Next
step](#644).

Review follow-up (2026-10-09): Restacked on the corrected preceding
branches. The SNMP status fix is introduced in #638, the
reindex/completion fixes in #640, and focused regression coverage in
#650.

Restacked tip validation: isolated Linux arm64 build and make check
passed (18 passed, 1 setuid prerequisite skip, no failures).
GitHub checks validate the updated published commits; earlier
profile/archive/sanitizer evidence above belongs to the previous
revisions.

---------

Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The branch includes out-of-scope later-series changes and has unresolved uptime, SNMP timeout, and UDP reachability defects.

4 open findings
What changed in this PR

Introduces unified poll-result classification and SNMP batch persistence, but also includes later runtime and build-series changes outside the stated scope.

Changes:

  • Centralizes result parsing, uptime validation, and SNMP batch flushing.
  • Adds regression, fault, and parser coverage.
  • Adds unrelated CLI, ping, configuration, C17, and hardening changes.
File Description
.github/​workflows/​ci.yml Prints failed test logs.
Makefile.am Registers new test programs.
Makefile.in Generated test build rules.
configure.ac Adds C17 and hardening probes.
docs/​adr/​0001-c17-language-standard.md Documents C17 adoption.
ping.c Changes probe timing and address parsing.
poller.c Unifies result and SNMP batch handling.
poller.h Removes obsolete parser declarations.
spine.c Validates numeric CLI options.
tests/​fixtures/​snmpd.conf Extends SNMP fixtures.
tests/​fuzz/​fuzz_output_parser.c Fuzzes unified classification.
tests/​test_spine_cli_contracts.c Tests maximum host lists.
tests/​test_spine_contracts.c Updates multipart contracts.
tests/​test_spine_faults.c Tests long-community batching.
tests/​test_spine_reindex_contracts.c Adds reindex result contracts.
tests/​test_spine_regressions.c Expands integration regressions.
tests/​unit/​test_classify_result.c Tests result classification.
tests/​unit/​test_linked.c Updates linked regressions.
tests/​unit/​test_log_config.c Tests logging and configuration bounds.
tests/​unit/​test_ping_timing.c Tests ping timing and UDP behavior.
tests/​unit/​test_poll_host.c Tests mocked polling paths.
util.c Implements classification and safer logging/configuration.
util.h Exposes classified-result types.

🧠 Review effort: Balanced


💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread ping.c
Comment thread poller.c Outdated
Comment thread poller.c Outdated
Comment thread configure.ac Outdated
TheWitness pushed a commit that referenced this pull request Oct 11, 2026
…lity (#656)

Polling workers currently return permits before the parent can prove
their threads have exited, and database pool slots have no lease
identity. Add joined worker ownership and thread-owned,
generation-checked database leases before shared-state cleanup. Preserve
early ping-only output behavior and keep the tested external-script
limits and destination-aware upserts from #655.

Add explicit local/remote database TLS modes, protocol/cipher
constraints, opt-in checked JSON logging, and the paired-build benchmark
harness with its recorded original measurements. Legacy connection
behavior and text logging remain defaults. Unsupported TLS policies fail
closed. The updated man page documents the implemented contracts and
connector limitations.

Recovered work is isolated from the original uncommitted checkouts. The
CMake fork behavior suite is preserved separately on
`somethingwithproof/spine` branch
`fix/recovered-cmake-behavior-contracts`; all 27 native behavior tests
passed. Its fork-specific interfaces are not imported into Cacti's
Autotools source layout.

Validation for head `3db06ab315041187680b896c3658db0fbe8c60e3`:

- All 34 GitHub checks passed: GCC/Clang with MariaDB/MySQL clients,
Fedora 44, Rocky Linux 9, macOS, release distcheck, configure contracts,
static analysis, integration regressions, sanitizers, nightly ASan/TSan,
Memcheck, Helgrind, and concurrency soak.
- GitHub and independently measured Linux coverage: **396/396 changed
executable production lines (100%)**. Worker ownership: **71/71 lines
and 56/56 branches**; database leases: **65/65 lines and 72/72
branches**. Local production-source hashes exactly match this revision.
- Native suite: **20 passed, 2 Linux-only prerequisite skips**. Linux
suite: **24 passed, 1 expected prerequisite skip**, followed by real
database/SNMP/SNMPv3 fault contracts.
- Disposable TLS fixtures passed trusted-certificate, wrong-identity,
untrusted-CA, protocol/cipher mismatch, and plaintext rejection
contracts for local and remote connections.
- JSON encoder: 3 tests passed; benchmark evidence self-tests: 7 passed;
original recorded benchmark verified. Mandatory commit checks and
man-page lint passed.
- New paired Linux ARM benchmark compares exact #655 head `a8bd78d7`
with this head using identical GCC/dependencies/`-O2 -g0`, 20 measured
pairs plus 2 warmup pairs, 64 devices, 16 items/device, 4 workers,
alternating order. All 44 cycles verified all 1,024 expected outputs.
Median elapsed time: baseline 0.752536 s, candidate 0.749570 s; median
CPU time: baseline 0.023159 s, candidate 0.023171 s. Median paired
elapsed delta is -0.002300 s. This noisy loopback fixture establishes no
general performance improvement. TLS is disabled for both benchmark
builds.

**Final-head one-hour memory soak passed:** 5,612 cycles, each exiting
successfully and writing all 200 results; 5,609 cycles fitted after 3
warmups. RSS slope -7.6 KB/hour; projected growth over the run -8 KB,
within the 1,024 KB limit. [Final soak
run](https://github.com/Cacti/spine/actions/runs/38128178635). This PR
is ready for review. The older committed benchmark record remains
explicitly scoped to its original revisions. Original uncommitted
checkouts were preserved, and the recovery archive includes
restored-and-verified Git objects for all 1,238 orphan commit heads
captured during the audit.

Depends on #655 and its earlier stack. No live polling endpoints,
installations, database settings, or production privilege configuration
were changed.

## Current merge dependencies

Current base: `fix/post-c17-execution-hardening`. Immediate dependency:
[#655](#655). Next open PR:
[#658](#658).

Merge order: #641 → #644 → #654 → #646 → #648 → #649 → #650 → #651 →
#655 → #656 → #658 → #659.

This repository uses squash-only merges and deletes merged branches.
Record the old parent SHA, squash one PR at a time, then restack and
retarget dependents before merging the next PR. Re-run merge validation
and checks on the resulting heads; an ordinary clean merge does not
replace squash/restack validation.

---------

Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
TheWitness
TheWitness previously approved these changes Oct 12, 2026
Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

SNMP batches can cross endpoint boundaries, and oversized script output can still be persisted from a truncated prefix.

2 open findings
4 resolved since last review

🧠 Review effort: Balanced

Comment thread poller.c
Comment thread util.c
@somethingwithproof somethingwithproof changed the title fix(results): normalize output and unify SNMP batch persistence fix(results): [develop] normalize output and unify SNMP batch persistence Oct 12, 2026
… output

Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
@somethingwithproof

Copy link
Copy Markdown
Member Author

Fixed the two open findings: SNMP batches include the hostname, and a single full-capacity script read rejects a full buffer or embedded NUL. Added endpoint and output-boundary regressions. The pre-fix production probes fail and the corrected probes pass. The one-read/no-EOF-wait contract is preserved. Both bugs also exist in 1.2.x and have a minimal backport.

The complete stack passes its native suite (22 programs, two platform skips), 32 Python tests and the flawfinder critical gate. The pushed order is #641 -> #660 -> #644 -> #654 -> #646 -> #648 -> #649 -> #650 -> #651 -> #655 -> #658 -> #659. Current CI is rerunning.

@TheWitness
TheWitness merged commit 9be5f8a into develop Oct 12, 2026
24 checks passed
@TheWitness
TheWitness deleted the refactor/c17-series-04-results branch October 12, 2026 22:39
TheWitness pushed a commit that referenced this pull request Oct 14, 2026
…utput (#661)

Consecutive SNMP items with the same credentials can target different
hostnames or require different timeouts. Include hostname and timeout in
the batch key so each endpoint gets its own session. External scripts
also read one byte less than the allocated buffer, which makes a full
truncated response indistinguishable from a complete value. Read the
full capacity once and return U for a full buffer or embedded NUL.

Uptimes must be whole unsigned counters. Reject decimal, negative and
overflowing responses and check seconds-to-centiseconds conversion
before multiplication. Invalid values cannot advance reindex state.

Backports the confirmed bugs fixed in #641. Keeps the maintenance
branch’s single-read response contract, timeout handling and nonblocking
child reaping. Changes poller.c and extends the existing maintenance CI
branch filter to stacked fix branches. Release notes are collected in
#664.

Validation: native build and version smoke test passed through mise.
Compiled production hostname comparisons reject the old behavior and
pass with the fix. A probe linked to the maintenance production objects
passed short output, the maximum terminated value, overflow, embedded
NUL and a response whose child keeps stdout open; the last case returned
in under 0.5 seconds. The probe substitutes only semaphore boundaries
because macOS does not support the legacy unnamed semaphore
implementation. No database or live device was contacted. The modified
file passed the flawfinder critical gate.

Additional validation: the compiled maintenance timeout comparison fails
before the fix and passes afterward. Eleven production uptime parser
boundary cases pass under ASan/UBSan. The native build and all five
existing script-output probes still pass.

---------

Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
TheWitness pushed a commit that referenced this pull request Oct 15, 2026
…644)

Polling workers need larger stacks on platforms such as musl. This step
initializes worker attributes with the measured minimum, preserves
larger platform defaults, and reports attribute failures. Compile-time
bounds reject result-buffer sizes that overflow multiplication,
addition, or rounding. The default minimum remains 384 KiB.

The diff contains step 7 only: nine files for worker stack sizing and
its existing fault/worker contracts. Native helper changes remain in
#654.

Immediate dependency: #660 (`refactor/c17-series-05-06-runtime-build`).
#641 is already merged. Merge order: #660 → #644 → #654 → #646 → #648 →
#649 → #650 → #651 → #655 → #658 → #659.

The branch includes the current #660 parent. This dependency repair
preserves the reviewed source tree. After each squash merge, restack and
retarget the next dependent PR before merging it; the repository deletes
merged branches.

Validation: macOS build and `make check` passed, with 13 passes and two
Linux-only prerequisite skips. Previous stack-bound compiler probes and
worker fault contracts remain in the branch. GitHub checks must pass on
the updated head before merge.

---------

Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants