Repository navigation
fix(results): [develop] normalize output and unify SNMP batch persistence - #641
Conversation
a4e0559 to
a93d363
Compare
2f89674 to
3d1511b
Compare
a93d363 to
49733e7
Compare
3d1511b to
07962c3
Compare
Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
07962c3 to
e29a4d0
Compare
49733e7 to
98be364
Compare
Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
e29a4d0 to
873c982
Compare
98be364 to
3355a4b
Compare
…638) Introduce bounded native interfaces and grouped runtime state, then link regression and fault contracts against the real poller implementations. The contracts capture CLI/configuration input, availability combinations, SNMP request/response ownership, script framing, reindexing, output persistence, worker completion and shutdown. This foundational change is larger because the shared interfaces and their existing production contracts must move together; later changes have separate review boundaries. Step 1/14 of the complete C17/native-correctness series. Base: develop. Earlier validation, before this review follow-up, in isolated copied snapshots on Linux arm64: - Clean GCC build and make check: 13 passed, 0 existing prerequisite skips, 0 failures. - ASan/UBSan: 12 passed, 1 existing privilege/allocator-interposition prerequisite skips, 0 failures; no sanitizer findings. - Production source/profile integrity verification against disposable MariaDB and loopback SNMP fixtures passed. - Release archive extracted into a separate source tree and built/tested out of tree; fuzz checks passed. Earlier local profiles retain source hashes for the copied validation tree; GitHub checks validate the published commit. No production database/device, host installation or live privilege change is part of validation. Existing prerequisite skips were retained; no test or finding was suppressed to obtain a pass. The worker contracts publish owned heap state and release it only after joining workers and restoring the runtime table. Coverage counters use atomic updates; cppcheck receives an explicit model of the fatal exit contract. Fixture images use public mirrors to avoid shared Docker Hub pull limits. Series navigation: review and merge in order. Every PR targets the preceding branch so its diff contains one step. 1. [refactor(core): add bounded native interfaces and polling contracts](#638) 2. [fix(privileges): drop root before input and retain owned ICMP sockets](#639) 3. [fix(poller): isolate worker failures and bound database reconnects](#640) 4. [fix(results): normalize output and unify SNMP batch persistence](#641) 5. [fix(runtime): bound probe retries and validate CLI and configuration](#642) 6. [build: select GNU C17 explicitly and probe hardening support](#643) 7. [fix(threads): size worker stacks and guard size arithmetic](#644) 8. [refactor(c17): simplify ownership guards and share test support](#645) 9. [refactor(core): extract configuration polling and runtime modules](#646) 10. [fix(boundaries): validate script slots and make cleanup portable](#647) 11. [refactor(layout): organize responsibilities and separate main from runtime](#648) 12. [docs(license): adopt SPDX notices and retain contributor credits](#649) 13. [test(runtime): measure all suites and cover protocol boundaries](#650) 14. [build(ci): validate connectors time width and native C17 modules](#651) Review follow-up (2026-10-09): Publish the decoded GET/GETNEXT status back to the host so availability checks observe exception and missing-response failures, while preserving the legacy transport-success behavior for agent PDU errors. Updated step 1 snapshot: isolated Linux arm64 build and make check passed (13 passed, no skips or failures). The restacked tip passes all five clang-tidy gate regression tests and real clang-tidy analysis (26 reviewed baseline findings, no new findings). Actionlint and matching DCO trailers pass. The setuid test also passes when run in the separate disposable root container. GitHub checks validate the updated published commits; earlier profile/archive/sanitizer evidence above belongs to the previous revisions. --------- Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
) Propagate database and device failures through worker completion rather than terminating the entire poller. Count reconnect attempts even with stale EINTR, apply session SQL mode after reconnect, preserve failed reindex assertions, keep completion writes outside the thread-accounting lock and redact script command arguments from logs. Linker fault injection drives the real connection, query, worker and startup paths. Step 3/14 of the complete C17/native-correctness series. Base: the preceding series branch; merge/review in order so this PR shows only its own step. Earlier validation, before this review follow-up, in isolated copied snapshots on Linux arm64: - Clean GCC build and make check: 13 passed, 1 existing prerequisite skips, 0 failures. - ASan/UBSan: 12 passed, 2 existing privilege/allocator-interposition prerequisite skips, 0 failures; no sanitizer findings. - Production source/profile integrity verification against disposable MariaDB and loopback SNMP fixtures passed. - Release archive extracted into a separate source tree and built/tested out of tree; fuzz checks passed. Earlier local profiles retain source hashes for the copied validation tree; GitHub checks validate the published commit. No production database/device, host installation or live privilege change is part of validation. Existing prerequisite skips were retained; no test or finding was suppressed to obtain a pass. Series navigation: [Series index](#638); [previous step](#639). [Next step](#641). Review follow-up (2026-10-09): Treat a failed reindex SELECT as a device polling failure and stop item collection. Finish each partition's host_errors write before publishing completion; record device polling time before the final schedule update, and mark the device complete only after all required writes succeed. Database waits remain outside LOCK_THDET. Updated step 3 snapshot: isolated Linux arm64 build and make check passed (13 passed, 1 setuid prerequisite skip, no failures). The restacked tip passes all five clang-tidy gate regression tests and real clang-tidy analysis (26 reviewed baseline findings, no new findings). Actionlint and matching DCO trailers pass. The setuid test also passes when run in the separate disposable root container. GitHub checks validate the updated published commits; earlier profile/archive/sanitizer evidence above belongs to the previous revisions. --------- Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
…642) Give each ICMP retry its own deadline, accept connected UDP replies and resolve transport-qualified/bracketed IPv6 names correctly. Reject non-numeric thread/host options and clamp thread limits. Avoid large logging buffers on thread stacks and reject truncated configuration keys/values/lines. Native socket/time interception and logging/configuration suites check retries, formatting, bounds and cleanup. Step 5/14 of the complete C17/native-correctness series. Base: the preceding series branch; merge/review in order so this PR shows only its own step. Earlier validation, before this review follow-up, in isolated copied snapshots on Linux arm64: - Clean GCC build and make check: 17 passed, 1 existing prerequisite skips, 0 failures. - ASan/UBSan: 16 passed, 2 existing privilege/allocator-interposition prerequisite skips, 0 failures; no sanitizer findings. - Production source/profile integrity verification against disposable MariaDB and loopback SNMP fixtures passed. - Release archive extracted into a separate source tree and built/tested out of tree; fuzz checks passed. Earlier local profiles retain source hashes for the copied validation tree; GitHub checks validate the published commit. No production database/device, host installation or live privilege change is part of validation. Existing prerequisite skips were retained; no test or finding was suppressed to obtain a pass. Series navigation: [Series index](#638); [previous step](#641). [Next step](#643). Review follow-up (2026-10-09): Restacked on the corrected preceding branches. The SNMP status fix is introduced in #638, the reindex/completion fixes in #640, and focused regression coverage in #650. Restacked tip validation: isolated Linux arm64 build and make check passed (18 passed, 1 setuid prerequisite skip, no failures). GitHub checks validate the updated published commits; earlier profile/archive/sanitizer evidence above belongs to the previous revisions. Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
Explicitly select GNU C17 when the caller has not chosen a language standard, preserving caller CC/CFLAGS precedence and warning about older explicit modes. Probe compiler/linker hardening before enabling it, retain preset fortify levels and support --disable-hardening. Autotools and pthreads remain in place; no Annex K interfaces are introduced. Step 6/14 of the complete C17/native-correctness series. Base: the preceding series branch; merge/review in order so this PR shows only its own step. Earlier validation, before this review follow-up, in isolated copied snapshots on Linux arm64: - Clean GCC build and make check: 17 passed, 1 existing prerequisite skips, 0 failures. - ASan/UBSan: 16 passed, 2 existing privilege/allocator-interposition prerequisite skips, 0 failures; no sanitizer findings. - Production source/profile integrity verification against disposable MariaDB and loopback SNMP fixtures passed. - Release archive extracted into a separate source tree and built/tested out of tree; fuzz checks passed. Earlier local profiles retain source hashes for the copied validation tree; GitHub checks validate the published commit. No production database/device, host installation or live privilege change is part of validation. Existing prerequisite skips were retained; no test or finding was suppressed to obtain a pass. Series navigation: [Series index](#638); [previous step](#642). [Next step](#644). Review follow-up (2026-10-09): Restacked on the corrected preceding branches. The SNMP status fix is introduced in #638, the reindex/completion fixes in #640, and focused regression coverage in #650. Restacked tip validation: isolated Linux arm64 build and make check passed (18 passed, 1 setuid prerequisite skip, no failures). GitHub checks validate the updated published commits; earlier profile/archive/sanitizer evidence above belongs to the previous revisions. --------- Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
There was a problem hiding this comment.
🟡 Changes recommended
The branch includes out-of-scope later-series changes and has unresolved uptime, SNMP timeout, and UDP reachability defects.
4 open findings
What changed in this PR
Introduces unified poll-result classification and SNMP batch persistence, but also includes later runtime and build-series changes outside the stated scope.
Changes:
- Centralizes result parsing, uptime validation, and SNMP batch flushing.
- Adds regression, fault, and parser coverage.
- Adds unrelated CLI, ping, configuration, C17, and hardening changes.
| File | Description |
|---|---|
.github/workflows/ci.yml |
Prints failed test logs. |
Makefile.am |
Registers new test programs. |
Makefile.in |
Generated test build rules. |
configure.ac |
Adds C17 and hardening probes. |
docs/adr/0001-c17-language-standard.md |
Documents C17 adoption. |
ping.c |
Changes probe timing and address parsing. |
poller.c |
Unifies result and SNMP batch handling. |
poller.h |
Removes obsolete parser declarations. |
spine.c |
Validates numeric CLI options. |
tests/fixtures/snmpd.conf |
Extends SNMP fixtures. |
tests/fuzz/fuzz_output_parser.c |
Fuzzes unified classification. |
tests/test_spine_cli_contracts.c |
Tests maximum host lists. |
tests/test_spine_contracts.c |
Updates multipart contracts. |
tests/test_spine_faults.c |
Tests long-community batching. |
tests/test_spine_reindex_contracts.c |
Adds reindex result contracts. |
tests/test_spine_regressions.c |
Expands integration regressions. |
tests/unit/test_classify_result.c |
Tests result classification. |
tests/unit/test_linked.c |
Updates linked regressions. |
tests/unit/test_log_config.c |
Tests logging and configuration bounds. |
tests/unit/test_ping_timing.c |
Tests ping timing and UDP behavior. |
tests/unit/test_poll_host.c |
Tests mocked polling paths. |
util.c |
Implements classification and safer logging/configuration. |
util.h |
Exposes classified-result types. |
🧠 Review effort: Balanced
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
…lity (#656) Polling workers currently return permits before the parent can prove their threads have exited, and database pool slots have no lease identity. Add joined worker ownership and thread-owned, generation-checked database leases before shared-state cleanup. Preserve early ping-only output behavior and keep the tested external-script limits and destination-aware upserts from #655. Add explicit local/remote database TLS modes, protocol/cipher constraints, opt-in checked JSON logging, and the paired-build benchmark harness with its recorded original measurements. Legacy connection behavior and text logging remain defaults. Unsupported TLS policies fail closed. The updated man page documents the implemented contracts and connector limitations. Recovered work is isolated from the original uncommitted checkouts. The CMake fork behavior suite is preserved separately on `somethingwithproof/spine` branch `fix/recovered-cmake-behavior-contracts`; all 27 native behavior tests passed. Its fork-specific interfaces are not imported into Cacti's Autotools source layout. Validation for head `3db06ab315041187680b896c3658db0fbe8c60e3`: - All 34 GitHub checks passed: GCC/Clang with MariaDB/MySQL clients, Fedora 44, Rocky Linux 9, macOS, release distcheck, configure contracts, static analysis, integration regressions, sanitizers, nightly ASan/TSan, Memcheck, Helgrind, and concurrency soak. - GitHub and independently measured Linux coverage: **396/396 changed executable production lines (100%)**. Worker ownership: **71/71 lines and 56/56 branches**; database leases: **65/65 lines and 72/72 branches**. Local production-source hashes exactly match this revision. - Native suite: **20 passed, 2 Linux-only prerequisite skips**. Linux suite: **24 passed, 1 expected prerequisite skip**, followed by real database/SNMP/SNMPv3 fault contracts. - Disposable TLS fixtures passed trusted-certificate, wrong-identity, untrusted-CA, protocol/cipher mismatch, and plaintext rejection contracts for local and remote connections. - JSON encoder: 3 tests passed; benchmark evidence self-tests: 7 passed; original recorded benchmark verified. Mandatory commit checks and man-page lint passed. - New paired Linux ARM benchmark compares exact #655 head `a8bd78d7` with this head using identical GCC/dependencies/`-O2 -g0`, 20 measured pairs plus 2 warmup pairs, 64 devices, 16 items/device, 4 workers, alternating order. All 44 cycles verified all 1,024 expected outputs. Median elapsed time: baseline 0.752536 s, candidate 0.749570 s; median CPU time: baseline 0.023159 s, candidate 0.023171 s. Median paired elapsed delta is -0.002300 s. This noisy loopback fixture establishes no general performance improvement. TLS is disabled for both benchmark builds. **Final-head one-hour memory soak passed:** 5,612 cycles, each exiting successfully and writing all 200 results; 5,609 cycles fitted after 3 warmups. RSS slope -7.6 KB/hour; projected growth over the run -8 KB, within the 1,024 KB limit. [Final soak run](https://github.com/Cacti/spine/actions/runs/38128178635). This PR is ready for review. The older committed benchmark record remains explicitly scoped to its original revisions. Original uncommitted checkouts were preserved, and the recovery archive includes restored-and-verified Git objects for all 1,238 orphan commit heads captured during the audit. Depends on #655 and its earlier stack. No live polling endpoints, installations, database settings, or production privilege configuration were changed. ## Current merge dependencies Current base: `fix/post-c17-execution-hardening`. Immediate dependency: [#655](#655). Next open PR: [#658](#658). Merge order: #641 → #644 → #654 → #646 → #648 → #649 → #650 → #651 → #655 → #656 → #658 → #659. This repository uses squash-only merges and deletes merged branches. Record the old parent SHA, squash one PR at a time, then restack and retarget dependents before merging the next PR. Re-run merge validation and checks on the resulting heads; an ordinary clean merge does not replace squash/restack validation. --------- Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
… output Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
|
Fixed the two open findings: SNMP batches include the hostname, and a single full-capacity script read rejects a full buffer or embedded NUL. Added endpoint and output-boundary regressions. The pre-fix production probes fail and the corrected probes pass. The one-read/no-EOF-wait contract is preserved. Both bugs also exist in 1.2.x and have a minimal backport. The complete stack passes its native suite (22 programs, two platform skips), 32 Python tests and the flawfinder critical gate. The pushed order is #641 -> #660 -> #644 -> #654 -> #646 -> #648 -> #649 -> #650 -> #651 -> #655 -> #658 -> #659. Current CI is rerunning. |
…utput (#661) Consecutive SNMP items with the same credentials can target different hostnames or require different timeouts. Include hostname and timeout in the batch key so each endpoint gets its own session. External scripts also read one byte less than the allocated buffer, which makes a full truncated response indistinguishable from a complete value. Read the full capacity once and return U for a full buffer or embedded NUL. Uptimes must be whole unsigned counters. Reject decimal, negative and overflowing responses and check seconds-to-centiseconds conversion before multiplication. Invalid values cannot advance reindex state. Backports the confirmed bugs fixed in #641. Keeps the maintenance branch’s single-read response contract, timeout handling and nonblocking child reaping. Changes poller.c and extends the existing maintenance CI branch filter to stacked fix branches. Release notes are collected in #664. Validation: native build and version smoke test passed through mise. Compiled production hostname comparisons reject the old behavior and pass with the fix. A probe linked to the maintenance production objects passed short output, the maximum terminated value, overflow, embedded NUL and a response whose child keeps stdout open; the last case returned in under 0.5 seconds. The probe substitutes only semaphore boundaries because macOS does not support the legacy unnamed semaphore implementation. No database or live device was contacted. The modified file passed the flawfinder critical gate. Additional validation: the compiled maintenance timeout comparison fails before the fix and passes afterward. Eleven production uptime parser boundary cases pass under ASan/UBSan. The native build and all five existing script-output probes still pass. --------- Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
…644) Polling workers need larger stacks on platforms such as musl. This step initializes worker attributes with the measured minimum, preserves larger platform defaults, and reports attribute failures. Compile-time bounds reject result-buffer sizes that overflow multiplication, addition, or rounding. The default minimum remains 384 KiB. The diff contains step 7 only: nine files for worker stack sizing and its existing fault/worker contracts. Native helper changes remain in #654. Immediate dependency: #660 (`refactor/c17-series-05-06-runtime-build`). #641 is already merged. Merge order: #660 → #644 → #654 → #646 → #648 → #649 → #650 → #651 → #655 → #658 → #659. The branch includes the current #660 parent. This dependency repair preserves the reviewed source tree. After each squash merge, restack and retarget the next dependent PR before merging it; the repository deletes merged branches. Validation: macOS build and `make check` passed, with 13 passes and two Linux-only prerequisite skips. Previous stack-bound compiler probes and worker fault contracts remain in the branch. GitHub checks must pass on the updated head before merge. --------- Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>



Classify numeric, undefined, multipart and Cacti octet results through one parser, normalize undefined agent values before persistence and flush every SNMP batch consistently. Validate whole non-negative uptimes, retain unknown reindex baselines and free script results through their owning pointer. The mocked poll_host suite checks session-profile boundaries, spike handling, output_regex input and completion writes.
Originally step 4/14 of the C17/native-correctness series; its preceding steps have merged. The current base is develop.
Earlier validation, before this review follow-up, in isolated copied snapshots on Linux arm64:
Earlier local profiles retain source hashes for the copied validation tree; GitHub checks validate the published commit. No production database/device, host installation or live privilege change is part of validation. Existing prerequisite skips were retained; no test or finding was suppressed to obtain a pass.
Review follow-up (2026-10-09): Restacked on the corrected preceding branches. The SNMP status fix is introduced in #638, the reindex/completion fixes in #640, and focused regression coverage in #650.
Restacked tip validation: isolated Linux arm64 build and make check passed (18 passed, 1 setuid prerequisite skip, no failures).
GitHub checks validate the updated published commits; earlier profile/archive/sanitizer evidence above belongs to the previous revisions.
Current merge dependencies
Current base:
develop. Immediate dependency:develop(earlier steps already merged). Next open PR: #644.Merge order: #641 → #644 → #654 → #646 → #648 → #649 → #650 → #651 → #655 → #658 → #659.
This repository uses squash-only merges and deletes merged branches. Record the old parent SHA, squash one PR at a time, then restack and retarget dependents before merging the next PR. Re-run merge validation and checks on the resulting heads; an ordinary clean merge does not replace squash/restack validation.