fix(poller): isolate worker failures and bound database reconnects by somethingwithproof · Pull Request #640 · Cacti/spine

somethingwithproof · 2026-10-09T21:30:31Z

Propagate database and device failures through worker completion rather than terminating the entire poller. Count reconnect attempts even with stale EINTR, apply session SQL mode after reconnect, preserve failed reindex assertions, keep completion writes outside the thread-accounting lock and redact script command arguments from logs. Linker fault injection drives the real connection, query, worker and startup paths.

Step 3/14 of the complete C17/native-correctness series. Base: the preceding series branch; merge/review in order so this PR shows only its own step.

Earlier validation, before this review follow-up, in isolated copied snapshots on Linux arm64:

Clean GCC build and make check: 13 passed, 1 existing prerequisite skips, 0 failures.
ASan/UBSan: 12 passed, 2 existing privilege/allocator-interposition prerequisite skips, 0 failures; no sanitizer findings.
Production source/profile integrity verification against disposable MariaDB and loopback SNMP fixtures passed.
Release archive extracted into a separate source tree and built/tested out of tree; fuzz checks passed.

Earlier local profiles retain source hashes for the copied validation tree; GitHub checks validate the published commit. No production database/device, host installation or live privilege change is part of validation. Existing prerequisite skips were retained; no test or finding was suppressed to obtain a pass.

Series navigation: Series index; previous step. Next step.

Review follow-up (2026-10-09): Treat a failed reindex SELECT as a device polling failure and stop item collection. Finish each partition's host_errors write before publishing completion; record device polling time before the final schedule update, and mark the device complete only after all required writes succeed. Database waits remain outside LOCK_THDET.

Updated step 3 snapshot: isolated Linux arm64 build and make check passed (13 passed, 1 setuid prerequisite skip, no failures).
The restacked tip passes all five clang-tidy gate regression tests and real clang-tidy analysis (26 reviewed baseline findings, no new findings). Actionlint and matching DCO trailers pass. The setuid test also passes when run in the separate disposable root container.
GitHub checks validate the updated published commits; earlier profile/archive/sanitizer evidence above belongs to the previous revisions.

Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>

…638) Introduce bounded native interfaces and grouped runtime state, then link regression and fault contracts against the real poller implementations. The contracts capture CLI/configuration input, availability combinations, SNMP request/response ownership, script framing, reindexing, output persistence, worker completion and shutdown. This foundational change is larger because the shared interfaces and their existing production contracts must move together; later changes have separate review boundaries. Step 1/14 of the complete C17/native-correctness series. Base: develop. Earlier validation, before this review follow-up, in isolated copied snapshots on Linux arm64: - Clean GCC build and make check: 13 passed, 0 existing prerequisite skips, 0 failures. - ASan/UBSan: 12 passed, 1 existing privilege/allocator-interposition prerequisite skips, 0 failures; no sanitizer findings. - Production source/profile integrity verification against disposable MariaDB and loopback SNMP fixtures passed. - Release archive extracted into a separate source tree and built/tested out of tree; fuzz checks passed. Earlier local profiles retain source hashes for the copied validation tree; GitHub checks validate the published commit. No production database/device, host installation or live privilege change is part of validation. Existing prerequisite skips were retained; no test or finding was suppressed to obtain a pass. The worker contracts publish owned heap state and release it only after joining workers and restoring the runtime table. Coverage counters use atomic updates; cppcheck receives an explicit model of the fatal exit contract. Fixture images use public mirrors to avoid shared Docker Hub pull limits. Series navigation: review and merge in order. Every PR targets the preceding branch so its diff contains one step. 1. [refactor(core): add bounded native interfaces and polling contracts](#638) 2. [fix(privileges): drop root before input and retain owned ICMP sockets](#639) 3. [fix(poller): isolate worker failures and bound database reconnects](#640) 4. [fix(results): normalize output and unify SNMP batch persistence](#641) 5. [fix(runtime): bound probe retries and validate CLI and configuration](#642) 6. [build: select GNU C17 explicitly and probe hardening support](#643) 7. [fix(threads): size worker stacks and guard size arithmetic](#644) 8. [refactor(c17): simplify ownership guards and share test support](#645) 9. [refactor(core): extract configuration polling and runtime modules](#646) 10. [fix(boundaries): validate script slots and make cleanup portable](#647) 11. [refactor(layout): organize responsibilities and separate main from runtime](#648) 12. [docs(license): adopt SPDX notices and retain contributor credits](#649) 13. [test(runtime): measure all suites and cover protocol boundaries](#650) 14. [build(ci): validate connectors time width and native C17 modules](#651) Review follow-up (2026-10-09): Publish the decoded GET/GETNEXT status back to the host so availability checks observe exception and missing-response failures, while preserving the legacy transport-success behavior for agent PDU errors. Updated step 1 snapshot: isolated Linux arm64 build and make check passed (13 passed, no skips or failures). The restacked tip passes all five clang-tidy gate regression tests and real clang-tidy analysis (26 reviewed baseline findings, no new findings). Actionlint and matching DCO trailers pass. The setuid test also passes when run in the separate disposable root container. GitHub checks validate the updated published commits; earlier profile/archive/sanitizer evidence above belongs to the previous revisions. --------- Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>

…642) Give each ICMP retry its own deadline, accept connected UDP replies and resolve transport-qualified/bracketed IPv6 names correctly. Reject non-numeric thread/host options and clamp thread limits. Avoid large logging buffers on thread stacks and reject truncated configuration keys/values/lines. Native socket/time interception and logging/configuration suites check retries, formatting, bounds and cleanup. Step 5/14 of the complete C17/native-correctness series. Base: the preceding series branch; merge/review in order so this PR shows only its own step. Earlier validation, before this review follow-up, in isolated copied snapshots on Linux arm64: - Clean GCC build and make check: 17 passed, 1 existing prerequisite skips, 0 failures. - ASan/UBSan: 16 passed, 2 existing privilege/allocator-interposition prerequisite skips, 0 failures; no sanitizer findings. - Production source/profile integrity verification against disposable MariaDB and loopback SNMP fixtures passed. - Release archive extracted into a separate source tree and built/tested out of tree; fuzz checks passed. Earlier local profiles retain source hashes for the copied validation tree; GitHub checks validate the published commit. No production database/device, host installation or live privilege change is part of validation. Existing prerequisite skips were retained; no test or finding was suppressed to obtain a pass. Series navigation: [Series index](#638); [previous step](#641). [Next step](#643). Review follow-up (2026-10-09): Restacked on the corrected preceding branches. The SNMP status fix is introduced in #638, the reindex/completion fixes in #640, and focused regression coverage in #650. Restacked tip validation: isolated Linux arm64 build and make check passed (18 passed, 1 setuid prerequisite skip, no failures). GitHub checks validate the updated published commits; earlier profile/archive/sanitizer evidence above belongs to the previous revisions. Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>

Explicitly select GNU C17 when the caller has not chosen a language standard, preserving caller CC/CFLAGS precedence and warning about older explicit modes. Probe compiler/linker hardening before enabling it, retain preset fortify levels and support --disable-hardening. Autotools and pthreads remain in place; no Annex K interfaces are introduced. Step 6/14 of the complete C17/native-correctness series. Base: the preceding series branch; merge/review in order so this PR shows only its own step. Earlier validation, before this review follow-up, in isolated copied snapshots on Linux arm64: - Clean GCC build and make check: 17 passed, 1 existing prerequisite skips, 0 failures. - ASan/UBSan: 16 passed, 2 existing privilege/allocator-interposition prerequisite skips, 0 failures; no sanitizer findings. - Production source/profile integrity verification against disposable MariaDB and loopback SNMP fixtures passed. - Release archive extracted into a separate source tree and built/tested out of tree; fuzz checks passed. Earlier local profiles retain source hashes for the copied validation tree; GitHub checks validate the published commit. No production database/device, host installation or live privilege change is part of validation. Existing prerequisite skips were retained; no test or finding was suppressed to obtain a pass. Series navigation: [Series index](#638); [previous step](#642). [Next step](#644). Review follow-up (2026-10-09): Restacked on the corrected preceding branches. The SNMP status fix is introduced in #638, the reindex/completion fixes in #640, and focused regression coverage in #650. Restacked tip validation: isolated Linux arm64 build and make check passed (18 passed, 1 setuid prerequisite skip, no failures). GitHub checks validate the updated published commits; earlier profile/archive/sanitizer evidence above belongs to the previous revisions. --------- Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>

Apply the behavior-preserving C17 cleanup to allocations, array sizing, declarations, guards and ownership. Preserve the captured runtime baseline and reorganize reusable test state, PHP child fixtures and release/debug helpers. Tests cover keyword locking, process selection, helper arguments and archive contents; the bounded memory-soak harness remains available. Step 8/14 of the complete C17/native-correctness series. Base: the preceding series branch; merge/review in order so this PR shows only its own step. Earlier validation, before this review follow-up, in isolated copied snapshots on Linux arm64: - Clean GCC build and make check: 18 passed, 1 existing prerequisite skips, 0 failures. - ASan/UBSan: 17 passed, 2 existing privilege/allocator-interposition prerequisite skips, 0 failures; no sanitizer findings. - Production source/profile integrity verification against disposable MariaDB and loopback SNMP fixtures passed. - Release archive extracted into a separate source tree and built/tested out of tree; fuzz checks passed. Earlier local profiles retain source hashes for the copied validation tree; GitHub checks validate the published commit. No production database/device, host installation or live privilege change is part of validation. Existing prerequisite skips were retained; no test or finding was suppressed to obtain a pass. Series navigation: [Series index](#638); [previous step](#644). [Next step](#646). Review follow-up (2026-10-09): Restacked on the corrected preceding branches. The SNMP status fix is introduced in #638, the reindex/completion fixes in #640, and focused regression coverage in #650. Restacked tip validation: isolated Linux arm64 build and make check passed (18 passed, 1 setuid prerequisite skip, no failures). GitHub checks validate the updated published commits; earlier profile/archive/sanitizer evidence above belongs to the previous revisions. Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>

Validate PHP reader process tables and configured/array slot bounds before descriptor access or recovery, and keep owned undefined-result behavior. Make ctype input unsigned, bound shared formatting, enforce monotonic child cleanup budgets and keep BSD spawn/descriptor behavior portable. Tests cover invalid slots, log boundaries, child cleanup, startup and release helpers. Step 10/14 of the complete C17/native-correctness series. Base: the preceding series branch; merge/review in order so this PR shows only its own step. Earlier validation, before this review follow-up, in isolated copied snapshots on Linux arm64: - Clean GCC build and make check: 18 passed, 1 existing prerequisite skips, 0 failures. - ASan/UBSan: 17 passed, 2 existing privilege/allocator-interposition prerequisite skips, 0 failures; no sanitizer findings. - Production source/profile integrity verification against disposable MariaDB and loopback SNMP fixtures passed. - Release archive extracted into a separate source tree and built/tested out of tree; fuzz checks passed. Earlier local profiles retain source hashes for the copied validation tree; GitHub checks validate the published commit. No production database/device, host installation or live privilege change is part of validation. Existing prerequisite skips were retained; no test or finding was suppressed to obtain a pass. Series navigation: [Series index](#638); [previous step](#646). [Next step](#648). Review follow-up (2026-10-09): Restacked on the corrected preceding branches. The SNMP status fix is introduced in #638, the reindex/completion fixes in #640, and focused regression coverage in #650. Restacked tip validation: isolated Linux arm64 build and make check passed (18 passed, 1 setuid prerequisite skip, no failures). GitHub checks validate the updated published commits; earlier profile/archive/sanitizer evidence above belongs to the previous revisions. Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>

…#639) Open and own the required ICMP sockets before permanently dropping setuid root, then parse input without elevated identity. Close-on-exec socket ownership, shared IPv4/IPv6 dispatch and fault-injected identity/capability calls are covered. The owned fixture tests check deny/allow behavior without contacting external devices. Step 2/14 of the complete C17/native-correctness series. Base: the preceding series branch; merge/review in order so this PR shows only its own step. Earlier validation, before this review follow-up, in isolated copied snapshots on Linux arm64: - Clean GCC build and make check: 13 passed, 1 existing prerequisite skips, 0 failures. - ASan/UBSan: 12 passed, 2 existing privilege/allocator-interposition prerequisite skips, 0 failures; no sanitizer findings. - Production source/profile integrity verification against disposable MariaDB and loopback SNMP fixtures passed. - Release archive extracted into a separate source tree and built/tested out of tree; fuzz checks passed. Earlier local profiles retain source hashes for the copied validation tree; GitHub checks validate the published commit. No production database/device, host installation or live privilege change is part of validation. Existing prerequisite skips were retained; no test or finding was suppressed to obtain a pass. Series navigation: [Series index](#638); [previous step](#638). [Next step](#640). Review follow-up (2026-10-09): Restacked on the corrected preceding branches. The SNMP status fix is introduced in #638, the reindex/completion fixes in #640, and focused regression coverage in #650. Restacked tip validation: isolated Linux arm64 build and make check passed (18 passed, 1 setuid prerequisite skip, no failures). GitHub checks validate the updated published commits; earlier profile/archive/sanitizer evidence above belongs to the previous revisions. --------- Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>

…ence (#641) Classify numeric, undefined, multipart and Cacti octet results through one parser, normalize undefined agent values before persistence and flush every SNMP batch consistently. Validate whole non-negative uptimes, retain unknown reindex baselines and free script results through their owning pointer. The mocked poll_host suite checks session-profile boundaries, spike handling, output_regex input and completion writes. Originally step 4/14 of the C17/native-correctness series; its preceding steps have merged. The current base is develop. Earlier validation, before this review follow-up, in isolated copied snapshots on Linux arm64: - Clean GCC build and make check: 15 passed, 1 existing prerequisite skips, 0 failures. - ASan/UBSan: 14 passed, 2 existing privilege/allocator-interposition prerequisite skips, 0 failures; no sanitizer findings. - Production source/profile integrity verification against disposable MariaDB and loopback SNMP fixtures passed. - Release archive extracted into a separate source tree and built/tested out of tree; fuzz checks passed. Earlier local profiles retain source hashes for the copied validation tree; GitHub checks validate the published commit. No production database/device, host installation or live privilege change is part of validation. Existing prerequisite skips were retained; no test or finding was suppressed to obtain a pass. Review follow-up (2026-10-09): Restacked on the corrected preceding branches. The SNMP status fix is introduced in #638, the reindex/completion fixes in #640, and focused regression coverage in #650. Restacked tip validation: isolated Linux arm64 build and make check passed (18 passed, 1 setuid prerequisite skip, no failures). GitHub checks validate the updated published commits; earlier profile/archive/sanitizer evidence above belongs to the previous revisions. ## Current merge dependencies Current base: `develop`. Immediate dependency: `develop` (earlier steps already merged). Next open PR: [#644](#644). Merge order: #641 → #644 → #654 → #646 → #648 → #649 → #650 → #651 → #655 → #658 → #659. This repository uses squash-only merges and deletes merged branches. Record the old parent SHA, squash one PR at a time, then restack and retarget dependents before merging the next PR. Re-run merge validation and checks on the resulting heads; an ordinary clean merge does not replace squash/restack validation. --------- Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>

somethingwithproof requested a review from a team as a code owner October 9, 2026 21:30

somethingwithproof requested review from cigamit and removed request for a team October 9, 2026 21:30

somethingwithproof force-pushed the refactor/c17-series-02-privileges branch from 43a54fb to de4477f Compare October 9, 2026 21:44

somethingwithproof force-pushed the refactor/c17-series-03-worker-failures branch from 2f89674 to 3d1511b Compare October 9, 2026 21:44

This was referenced Oct 9, 2026

refactor(core): add bounded native interfaces and polling contracts #638

Merged

fix(privileges): drop root before input and retain owned ICMP sockets #639

Merged

fix(results): [develop] normalize output and unify SNMP batch persistence #641

Merged

somethingwithproof force-pushed the refactor/c17-series-02-privileges branch from de4477f to ccb45cd Compare October 9, 2026 21:54

somethingwithproof force-pushed the refactor/c17-series-03-worker-failures branch 2 times, most recently from 07962c3 to e29a4d0 Compare October 9, 2026 23:01

somethingwithproof force-pushed the refactor/c17-series-02-privileges branch from ccb45cd to 8573497 Compare October 9, 2026 23:01

somethingwithproof added 2 commits October 9, 2026 16:27

fix(poller): isolate worker failures and bound database reconnects

cafb1ef

Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>

fix(poller): fail reindex reads and settle writes before scheduling

873c982

Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>

somethingwithproof force-pushed the refactor/c17-series-03-worker-failures branch from e29a4d0 to 873c982 Compare October 10, 2026 00:17

somethingwithproof force-pushed the refactor/c17-series-02-privileges branch from 8573497 to 11ad497 Compare October 10, 2026 00:17

somethingwithproof added the bug label Oct 10, 2026

somethingwithproof self-assigned this Oct 10, 2026

TheWitness approved these changes Oct 11, 2026

View reviewed changes

TheWitness merged commit c256e5b into refactor/c17-series-02-privileges Oct 11, 2026
20 of 38 checks passed

TheWitness deleted the refactor/c17-series-03-worker-failures branch October 11, 2026 00:04

Navigation Menu

Repository navigation

fix(poller): isolate worker failures and bound database reconnects - #640

fix(poller): isolate worker failures and bound database reconnects#640
TheWitness merged 2 commits into
refactor/c17-series-02-privilegesfrom
refactor/c17-series-03-worker-failures

somethingwithproof commented Oct 9, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

somethingwithproof commented Oct 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

somethingwithproof commented Oct 9, 2026 •

edited

Loading