A .NET 8 IT operations and incident-response platform for monitoring distributed Windows and Linux systems, correlating failures, managing incidents, and measuring service reliability.
Distributed agents · Incident correlation · RBAC · Audit logging · Reliability analytics · Controlled remediation
OpsForge is a self-contained IT operations / NOC lab built with C# and .NET 8.
It combines a central ASP.NET Core server, authenticated Windows agents, persistent telemetry, topology discovery, incident correlation, operator workflows, reliability analytics, and a constrained remediation channel into a single runnable environment.
The project models the operational lifecycle around a failure:
observe → correlate → acknowledge → assign → investigate → preview remediation → execute → verify recovery → measure reliability
OpsForge is a portfolio/lab platform with increasingly production-oriented controls. It is not presented as a finished enterprise monitoring or security product.
OpsForge is designed around explicit operational invariants rather than treating monitoring, remediation, and reliability as independent features.
| Decision | Why it matters |
|---|---|
| ADR-0001 — Preview → Execute → Verify remediation | A successful command does not prove service recovery. Later monitoring evidence must verify the outcome. |
| ADR-0002 — Deterministic incident correlation | Related process, TCP, HTTP, and dependency failures can be represented as one explainable Primary Incident without discarding derivative evidence. |
| ADR-0003 — Evidence-based reliability accounting | Missing telemetry is not silently counted as healthy uptime, preventing optimistic SLA and error-budget reporting. |
| ADR-0004 — Explicit fleet service identity | Source and observer failures join only when their authenticated heartbeats declare the same service and rule; recovery requires fresh measurements. |
These decisions reflect three recurring principles:
- Evidence over assumption — recovery and availability must be supported by observed telemetry.
- Explainability over opaque automation — incident correlation and remediation decisions should be reproducible and auditable.
- Human control over destructive actions — diagnosis and remediation execution remain separate operational boundaries.
The ADRs document decisions already implemented by OpsForge rather than aspirational architecture.
OpsForge operations dashboard showing monitored systems, incidents, telemetry, reliability, and operator workflows.
- Distributed system monitoring
- Authenticated agent/server communication
- Windows process and service telemetry
- HTTP, TCP, and DNS probes
- Incident correlation and probable root cause
- Cross-agent service incident correlation with per-agent traces and recovery evidence
- Dependency topology and blast-radius analysis
- Role-based access control
- Operator acknowledgement and incident ownership
- Persistent audit logging
- Maintenance windows
- SLA and error-budget calculations
- MTTR and reliability analytics
- Persistent telemetry history
- Constrained remote remediation
- Remediation verification
- SQLite schema evolution
- Security-conscious agent enrollment and key rotation
- Optional certificate-bound agent identity
- Distributed HTTP tracing with optional OTLP export
- Notification routing, escalation, delivery history, generic webhooks, Slack/Teams-compatible webhooks, and SMTP email
- Windows 10/11 or Windows Server for the complete local demo
- .NET 8 SDK available through
dotnet - PowerShell 5.1 or newer
- A modern web browser
No Node.js, Python, external database server, or previous OpsForge version is required.
Clone or download OpsForge, open the repository root, and double-click:
START-HERE.cmd
The startup script:
- Performs a real
dotnet build - Stops if compilation fails
- Initializes a fresh SQLite database when needed
- Starts the OpsForge server
- Starts the deliberately killable demo HTTP service
- Starts a local Windows monitoring agent
- Opens the dashboard at:
http://localhost:5080
On the first run, open:
data/security/admin-bootstrap.txt
Sign in as:
admin
using the temporary password in that file.
OpsForge requires the bootstrap password to be replaced with a permanent password. The obsolete bootstrap credential file is removed after the password change succeeds.
Once signed in:
- Use Inject Demo Failure
- Watch multiple symptoms become a correlated Primary Incident
- Acknowledge the incident
- Take ownership
- Preview the proposed restart
- Execute the remediation
- Wait for later telemetry to verify recovery
- Open Executive Reliability to review availability, SLA/error budget, MTTR, trends, resource history, and per-node reliability
This provides a complete working demonstration without requiring external infrastructure.
flowchart LR
A1[Windows Agent] -->|Authenticated telemetry| S[OpsForge Server]
A2[Remote Windows Agent] -->|API key / optional mTLS| S
S --> DB[(SQLite)]
S --> CORR[Correlation Engine]
S --> TOPO[Topology Engine]
S --> REL[Reliability Analytics]
S --> AUDIT[Audit Trail]
S --> UI[Browser NOC UI]
CORR --> INC[Primary Incidents]
TOPO --> INC
REL --> UI
INC --> UI
UI -->|Preview / Execute| CMD[Constrained Command Channel]
CMD --> A1
CMD --> A2
A1 -->|Later telemetry verifies recovery| S
OpsForge/
├── OpsForge.Server/ # ASP.NET Core server, UI, persistence,
│ # correlation, topology and reliability
├── OpsForge.Agent/ # Windows telemetry, probes and command agent
├── OpsForge.Contracts/ # Shared DTOs / contracts
├── OpsForge.DemoService/ # Deliberately killable demonstration service
├── OpsForge.sln # Complete .NET solution
├── START-HERE.cmd # Build + launch complete local lab
├── STOP-OPSFORGE.cmd # Stop local OpsForge processes
├── Test-OpsForge.ps1 # Full-build compiler/runtime smoke test
├── Publish-Remote-Agent.ps1 # Build a copyable remote-agent package
├── CHANGELOG.md
└── LICENSE.txt
Agents can collect or evaluate:
- CPU and memory usage
- Process state
- Windows service state
- HTTP endpoints
- TCP endpoints
- DNS resolution
- Agent heartbeat / availability
- Inventory and status information
Telemetry is persisted so the UI can show both current state and historical reliability.
OpsForge supports multiple enrolled agents rather than assuming a single monitored machine.
The server maintains agent inventory, live status, persistent telemetry, agent-specific credentials, API-key fingerprints, revocation state, historical reliability, and topology relationships.
OpsForge does not treat every failed probe as an unrelated alert.
The correlation engine can combine multiple symptoms into a Primary Incident using deterministic reasoning based on system relationships and evidence.
Capabilities include:
- Probable root cause
- Confidence information
- Dependency traversal
- Blast-radius analysis
- Derivative-signal suppression
- Incident reassessment as telemetry changes
- Persistent incident history
- Markdown incident reports
The default local demo models:
OpsForge.DemoService process
↓
TCP listener :5091
↓
HTTP /health
Killing the demo process therefore creates multiple symptoms that can be correlated into one operational incident rather than three unrelated alerts.
Records that someone has seen and accepted responsibility for investigating the incident.
Associates a specific Operator or Administrator with the incident.
Returns an incident to the unowned queue.
Keeps diagnostic evidence visible while removing planned failures from actionable NOC counts during a valid maintenance window.
OpsForge intentionally separates command execution from recovery.
Preview remediation
↓
Operator approval
↓
Execute constrained command
↓
Wait for subsequent telemetry
↓
Verify recovery
A command returning successfully does not automatically mark a system healthy. Later monitoring evidence must verify recovery.
The v0.7 line adds persistent reliability analytics.
Available views include:
- 24-hour, 7-day, and 30-day reliability
- Fleet and per-agent availability
- Configurable SLA target
- Remaining error budget
- CPU and memory history
- Probe success rate and average latency
- Incident-open and resolution trends
- Mean time to recovery (MTTR)
- Per-node telemetry history
- Maintenance-excluded monitored time
- Active maintenance counts
- Acknowledged and unowned incident counts
Telemetry samples are retained for 30 days and throttled to keep the local SQLite database compact.
OpsForge does not silently interpret missing telemetry as healthy.
A heartbeat creates a short healthy interval. Once the offline threshold is exceeded, the remaining gap is counted as unavailable until communication resumes.
Planned maintenance is excluded from the SLA denominator.
Maintenance cannot be backdated by more than five minutes, preventing an operator from rewriting a historical outage as planned maintenance after the fact.
Read-only access to NOC status, reliability, incidents, topology, telemetry history, reports, inventory, and maintenance visibility.
Includes Viewer permissions plus incident acknowledgement/ownership, maintenance scheduling/cancellation, failure injection, and remediation.
Includes Operator permissions plus user management, agent enrollment, API-key rotation/revocation, certificate binding, and security audit visibility.
Operator credentials use:
- PBKDF2-SHA256 password hashing
- Per-user random salts
- 210,000 PBKDF2 iterations
- Forced replacement of temporary passwords
- Server-side sessions
- 8-hour interactive session lifetime
- HttpOnly session cookies
- SameSite=Strict session behavior
- CSRF protection on browser mutation requests
- Login throttling
Password reset revokes active sessions and forces another password change.
Security and operational changes are persisted with attribution.
Audit records include information such as actor, role, action, target, outcome, source IP, detail, and UTC timestamp.
Examples include:
- Authentication events
- Incident acknowledgement
- Ownership changes
- Maintenance scheduling/cancellation
- Remediation actions
- Agent enrollment/security operations
- Key rotation and revocation
Each non-revoked agent receives its own high-entropy API key.
The server stores the key hash and fingerprint rather than the plaintext credential.
Authenticated operations include heartbeat, telemetry submission, command polling, and command-result submission.
The initial enrollment token is stored locally at:
data/security/enrollment-token.txt
A remote agent uses that token only for initial enrollment.
Administrators can rotate an agent API key or revoke an agent. Rotation invalidates the old key immediately.
OpsForge supports optional certificate-bound agent identity.
When enabled, remote agent requests can require both the valid API key and the client certificate associated with that agent.
Example:
$env:OPSFORGE_AGENT_MTLS = '1'
$env:OPSFORGE_LISTEN_URL = 'https://0.0.0.0:5443'Configure the normal ASP.NET Core/Kestrel server certificate separately.
When mandatory agent mTLS is enabled, remote agent traffic cannot silently downgrade to plain LAN HTTP.
Build a copyable remote-agent package:
.\Publish-Remote-Agent.ps1This creates:
dist\OpsForge.Agent
Copy that folder to another Windows machine and configure its agent.json.
For first enrollment:
$env:OPSFORGE_AGENT_ENROLLMENT_TOKEN = '<server enrollment token>'
.\RUN-AGENT.cmdFor multi-machine use, deploy the OpsForge server behind HTTPS and appropriate firewall/reverse-proxy controls.
Operators can schedule maintenance for one enrolled agent or the entire fleet.
Maintenance records include the name, reason, start/end times, creator, creation time, cancellation state, and cancellation attribution.
During active maintenance:
- Diagnostic evidence remains available
- Matching incidents can be removed from actionable counts
- Matching time is excluded from reliability/SLA calculations
This preserves operational history without penalizing planned maintenance.
OpsForge uses SQLite for local persistence.
Default database:
data/opsforge.db
The v0.7 schema persists telemetry samples, maintenance windows, incident workflow, incidents, commands, inventory, agent state, authentication, sessions, and audit events.
Generated databases and credentials are excluded from source control.
The repository's .gitignore excludes:
data/*.db
data/*.db-shm
data/*.db-wal
data/security/
data/agents/
*.credentials.json
*.pfx
Run the full-build smoke test:
.\Test-OpsForge.ps1It performs a real dotnet build and exercises a clean temporary environment, including:
- Compilation
- Database initialization
- Bootstrap login/password replacement
- RBAC denial
- Agent enrollment
- Authenticated telemetry
- Deterministic incident correlation
- Incident acknowledgement/ownership
- Maintenance suppression
- Reliability/history analytics
- Maintenance cancellation
- Inventory
- Audit attribution
- Fleet-wide correlation, per-agent trace evidence, restart persistence, and partial recovery
The temporary test environment is removed afterward and does not overwrite the normal OpsForge database.
dotnet restore
dotnet build OpsForge.slnThen use the included startup scripts for the integrated demonstration environment.
See the detailed CHANGELOG.md.
An agent can declare fleetServiceId, fleetRuleId, and fleetRole (source or observer) in its OpsForge.Agent/appsettings.json. Assign the same service ID and a configured rule ID to one source and any number of observers. The source reports the configured process or Windows service and probes; observers report the rule's probes. Once the source reports a configured failure, observer failures for the same service and rule within the rule's time window join one primary incident. Observer-only failures do not create a fleet primary incident. Other services and unrelated CPU or memory warnings remain separate.
The dashboard and incident report list affected agents, their signals, and each agent's opening and recovery trace IDs. The opening incident trace ID stays stable. A fleet incident closes only after every affected agent reports a fresh complete healthy snapshot; missing or stale measurements leave it open. The server saves agent snapshots and fleet evidence in SQLite schema 9.0, including across restarts. Existing databases migrate in place.
The included ChaosLab demonstrates the incident with synthetic agents:
.\Start-OpsForge-ChaosLab.ps1 -Scenario CascadingFailure -Agents 10 -HoldSeconds 20Tags come from agents holding valid credentials. Assign one source per service/rule identity in a trusted deployment; the server does not independently verify service membership. Each agent currently supports one fleet service/rule pair. See ADR-0004 for the correlation and recovery rules.
The agent now creates a trace for each heartbeat cycle. HTTP probes and authenticated heartbeats propagate W3C trace context to the demo service and server. The server records a child span for heartbeat ingestion and each correlation rule; the trace ID that opens a primary incident is stored in SQLite, returned by the API, and shown in the dashboard and downloadable incident report. Reassessment keeps the opening trace ID.
Tracing works locally without a collector, but traces are only retained when an OTLP collector is configured. For a local Jaeger trace viewer, start the collector before OpsForge:
docker run --rm --name opsforge-jaeger -p 16686:16686 -p 4317:4317 cr.jaegertracing.io/jaegertracing/jaeger:2.21.0
$env:OPSFORGE_OTLP_ENDPOINT = 'http://localhost:4317'
.\START-HERE.cmdOpen http://localhost:16686/trace/<Trace ID> using the Trace ID displayed on a primary incident; the Jaeger search page also lets you select a service. Set OPSFORGE_OTLP_ENDPOINT in the environment of each process to export all three services. When no endpoint is set, spans are generated for correlation but no exporter is started. The local Jaeger container uses temporary storage; use a protected OTLP endpoint and appropriate collector retention for remote environments. The incident trace ID points to the heartbeat that first opened it, while later reassessments are separate traces.
Configurable correlation rules, event-time windows, stale-evidence handling, deterministic rule checks, and a synthetic multi-agent ChaosLab.
Configure relationships in OpsForge.Server/correlation-rules.json, then rebuild and restart the server. Each rule names a process, a Windows service, or probe IDs that belong to one failure domain. At least MinimumFailures distinct configured observations must fail within WindowSeconds; an unrelated probe never joins the group merely because it failed at the same time. Rule IDs and selectors must be unique. For example:
{
"Id": "payments",
"Title": "Payments unavailable",
"Process": "Payments.Worker",
"ProbeIds": ["payments-tcp", "payments-http"],
"WindowSeconds": 30,
"MinimumFailures": 2
}Add this object to the Correlation.Rules array and configure the agent to collect matching process and probe IDs. A rule may use Service instead of Process. The default demo rule remains in the file. A complete healthy or single-failure snapshot closes a prior primary incident; missing or stale observations leave it open. Incoming duplicate or older heartbeats are ignored. The current rule engine correlates within each agent and rule; it does not yet form one cross-agent root-cause incident or buffer late events for replay.
With OpsForge running locally, simulate ten agents, three related node failures, an independent memory warning, and recovery:
.\Start-OpsForge-ChaosLab.ps1 -Scenario CascadingFailure -Agents 10 -HoldSeconds 20ChaosLab enrolls synthetic agents through the local authenticated API; it does not stop services or probe real systems. Each run creates new synthetic agent IDs, which later appear offline until retention or cleanup. Run the focused checks with dotnet run --project .\OpsForge.CorrelationChecks\OpsForge.CorrelationChecks.csproj and the Windows smoke test with .\TEST-OPSFORGE.cmd. The CMD wrapper invokes PowerShell with a bypass limited to that process, as START-HERE.cmd does.
- Added Windows GitHub Actions CI
- Automated .NET 8 Release build and integration smoke testing
- Fixed PowerShell 5.1 incident-array enumeration in the smoke test
- Added professional project documentation and dashboard presentation
- Fixed build issues discovered after the v0.7 reliability release
- Preserves the complete v0.7 feature set
- Ships as a standalone/full source build
Added reliability history, SLA/error-budget calculations, MTTR analytics, Executive Reliability dashboard, incident acknowledgement and ownership, maintenance windows, maintenance-aware SLA calculations, and expanded audit attribution.
Added named operator accounts, Viewer/Operator/Administrator RBAC, password hashing, sessions, CSRF protection, login throttling, persistent audit logging, and optional agent mTLS.
Added authenticated enrollment, per-agent API keys, inventory/status persistence, and key rotation/revocation.
Added multi-node topology, cross-machine dependencies, blast-radius analysis, and derivative-alert suppression.
Added deterministic multi-signal correlation, root-cause reasoning, confidence information, and Primary Incident reporting.
Added SQLite persistence, HTTP/TCP/DNS probes, Windows service monitoring, MTTR, timeline history, and Preview → Execute → Verify remediation.
Introduced the .NET 8 server, Windows telemetry agent, browser dashboard, constrained command channel, and demo service.
OpsForge is a portfolio/lab platform with production-oriented controls.
It does not currently claim to provide every control expected from a hardened enterprise monitoring platform.
Not currently included:
- SSO / OIDC / SAML
- MFA
- Centralized secrets-vault integration
- Complete PKI lifecycle management
- CRL / OCSP handling
- Signed automatic updates
- High-availability database/session storage
- Hardened enterprise installer
- Full production deployment automation
For network-accessible deployment, use HTTPS, firewall restrictions, least-privilege service accounts, appropriate reverse-proxy controls, secure certificate management, protected secrets, and appropriate host hardening.
OpsForge is actively developed as a portfolio and systems-engineering project.
It demonstrates practical work across C#/.NET 8, ASP.NET Core, distributed-agent architecture, Windows monitoring, incident-response workflows, observability/reliability concepts, SQLite, authentication/RBAC, API-key lifecycle management, security auditing, network probes, topology modeling, automated smoke testing, and release management.
OpsForge is released under the MIT License.
Daniel Fuhr
- GitHub: github.com/fuhrdan
- LinkedIn: linkedin.com/in/danielfuhr
- Portfolio: lakehousesoftware.com
OpsForge demonstrates the kind of systems work that sits between software development and IT operations.
Rather than building only a dashboard or only a monitoring agent, the project carries an operational event through telemetry, authentication, dependency modeling, correlation, human ownership, remediation, verification, historical persistence, auditability, and reliability measurement.
That end-to-end operational lifecycle is the core of the project.