Skip to content

cli.md

Latest commit

 

History

History
357 lines (291 loc) · 17 KB

cli.md

File metadata and controls

357 lines (291 loc) · 17 KB
Raw

CLI

sermoctl is the operator and scripting interface. Run it with no arguments or --help for the command index, and use sermoctl help COMMAND or sermoctl COMMAND --help for focused usage, flags and examples.

Root flags

--config /etc/sermo/sermo.yml
--backend auto|systemd|openrc
--json
--quiet / -q
--timeout duration
--version / -V
--help / -h

Global flags may be placed before or after the command. Command-specific flags are shown by sermoctl help COMMAND.

Without --timeout, live service queries (status and is-active) use the 10-second engine check budget; service operations use the 90-second operation budget. Other short probe commands keep their 2-second CLI budget.

sermod daemon flags

sermod is the long-running monitoring daemon. Packaged units normally start it with the standard config path:

sermod run --config /etc/sermo/sermo.yml

Manual runs support these flags:

sermod run [--config PATH] [--verbose|-v]
sermod version
sermod --version
  • --config PATH loads the global config file. The default is /etc/sermo/sermo.yml. Use the same path with sermoctl --config when validating or reloading a non-standard tree.
  • --verbose / -v enables debug logging, including config load details, backend detection and monitor-target counts.

Use sermoctl daemon reload to ask a running daemon to re-read the config file it was started with.

Command surface

sermoctl help [COMMAND]
sermoctl backend
sermoctl version
sermoctl status SERVICE
sermoctl is-active SERVICE
sermoctl watch status WATCH
sermoctl watch monitor WATCH
sermoctl watch unmonitor WATCH
sermoctl watch probe WATCH
sermoctl watch pause RAID_WATCH --confirm MD_ARRAY
sermoctl watch resume RAID_WATCH
sermoctl start SERVICE [--no-cascade]
sermoctl stop SERVICE [--no-cascade]
sermoctl restart SERVICE [--no-cascade]
sermoctl resume SERVICE
sermoctl reload SERVICE

sermoctl mount TARGET                 # TARGET is a configured mount name or absolute path
sermoctl umount TARGET
sermoctl mount status TARGET
sermoctl mount list

sermoctl preflight SERVICE
sermoctl processes SERVICE
sermoctl reap SERVICE [--apply]        # list the service's stray processes; --apply signals the authorized ones
sermoctl locks SERVICE
sermoctl monitor SERVICE
sermoctl unmonitor SERVICE

sermoctl panic on|off|status          # daemon-wide emergency switch (see Panic mode)

sermoctl config validate

sermoctl web hash-password [--stdin|--generate] [--hash bcrypt|sha256] [--cost N] [--name LABEL]
                                       # print one credential line for web.password_file

sermoctl daemon reload                 # reload sermod config, not services
sermoctl notifier test NAME            # send an explicit test message through one notifier

sermoctl services [all] [--long] [--notify NAME[,NAME]|all]   # catalog inventory, not runtime config
sermoctl apps [all] [--long]                                  # catalog apps (see Catalog inventory)
sermoctl libs [all] [--long]
sermoctl patterns

sermoctl sla [TARGET]                   # availability windows for every service and availability watch, or one
sermoctl sla --series TARGET [--since DURATION]   # per-minute series; --since default 24h

sermoctl events [SERVICE] [--limit N]   # list recent events (global or for SERVICE)
sermoctl events clear [--before TIME]   # omit TIME to clear all; TIME may be non-future RFC3339 or positive duration
                                        # only events strictly before the timestamp are removed
sermoctl activity clear [--before TIME] # clears the same log shown in Events

sermoctl state compact [--before TIME]  # consolidates and prunes stored history, then vacuums the state database
                                        # omit TIME for the configured retention; TIME additionally drops older history (non-future RFC3339 or positive duration)

sermoctl lock SERVICE [--name NAME] --reason REASON --ttl DURATION -- COMMAND...
sermoctl lock acquire SERVICE [--name NAME] --reason REASON --ttl DURATION
sermoctl lock release SERVICE [--name NAME]

sermoctl wizard
sermoctl wizard service|docker|vm|mount|volume|net|uplink

Availability

sermoctl sla is observed check availability: it only counts monitored daemon cycles. A window with no observed cycles reads n/a, not downtime — daemon downtime or missing data never becomes observed downtime. sermoctl sla --series TARGET emits that target's stored per-minute availability series (the raw data a graph is built from).

A target is a configured service or a host watch whose check asserts availability — tcp, ports, http, route, the state metric of net and icmp, and the endpoint form of cert. Those are the checks whose failing half is genuinely something not answering; a cert check reading a file on disk is a certificate nearing expiry, which is a condition, not a host being unreachable. A condition watch keeps no series: a filesystem crossing 90% used is a threshold being met, not an outage, and reporting it as availability would give a percentage that reads like uptime while meaning something else. The same exclusions the services already apply carry over — a verdictless watch (reports: state) is a sensor and an advisory (severity: warning) is a thing to look at, so neither is downtime. A name is resolved as a service first, so an existing service name never changes meaning.

Examples:

sermoctl help restart
sermoctl restart mysql-main
sermoctl services --notify ops-email
sermoctl notifier test ops-email
sermoctl daemon reload
sermoctl state compact --before 720h

Panic mode

Panic mode is a daemon-wide emergency switch for maintenance windows, attacks, denial-of-service, system malfunction or overload. While it is on, the daemon keeps running its checks (so status stays visible) but suspends all hooks, alert notifications and automatic remediation. Manual operations (start, stop, restart, reload, resume) stay available, so you can drive services by hand without the daemon fighting you.

sermoctl panic on        # suspend hooks, alerts and automatic remediation
sermoctl panic status    # show the current state (default when no argument)
sermoctl panic off       # resume normal operation

The flag is persisted in the state database (paths.state), so it survives daemon restarts until you turn it off, and the CLI works without the web UI enabled. The running daemon picks up a change within ~1 second. While active, the daemon status reported by /readyz and the web header shows panic mode. In the web UI the same toggle is the red panic mode button in the footer (it asks for confirmation in both directions so it is not triggered by accident). The CLI applies the change immediately without a prompt.

Service target resolution

For a configured service, sermoctl status, is-active and service operations resolve the same control target that sermod and the web UI use. When sermod is running with web enabled, sermoctl status prefers the daemon's computed state (including starting during startup settling); if the web API is unreachable it falls back to the init backend plus local monitor metadata, as before. Service states are: disabled, stopped, started (backend active but not monitored), starting (startup/operation settling), collecting (active and monitored, but graphs/indicators are not complete yet), warning (active with an advisory problem such as an invalid application configuration, an unattributed process tree, or an init unit failed while an exact process and its functional checks remain healthy), restart_required (active and observed but running a binary that was replaced on disk), monitored (active, monitored and observability-ready) and failed. Without the daemon view, a configured active monitored service falls back to collecting; an active service that is not known to be monitored falls back to started.

For the second warning case, status remains failed in the API: operations still follow the init backend and retain their normal locks, guards and preflight gates. The warning only prevents a working workload from being presented as an application outage.

sermoctl status SERVICE exposes warning and restart_required directly. For a configuration warning, sermoctl preflight SERVICE reruns the same bounded preflight.config command and prints its current output. is-active continues to report only the init backend's active/inactive verdict.

A backend status of unknown is not a verdict of "down" — a transitional systemd state such as activating/deactivating, an init script that replaces status with its own report, or a query that timed out can read unknown while the service runs normally — so it never yields failed on its own. The service's own checks decide instead: a failing required check still reads failed, and healthy checks read active or collecting rather than monitored, because a backend that would not answer cannot underwrite the full-observability claim.

Each manual service operation persists exactly one result in the shared event feed, so sermoctl events and the Web UI show the same action outcome. Sermoctl does not start an operation when the state database cannot be opened for that audit record.

sermoctl is-active is different: it always probes the init backend (active / inactive / paused) for the exit code and plain-text output. A monitored service still settling with an inactive backend therefore shows state=starting in status but exits 1 from is-active until the unit reports active.

The same preference applies to sermoctl watch status WATCH and to the STATUS column of sermoctl apps for installed applications monitored by the daemon. Catalog apps whose binary is not installed are omitted from sermoctl apps and do not participate in startup settling.

When the daemon has current watch readings, sermoctl watch status WATCH also prints them (including RAID operation and rebuild percentage) and the separate last-check timestamp; --json exposes the same readings in a readings array.

sermoctl watch monitor|unmonitor WATCH pauses or resumes a single watch, persisted under paths.state and read live by the daemon. WATCH is a host watch name or a service-embedded watch "<service>:<watch>"; a watch's monitor state is independent of its service's, so unmonitor on a service never pauses its watches.

sermoctl watch probe WATCH asks the running daemon to run one fresh sample for a host diskio, hdparm, lvm, raid, smart, storcli or ssacli watch and prints the resulting readings when available (for hardware RAID this includes one reading per controller, cache, virtual volume and physical drive, with identity/capacity, controller RAM, controller SMART and rebuild progress where reported). Every probe except smart is read-only. A smart probe starts the device's short SMART self-test with smartctl --test=short DEVICE; success means the device accepted the test, not that it has passed it. Normal scheduled SMART checks remain read-only health/attribute reads. The command records a probe event and last-check time, but does not run rules, notifications or remediation. A RAID watch with raid_control.pause_resume: true and an explicit check.array also supports watch pause and watch resume. Pausing requires --confirm MD_ARRAY in addition to naming the watch; both actions re-check the array, use an exclusive runtime operation lock and verify the resulting kernel state. Resume accepts any currently paused configured array, including one paused outside Sermo.

The daemon records both probe/running when a manual sample starts and its probe/ok or probe/failed completion event with the elapsed time. A SMART self-test remains testing in watch status until the device reports it has ended. RAID/LVM device work is also reported as testing, recovering, rebuilding, repairing, moving or merging, including the reported percentage where available; those states describe work, not health. Only one manual sample for a watch may run at once; sermoctl watch probe waits for that same daemon task and reports an already-running sample instead of starting a second disk, LVM, RAID or SMART command. Sermo reads the service's service: candidates, picks the first unit known by the active backend, and normalizes systemd names with .service when needed.

If the backend probe cannot surface a configured init unit but the service still has a usable configured seed, Sermo falls back to that unit and prints a warning, matching the daemon/web behavior used for historic init-service setups. There is no fallback for invalid control: targets or a per-backend service: map with no candidate for the active backend; those are configuration errors.

Catalog inventory

sermoctl services, sermoctl apps, sermoctl libs and sermoctl patterns list catalog definitions shipped in the packaged catalog (see services.md): which profiles are installed, the version their version command reports, and whether they resolve. Add all to include entries whose binary or library file is not present on the host.

This is not the list of configured runtime targets that sermod monitors. Those are the service files under paths.services (and the matching names in the global config tree).

Question Where to look
Which catalog service profiles exist / are installed? sermoctl services [all]
Which catalog apps / libs / pattern sets exist? sermoctl apps, sermoctl libs, sermoctl patterns
Which services are enabled in my config right now? YAML under paths.services, or the web UI Services panel (GET /api/services)
One configured service's live state sermoctl status SERVICE, sermoctl is-active SERVICE
Availability history for services and availability watches sermoctl sla [TARGET]

The web UI uses the same split: Services shows configured runtime services; Applications (GET /api/applications) and Libraries (GET /api/libraries) are installed catalog inventories, aligned with sermoctl apps and sermoctl libs, not sermoctl services.

Reaping stray processes

A stray is a process the init backend attributes to the service's control group that no processes: selector or pidfile claims and that no longer hangs off the unit's principal process — a probe that daemonized, a child the daemon never reaped, a survivor of an earlier incarnation. sermoctl processes SERVICE flags them with stray=true, and the injected strays check reports them every cycle.

A stop never reaps, so a restart that a stray blocks ends in orphan_processes and names it; clearing it is this command's job.

sermoctl reap SERVICE is a preview: it lists every stray, reports how many would be signalled, and touches nothing — no operation lock, no event.

sermoctl reap SERVICE --apply signals them, gated by the service's own reap.kill_only_if selector. With no such block nothing is authorized, so the command reports every stray, signals none and exits 75. Otherwise the exit code follows the usual operation mapping: 0 when no stray remains, 1 when one was spared or outlived SIGKILL (orphan_processes), 2 on a failure.

--apply is rejected by every other command, and no rule action can reap — see safety.md for the whole contract and services.md for the reap: block.

Exit codes

0   success / active / allowed
1   expected false condition, such as inactive or a failed check
2   internal or runtime error / backend not detected
64  usage error (bad flags or arguments)
75  temporarily blocked action, such as an active backup lock or guard
78  configuration invalid (syntax, schema or `config validate` failure)

The 2 vs 78 distinction: use 78 whenever the problem is in the config files the operator can fix (YAML syntax, missing kind/name, unknown variable, unresolved uses/clone, failed config validate). 2 is everything else that is not a clean false (1), a usage error (64) or a temporary block (75): I/O errors, backend not detected, an exec that could not be launched, an unexpected panic recovered at the top level.

is-active maps directly: 0 active, 1 not active (including paused), 2 error.

Mounts

Mount actions are fstab-backed and use storage watch files with a mount: block from directories listed in paths.watches (the wizard writes /etc/sermo/mounts by default). A path target that is not configured is still accepted, but it uses safe defaults and must exist in /etc/fstab. See storage and mount units. sermoctl umount / is always rejected; Sermo never unmounts the root filesystem. sermoctl umount TARGET --force permits umount -f after the normal unmount fails, --lazy permits umount -l as the last fallback, and --kill-blockers signals only blockers that match mount.stop_policy.kill_only_if.

sermoctl wizard mount lists mount points declared in /etc/fstab and writes safe storage watch files under mounts/, adding that directory to paths.watches; it does not execute mount or umount while generating the config.