docs: clarify current leak detection health workflows #5480
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,97 @@ | ||
| # Leak Detection, Health, and Allocation Protection | ||
|
|
||
| ## Overview | ||
|
|
||
| NICo evaluates leak-related conditions from compute-tray sensor health and BMS leak events. It turns active conditions into health signals and, where configured, allocation protection. | ||
|
Author
There was a problem hiding this comment. @zhaozhongn - Is it just compute tray or there are other sensors in rack that NICo can access through redfish ? |
||
|
|
||
| BMS owns the physical leak sensors and remediation actions. NICo provides the infrastructure-management health and allocation-protection surface. | ||
|
|
||
| ## Current capability | ||
|
|
||
| ### Compute-tray sensor health | ||
|
|
||
| NICo hardware-health monitoring discovers configured BMC endpoints and queries them through Redfish. When compute-tray BMCs expose leak-related sensor data and the relevant health collector is configured, NICo considers that data as part of hardware health. | ||
|
|
||
| ```text | ||
| Compute-tray leak sensor | ||
| ↓ | ||
| Compute-tray BMC | ||
| ↓ Redfish | ||
| NICo hardware-health service | ||
| ↓ | ||
| NICo hardware health | ||
| ↓ | ||
| Aggregate host health | ||
| ``` | ||
|
|
||
| The alert classification and operational effect for this path depend on the site's health-processor configuration. | ||
|
|
||
| ### BMS leak-event integration | ||
|
|
||
| NICo also supports a BMS event path through `nico-dsx-exchange-consumer`. | ||
|
|
||
| ```text | ||
| BMS detects or clears a leak condition | ||
| ↓ | ||
| BMS publishes MQTT metadata and value events | ||
| ↓ | ||
| NICo DSX Exchange consumer | ||
| ↓ | ||
| NICo rack health report | ||
| ↓ | ||
| Aggregate host health and allocation protection | ||
| ``` | ||
|
|
||
| The consumer supports these BMS event types: | ||
|
|
||
| - Rack leak detection (`LeakDetectRack`) | ||
| - Rack leak-sensor fault (`LeakSensorFaultRack`) | ||
| - Rack-tray leak detection (`LeakDetectRackTray`) | ||
|
|
||
| For an active supported BMS event, NICo creates a rack health alert with these classifications: | ||
|
|
||
| - `PreventAllocations` | ||
| - `SensorCritical` | ||
| - `Hardware` | ||
|
|
||
| `PreventAllocations` blocks new allocations for hosts affected by the active rack-health condition. When BMS publishes a clear event, NICo removes the corresponding rack health report. NICo then recalculates aggregate health; allocation eligibility can recover when no other active health condition prevents allocation. | ||
|
There was a problem hiding this comment.
When multiple supported BMS points are active for the same rack, this recovery description overstates the protection provided. AGENTS.md reference: AGENTS.md:L333-L343 Useful? React with 👍 / 👎.
Contributor
There was a problem hiding this comment. 🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift 🔎 Supported by static analysis🏁 Script executed: printf '%s\n' '--- applicable repository guidance ---'
find /tmp/coderabbit-repo-knowledge/nvidia-infra-controller-80992b25 -type f -name '*.md' -maxdepth 3 -print
printf '%s\n' '--- target document ---'
cat -n docs/operations/leak-detection.md | sed -n '45,65p'
printf '%s\n' '--- relevant source files ---'
fd -i 'health_updater|leak-detection' .
rg -n -C 8 'point_path|rack_id|PreventAllocations|rack health|clear event|health report' crates docs 2>/dev/null | head -240Repository: NVIDIA/infra-controller Length of output: 27325 🏁 Script executed: printf '%s\n' '--- documentation conventions ---'
cat /tmp/coderabbit-repo-knowledge/nvidia-infra-controller-80992b25/conventions/docs.md
printf '%s\n' '--- relevant learnings ---'
cat /tmp/coderabbit-repo-knowledge/nvidia-infra-controller-80992b25/learnings/crates-health-src.md 2>/dev/null || true
printf '%s\n' '--- health updater outline ---'
ast-grep outline crates/dsx-exchange-consumer/src/health_updater.rs
printf '%s\n' '--- health updater references ---'
rg -n -C 14 'point_path|rack_id|PreventAllocations|clear|remove|health report|aggregate' crates/dsx-exchange-consumer/src/health_updater.rsRepository: NVIDIA/infra-controller Length of output: 21398 🏁 Script executed: printf '%s\n' '--- sink trait and API binding ---'
rg -n -C 12 'trait RackHealthReportSink|impl RackHealthReportSink|remove_rack_health_report|insert_rack_health_report' crates/dsx-exchange-consumer crates 2>/dev/null | head -260
printf '%s\n' '--- health updater imports and type bounds ---'
cat -n crates/dsx-exchange-consumer/src/health_updater.rs | sed -n '1,70p'Repository: NVIDIA/infra-controller Length of output: 25307 Retain rack protection until all active BMS points clear.
🤖 Prompt for AI AgentsSources: Path instructions, MCP tools |
||
|
|
||
| ## Deployment requirements for BMS event integration | ||
|
|
||
| The BMS event path is not enabled automatically in every NICo deployment. To use it, a deployment must have all of the following: | ||
|
|
||
| 1. `nico-dsx-exchange-consumer` enabled. This Helm subchart is disabled by default. | ||
|
There was a problem hiding this comment.
For the umbrella chart at this revision, AGENTS.md reference: AGENTS.md:L272-L276 Useful? React with 👍 / 👎. |
||
| 2. Connectivity and configuration for the BMS MQTT broker and event topics. | ||
| 3. BMS metadata and value events that use supported point types and identify the affected rack. | ||
| 4. The consumer configured to call the NICo API. | ||
| 5. Health aggregation configured for the relevant racks and hosts. | ||
|
|
||
| Without these prerequisites, BMS events do not create NICo rack health reports. | ||
|
Comment on lines
+61
to
+69
Contributor
There was a problem hiding this comment. 🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win Separate report creation from allocation protection prerequisites. Health aggregation is not required for the consumer to create a rack health report. The consumer persists the report after matching supported metadata and value events. (raw.githubusercontent.com) Health aggregation determines whether the report affects aggregate host health and allocation decisions. (github.com) Rewrite this section into two groups: prerequisites for report creation and prerequisites for allocation protection. Line 69 currently says that no report exists when the actual failure may be missing allocation propagation. As per path instructions: Review Markdown for technical correctness, clarity, spelling, grammar, working links, and operator usability. 🧰 Tools🪛 LanguageTool[style] ~61-~61: Consider removing “of” to be more concise (ALL_OF_THE) 🤖 Prompt for AI AgentsSources: Path instructions, MCP tools |
||
|
|
||
| ## Operational meaning | ||
|
|
||
| NICo provides **health visibility and allocation protection** for leak-related conditions. The current BMS event workflow creates a rack health alert that propagates through NICo health aggregation, while BMS handles physical remediation. | ||
|
|
||
| Current operational visibility is provided through NICo's health data, health alert details, logs, and metrics. | ||
|
|
||
| ## Next phase: API-based leak workflows | ||
|
|
||
| The next phase follows [issue #5018](https://github.com/NVIDIA/infra-controller/issues/5018) by adding API-based workflows for rack and tray leak detection and leak-handling state. It builds on the current health and allocation-protection workflow; it does not replace it. | ||
|
|
||
| The API workflows expose: | ||
|
|
||
| - rack and tray leak status; | ||
| - tray and BMS sensor details, including optional readouts and thresholds; | ||
| - BMS leak-handling status; | ||
| - ongoing handling operations, such as tray shutdown or rack isolation; | ||
| - previous handling results and details. | ||
|
|
||
| This phase makes leak state and handling workflow available through the NICo API. BMS remains responsible for physical detection and remediation, while NICo provides the infrastructure-management health and lifecycle surface. | ||
|
Comment on lines
+81
to
+89
Contributor
There was a problem hiding this comment. 🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win Mark the API surface as planned. As of August 27, 2026, issue Change “The API workflows expose” to “The planned API workflows will expose” and keep the remaining description in future tense. As per path instructions: Review Markdown for technical correctness, clarity, spelling, grammar, working links, and operator usability. 🧰 Tools🪛 LanguageTool[grammar] ~89-~89: Ensure spelling is correct (QB_NEW_EN_ORTHOGRAPHY_ERROR_IDS_1) 🤖 Prompt for AI AgentsSources: Path instructions, MCP tools |
||
|
|
||
| ## Related documentation and implementation | ||
|
|
||
| - [Monitoring and Health](monitoring-health.md) | ||
| - `crates/dsx-exchange-consumer/README.md` | ||
| - `crates/dsx-exchange-consumer/src/health_updater.rs` | ||
| - `helm/README.md` | ||
| - [Issue #5018](https://github.com/NVIDIA/infra-controller/issues/5018) | ||
|
Comment on lines
+91
to
+97
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This new public operations page is not referenced anywhere in
docs/index.yml, so it will not appear in the published Fern documentation navigation and readers cannot discover it through the operations section. Addoperations/leak-detection.mdto the appropriate navigation contents.AGENTS.md reference: AGENTS.md:L396-L397
Useful? React with 👍 / 👎.