From 342422f14ec056547aff09483ebc04490ef4b659 Mon Sep 17 00:00:00 2001 From: Deepak Poornachandra Date: Thu, 27 Aug 2026 16:50:42 -0500 Subject: [PATCH 01/28] docs: describe leak detection health workflow Signed-off-by: Zhaozhong Ni --- docs/operations/leak-detection.md | 97 +++++++++++++++++++++++++++++++ 1 file changed, 97 insertions(+) create mode 100644 docs/operations/leak-detection.md diff --git a/docs/operations/leak-detection.md b/docs/operations/leak-detection.md new file mode 100644 index 0000000000..27590724d3 --- /dev/null +++ b/docs/operations/leak-detection.md @@ -0,0 +1,97 @@ +# Leak Detection, Health, and Allocation Protection + +## Overview + +NICo evaluates leak-related conditions from compute-tray sensor health and BMS leak events. It turns active conditions into health signals and, where configured, allocation protection. + +BMS owns the physical leak sensors and remediation actions. NICo provides the infrastructure-management health and allocation-protection surface. + +## Current capability + +### Compute-tray sensor health + +NICo hardware-health monitoring discovers configured BMC endpoints and queries them through Redfish. When compute-tray BMCs expose leak-related sensor data and the relevant health collector is configured, NICo considers that data as part of hardware health. + +```text +Compute-tray leak sensor + ↓ +Compute-tray BMC + ↓ Redfish +NICo hardware-health service + ↓ +NICo hardware health + ↓ +Aggregate host health +``` + +The alert classification and operational effect for this path depend on the site's health-processor configuration. + +### BMS leak-event integration + +NICo also supports a BMS event path through `nico-dsx-exchange-consumer`. + +```text +BMS detects or clears a leak condition + ↓ +BMS publishes MQTT metadata and value events + ↓ +NICo DSX Exchange consumer + ↓ +NICo rack health report + ↓ +Aggregate host health and allocation protection +``` + +The consumer supports these BMS event types: + +- Rack leak detection (`LeakDetectRack`) +- Rack leak-sensor fault (`LeakSensorFaultRack`) +- Rack-tray leak detection (`LeakDetectRackTray`) + +For an active supported BMS event, NICo creates a rack health alert with these classifications: + +- `PreventAllocations` +- `SensorCritical` +- `Hardware` + +`PreventAllocations` blocks new allocations for hosts affected by the active rack-health condition. When BMS publishes a clear event, NICo removes the corresponding rack health report. NICo then recalculates aggregate health; allocation eligibility can recover when no other active health condition prevents allocation. + +## Deployment requirements for BMS event integration + +The BMS event path is not enabled automatically in every NICo deployment. To use it, a deployment must have all of the following: + +1. `nico-dsx-exchange-consumer` enabled. This Helm subchart is disabled by default. +2. Connectivity and configuration for the BMS MQTT broker and event topics. +3. BMS metadata and value events that use supported point types and identify the affected rack. +4. The consumer configured to call the NICo API. +5. Health aggregation configured for the relevant racks and hosts. + +Without these prerequisites, BMS events do not create NICo rack health reports. + +## Operational meaning + +NICo provides **health visibility and allocation protection** for leak-related conditions. The current BMS event workflow creates a rack health alert that propagates through NICo health aggregation, while BMS handles physical remediation. + +Current operational visibility is provided through NICo's health data, health alert details, logs, and metrics. + +## Next phase: API-based leak workflows + +The next phase follows [issue #5018](https://github.com/NVIDIA/infra-controller/issues/5018) by adding API-based workflows for rack and tray leak detection and leak-handling state. It builds on the current health and allocation-protection workflow; it does not replace it. + +The API workflows expose: + +- rack and tray leak status; +- tray and BMS sensor details, including optional readouts and thresholds; +- BMS leak-handling status; +- ongoing handling operations, such as tray shutdown or rack isolation; +- previous handling results and details. + +This phase makes leak state and handling workflow available through the NICo API. BMS remains responsible for physical detection and remediation, while NICo provides the infrastructure-management health and lifecycle surface. + +## Related documentation and implementation + +- [Monitoring and Health](monitoring-health.md) +- `crates/dsx-exchange-consumer/README.md` +- `crates/dsx-exchange-consumer/src/health_updater.rs` +- `helm/README.md` +- [Issue #5018](https://github.com/NVIDIA/infra-controller/issues/5018) From f32f5129c0e769d35d663466449b3136e987b9fa Mon Sep 17 00:00:00 2001 From: zhaozhongn <38698630+zhaozhongn@users.noreply.github.com> Date: Fri, 28 Aug 2026 02:59:10 -0700 Subject: [PATCH 02/28] Update and rename leak-detection.md to leak-detection-handling.md Signed-off-by: zhaozhongn <38698630+zhaozhongn@users.noreply.github.com> Signed-off-by: zhaozhongn Signed-off-by: Zhaozhong Ni --- docs/operations/leak-detection-handling.md | 147 +++++++++++++++++++++ docs/operations/leak-detection.md | 97 -------------- 2 files changed, 147 insertions(+), 97 deletions(-) create mode 100644 docs/operations/leak-detection-handling.md delete mode 100644 docs/operations/leak-detection.md diff --git a/docs/operations/leak-detection-handling.md b/docs/operations/leak-detection-handling.md new file mode 100644 index 0000000000..dbbdaa8a74 --- /dev/null +++ b/docs/operations/leak-detection-handling.md @@ -0,0 +1,147 @@ +# Leak Detection and Handling + +## Overview + +In rack-scale system, there are two sets of leak sensors, with NICo and BMS each managing one set and detects leak based on them. Additionally, BMS controls the rack AC power breaker and coolant valve for rack isolation remediation, while NICo provides the infrastructure-management health reporting, allocation protection, and safe handling. + +NICo evaluates leak-related conditions from compute and NVSwitch tray sensors managed by it to detect leak events. NICo also listens via DSX Exchange (MQTT event bus) for BMS-issued leak events based on BMS-managed sensors such as CDU, rope, and dripping pan. A leak event results in tray and rack health warnings and alerts in NICo, which prevent the machines from being allocated. Depending on the leak type and severity, NICo takes automated handling actions based on configuration and policy. + +## Current capability: automated e2e leak detection, reporting, protection, and handling + +### Compute and NVSwitch tray sensor path + +NICo hardware-health monitoring discovers configured BMC endpoints and queries them through Redfish. When compute or NVSwitch tray BMCs expose leak-related sensor data and the relevant health collector is configured, NICo considers that data as part of hardware health. + +```text +Compute/NVSwitch-tray leak sensor + ↓ +Compute/NVSwitch-tray BMC + ↓ Redfish +NICo hardware-health service + ↓ +NICo tray health report +``` + +Health report with leakage warnings or alerts are generated for the leaking trays. The alert classification and operational effect for this path depend on the site's health-processor configuration. + +### BMS leak-event integration + +NICo also supports a BMS leak event (based on BMS-managed sensors such as CDU, rope, dripping pan, etc.) path through the `nico-dsx-exchange-consumer` service. + +```text +BMS detects / clears a leak condition + ↓ +BMS publishes MQTT metadata and value events + ↓ +NICo DSX Exchange consumer + ↓ +NICo rack health report +``` + +The consumer supports these BMS event types: + +- Rack leak detection (`LeakDetectRack`) +- Rack leak-sensor fault (`LeakSensorFaultRack`) +- Rack-tray leak detection (`LeakDetectRackTray`) + +For an active supported BMS event, NICo creates rack health report with a leak alert. When BMS publishes a clear event, NICo removes the corresponding rack health report. + +### Deployment requirements for BMS event integration + +The BMS event path is not enabled automatically in every NICo deployment. To use it, a deployment must have all of the following: + +1. `nico-dsx-exchange-consumer` enabled. This Helm subchart is disabled by default. +2. Connectivity and configuration for the BMS MQTT broker and event topics. +3. BMS metadata and value events that use supported point types and identify the affected rack. +4. The consumer configured to call the NICo API. +5. Health aggregation configured for the relevant racks and hosts. + +Without these prerequisites, BMS events do not create NICo rack health reports. + +### Health Reporting and Allocation Protection + +NICo provides **health visibility and allocation protection** for leak-related conditions. Regardless of the source of leak detection and the path of health report creation, leak health alert comes with these classifications: + +- `PreventAllocations` +- `SensorCritical` + +`PreventAllocations` blocks new allocations for hosts affected by the active tray- or rack-health condition. When the health alert clears later, NICo recalculates aggregate health; allocation eligibility can recover when no other active health condition prevents allocation. + +Current operational visibility is provided through NICo's health data, health alert details, logs, and metrics. To obtain the current health reports of a machine: +`GET /nico/machine/{id}/health-report` + +### Three Tiers of Automated Leak Handling + +NICo and BMS combined together automatically take leak handling actions in three tiers. + +#### Critical Leakage Handling + +Critical leakages are those detected by BMS based on BMS-managed sensors. Because these sensors are often at rack or even larger scope, any leaks detected from them may have very large and serious impact, and rack electric and liquid isolation often need to be performed ASAP, by BMS via its control of AC power breaker and coolant valve. + +After NICo received the leak event from BMS via DSX Exchange, if, for any reason, the rack still has AC power on and there are still trays on, NICo will turn them off through its general leakage handling automation. + +#### Severe Leakage Handling + +When multiple trays in a rack are detected to be leaking based on in-tray NICo-managed sensors, if it goes beyond a configured threshold (default to 2 for NVL72 racks currently), the rack is considered to have a severe leakage. To prevent more trays in the rack from being impacted by the leak, NICo will immediately request BMS, via DSX Exchange, to perform electric and liquid isolation to the rack. + +Upon receiving the isolation request, BMS will trigger the AC power breaker and shut off coolant valve, and broadcast the isolation result back to NICo via DSX Exchange, which NICo waits for. + +#### General Leakage Handling + +When there are leaking trays but not enough to trigger a critical or severe leak, NICo handles the general leakage via policy-based automation. + +Currently, the default handling policy is to shut down all leaking trays. For a leaking tray that is still powered on, NICo will first gracefully and then forcefully shut it down. + +To see the handling operation task status +`GET /nico/tray/{id}/task` + +## Next phase: API-manageable, customizable, whole-life leak detection and handling + +The overall focus for the next phase of NICo leak detection and handling is to make the feature manageable from API and supporting customized policy. + +### Dedicated leak status and leak handling status API + +Instead of relying on low-level APIs for health report and task, [issue #5018](https://github.com/NVIDIA/infra-controller/issues/5018) aims to provide a set of dedicated, easier-to-use leak status and leak handling status reporting API, such as + +``` +GET /nico/rack/{id}/leak +GET /nico/tray/{id}/leak +GET /nico/leak +``` + +Which should report +- Tray leak sensor status +- Tray leak sensor leakage status +- BMS rack leak sensor status +- BMS rack leak handling status +- Ongoing leak handling (graceful tray shutdown, forceful tray shutdown, rack isolation) +- If not ongoing leak handling, last leak handling and results + +### Customizable leak handling policies + +The current default general leak handling is built on top of NICo policy and automation engine. [issue #2076](https://github.com/NVIDIA/infra-controller/issues/2076) aims to provide a set of API to allow easy creation and management of customized general leakage handling policies at runtime. This is critical for fine-tuning for site-specific needs, especially as hardware become denser and more complex. + +Examples of potential customized policies in the future: +- Preventively turn off all trays beneath any leaking trays (rack topology) +- Turn off a rack if trays are leaking and flapping rapidly (time series) +- Turn off a rack based on combination of BMS sensor metrics + NICo tray detection (more sensitive yet robust detection) +- Turn off racks in a coolant loop with 2+ leaking racks (site topology) + +### Whole-life leak detection and handling + +Currently, NICo only detects from in-tray BMC sensors of ingested machines and switches. This does not cover the entire lifespan of liquid-cooled hardware, which not only exposes risks but will eventually become blockers for the scaling of AI factory. + +For trays with BMC powered on and visible from NICo but have not yet been fully ingested, [issue #5391](https://github.com/NVIDIA/infra-controller/issues/5391) aims to expand the collection, detection, reporting, and handling of leakage to those trays, which not only expands the leak detection and handling coverage, but is also a dependency for the enhancement below. + +For hosts that were known to be leaking before previously turned off, and for hosts that belong to a previously known-leaking rack, [issue #5510](https://github.com/NVIDIA/infra-controller/issues/5510) aims to prevent them from being automatically turned on, until their BMC or the whole rack's BMCs can give a definitive clearance of leakage. This is + +## Related documentation and implementation + +- [Monitoring and Health](monitoring-health.md) +- `crates/dsx-exchange-consumer/README.md` +- `crates/dsx-exchange-consumer/src/health_updater.rs` +- `helm/README.md` +- [issue #2076](https://github.com/NVIDIA/infra-controller/issues/2076) +- [Issue #5018](https://github.com/NVIDIA/infra-controller/issues/5018) +- [issue #5391](https://github.com/NVIDIA/infra-controller/issues/5391) +- [issue #5510](https://github.com/NVIDIA/infra-controller/issues/5510) diff --git a/docs/operations/leak-detection.md b/docs/operations/leak-detection.md deleted file mode 100644 index 27590724d3..0000000000 --- a/docs/operations/leak-detection.md +++ /dev/null @@ -1,97 +0,0 @@ -# Leak Detection, Health, and Allocation Protection - -## Overview - -NICo evaluates leak-related conditions from compute-tray sensor health and BMS leak events. It turns active conditions into health signals and, where configured, allocation protection. - -BMS owns the physical leak sensors and remediation actions. NICo provides the infrastructure-management health and allocation-protection surface. - -## Current capability - -### Compute-tray sensor health - -NICo hardware-health monitoring discovers configured BMC endpoints and queries them through Redfish. When compute-tray BMCs expose leak-related sensor data and the relevant health collector is configured, NICo considers that data as part of hardware health. - -```text -Compute-tray leak sensor - ↓ -Compute-tray BMC - ↓ Redfish -NICo hardware-health service - ↓ -NICo hardware health - ↓ -Aggregate host health -``` - -The alert classification and operational effect for this path depend on the site's health-processor configuration. - -### BMS leak-event integration - -NICo also supports a BMS event path through `nico-dsx-exchange-consumer`. - -```text -BMS detects or clears a leak condition - ↓ -BMS publishes MQTT metadata and value events - ↓ -NICo DSX Exchange consumer - ↓ -NICo rack health report - ↓ -Aggregate host health and allocation protection -``` - -The consumer supports these BMS event types: - -- Rack leak detection (`LeakDetectRack`) -- Rack leak-sensor fault (`LeakSensorFaultRack`) -- Rack-tray leak detection (`LeakDetectRackTray`) - -For an active supported BMS event, NICo creates a rack health alert with these classifications: - -- `PreventAllocations` -- `SensorCritical` -- `Hardware` - -`PreventAllocations` blocks new allocations for hosts affected by the active rack-health condition. When BMS publishes a clear event, NICo removes the corresponding rack health report. NICo then recalculates aggregate health; allocation eligibility can recover when no other active health condition prevents allocation. - -## Deployment requirements for BMS event integration - -The BMS event path is not enabled automatically in every NICo deployment. To use it, a deployment must have all of the following: - -1. `nico-dsx-exchange-consumer` enabled. This Helm subchart is disabled by default. -2. Connectivity and configuration for the BMS MQTT broker and event topics. -3. BMS metadata and value events that use supported point types and identify the affected rack. -4. The consumer configured to call the NICo API. -5. Health aggregation configured for the relevant racks and hosts. - -Without these prerequisites, BMS events do not create NICo rack health reports. - -## Operational meaning - -NICo provides **health visibility and allocation protection** for leak-related conditions. The current BMS event workflow creates a rack health alert that propagates through NICo health aggregation, while BMS handles physical remediation. - -Current operational visibility is provided through NICo's health data, health alert details, logs, and metrics. - -## Next phase: API-based leak workflows - -The next phase follows [issue #5018](https://github.com/NVIDIA/infra-controller/issues/5018) by adding API-based workflows for rack and tray leak detection and leak-handling state. It builds on the current health and allocation-protection workflow; it does not replace it. - -The API workflows expose: - -- rack and tray leak status; -- tray and BMS sensor details, including optional readouts and thresholds; -- BMS leak-handling status; -- ongoing handling operations, such as tray shutdown or rack isolation; -- previous handling results and details. - -This phase makes leak state and handling workflow available through the NICo API. BMS remains responsible for physical detection and remediation, while NICo provides the infrastructure-management health and lifecycle surface. - -## Related documentation and implementation - -- [Monitoring and Health](monitoring-health.md) -- `crates/dsx-exchange-consumer/README.md` -- `crates/dsx-exchange-consumer/src/health_updater.rs` -- `helm/README.md` -- [Issue #5018](https://github.com/NVIDIA/infra-controller/issues/5018) From 59ff3c751dee8653ae850f5ce00e430497854872 Mon Sep 17 00:00:00 2001 From: zhaozhongn <38698630+zhaozhongn@users.noreply.github.com> Date: Fri, 28 Aug 2026 10:13:21 -0700 Subject: [PATCH 03/28] Update leak-detection-handling.md Signed-off-by: zhaozhongn <38698630+zhaozhongn@users.noreply.github.com> Signed-off-by: zhaozhongn Signed-off-by: Zhaozhong Ni --- docs/operations/leak-detection-handling.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/operations/leak-detection-handling.md b/docs/operations/leak-detection-handling.md index dbbdaa8a74..32aac25e80 100644 --- a/docs/operations/leak-detection-handling.md +++ b/docs/operations/leak-detection-handling.md @@ -97,7 +97,7 @@ To see the handling operation task status ## Next phase: API-manageable, customizable, whole-life leak detection and handling -The overall focus for the next phase of NICo leak detection and handling is to make the feature manageable from API and supporting customized policy. +The overall focus for the next phase of NICo leak detection and handling is to make the feature manageable from API, supporting customized polices, and cover the whole lifespan of liquid-cooled hardware. ### Dedicated leak status and leak handling status API @@ -133,7 +133,7 @@ Currently, NICo only detects from in-tray BMC sensors of ingested machines and s For trays with BMC powered on and visible from NICo but have not yet been fully ingested, [issue #5391](https://github.com/NVIDIA/infra-controller/issues/5391) aims to expand the collection, detection, reporting, and handling of leakage to those trays, which not only expands the leak detection and handling coverage, but is also a dependency for the enhancement below. -For hosts that were known to be leaking before previously turned off, and for hosts that belong to a previously known-leaking rack, [issue #5510](https://github.com/NVIDIA/infra-controller/issues/5510) aims to prevent them from being automatically turned on, until their BMC or the whole rack's BMCs can give a definitive clearance of leakage. This is +For hosts that were known to be leaking before previously turned off, and for hosts that belong to a previously known-leaking rack, [issue #5510](https://github.com/NVIDIA/infra-controller/issues/5510) aims to prevent them from being automatically turned on, until their BMC or the whole rack's BMCs can give a definitive clearance of leakage. This will complete the coverage of the full lifespan of liquid-cooled hardware. ## Related documentation and implementation From 3f1bf723f687017684f633bc7457c3c2d196a0f6 Mon Sep 17 00:00:00 2001 From: zhaozhongn <38698630+zhaozhongn@users.noreply.github.com> Date: Fri, 28 Aug 2026 13:16:14 -0700 Subject: [PATCH 04/28] Update leak-detection-handling.md Signed-off-by: zhaozhongn <38698630+zhaozhongn@users.noreply.github.com> Signed-off-by: zhaozhongn Signed-off-by: Zhaozhong Ni --- docs/operations/leak-detection-handling.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/operations/leak-detection-handling.md b/docs/operations/leak-detection-handling.md index 32aac25e80..37926d7213 100644 --- a/docs/operations/leak-detection-handling.md +++ b/docs/operations/leak-detection-handling.md @@ -6,7 +6,7 @@ In rack-scale system, there are two sets of leak sensors, with NICo and BMS each NICo evaluates leak-related conditions from compute and NVSwitch tray sensors managed by it to detect leak events. NICo also listens via DSX Exchange (MQTT event bus) for BMS-issued leak events based on BMS-managed sensors such as CDU, rope, and dripping pan. A leak event results in tray and rack health warnings and alerts in NICo, which prevent the machines from being allocated. Depending on the leak type and severity, NICo takes automated handling actions based on configuration and policy. -## Current capability: automated e2e leak detection, reporting, protection, and handling +## Current capability: automated e2e leak detection, reporting, allocation protection, and handling ### Compute and NVSwitch tray sensor path @@ -95,9 +95,9 @@ Currently, the default handling policy is to shut down all leaking trays. For a To see the handling operation task status `GET /nico/tray/{id}/task` -## Next phase: API-manageable, customizable, whole-life leak detection and handling +## Next phase: API-manageable, customizable, full lifecycle leak detection and handling -The overall focus for the next phase of NICo leak detection and handling is to make the feature manageable from API, supporting customized polices, and cover the whole lifespan of liquid-cooled hardware. +The overall focus for the next phase of NICo leak detection and handling is to make the feature manageable from API, supporting customized polices, and cover the full lifecycle of liquid-cooled hardware. ### Dedicated leak status and leak handling status API @@ -127,13 +127,13 @@ Examples of potential customized policies in the future: - Turn off a rack based on combination of BMS sensor metrics + NICo tray detection (more sensitive yet robust detection) - Turn off racks in a coolant loop with 2+ leaking racks (site topology) -### Whole-life leak detection and handling +### Full lifecycle leak detection and handling -Currently, NICo only detects from in-tray BMC sensors of ingested machines and switches. This does not cover the entire lifespan of liquid-cooled hardware, which not only exposes risks but will eventually become blockers for the scaling of AI factory. +Currently, NICo only detects from in-tray BMC sensors of ingested machines and switches. This does not cover the full lifecycle of liquid-cooled hardware, which not only exposes risks but will eventually become blockers for the scaling of AI factory. For trays with BMC powered on and visible from NICo but have not yet been fully ingested, [issue #5391](https://github.com/NVIDIA/infra-controller/issues/5391) aims to expand the collection, detection, reporting, and handling of leakage to those trays, which not only expands the leak detection and handling coverage, but is also a dependency for the enhancement below. -For hosts that were known to be leaking before previously turned off, and for hosts that belong to a previously known-leaking rack, [issue #5510](https://github.com/NVIDIA/infra-controller/issues/5510) aims to prevent them from being automatically turned on, until their BMC or the whole rack's BMCs can give a definitive clearance of leakage. This will complete the coverage of the full lifespan of liquid-cooled hardware. +For hosts that were known to be leaking before previously turned off, and for hosts that belong to a previously known-leaking rack, [issue #5510](https://github.com/NVIDIA/infra-controller/issues/5510) aims to prevent them from being automatically turned on, until their BMC or the whole rack's BMCs can give a definitive clearance of leakage. This will complete the coverage of the full lifecycle of liquid-cooled hardware. ## Related documentation and implementation From 22a16d96056a1f1f8842e076f8bf8b12ff9fb926 Mon Sep 17 00:00:00 2001 From: zhaozhongn Date: Sat, 29 Aug 2026 14:41:13 -0700 Subject: [PATCH 05/28] Update leak-detection-handling.md Signed-off-by: zhaozhongn --- docs/operations/leak-detection-handling.md | 16 +++++++--------- 1 file changed, 7 insertions(+), 9 deletions(-) diff --git a/docs/operations/leak-detection-handling.md b/docs/operations/leak-detection-handling.md index 37926d7213..c6678835fa 100644 --- a/docs/operations/leak-detection-handling.md +++ b/docs/operations/leak-detection-handling.md @@ -4,7 +4,7 @@ In rack-scale system, there are two sets of leak sensors, with NICo and BMS each managing one set and detects leak based on them. Additionally, BMS controls the rack AC power breaker and coolant valve for rack isolation remediation, while NICo provides the infrastructure-management health reporting, allocation protection, and safe handling. -NICo evaluates leak-related conditions from compute and NVSwitch tray sensors managed by it to detect leak events. NICo also listens via DSX Exchange (MQTT event bus) for BMS-issued leak events based on BMS-managed sensors such as CDU, rope, and dripping pan. A leak event results in tray and rack health warnings and alerts in NICo, which prevent the machines from being allocated. Depending on the leak type and severity, NICo takes automated handling actions based on configuration and policy. +NICo evaluates leak-related conditions from compute and NVSwitch tray sensors managed by it to detect leak events. NICo also listens via DSX Exchange (MQTT event bus) for BMS-issued leak events based on BMS-managed sensors such as CDU, rope, and dripping pan. A leak event results in tray or rack health warnings and alerts in NICo, which prevent the machines from being allocated. Depending on the leak type and severity, NICo takes automated handling actions based on configuration and policy. ## Current capability: automated e2e leak detection, reporting, allocation protection, and handling @@ -60,7 +60,7 @@ Without these prerequisites, BMS events do not create NICo rack health reports. ### Health Reporting and Allocation Protection -NICo provides **health visibility and allocation protection** for leak-related conditions. Regardless of the source of leak detection and the path of health report creation, leak health alert comes with these classifications: +NICo provides **health visibility and allocation protection** for leak-related conditions. For BMS-based leak detection, rack leak health alert comes with these classifications: - `PreventAllocations` - `SensorCritical` @@ -78,24 +78,22 @@ NICo and BMS combined together automatically take leak handling actions in three Critical leakages are those detected by BMS based on BMS-managed sensors. Because these sensors are often at rack or even larger scope, any leaks detected from them may have very large and serious impact, and rack electric and liquid isolation often need to be performed ASAP, by BMS via its control of AC power breaker and coolant valve. -After NICo received the leak event from BMS via DSX Exchange, if, for any reason, the rack still has AC power on and there are still trays on, NICo will turn them off through its general leakage handling automation. - #### Severe Leakage Handling When multiple trays in a rack are detected to be leaking based on in-tray NICo-managed sensors, if it goes beyond a configured threshold (default to 2 for NVL72 racks currently), the rack is considered to have a severe leakage. To prevent more trays in the rack from being impacted by the leak, NICo will immediately request BMS, via DSX Exchange, to perform electric and liquid isolation to the rack. -Upon receiving the isolation request, BMS will trigger the AC power breaker and shut off coolant valve, and broadcast the isolation result back to NICo via DSX Exchange, which NICo waits for. +Upon receiving the isolation request, BMS will trigger the AC power breaker and shut off coolant valve, and broadcast the isolation result back DSX Exchange. #### General Leakage Handling When there are leaking trays but not enough to trigger a critical or severe leak, NICo handles the general leakage via policy-based automation. -Currently, the default handling policy is to shut down all leaking trays. For a leaking tray that is still powered on, NICo will first gracefully and then forcefully shut it down. +Currently, the default handling policy is to shut down all leaking trays. For a leaking tray that is still powered on, NICo will forcefully shut it down. To see the handling operation task status `GET /nico/tray/{id}/task` -## Next phase: API-manageable, customizable, full lifecycle leak detection and handling +## Next phase: API-manageable, customizable, full-lifecycle leak detection and handling The overall focus for the next phase of NICo leak detection and handling is to make the feature manageable from API, supporting customized polices, and cover the full lifecycle of liquid-cooled hardware. @@ -127,13 +125,13 @@ Examples of potential customized policies in the future: - Turn off a rack based on combination of BMS sensor metrics + NICo tray detection (more sensitive yet robust detection) - Turn off racks in a coolant loop with 2+ leaking racks (site topology) -### Full lifecycle leak detection and handling +### Full-lifecycle leak detection and handling Currently, NICo only detects from in-tray BMC sensors of ingested machines and switches. This does not cover the full lifecycle of liquid-cooled hardware, which not only exposes risks but will eventually become blockers for the scaling of AI factory. For trays with BMC powered on and visible from NICo but have not yet been fully ingested, [issue #5391](https://github.com/NVIDIA/infra-controller/issues/5391) aims to expand the collection, detection, reporting, and handling of leakage to those trays, which not only expands the leak detection and handling coverage, but is also a dependency for the enhancement below. -For hosts that were known to be leaking before previously turned off, and for hosts that belong to a previously known-leaking rack, [issue #5510](https://github.com/NVIDIA/infra-controller/issues/5510) aims to prevent them from being automatically turned on, until their BMC or the whole rack's BMCs can give a definitive clearance of leakage. This will complete the coverage of the full lifecycle of liquid-cooled hardware. +For hosts that were known to be leaking before being previously turned off, and for hosts that belong to a previously known-leaking rack, [issue #5510](https://github.com/NVIDIA/infra-controller/issues/5510) aims to prevent them from being automatically turned on, until their BMC or the whole rack's BMCs can give a definitive clearance of leakage. This will complete the coverage of the full lifecycle of liquid-cooled hardware. ## Related documentation and implementation From 32aafb6a16583e720ff1dfb39765d0b3ea79c40c Mon Sep 17 00:00:00 2001 From: zhaozhongn Date: Sat, 29 Aug 2026 14:50:40 -0700 Subject: [PATCH 06/28] Update index.yml Signed-off-by: zhaozhongn --- docs/index.yml | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/index.yml b/docs/index.yml index 5410548a09..65db3e0aac 100644 --- a/docs/index.yml +++ b/docs/index.yml @@ -206,8 +206,10 @@ navigation: path: manuals/machine_identity_signing_key_rotation.md - page: NVLink Partitioning path: manuals/nvlink_partitioning.md - - page: Rack-Level Administration (RLA) + - page: Rack-Level Administration path: manuals/rack_level_admin.md + - page: Leak Detection & Handling + path: operations/leak-detection-handling.md - section: Managing VPCs contents: - page: VPC Routing Profiles From 86afb0199a3282d8f9885287b7c076f19ce8cc13 Mon Sep 17 00:00:00 2001 From: zhaozhongn Date: Mon, 31 Aug 2026 11:39:34 -0700 Subject: [PATCH 07/28] Update docs/operations/leak-detection-handling.md Co-authored-by: Alex Ball Signed-off-by: zhaozhongn --- docs/operations/leak-detection-handling.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/operations/leak-detection-handling.md b/docs/operations/leak-detection-handling.md index c6678835fa..1ce00440fb 100644 --- a/docs/operations/leak-detection-handling.md +++ b/docs/operations/leak-detection-handling.md @@ -2,7 +2,7 @@ ## Overview -In rack-scale system, there are two sets of leak sensors, with NICo and BMS each managing one set and detects leak based on them. Additionally, BMS controls the rack AC power breaker and coolant valve for rack isolation remediation, while NICo provides the infrastructure-management health reporting, allocation protection, and safe handling. +In a rack-scale system, there are two sets of leak sensors: NICo and the BMS each manage one set, and detect leaks based on them. Additionally, the BMS controls the rack AC power breaker and coolant valve for rack isolation remediation, while NICo provides infrastructure-management health reporting, allocation protection, and safe handling. NICo evaluates leak-related conditions from compute and NVSwitch tray sensors managed by it to detect leak events. NICo also listens via DSX Exchange (MQTT event bus) for BMS-issued leak events based on BMS-managed sensors such as CDU, rope, and dripping pan. A leak event results in tray or rack health warnings and alerts in NICo, which prevent the machines from being allocated. Depending on the leak type and severity, NICo takes automated handling actions based on configuration and policy. From 32b63befaada445810f9a7cf329962c5c45b47b8 Mon Sep 17 00:00:00 2001 From: zhaozhongn Date: Mon, 31 Aug 2026 11:39:47 -0700 Subject: [PATCH 08/28] Update docs/operations/leak-detection-handling.md Co-authored-by: Alex Ball Signed-off-by: zhaozhongn --- docs/operations/leak-detection-handling.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/operations/leak-detection-handling.md b/docs/operations/leak-detection-handling.md index 1ce00440fb..fa2f11fa7e 100644 --- a/docs/operations/leak-detection-handling.md +++ b/docs/operations/leak-detection-handling.md @@ -4,7 +4,7 @@ In a rack-scale system, there are two sets of leak sensors: NICo and the BMS each manage one set, and detect leaks based on them. Additionally, the BMS controls the rack AC power breaker and coolant valve for rack isolation remediation, while NICo provides infrastructure-management health reporting, allocation protection, and safe handling. -NICo evaluates leak-related conditions from compute and NVSwitch tray sensors managed by it to detect leak events. NICo also listens via DSX Exchange (MQTT event bus) for BMS-issued leak events based on BMS-managed sensors such as CDU, rope, and dripping pan. A leak event results in tray or rack health warnings and alerts in NICo, which prevent the machines from being allocated. Depending on the leak type and severity, NICo takes automated handling actions based on configuration and policy. +NICo uses the sensors it manages in compute and NVSwitch trays to detect leak events. NICo also listens via the [DSX Exchange](https://docs.nvidia.com/dsx-exchange) MQTT event bus for BMS-issued leak events from BMS-managed sensors such as CDU, rope, and dripping pan. A leak event results in tray or rack health warnings and alerts in NICo, which prevent the machines from being allocated. Depending on the leak type and severity, NICo takes automated handling actions based on configuration and policy. ## Current capability: automated e2e leak detection, reporting, allocation protection, and handling From 7eef31be19ae108a43270eb97097eaa817782892 Mon Sep 17 00:00:00 2001 From: zhaozhongn Date: Mon, 31 Aug 2026 11:39:57 -0700 Subject: [PATCH 09/28] Update docs/operations/leak-detection-handling.md Co-authored-by: Alex Ball Signed-off-by: zhaozhongn --- docs/operations/leak-detection-handling.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/operations/leak-detection-handling.md b/docs/operations/leak-detection-handling.md index fa2f11fa7e..f4b7b71937 100644 --- a/docs/operations/leak-detection-handling.md +++ b/docs/operations/leak-detection-handling.md @@ -26,7 +26,7 @@ Health report with leakage warnings or alerts are generated for the leaking tray ### BMS leak-event integration -NICo also supports a BMS leak event (based on BMS-managed sensors such as CDU, rope, dripping pan, etc.) path through the `nico-dsx-exchange-consumer` service. +NICo also supports a BMS leak event path (based on BMS-managed sensors such as CDU, rope, dripping pan, and others) through the `nico-dsx-exchange-consumer` service. ```text BMS detects / clears a leak condition From fd4408fd5f68b8c79492a138175786b350bfb7d1 Mon Sep 17 00:00:00 2001 From: zhaozhongn Date: Mon, 31 Aug 2026 11:40:06 -0700 Subject: [PATCH 10/28] Update docs/operations/leak-detection-handling.md Co-authored-by: Alex Ball Signed-off-by: zhaozhongn --- docs/operations/leak-detection-handling.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/docs/operations/leak-detection-handling.md b/docs/operations/leak-detection-handling.md index f4b7b71937..3cbc060744 100644 --- a/docs/operations/leak-detection-handling.md +++ b/docs/operations/leak-detection-handling.md @@ -107,13 +107,14 @@ GET /nico/tray/{id}/leak GET /nico/leak ``` -Which should report +These endpoints should report: + - Tray leak sensor status - Tray leak sensor leakage status - BMS rack leak sensor status - BMS rack leak handling status -- Ongoing leak handling (graceful tray shutdown, forceful tray shutdown, rack isolation) -- If not ongoing leak handling, last leak handling and results +- Ongoing leak handling actions (graceful tray shutdown, forceful tray shutdown, rack isolation) +- If there is no ongoing leak handling, last leak-handling actions and results ### Customizable leak handling policies From 8a4f2f7c022969b43860244cdc9f8ab2beff92cc Mon Sep 17 00:00:00 2001 From: zhaozhongn Date: Mon, 31 Aug 2026 11:40:24 -0700 Subject: [PATCH 11/28] Update docs/operations/leak-detection-handling.md Co-authored-by: Alex Ball Signed-off-by: zhaozhongn --- docs/operations/leak-detection-handling.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/operations/leak-detection-handling.md b/docs/operations/leak-detection-handling.md index 3cbc060744..8f1d6809fd 100644 --- a/docs/operations/leak-detection-handling.md +++ b/docs/operations/leak-detection-handling.md @@ -22,7 +22,7 @@ NICo hardware-health service NICo tray health report ``` -Health report with leakage warnings or alerts are generated for the leaking trays. The alert classification and operational effect for this path depend on the site's health-processor configuration. +NICo generates health reports with leakage warnings or alerts for the leaking trays. The alert classification and operational effect for this path depend on the site's health-processor configuration. ### BMS leak-event integration From 39c3f63bd337bf8997e906e98ba84b02e60799e3 Mon Sep 17 00:00:00 2001 From: zhaozhongn Date: Mon, 31 Aug 2026 11:40:35 -0700 Subject: [PATCH 12/28] Update docs/operations/leak-detection-handling.md Co-authored-by: Alex Ball Signed-off-by: zhaozhongn --- docs/operations/leak-detection-handling.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/operations/leak-detection-handling.md b/docs/operations/leak-detection-handling.md index 8f1d6809fd..28f0f64555 100644 --- a/docs/operations/leak-detection-handling.md +++ b/docs/operations/leak-detection-handling.md @@ -44,7 +44,7 @@ The consumer supports these BMS event types: - Rack leak-sensor fault (`LeakSensorFaultRack`) - Rack-tray leak detection (`LeakDetectRackTray`) -For an active supported BMS event, NICo creates rack health report with a leak alert. When BMS publishes a clear event, NICo removes the corresponding rack health report. +For an active supported BMS event, NICo creates a rack health report with a leak alert. When the BMS publishes a clear event, NICo removes the corresponding rack health report. ### Deployment requirements for BMS event integration From 54597b510fe80d533e4e41d618bf0b2cd4b96ef7 Mon Sep 17 00:00:00 2001 From: zhaozhongn Date: Mon, 31 Aug 2026 11:40:48 -0700 Subject: [PATCH 13/28] Update docs/operations/leak-detection-handling.md Co-authored-by: Alex Ball Signed-off-by: zhaozhongn --- docs/operations/leak-detection-handling.md | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/docs/operations/leak-detection-handling.md b/docs/operations/leak-detection-handling.md index 28f0f64555..f03cc20eee 100644 --- a/docs/operations/leak-detection-handling.md +++ b/docs/operations/leak-detection-handling.md @@ -118,13 +118,14 @@ These endpoints should report: ### Customizable leak handling policies -The current default general leak handling is built on top of NICo policy and automation engine. [issue #2076](https://github.com/NVIDIA/infra-controller/issues/2076) aims to provide a set of API to allow easy creation and management of customized general leakage handling policies at runtime. This is critical for fine-tuning for site-specific needs, especially as hardware become denser and more complex. +NICo's current default general leak handling uses the NICo policy and automation engine. [Issue #2076](https://github.com/NVIDIA/infra-controller/issues/2076) aims to provide a set of APIs to allow you to create and manage customized general leakage-handling policies at runtime. This is critical for fine-tuning for site-specific needs, especially as hardware becomes denser and more complex. -Examples of potential customized policies in the future: -- Preventively turn off all trays beneath any leaking trays (rack topology) +The following are examples of _potential_ customized future policies: + +- Preemptively turn off all trays located below a leaking tray (based on rack topology) - Turn off a rack if trays are leaking and flapping rapidly (time series) -- Turn off a rack based on combination of BMS sensor metrics + NICo tray detection (more sensitive yet robust detection) -- Turn off racks in a coolant loop with 2+ leaking racks (site topology) +- Turn off a rack based on a combination of BMS sensor metrics and NICo tray detection (more sensitive yet robust detection) +- Turn off racks in a coolant loop with two or more leaking racks (based on site topology) ### Full-lifecycle leak detection and handling From 3effc8929d4661c3ec873404de6aad8ac4931297 Mon Sep 17 00:00:00 2001 From: zhaozhongn Date: Mon, 31 Aug 2026 11:41:07 -0700 Subject: [PATCH 14/28] Update docs/operations/leak-detection-handling.md Co-authored-by: Alex Ball Signed-off-by: zhaozhongn --- docs/operations/leak-detection-handling.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/operations/leak-detection-handling.md b/docs/operations/leak-detection-handling.md index f03cc20eee..d029028c52 100644 --- a/docs/operations/leak-detection-handling.md +++ b/docs/operations/leak-detection-handling.md @@ -129,11 +129,11 @@ The following are examples of _potential_ customized future policies: ### Full-lifecycle leak detection and handling -Currently, NICo only detects from in-tray BMC sensors of ingested machines and switches. This does not cover the full lifecycle of liquid-cooled hardware, which not only exposes risks but will eventually become blockers for the scaling of AI factory. +Currently, NICo only detects from in-tray BMC sensors of _ingested_ machines and switches. This does not cover the full lifecycle of liquid-cooled hardware, which not only exposes risks, but will eventually become a blocker for scaling AI factories. -For trays with BMC powered on and visible from NICo but have not yet been fully ingested, [issue #5391](https://github.com/NVIDIA/infra-controller/issues/5391) aims to expand the collection, detection, reporting, and handling of leakage to those trays, which not only expands the leak detection and handling coverage, but is also a dependency for the enhancement below. +For trays with a BMC powered on and visible from NICo, but that have not yet been fully ingested, [issue #5391](https://github.com/NVIDIA/infra-controller/issues/5391) aims to expand the collection, detection, reporting, and handling of leakage to those trays. This not only expands leak detection and handling coverage, but is also a dependency for the following enhancement. -For hosts that were known to be leaking before being previously turned off, and for hosts that belong to a previously known-leaking rack, [issue #5510](https://github.com/NVIDIA/infra-controller/issues/5510) aims to prevent them from being automatically turned on, until their BMC or the whole rack's BMCs can give a definitive clearance of leakage. This will complete the coverage of the full lifecycle of liquid-cooled hardware. +For hosts known to be leaking before being turned off, and for hosts in a previously known-leaking rack, [issue #5510](https://github.com/NVIDIA/infra-controller/issues/5510) aims to prevent these hosts from being automatically turned on until their BMCs (or the whole rack's BMCs) can give a definitive clearance of leakage. This will complete the coverage of the full lifecycle of liquid-cooled hardware. ## Related documentation and implementation From ae1bd3e336329e7d6fae9d140508f208aa16ec91 Mon Sep 17 00:00:00 2001 From: zhaozhongn Date: Mon, 31 Aug 2026 11:41:20 -0700 Subject: [PATCH 15/28] Update docs/operations/leak-detection-handling.md Co-authored-by: Alex Ball Signed-off-by: zhaozhongn --- docs/operations/leak-detection-handling.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/operations/leak-detection-handling.md b/docs/operations/leak-detection-handling.md index d029028c52..72eeb197d0 100644 --- a/docs/operations/leak-detection-handling.md +++ b/docs/operations/leak-detection-handling.md @@ -138,9 +138,9 @@ For hosts known to be leaking before being turned off, and for hosts in a previo ## Related documentation and implementation - [Monitoring and Health](monitoring-health.md) -- `crates/dsx-exchange-consumer/README.md` -- `crates/dsx-exchange-consumer/src/health_updater.rs` -- `helm/README.md` +- [`crates/dsx-exchange-consumer/README.md`](https://github.com/NVIDIA/infra-controller/blob/main/crates/dsx-exchange-consumer/README.md) +- [`crates/dsx-exchange-consumer/src/health_updater.rs`](https://github.com/NVIDIA/infra-controller/blob/main/crates/dsx-exchange-consumer/src/health_updater.rs) +- [`helm/README.md`](https://github.com/NVIDIA/infra-controller/blob/main/helm/README.md) - [issue #2076](https://github.com/NVIDIA/infra-controller/issues/2076) - [Issue #5018](https://github.com/NVIDIA/infra-controller/issues/5018) - [issue #5391](https://github.com/NVIDIA/infra-controller/issues/5391) From 8d6a5cae50235984df44abdf9ff4c10a84aa74c4 Mon Sep 17 00:00:00 2001 From: zhaozhongn Date: Mon, 31 Aug 2026 11:41:47 -0700 Subject: [PATCH 16/28] Update docs/operations/leak-detection-handling.md Co-authored-by: Alex Ball Signed-off-by: zhaozhongn --- docs/operations/leak-detection-handling.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/operations/leak-detection-handling.md b/docs/operations/leak-detection-handling.md index 72eeb197d0..7af98bf3e9 100644 --- a/docs/operations/leak-detection-handling.md +++ b/docs/operations/leak-detection-handling.md @@ -141,7 +141,7 @@ For hosts known to be leaking before being turned off, and for hosts in a previo - [`crates/dsx-exchange-consumer/README.md`](https://github.com/NVIDIA/infra-controller/blob/main/crates/dsx-exchange-consumer/README.md) - [`crates/dsx-exchange-consumer/src/health_updater.rs`](https://github.com/NVIDIA/infra-controller/blob/main/crates/dsx-exchange-consumer/src/health_updater.rs) - [`helm/README.md`](https://github.com/NVIDIA/infra-controller/blob/main/helm/README.md) -- [issue #2076](https://github.com/NVIDIA/infra-controller/issues/2076) +- [Issue #2076](https://github.com/NVIDIA/infra-controller/issues/2076) - [Issue #5018](https://github.com/NVIDIA/infra-controller/issues/5018) -- [issue #5391](https://github.com/NVIDIA/infra-controller/issues/5391) -- [issue #5510](https://github.com/NVIDIA/infra-controller/issues/5510) +- [Issue #5391](https://github.com/NVIDIA/infra-controller/issues/5391) +- [Issue #5510](https://github.com/NVIDIA/infra-controller/issues/5510) From f2aaa749edb952d5902ecf784510ac3867318365 Mon Sep 17 00:00:00 2001 From: zhaozhongn Date: Mon, 31 Aug 2026 11:52:52 -0700 Subject: [PATCH 17/28] Update docs/operations/leak-detection-handling.md Co-authored-by: Alex Ball Signed-off-by: zhaozhongn --- docs/operations/leak-detection-handling.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/operations/leak-detection-handling.md b/docs/operations/leak-detection-handling.md index 7af98bf3e9..4270b3da68 100644 --- a/docs/operations/leak-detection-handling.md +++ b/docs/operations/leak-detection-handling.md @@ -48,7 +48,7 @@ For an active supported BMS event, NICo creates a rack health report with a leak ### Deployment requirements for BMS event integration -The BMS event path is not enabled automatically in every NICo deployment. To use it, a deployment must have all of the following: +The BMS event path is not enabled automatically in every NICo deployment. To use the BMS event path, a deployment must have all of the following: 1. `nico-dsx-exchange-consumer` enabled. This Helm subchart is disabled by default. 2. Connectivity and configuration for the BMS MQTT broker and event topics. From 66c062ba88d577fda2de9e8ef2d6433e831bc6a2 Mon Sep 17 00:00:00 2001 From: zhaozhongn Date: Mon, 31 Aug 2026 11:53:11 -0700 Subject: [PATCH 18/28] Update docs/operations/leak-detection-handling.md Co-authored-by: Alex Ball Signed-off-by: zhaozhongn --- docs/operations/leak-detection-handling.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/operations/leak-detection-handling.md b/docs/operations/leak-detection-handling.md index 4270b3da68..9eda2bb039 100644 --- a/docs/operations/leak-detection-handling.md +++ b/docs/operations/leak-detection-handling.md @@ -50,11 +50,11 @@ For an active supported BMS event, NICo creates a rack health report with a leak The BMS event path is not enabled automatically in every NICo deployment. To use the BMS event path, a deployment must have all of the following: -1. `nico-dsx-exchange-consumer` enabled. This Helm subchart is disabled by default. -2. Connectivity and configuration for the BMS MQTT broker and event topics. -3. BMS metadata and value events that use supported point types and identify the affected rack. -4. The consumer configured to call the NICo API. -5. Health aggregation configured for the relevant racks and hosts. +1. `nico-dsx-exchange-consumer` enabled (this Helm subchart is disabled by default) +1. Connectivity and configuration for the BMS MQTT broker and event topics +1. BMS metadata and value events that use supported point types and identify the affected rack +1. The consumer configured to call the NICo API +1. Health aggregation configured for the relevant racks and hosts Without these prerequisites, BMS events do not create NICo rack health reports. From 3668e389598b59c3be0cf50e77527fcdf3ed7c22 Mon Sep 17 00:00:00 2001 From: zhaozhongn Date: Mon, 31 Aug 2026 11:53:30 -0700 Subject: [PATCH 19/28] Update docs/operations/leak-detection-handling.md Co-authored-by: Alex Ball Signed-off-by: zhaozhongn --- docs/operations/leak-detection-handling.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/operations/leak-detection-handling.md b/docs/operations/leak-detection-handling.md index 9eda2bb039..e59e79ec91 100644 --- a/docs/operations/leak-detection-handling.md +++ b/docs/operations/leak-detection-handling.md @@ -60,7 +60,7 @@ Without these prerequisites, BMS events do not create NICo rack health reports. ### Health Reporting and Allocation Protection -NICo provides **health visibility and allocation protection** for leak-related conditions. For BMS-based leak detection, rack leak health alert comes with these classifications: +NICo provides **health visibility and allocation protection** for leak-related conditions. For BMS-based leak detection, rack leak health alerts come with these classifications: - `PreventAllocations` - `SensorCritical` From 2f8996669b1bc6b68402c469eb48ca546b535c13 Mon Sep 17 00:00:00 2001 From: zhaozhongn Date: Mon, 31 Aug 2026 11:54:24 -0700 Subject: [PATCH 20/28] Update docs/operations/leak-detection-handling.md Co-authored-by: Alex Ball Signed-off-by: zhaozhongn --- docs/operations/leak-detection-handling.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/docs/operations/leak-detection-handling.md b/docs/operations/leak-detection-handling.md index e59e79ec91..2f59ef2f89 100644 --- a/docs/operations/leak-detection-handling.md +++ b/docs/operations/leak-detection-handling.md @@ -67,8 +67,7 @@ NICo provides **health visibility and allocation protection** for leak-related c `PreventAllocations` blocks new allocations for hosts affected by the active tray- or rack-health condition. When the health alert clears later, NICo recalculates aggregate health; allocation eligibility can recover when no other active health condition prevents allocation. -Current operational visibility is provided through NICo's health data, health alert details, logs, and metrics. To obtain the current health reports of a machine: -`GET /nico/machine/{id}/health-report` +Current operational visibility is provided through NICo's health data, health alert details, logs, and metrics. To obtain a machine's current health reports, use the `GET /v2/org/{org}/nico/machine/{id}/health-report` [REST API endpoint](api:GET/v2/org/:org/nico/machine/:machineId/health-report). ### Three Tiers of Automated Leak Handling From 69c1806736aba46f72b0307391c08e65aff0f177 Mon Sep 17 00:00:00 2001 From: zhaozhongn Date: Mon, 31 Aug 2026 11:54:42 -0700 Subject: [PATCH 21/28] Update docs/operations/leak-detection-handling.md Co-authored-by: Alex Ball Signed-off-by: zhaozhongn --- docs/operations/leak-detection-handling.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/operations/leak-detection-handling.md b/docs/operations/leak-detection-handling.md index 2f59ef2f89..8f34673ddb 100644 --- a/docs/operations/leak-detection-handling.md +++ b/docs/operations/leak-detection-handling.md @@ -71,7 +71,7 @@ Current operational visibility is provided through NICo's health data, health al ### Three Tiers of Automated Leak Handling -NICo and BMS combined together automatically take leak handling actions in three tiers. +NICo and the BMS together automatically take leak handling actions in three tiers. #### Critical Leakage Handling From d5d0a515fc968ae9d3b76955d2dc4c156867f57d Mon Sep 17 00:00:00 2001 From: zhaozhongn Date: Mon, 31 Aug 2026 11:54:58 -0700 Subject: [PATCH 22/28] Update docs/operations/leak-detection-handling.md Co-authored-by: Alex Ball Signed-off-by: zhaozhongn --- docs/operations/leak-detection-handling.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/operations/leak-detection-handling.md b/docs/operations/leak-detection-handling.md index 8f34673ddb..09b706c3ac 100644 --- a/docs/operations/leak-detection-handling.md +++ b/docs/operations/leak-detection-handling.md @@ -75,7 +75,7 @@ NICo and the BMS together automatically take leak handling actions in three tier #### Critical Leakage Handling -Critical leakages are those detected by BMS based on BMS-managed sensors. Because these sensors are often at rack or even larger scope, any leaks detected from them may have very large and serious impact, and rack electric and liquid isolation often need to be performed ASAP, by BMS via its control of AC power breaker and coolant valve. +Critical leaks are those detected by the BMS based on BMS-managed sensors. Because these sensors are often at rack or even larger scope, any leaks detected from them can have a very large and serious impact. Critical leaks often require immediate rack electric and liquid isolation; the BMS does this using its control of AC power breakers and coolant valves. #### Severe Leakage Handling From a2cf2dcd26fd4c75f0f959f18abccdf0c5db2e7d Mon Sep 17 00:00:00 2001 From: zhaozhongn Date: Mon, 31 Aug 2026 11:55:13 -0700 Subject: [PATCH 23/28] Update docs/operations/leak-detection-handling.md Co-authored-by: Alex Ball Signed-off-by: zhaozhongn --- docs/operations/leak-detection-handling.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/operations/leak-detection-handling.md b/docs/operations/leak-detection-handling.md index 09b706c3ac..18e8a5b139 100644 --- a/docs/operations/leak-detection-handling.md +++ b/docs/operations/leak-detection-handling.md @@ -79,9 +79,9 @@ Critical leaks are those detected by the BMS based on BMS-managed sensors. Becau #### Severe Leakage Handling -When multiple trays in a rack are detected to be leaking based on in-tray NICo-managed sensors, if it goes beyond a configured threshold (default to 2 for NVL72 racks currently), the rack is considered to have a severe leakage. To prevent more trays in the rack from being impacted by the leak, NICo will immediately request BMS, via DSX Exchange, to perform electric and liquid isolation to the rack. +If in-tray NICo-managed sensors indicate that multiple trays in a rack are leaking, and the number of leaking trays exceeds a configured threshold (the default for NVL72 racks is two leaking trays), the rack is considered to have a severe leakage. To prevent more trays in the rack from being impacted by the leak, NICo immediately requests the BMS, using DSX Exchange, to perform electric and liquid isolation for the rack. -Upon receiving the isolation request, BMS will trigger the AC power breaker and shut off coolant valve, and broadcast the isolation result back DSX Exchange. +Upon receiving the isolation request, the BMS triggers the AC power breaker and coolant shutoff valve, and broadcasts the isolation result over DSX Exchange. #### General Leakage Handling From 2bb1f0584079274eddf365378c4efe519520fe56 Mon Sep 17 00:00:00 2001 From: zhaozhongn Date: Mon, 31 Aug 2026 11:55:26 -0700 Subject: [PATCH 24/28] Update docs/operations/leak-detection-handling.md Co-authored-by: Alex Ball Signed-off-by: zhaozhongn --- docs/operations/leak-detection-handling.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/operations/leak-detection-handling.md b/docs/operations/leak-detection-handling.md index 18e8a5b139..6236dd7ab7 100644 --- a/docs/operations/leak-detection-handling.md +++ b/docs/operations/leak-detection-handling.md @@ -87,7 +87,7 @@ Upon receiving the isolation request, the BMS triggers the AC power breaker and When there are leaking trays but not enough to trigger a critical or severe leak, NICo handles the general leakage via policy-based automation. -Currently, the default handling policy is to shut down all leaking trays. For a leaking tray that is still powered on, NICo will forcefully shut it down. +The default handling policy is to shut down all leaking trays. For a leaking tray that is still powered on, NICo forcefully shuts it down. To see the handling operation task status `GET /nico/tray/{id}/task` From 19df9cc2e52770321687ea6a6af7b7879f777e2c Mon Sep 17 00:00:00 2001 From: zhaozhongn Date: Mon, 31 Aug 2026 11:55:50 -0700 Subject: [PATCH 25/28] Update docs/operations/leak-detection-handling.md Co-authored-by: Alex Ball Signed-off-by: zhaozhongn --- docs/operations/leak-detection-handling.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/docs/operations/leak-detection-handling.md b/docs/operations/leak-detection-handling.md index 6236dd7ab7..a00e19d1b9 100644 --- a/docs/operations/leak-detection-handling.md +++ b/docs/operations/leak-detection-handling.md @@ -89,8 +89,7 @@ When there are leaking trays but not enough to trigger a critical or severe leak The default handling policy is to shut down all leaking trays. For a leaking tray that is still powered on, NICo forcefully shuts it down. -To see the handling operation task status -`GET /nico/tray/{id}/task` +Use the `GET /nico/tray/{id}/task` [REST API endpoint](api:GET/v2/org/:org/nico/tray/:id/task) to see the handling operation task status ## Next phase: API-manageable, customizable, full-lifecycle leak detection and handling From 7e761d1aa594ff07901ac6cfb48cbb8fc1e1ddf4 Mon Sep 17 00:00:00 2001 From: zhaozhongn Date: Mon, 31 Aug 2026 11:56:12 -0700 Subject: [PATCH 26/28] Update docs/operations/leak-detection-handling.md Co-authored-by: Alex Ball Signed-off-by: zhaozhongn --- docs/operations/leak-detection-handling.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/operations/leak-detection-handling.md b/docs/operations/leak-detection-handling.md index a00e19d1b9..ab26cb709d 100644 --- a/docs/operations/leak-detection-handling.md +++ b/docs/operations/leak-detection-handling.md @@ -91,7 +91,7 @@ The default handling policy is to shut down all leaking trays. For a leaking tra Use the `GET /nico/tray/{id}/task` [REST API endpoint](api:GET/v2/org/:org/nico/tray/:id/task) to see the handling operation task status -## Next phase: API-manageable, customizable, full-lifecycle leak detection and handling +## Future work: API-manageable, customizable, full-lifecycle leak detection and handling The overall focus for the next phase of NICo leak detection and handling is to make the feature manageable from API, supporting customized polices, and cover the full lifecycle of liquid-cooled hardware. From a5b245258d47b04982b72fec5daa24abaefc28f4 Mon Sep 17 00:00:00 2001 From: zhaozhongn Date: Mon, 31 Aug 2026 11:56:26 -0700 Subject: [PATCH 27/28] Update docs/operations/leak-detection-handling.md Co-authored-by: Alex Ball Signed-off-by: zhaozhongn --- docs/operations/leak-detection-handling.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/operations/leak-detection-handling.md b/docs/operations/leak-detection-handling.md index ab26cb709d..fd080323f7 100644 --- a/docs/operations/leak-detection-handling.md +++ b/docs/operations/leak-detection-handling.md @@ -93,7 +93,7 @@ Use the `GET /nico/tray/{id}/task` [REST API endpoint](api:GET/v2/org/:org/nico/ ## Future work: API-manageable, customizable, full-lifecycle leak detection and handling -The overall focus for the next phase of NICo leak detection and handling is to make the feature manageable from API, supporting customized polices, and cover the full lifecycle of liquid-cooled hardware. +The overall focus for the next phase of NICo leak detection and handling is to make the feature manageable using APIs, support customized polices, and cover the full lifecycle of liquid-cooled hardware. ### Dedicated leak status and leak handling status API From c994d7484532a63c2f34bf4f5637591c30dfd5ea Mon Sep 17 00:00:00 2001 From: zhaozhongn Date: Mon, 31 Aug 2026 11:56:43 -0700 Subject: [PATCH 28/28] Update docs/operations/leak-detection-handling.md Co-authored-by: Alex Ball Signed-off-by: zhaozhongn --- docs/operations/leak-detection-handling.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/operations/leak-detection-handling.md b/docs/operations/leak-detection-handling.md index fd080323f7..39ca75e2a9 100644 --- a/docs/operations/leak-detection-handling.md +++ b/docs/operations/leak-detection-handling.md @@ -99,7 +99,7 @@ The overall focus for the next phase of NICo leak detection and handling is to m Instead of relying on low-level APIs for health report and task, [issue #5018](https://github.com/NVIDIA/infra-controller/issues/5018) aims to provide a set of dedicated, easier-to-use leak status and leak handling status reporting API, such as -``` +```http GET /nico/rack/{id}/leak GET /nico/tray/{id}/leak GET /nico/leak