Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
52 changes: 50 additions & 2 deletions crates/api-core/src/cfg/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ Use `site_explorer.dpu_policy` instead.
| `enable_route_servers` | `bool` | `false` | `networking` | Enables route server injection into DPU FRR configs for L2VPN. |
| `deny_prefixes` | `Vec<IpNetwork>` | `[]` | `networking` | IPv4 and IPv6 CIDR prefixes that tenant instances are blocked from reaching. FNN generates family-specific NVUE ACL policies; all non-FNN virtualizers apply the IPv4 prefixes only. |
| `site_fabric_prefixes` | `Vec<IpNetwork>` | `[]` | `networking` | IP prefixes (v4/v6) assigned for tenant use within this site. |
| `tenant_prefix_overlap_enabled` | `bool` | `false` | `networking` | Site opt-in for tenant prefix overlap admission. This setting has no effect until [#3890](https://github.com/NVIDIA/infra-controller/issues/3890) lands and does not change the existing database prefix constraints. Admission will also require site-wide `vpc_isolation_behavior = "mutual_isolation"` and participating FNN base profiles with `tenant_prefix_overlap_eligible = true`. |
| `tenant_prefix_overlap_enabled` | `bool` | `false` | `networking` | Site opt-in for [tenant prefix overlap checks](#tenant-prefix-overlap-checks). The existing `VpcPrefix` exclusion continues to prevent overlapping `VpcPrefix` persistence until the cutover tracked by [#3892](https://github.com/NVIDIA/infra-controller/issues/3892). |
| `max_site_prefixes_per_tenant` | `u32` | `8` | `networking` | Maximum tenant-managed SitePrefixes retained for one tenant at this site. Prefixes awaiting removal still count against this limit and keep their CIDR reserved. |
| `anycast_site_prefixes` | `Vec<Ipv4Network>` | `[]` | `networking` | Aggregate IPv4 prefixes containing tenant-announced prefixes (e.g., BYOIP). **Deprecated.** Use [`routing_profiles.allowed_anycast_prefixes`](#fnnroutingprofileconfig) instead. |
| `common_tenant_host_asn` | `Option<u32>` | — | `networking` | ASN that tenants use to peer with the DPU. If unset, any ASN is accepted. |
Expand Down Expand Up @@ -705,7 +705,7 @@ client-certificate authentication is not used.
| `route_target_imports` | `Option<Vec<RouteTargetConfig>>` | — (effective `[]`) | Route targets imported into DPU VRFs for VPC routes. |
| `route_targets_on_exports` | `Option<Vec<RouteTargetConfig>>` | — (effective `[]`) | Route targets added to routes exported by the DPU. |
| `internal` | `Option<bool>` | — (effective `false`) | Whether the profile uses internal VNI allocation. This property cannot be overridden on a VPC. |
| `tenant_prefix_overlap_eligible` | `bool` | `false` | Base-profile opt-in for tenant prefix overlap admission. This setting has no effect until [#3890](https://github.com/NVIDIA/infra-controller/issues/3890) lands and cannot be overridden on a VPC. |
| `tenant_prefix_overlap_eligible` | `bool` | `false` | Base routing profile opt-in for [tenant prefix overlap checks](#tenant-prefix-overlap-checks). This setting cannot be overridden on a VPC. |
| `leak_default_route_from_underlay` | `Option<bool>` | — (effective `false`) | Leak the default route from the underlay/default VRF into tenant VRFs. |
| `leak_tenant_host_routes_to_underlay` | `Option<bool>` | — (effective `false`) | Leak tenant host routes into the underlay/default VRF. |
| `tenant_leak_communities_accepted` | `Option<bool>` | — (effective `false`) | Honor route-leak communities sent by the tenant host OS. |
Expand All @@ -717,6 +717,54 @@ Unset properties retain presence information so a VPC's inline
`routing_profile_overrides` can inherit them. After the named profile and VPC
override are combined, properties still unset use the effective defaults above.

### Tenant prefix overlap checks

`tenant_prefix_overlap_enabled` defaults to `false`. When set to `true`, NICo
checks whether two `VpcPrefix` records may reuse the same CIDR. It does not
permit direct `NetworkPrefix` reuse or change the database constraints.

An overlapping `VpcPrefix` pair is eligible only when all of these conditions
are true:

- The requested and existing CIDRs are identical, the existing `VpcPrefix` is
not deleted, and the `VpcPrefix` records belong to different VPCs and tenant
organizations.
- Both VPCs use FNN and have assigned, distinct status VNIs.
- Each `VpcPrefix` is linked to a tenant-managed, `DatacenterOnly` `SitePrefix`
owned by its VPC tenant and containing the `VpcPrefix` CIDR. The requested
`SitePrefix` must be `Ready`; the existing `SitePrefix` may be `Ready` or
`Deleting`.
- Site-wide `vpc_isolation_behavior` is `"mutual_isolation"`.
- `site_global_vpc_vni` and `common_internal_route_target` are unset, and

@bcavnvidia bcavnvidia Aug 31, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

site_global_vpc_vni - The imports/exports should be driven by the actual RTs, not this value. Though we should check if there is an auto somewhere that could let site_global_vpc_vni do more than it should. We should remove that if so.

`additional_route_target_imports` is empty, so they cannot bridge the VPCs.
- Each resolved FNN profile, after applying its VPC overrides, has
`tenant_prefix_overlap_eligible = true` and `internal = true`; has no import
or export route targets; disables default-route leakage, tenant-host-route
leakage, and tenant leak communities; and has no accepted underlay leaks or

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why care about tenant leak communities? These should only be used to control whether we honor communities that the tenant attached to routes it advertises to the DPU. 🤔

allowed anycast prefixes.

The gRPC `CreateNetworkSegment` and `AttachNetworkSegmentToVpc` handlers reject
any direct prefix that overlaps a `VpcPrefix`, regardless of the site gate. The
gRPC `CreateVpcPrefix` handler considers prefixes on attached segments. It may
adopt only direct Tenant segment prefixes in the same VPC that are not already
linked to a `VpcPrefix`; every other direct `NetworkPrefix` overlap on an
attached segment is rejected. An unattached `CreateNetworkSegment` request does
not run these checks, but a later attachment does.

These handlers do not validate changes to peering or VPC policy, or Instance
paths that retain routing state. They also do not cover startup or audit every
writer. Those checks are tracked in
[#5114](https://github.com/NVIDIA/infra-controller/issues/5114) and
[#5115](https://github.com/NVIDIA/infra-controller/issues/5115), while startup
and complete writer coverage are tracked in
[#5116](https://github.com/NVIDIA/infra-controller/issues/5116). All three must
land before the database cutover in
[#3892](https://github.com/NVIDIA/infra-controller/issues/3892).

Even when the application accepts an eligible pair, the existing `VpcPrefix`
exclusion rejects overlapping `VpcPrefix` persistence until the cutover tracked
by [#3892](https://github.com/NVIDIA/infra-controller/issues/3892).

### `VpcDefinition`

| Field | Type | Default | Description |
Expand Down
35 changes: 20 additions & 15 deletions crates/api-core/src/cfg/file.rs
Original file line number Diff line number Diff line change
Expand Up @@ -231,11 +231,11 @@ pub struct CarbideConfig {
#[serde(default)]
pub site_fabric_prefixes: Vec<IpNetwork>,

/// Opts this site into tenant prefix overlap admission.
/// Opts this site into exact tenant VpcPrefix reuse checks.
///
/// Defaults to `false`. Participating FNN base profiles must separately
/// set `tenant_prefix_overlap_eligible`, and configuration alone does not permit
/// duplicate prefix persistence while database exclusions remain active.
/// Defaults to `false`. The complete eligibility, rejection, and database
/// fallback contract is documented under "Tenant prefix overlap checks"
/// in `crates/api-core/src/cfg/README.md`.
#[serde(default)]
pub tenant_prefix_overlap_enabled: bool,

Expand Down Expand Up @@ -2793,12 +2793,12 @@ pub struct FnnRoutingProfileConfig {
#[serde(default)]
pub internal: Option<bool>,

/// Opts VPCs based on this profile into future tenant prefix overlap admission.
/// Opts VPCs based on this profile into exact tenant VpcPrefix reuse checks.
///
/// This base-profile setting defaults to `false` and cannot be overridden
/// by a VPC. Admission support is tracked by
/// <https://github.com/NVIDIA/infra-controller/issues/3890>; this value
/// alone changes neither routing nor prefix persistence.
/// by a VPC. The complete eligibility, rejection, and database fallback
/// contract is documented under "Tenant prefix overlap checks" in
/// `crates/api-core/src/cfg/README.md`.
#[serde(default)]
pub tenant_prefix_overlap_eligible: bool,

Expand Down Expand Up @@ -2845,22 +2845,27 @@ pub struct FnnRoutingProfileConfig {
}

impl FnnRoutingProfileConfig {
/// Returns whether this resolved profile satisfies the profile-local overlap policy.
/// `is_eligible_for_tenant_prefix_overlap` returns whether the resolved
/// profile meets every profile condition for exact prefix reuse.
///
/// Evaluate the profile returned by [`FnnConfig::resolve_vpc_routing_profile`],
/// not the raw base profile, so VPC overrides participate in the decision.
/// This check cannot see site-wide route targets, additional FNN imports,
/// VPC peering, or retained routing state. Callers must reject those paths
/// between overlapping VPCs and separately require the site gate and
/// site-wide `vpc_isolation_behavior = "mutual_isolation"`.
#[allow(dead_code)] // Staged for https://github.com/NVIDIA/infra-controller/issues/3890.
/// The caller adds the site-wide conditions for these prefix writers.
/// Peering and VPC policy changes are tracked in
/// <https://github.com/NVIDIA/infra-controller/issues/5114>, and retained
/// Instance paths are tracked in
/// <https://github.com/NVIDIA/infra-controller/issues/5115>. Startup and
/// complete writer coverage are tracked in
/// <https://github.com/NVIDIA/infra-controller/issues/5116>. All three must
/// land before the database cutover in
/// <https://github.com/NVIDIA/infra-controller/issues/3892>.
pub(crate) fn is_eligible_for_tenant_prefix_overlap(&self) -> bool {
// Keep this exhaustive so new profile fields require an explicit eligibility decision.
let Self {
tenant_prefix_overlap_eligible,
route_target_imports,
route_targets_on_exports,
// External profiles are outside the initial overlap-admission scope.
// External profiles cannot participate in exact prefix reuse.
internal,
leak_default_route_from_underlay,
leak_tenant_host_routes_to_underlay,
Expand Down
1 change: 1 addition & 0 deletions crates/api-core/src/handlers/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -90,6 +90,7 @@ pub(super) mod switch;
pub(super) mod tenant;
pub(super) mod tenant_identity_config;
pub(super) mod tenant_keyset;
mod tenant_prefix_overlap;
pub(super) mod tpm_ca;
pub(super) mod uefi;
pub(super) mod uefi_credential_rotation;
Expand Down
45 changes: 43 additions & 2 deletions crates/api-core/src/handlers/network_segment.rs
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@
use ::rpc::forge as rpc;
use db::resource_pool::ResourcePoolDatabaseError;
use db::{AnnotatedSqlxError, DatabaseError, ObjectColumnFilter, network_segment};
use ipnetwork::IpNetwork;
use model::network_segment::{
NetworkSegment, NetworkSegmentControllerState, NetworkSegmentSearchConfig, NetworkSegmentType,
NewNetworkSegment,
Expand All @@ -25,8 +26,8 @@ use model::vpc::VpcVirtualizationTypeCapabilities;
use sqlx::{PgConnection, PgTransaction};
use tonic::{Request, Response, Status};

use crate::CarbideError;
use crate::api::{Api, log_request_data};
use crate::{CarbideError, CarbideResult};

pub(crate) async fn find_ids(
api: &Api,
Expand Down Expand Up @@ -87,6 +88,24 @@ pub(crate) async fn find_by_ids(
}))
}

/// `reject_vpc_prefix_overlaps` rejects direct `NetworkPrefix` records that
/// overlap a `VpcPrefix`.
///
/// The caller holds the overlap transaction lock from this probe through the
/// `NetworkSegment` write, so another participating `VpcPrefix` request cannot
/// commit between them.
async fn reject_vpc_prefix_overlaps(
txn: &mut PgConnection,
prefixes: &[IpNetwork],
) -> CarbideResult<()> {
for prefix in prefixes {
if !db::vpc_prefix::probe(*prefix, &mut *txn).await?.is_empty() {
return Err(super::tenant_prefix_overlap::overlap_error());
}
}
Ok(())
}

pub(crate) async fn create(
api: &Api,
request: Request<rpc::NetworkSegmentCreationRequest>,
Expand Down Expand Up @@ -131,6 +150,9 @@ pub(crate) async fn create(
}

let mut txn = api.txn_begin().await?;
if new_network_segment.vpc_id.is_some() {
db::tenant_prefix_overlap::lock_checks(txn.as_mut()).await?;
}
Comment thread
coderabbitai[bot] marked this conversation as resolved.

let allocate_svi_ip = if let Some(vpc_id) = new_network_segment.vpc_id {
let vpcs = db::vpc::find_by(
Expand All @@ -156,6 +178,15 @@ pub(crate) async fn create(
false
};

if new_network_segment.vpc_id.is_some() {
let prefixes = new_network_segment
.prefixes
.iter()
.map(|prefix| prefix.prefix)
.collect::<Vec<_>>();
reject_vpc_prefix_overlaps(&mut txn, &prefixes).await?;
}

let network_segment = save(api, &mut txn, new_network_segment, false, allocate_svi_ip).await?;

txn.commit().await?;
Expand All @@ -181,6 +212,7 @@ pub(crate) async fn attach_to_vpc(
let vpc_id = vpc_id.ok_or(CarbideError::MissingArgument("vpc_id"))?;

let mut txn = api.txn_begin().await?;
db::tenant_prefix_overlap::lock_checks(txn.as_mut()).await?;

let vpcs = db::vpc::find_by_with_lock(
txn.as_mut(),
Expand Down Expand Up @@ -228,7 +260,16 @@ pub(crate) async fn attach_to_vpc(
))
.into());
}
_ => db::network_segment::attach_to_vpc(&segment, txn.as_mut(), vpc_id).await?,
_ => {
let prefixes = segment
.prefixes
.iter()
.filter(|prefix| prefix.vpc_prefix_id.is_none())
.map(|prefix| prefix.prefix)
.collect::<Vec<_>>();
reject_vpc_prefix_overlaps(&mut txn, &prefixes).await?;
db::network_segment::attach_to_vpc(&segment, txn.as_mut(), vpc_id).await?
}
};

txn.commit().await?;
Expand Down
Loading
Loading