Add node pool limits to ENT resource quotas

[henrikjohansen]

Currently, Nomad ENT customers can only define resource quotas per region - which is rather coarse grained.
Since the introduction of node pools in v1.6 we now have a proper way to group resources - this really should be reflected in the resource quota spec :)

It should be possible to add limits (cpu / cores / memory / memory_max / devices) for one or more node pools under a region limit - the combined resources of those node pool limits should not be able to exceed the given region limit.

Node pools are a great addition to Nomad - being able to guard them using both namespace node pool configuration and resource quotas would make them even better for ENT customers ...

[tgross]

Hi @henrikjohansen! The intended interaction between node pools and quotas was that you'd attach quotas to namespaces and then limit those namespaces to the node pool via the Node Pool Governance features.

For example, suppose you have a prod and dev node pool, and namespaces devA and devB, and various prod namespaces. You could define a dev quota, and then use the following namespace spec (with an equivalent spec for "devB"):

name = "devA"
node_pool_config {
  default = "dev"
  allowed = ["dev"]
}

Define that "devA" namespace with the quota nomad namespace -quota dev devA ./devA.spec.hcl. That effectively ends up with a quota that applies to only the dev node pool, but lets you have more fine-grained control by having namespaces that don't have the quota (ex. a `"system" namespace that can deploy to any node pool).

[henrikjohansen]

Hi @tgross. Sure - that model works fine as long as you have a 1:1 relationship between namespaces and node pools. How about a situation where you have multiple node pools you want to grant access to AND govern using resource quotas?

name = "devA"
node_pool_config {
  default = "dedicated_pool_for_dev_a"
  allowed = ["dedicated_pool_for_dev_a", "some_shared_pool"]
}

The devA namespace has access to a dedicated pool but also needs access to some shared ressources. Given the 1:1 model I cannot control how much of some_shared_pool can be allocated by the devA namespace?

Or perhaps something like this :

name = "devA"
node_pool_config {
  default = "cheap_compute_nodes"
  allowed = ["cheap_compute_nodes", "expensive_compute_nodes"]
}

In this case I would like to ensure that the devA namespace can consume 5000 cores in the cheap_compute_nodes node pool but only 1500 in the expensive_compute_nodes pool.

[tgross]

Sure - that model works fine as long as you have a 1:1 relationship between namespaces and node pools

Or an N:1 relationship, but point taken.

Thinking a bit about how we might model this in the quota spec... node pools are replicated objects (like namepsaces, ACLs, and quotas) so they're duplicated across regions.

We could implement this so that we have a new top-level limit block for node pools that's a peer of the region block. That would allow setting a quota that applies to the node pool no matter what region it's in (but still tracked separately within each region) without having to set an overall quota for the region. For example:

name        = "myquota"
description = "Something something"

limit {
  region = "global"
  region_limit {
    cores      = 0
    cpu        = 2500
    memory     = 1000
    memory_max = 1000
  }
}

limit {
  node_pool = "prod"
  node_pool_limit {
    cores      = 0
    cpu        = 2500
    memory     = 1000
    memory_max = 1000
  }
}

or we could embed the node pool limits inside the region limits, which would give more fine-grained control at the cost of having to add node pool limits individually for each region.

name        = "myquota"
description = "Something something"

limit {
  region = "global"
  region_limit {
    cores      = 0
    cpu        = 2500
    memory     = 1000
    memory_max = 1000

    node_pool "prod" {
      cores      = 0
      cpu        = 2500
      memory     = 1000
      memory_max = 1000
    } 
  }
}

Something for whomever does the implementation to think about. I'll mark this for roadmapping.