Balloons Policy

Overview

The balloons policy implements workload placement into “balloons” that are disjoint CPU pools. Balloons can be inflated and deflated, that is CPUs added and removed, based on the CPU resource requests of containers. Balloons can be static or dynamically created and destroyed. CPUs in balloons can be configured, for example, by setting min and max frequencies on CPU cores and uncore.

How It Works

  1. User configures balloon types from which the policy instantiates balloons.

  2. A balloon has a set of CPUs and a set of containers that run on the CPUs.

  3. Every container is assigned to exactly one balloon. A container is allowed to use all CPUs of its balloon and no other CPUs.

  4. Every logical CPU belongs to at most one balloon. There can be CPUs that do not belong to any balloon.

  5. The number of CPUs in a balloon can change during the lifetime of the balloon. If a balloon inflates, that is CPUs are added to it, all containers in the balloon are allowed to use more CPUs. If a balloon deflates, the opposite is true.

  6. When a new container is created on a Kubernetes node, the policy first decides the type of the balloon that will run the container. The decision is based on annotations of the pod, or the namespace if annotations are not given.

  7. Next the policy decides which balloon of the decided type will run the container. Options are:

    • an existing balloon that already has enough CPUs to run its current and new containers

    • an existing balloon that can be inflated to fit its current and new containers

    • new balloon.

  8. When a CPU is added to a balloon or removed from it, the CPU is reconfigured based on balloon’s CPU class attributes, or idle CPU class attributes.

Deployment

Install cri-resmgr

Deploy cri-resmgr on each node as you would for any other policy. See installation for more details.

Configuration

The balloons policy is configured using the yaml-based configuration system of CRI-RM. See setup and usage for more details on managing the configuration.

Parameters

Balloons policy parameters:

  • PinCPU controls pinning a container to CPUs of its balloon. The default is true: the container cannot use other CPUs.

  • PinMemory controls pinning a container to the memories that are closest to the CPUs of its balloon. The default is true: allow using memory only from the closest NUMA nodes. Warning: this may cause kernel to kill workloads due to out-of-memory error when closest NUMA nodes do not have enough memory. In this situation consider switching this option false.

  • IdleCPUClass specifies the CPU class of those CPUs that do not belong to any balloon.

  • ReservedPoolNamespaces is a list of namespaces (wildcards allowed) that are assigned to the special reserved balloon, that is, will run on reserved CPUs. This always includes the kube-system namespace.

  • AllocatorTopologyBalancing affects selecting CPUs for new balloons. If true, new balloons are created using CPUs on NUMA/die/package with most free CPUs, that is, balloons are spread across the hardware topology. This helps inflating balloons within the same NUMA/die/package and reduces interference between workloads in balloons when system is not fully loaded. The default is false: pack new balloons tightly into the same NUMAs/dies/packages. This helps keeping large portions of hardware idle and entering into deep power saving states.

  • PreferSpreadOnPhysicalCores prefers allocating logical CPUs (possibly hyperthreads) for a balloon from separate physical CPU cores. This prevents workloads in the balloon from interfering with themselves as they do not compete on the resources of the same CPU cores. On the other hand, it allows more interference between workloads in different balloons. The default is false: balloons are packed tightly to a minimum number of physical CPU cores. The value set here is the default for all balloon types, but it can be overridden with the balloon type specific setting with the same name.

  • BalloonTypes is a list of balloon type definitions. Each type can be configured with the following parameters:

    • Name of the balloon type. This is used in pod annotations to assign containers to balloons of this type.

    • Namespaces is a list of namespaces (wildcards allowed) whose pods should be assigned to this balloon type, unless overridden by pod annotations.

    • MinBalloons is the minimum number of balloons of this type that is always present, even if the balloons would not have any containers. The default is 0: if a balloon has no containers, it can be destroyed.

    • MaxBalloons is the maximum number of balloons of this type that is allowed to co-exist. The default is 0: creating new balloons is not limited by the number of existing balloons.

    • MaxCPUs specifies the maximum number of CPUs in any balloon of this type. Balloons will not be inflated larger than this. 0 means unlimited.

    • MinCPUs specifies the minimum number of CPUs in any balloon of this type. When a balloon is created or deflated, it will always have at least this many CPUs, even if containers in the balloon request less.

    • CpuClass specifies the name of the CPU class according to which CPUs of balloons are configured.

    • PreferSpreadingPods: if true, containers of the same pod should be spread to different balloons of this type. The default is false: prefer placing containers of the same pod to the same balloon(s).

    • PreferPerNamespaceBalloon: if true, containers in the same namespace will be placed in the same balloon(s). On the other hand, containers in different namespaces are preferrably placed in different balloons. The default is false: namespace has no effect on choosing the balloon of this type.

    • PreferNewBalloons: if true, prefer creating new balloons over placing containers to existing balloons. This results in preferring exclusive CPUs, as long as there are enough free CPUs. The default is false: prefer filling and inflating existing balloons over creating new ones.

    • ShareIdleCPUsInSame: Whenever the number of or sizes of balloons change, idle CPUs (that do not belong to any balloon) are reshared as extra CPUs to workloads in balloons with this option. The value sets locality of allowed extra CPUs that will be common to these workloads.

      • system: workloads are allowed to use idle CPUs available anywhere in the system.

      • package: …allowed to use idle CPUs in the same package(s) (sockets) as the balloon.

      • die: …in the same die(s) as the balloon.

      • numa: …in the same numa node(s) as the balloon.

      • core: …allowed to use idle CPU threads in the same cores with the balloon.

    • PreferSpreadOnPhysicalCores overrides the policy level option with the same name in the scope of this balloon type.

    • AllocatorPriority (0: High, 1: Normal, 2: Low, 3: None). CPU allocator parameter, used when creating new or resizing existing balloons. If there are balloon types with pre-created balloons (MinBalloons > 0), balloons of the type with the highest AllocatorPriority are created first.

Related configuration parameters:

  • policy.ReservedResources.CPU specifies the (number of) CPUs in the special reserved balloon. By default all containers in the kube-system namespace are assigned to the reserved balloon.

  • cpu.classes defines CPU classes and their parameters (such as minFreq, maxFreq, uncoreMinFreq and uncoreMaxFreq).

Example

Example configuration that runs all pods in balloons of 1-4 CPUs.

policy:
  Active: balloons
  ReservedResources:
    CPU: 1
  balloons:
    PinCPU: true
    PinMemory: true
    IdleCPUClass: lowpower
    BalloonTypes:
      - Name: "quad"
        MinCpus: 1
        MaxCPUs: 4
        CPUClass: dynamic
        Namespaces:
          - "*"
cpu:
  classes:
    lowpower:
      minFreq: 800
      maxFreq: 800
    dynamic:
      minFreq: 800
      maxFreq: 3600
    turbo:
      minFreq: 3000
      maxFreq: 3600
      uncoreMinFreq: 2000
      uncoreMaxFreq: 2400

See the sample configmap for a complete example.

Assigning a Container to a Balloon

The balloon type of a container can be defined in pod annotations. In the example below, the first annotation sets the balloon type (BT) of a single container (CONTAINER_NAME). The last two annotations set the default balloon type for all containers in the pod.

balloon.balloons.cri-resource-manager.intel.com/container.CONTAINER_NAME: BT
balloon.balloons.cri-resource-manager.intel.com/pod: BT
balloon.balloons.cri-resource-manager.intel.com: BT

If a pod has no annotations, its namespace is matched to the Namespaces of balloon types. The first matching balloon type is used.

If the namespace does not match, the container is assigned to the special default balloon, that means reserved CPUs unless MinCPUs or MaxCPUs of the default balloon type are explicitely defined in the BalloonTypes configuration.

Metrics and Debugging

In order to enable more verbose logging and metrics exporting from the balloons policy, enable instrumentation and policy debugging from the CRI-RM global config:

instrumentation:
  # The balloons policy exports containers running in each balloon,
  # and cpusets of balloons. Accessible in command line:
  # curl --silent http://localhost:8891/metrics
  HTTPEndpoint: :8891
  PrometheusExport: true
logger:
  Debug: policy