Balloons Policy

Overview

The balloons policy implements workload placement into “balloons” that are disjoint CPU pools. Balloons can be inflated and deflated, that is CPUs added and removed, based on the CPU resource requests of containers. Balloons can be static or dynamically created and destroyed. CPUs in balloons can be configured, for example, by setting min and max frequencies on CPU cores and uncore.

How It Works

  1. User configures balloon types from which the policy instantiates balloons.

  2. A balloon has a set of CPUs and a set of containers that run on the CPUs.

  3. Every container is assigned to exactly one balloon. A container is allowed to use all CPUs of its balloon and no other CPUs.

  4. Every logical CPU belongs to at most one balloon. There can be CPUs that do not belong to any balloon.

  5. The number of CPUs in a balloon can change during the lifetime of the balloon. If a balloon inflates, that is CPUs are added to it, all containers in the balloon are allowed to use more CPUs. If a balloon deflates, the opposite is true.

  6. When a new container is created on a Kubernetes node, the policy first decides the type of the balloon that will run the container. The decision is based on annotations of the pod, or the namespace if annotations are not given.

  7. Next the policy decides which balloon of the decided type will run the container. Options are:

    • an existing balloon that already has enough CPUs to run its current and new containers

    • an existing balloon that can be inflated to fit its current and new containers

    • new balloon.

  8. When a CPU is added to a balloon or removed from it, the CPU is reconfigured based on balloon’s CPU class attributes, or idle CPU class attributes.

Deployment

Install cri-resmgr

Deploy cri-resmgr on each node as you would for any other policy. See installation for more details.

Configuration

The balloons policy is configured using the yaml-based configuration system of CRI-RM. See setup and usage for more details on managing the configuration.

Parameters

Balloons policy parameters:

  • PinCPU controls pinning a container to CPUs of its balloon. The default is true: the container cannot use other CPUs.

  • PinMemory controls pinning a container to the memories that are closest to the CPUs of its balloon. Pinning memory disallows using memory from other NUMA nodes.

  • IdleCPUClass specifies the CPU class of those CPUs that do not belong to any balloon.

  • ReservedPoolNamespaces is a list of namespaces (wildcards allowed) that are assigned to the special reserved balloon, that is, will run on reserved CPUs. This always includes the kube-system namespace.

  • BalloonTypes is a list of balloon type definitions. Each type can be configured with the following parameters:

    • Name of the balloon type. This is used in pod annotations to assign containers to balloons of this type.

    • Namespaces is a list of namespaces (wildcards allowed) whose pods should be assigned to this balloon type, unless overridden by pod annotations.

    • MinBalloons is the minimum number of balloons of this type that is always present, even if the balloons would not have any containers. The default is 0: if a balloon has no containers, it can be destroyed.

    • MaxCPUs specifies the maximum number of CPUs in any balloon of this type. Balloons will not be inflated larger than this. 0 means unlimited.

    • MinCPUs specifies the minimum number of CPUs in any balloon of this type. When a balloon is created or deflated, it will always have at least this many CPUs, even if containers in the balloon request less.

    • CpuClass specifies the name of the CPU class according to which CPUs of balloons are configured.

    • PreferSpreadingPods: if true, containers of the same pod should be spread to different balloons of this type. The default is false: prefer placing containers of the same pod to the same balloon(s).

    • PreferPerNamespaceBalloon: if true, containers in the same namespace will be placed in the same balloon(s). On the other hand, containers in different namespaces are preferrably placed in different balloons. The default is false: namespace has no effect on choosing the balloon of this type.

    • PreferNewBalloons: if true, prefer creating new balloons over placing containers to existing balloons. This results in preferring exclusive CPUs, as long as there are enough free CPUs. The default is false: prefer filling and inflating existing balloons over creating new ones.

    • AllocatorPriority (0: High, 1: Normal, 2: Low, 3: None). CPU allocator parameter, used when creating new or resizing existing balloons. If there are balloon types with pre-created balloons (MinBalloons > 0), balloons of the type with the highest AllocatorPriority are created first.

Related configuration parameters:

  • policy.ReservedResources.CPU specifies the (number of) CPUs in the special reserved balloon. By default all containers in the kube-system namespace are assigned to the reserved balloon.

  • cpu.classes defines CPU classes and their parameters (such as minFreq, maxFreq, uncoreMinFreq and uncoreMaxFreq).

Example

Example configuration that runs all pods in balloons of 1-4 CPUs.

policy:
  Active: balloons
  ReservedResources:
    CPU: 1
  balloons:
    PinCPU: true
    PinMemory: true
    IdleCPUClass: lowpower
    BalloonTypes:
      - Name: "quad"
        MinCpus: 1
        MaxCPUs: 4
        CPUClass: dynamic
        Namespaces:
          - "*"
cpu:
  classes:
    lowpower:
      minFreq: 800
      maxFreq: 800
    dynamic:
      minFreq: 800
      maxFreq: 3600
    turbo:
      minFreq: 3000
      maxFreq: 3600
      uncoreMinFreq: 2000
      uncoreMaxFreq: 2400

See the sample configmap for a complete example.

Assigning a Container to a Balloon

The balloon type of a container can be defined in pod annotations. In the example below, the first annotation sets the balloon type (BT) of a single container (CONTAINER_NAME). The last two annotations set the default balloon type for all containers in the pod.

balloon.balloons.cri-resource-manager.intel.com/container.CONTAINER_NAME: BT
balloon.balloons.cri-resource-manager.intel.com/pod: BT
balloon.balloons.cri-resource-manager.intel.com: BT

If a pod has no annotations, its namespace is matched to the Namespaces of balloon types. The first matching balloon type is used.

If the namespace does not match, the container is assigned to the special default balloon, that means reserved CPUs unless MinCPUs or MaxCPUs of the default balloon type are explicitely defined in the BalloonTypes configuration.

Metrics and Debugging

In order to enable more verbose logging and metrics exporting from the balloons policy, enable instrumentation and policy debugging from the CRI-RM global config:

instrumentation:
  # The balloons policy exports containers running in each balloon,
  # and cpusets of balloons. Accessible in command line:
  # curl --silent http://localhost:8891/metrics
  HTTPEndpoint: :8891
  PrometheusExport: true
logger:
  Debug: policy