Trusted Execution Cluster Operator (trusted-cluster-operator)

This repository contains a Kubernetes operator for managing Trusted Execution Clusters. The operator introduces a TrustedExecutionCluster Custom Resource Definition (CRD) which allows users to declaratively manage the configuration of a trusted execution cluster and Trustee server, a core component which handles the attestation process.

The operator watches for TrustedExecutionCluster resources and ensures that the necessary configurations for the Trustee (such as KBS configuration, attestation policies, and resource policies) are correctly set up and maintained within the cluster.

Repository Structure

• /api: Defines the TrustedExecutionCluster Custom Resource Definition (CRD) and associated CRDs and RBAC definitions in Go. Also contains a program to generate a TrustedExecutionCluster CR and associated deployment.
• /operator: Contains the source code for the Kubernetes operator itself.
• /register-server: A server that provides Clevis PINs for key retrieval with random UUIDs.
• /compute-pcrs: A program to compute PCR reference values using the compute-pcrs library and insert them into a ConfigMap, run as a Job.
• /lib: Shared Rust definitions, including translated CRDs
• /scripts: Helper scripts for managing a local kind development cluster.
• /config: The default output directory for generated manifests. This directory is not checked into source control.

Getting Started

Prerequisites

• Rust toolchain
• podman or docker (set CONTAINER_CLI and RUNTIME environment variables accordingly)
• kubectl
• kind
• operator-sdk (refer to its official documentation for installation instructions)

Quick Start

Create the cluster and deploy the operator.

Provide an address where the VM you will attest from can access the cluster. When using a local kind & libvirt VM, this may be your gateway address (default via … in ip route) for user libvirt or bridge (virbr0 in ip route) for system libvirt.

$ ip route
...
192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1
...
$ ip=192.168.122.1

To use Docker:

export CONTAINER_CLI=docker
export RUNTIME=docker

To use Podman (these exports can be omitted as Podman is the default):

export CONTAINER_CLI=podman
export RUNTIME=podman

Then run the following commands:

make cluster-up
make REGISTRY=localhost:5000 PUSH_FLAGS="--tls-verify=false" push # optional: use BUILD_TYPE=debug
make REGISTRY=localhost:5000 manifests
make TRUSTEE_ADDR=$ip install

The KBS port will be forwarded to 8080 on your machine; the node register server to 8000, where new Ignition configs are served at /register.

Test

Run a VM as described in the investigations repository.

OLM Bundle Workflow

This operator can be packaged and deployed as an OLM bundle. This workflow supports deploying to any Kubernetes cluster with access to a container registry.

1. Prerequisites

• Setup Cluster: Ensure your kubectl context points to your target cluster. For local development, you can create a kind cluster by running:

  # Set RUNTIME=docker if using Docker instead of Podman.
  make cluster-up

• Login to Registry:

  # Login to your remote container registry (e.g., quay.io)
  docker login quay.io

• Install OLM:

  # Install OLM on your target cluster
  (cd /tmp && operator-sdk olm install)

2. Set Environment Variables

Define the container registry, image tag, and CLI.

export REGISTRY=quay.io/<your-username>
export TAG=0.1.0
export CONTAINER_CLI=docker # or podman

Note: The TAG must be a valid semantic version (e.g., 0.1.0). OLM does not support tags like latest for bundle versions.

3. Build, Validate, and Push

The push-all target builds all operator images, generates the bundle, builds the bundle image, and pushes everything to the specified $REGISTRY.

make push-all

You can optionally validate the generated bundle manifests at any time after the bundle has been generated:

(cd ./bundle && operator-sdk bundle validate .)

4. Deploy the Bundle

Deploy the bundle to your cluster. We use trusted-execution-clusters as an example namespace.

kubectl create namespace trusted-execution-clusters || true
(cd /tmp && operator-sdk run bundle ${REGISTRY}/trusted-cluster-operator-bundle:${TAG} --namespace trusted-execution-clusters)

5. Create the Custom Resource

Once the operator is running, you need to create a TrustedExecutionCluster custom resource to make it functional.

First, you must update the example CR with the correct public address for the Trustee service, which must be accessible from your worker nodes or VMs.

# Provide an address where your VMs can access the cluster.
# When using a local kind cluster, this is often the kind bridge IP.
$ ip route
...
192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1
...
$ export TRUSTEE_ADDR=192.168.122.1

# Use yq (or manually edit) to set the address in the CR.
# Note: yq is installed via 'make build-tools'.
$ yq -i '.spec.publicTrusteeAddr = "'$TRUSTEE_ADDR':8080"' config/deploy/trusted_execution_cluster_cr.yaml

# Now, apply the configured CR
$ kubectl apply -f config/deploy/trusted_execution_cluster_cr.yaml

Cleaning Up the Bundle Deployment

To remove all resources deployed by the bundle, use the cleanup command with the operator's package name:

(cd /tmp && operator-sdk cleanup trusted-cluster-operator --namespace trusted-execution-clusters)

Quick Start Cleanup

To clean up your environment after running the non-OLM Quick Start method, execute the following commands:

make cluster-cleanup
# Note: You must use the same RUNTIME environment variable for `cluster-down`
# that you used for `cluster-up`. For example:
#
# RUNTIME=docker make cluster-down
make cluster-down
make clean

Licenses

See LICENSES.