NVIDIA tried to address this with Multi-Instance GPU (MIG) technology, which can physically partition an A100 or H100 into up to seven isolated instances. But the Kubernetes scheduler didn't understand MIG. You could partition the hardware, but getting workloads onto those partitions required awkward workarounds and custom tooling.

That changed with Kubernetes 1.34, where Dynamic Resource Allocation (DRA) went GA. DRA replaces the "give me one GPU" model with structured, attribute-based requests. You can now ask for "a 10GB MIG partition with at least 1/7th compute" and the scheduler knows what to do with that request. Device plugins just count GPUs. DRA actually knows what's inside them.

This post walks through building GPU infrastructure on Amazon EKS with Pulumi that puts DRA and MIG to work. We'll set up a cluster with GPU node groups that partition A100 GPUs into different-sized slices, then deploy mixed workloads (training jobs and inference services) that share a single GPU with full hardware isolation. Along the way, we'll cover real-world implementation challenges including GPU compatibility, Spot capacity constraints, AL2023 AMI configuration, and MIG strategy pitfalls.

How DRA works

The problem with device plugins

The device plugin API was added to Kubernetes in version 1.8 to handle hardware like GPUs. It works, but it has a fundamental limitation: devices are opaque integers. When you write this:

resources:
  limits:
    nvidia.com/gpu: 1

Kubernetes sees "1" and nothing else. It doesn't know what kind of GPU, how much memory it has, whether it supports MIG, or what interconnects it has. The scheduler counts GPUs the same way it counts CPU millicores - as fungible units. This works fine when all your GPUs are identical and you always need a whole one. It falls apart when you have mixed GPU types, need partial GPUs, or want topology-aware placement.

Device plugins also can't express relationships between devices. If you need two GPUs connected via NVLink for a training job, there's no way to say that. You get two GPUs, and you hope they're connected.

How DRA changes the model

DRA flips the model. Instead of counting GPUs like CPU cores, the scheduler now sees what each device actually is. A DRA driver publishes memory size, compute capability, MIG profiles, interconnect topology. The scheduler matches workloads to devices based on real requirements, not just "give me one."

Here's what the components look like:

ResourceSlice - The DRA driver on each node publishes ResourceSlices to advertise what's available. Here's what one looks like from a running cluster with MIG-partitioned A100s:

apiVersion: resource.k8s.io/v1
kind: ResourceSlice
metadata:
  name: ip-10-0-170-169.ec2.internal-gpu.nvidia.com-qbgxv
  ownerReferences:
  - apiVersion: v1
    kind: Node
    name: ip-10-0-170-169.ec2.internal
spec:
  devices:
  - name: gpu-0-mig-19-2-1
    attributes:
      architecture:
        string: Ampere
      brand:
        string: Nvidia
      cudaComputeCapability:
        version: 8.0.0
      cudaDriverVersion:
        version: 13.0.0
      driverVersion:
        version: 580.126.9
      productName:
        string: NVIDIA A100-SXM4-40GB
      profile:
        string: 1g.5gb
      type:
        string: mig
      uuid:
        string: MIG-53146931-c7ac-5a72-878e-15adb49f83ff
    capacity:
      memory:
        value: 4864Mi
      multiprocessors:
        value: "14"

The scheduler sees everything: it's an Ampere A100-SXM4-40GB, partitioned as a 1g.5gb MIG instance with 4864Mi memory and 14 multiprocessors. The profile attribute is what you'll match in CEL expressions.

DeviceClass - A DeviceClass is like a StorageClass but for devices. It defines a category of hardware that workloads can request. Cluster admins create DeviceClasses to expose different tiers of resources: "give me any GPU" vs "give me a small MIG partition" vs "give me a full A100 for training."

The NVIDIA DRA driver creates a gpu.nvidia.com DeviceClass automatically:

apiVersion: resource.k8s.io/v1
kind: DeviceClass
metadata:
  name: gpu.nvidia.com
spec:
  selectors:
  - cel:
      expression: device.driver == 'gpu.nvidia.com' && device.attributes['gpu.nvidia.com'].type == 'gpu'

The selectors field uses CEL expressions to define which devices from ResourceSlices are available through this class. When a ResourceClaim references a DeviceClass, Kubernetes only considers devices that match the class's selectors. The claim can add more selectors to narrow things down further, but it can't escape the class's constraints.

This is where platform teams set guardrails. Create a mig-small.nvidia.com class that only matches 1g.5gb profiles, and users requesting from that class can't accidentally (or intentionally) grab a full GPU.

ResourceClaim - Workloads request devices through ResourceClaims. Unlike the device plugin model where you ask for a count, claims use CEL (Common Expression Language) to express requirements:

apiVersion: resource.k8s.io/v1
kind: ResourceClaim
metadata:
  name: my-inference-gpu
spec:
  devices:
    requests:
    - name: gpu
      exactly:
        deviceClassName: mig.nvidia.com
        count: 1
        selectors:
        - cel:
            expression: |
              device.attributes["gpu.nvidia.com"].type == "mig" &&
              device.attributes["gpu.nvidia.com"].profile == "1g.5gb"

The exactly block specifies that the claim needs exactly 1 device matching the criteria. The CEL expression filters for MIG devices with the 1g.5gb profile. You can write more complex requirements: "a GPU with Ampere architecture" or "two GPUs on the same PCIe root."

A ResourceClaim exists independently of any pod. You create it once, and multiple pods can reference the same claim to share access to the allocated device. Think shared inference servers where several client pods hit the same GPU. The lifecycle is manual: you create the claim, pods use it, you delete it when done.

ResourceClaimTemplate - For Deployments and StatefulSets, you typically use templates so each pod gets its own claim:

apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
  name: inference-gpu-template
spec:
  spec:
    devices:
      requests:
      - name: gpu
        exactly:
          deviceClassName: mig.nvidia.com
          count: 1
          selectors:
          - cel:
              expression: |
                device.attributes["gpu.nvidia.com"].type == "mig" &&
                device.attributes["gpu.nvidia.com"].profile == "1g.5gb"

The CEL expression matches the type and profile attributes from the ResourceSlice. Checking for type == "mig" ensures you get a MIG partition, not a full GPU.

Unlike a ResourceClaim, a ResourceClaimTemplate doesn't allocate anything by itself. When a pod references the template, Kubernetes generates a new ResourceClaim for that pod. Each replica in a Deployment gets its own claim, its own device. When the pod terminates, Kubernetes cleans up the generated claim.

The short version: ResourceClaim = shared device, manual lifecycle. ResourceClaimTemplate = dedicated device per pod, automatic lifecycle.

The allocation flow

When you deploy a pod that references a ResourceClaim, here's what happens:

1. Scheduling - The scheduler reads the claim's requirements and evaluates them against all ResourceSlices in the cluster. It finds nodes where matching devices are available. This is different from device plugins, where the scheduler just checks if nvidia.com/gpu >= 1.

2. Allocation - Once a node is selected, the scheduler marks specific devices as allocated in the ResourceClaim's status. The claim now has a binding to actual hardware.

3. Node preparation - The kubelet on the selected node calls the DRA driver to prepare the device. For GPUs, this might mean configuring MIG partitions, setting up container device interface (CDI) specs, or configuring CUDA visible devices.

4. Container startup - The container runs with the allocated device exposed according to the CDI spec. The workload sees exactly the GPU resources it requested.

5. Cleanup - When the pod terminates, the driver releases the allocation. The device goes back into the pool for other workloads.

Notice that allocation happens at scheduling time, not runtime. The scheduler knows exactly which device a pod will get before it even starts. No more race conditions when multiple pods compete for GPUs.

GPU sharing strategies

Before diving into MIG, it helps to understand what else is out there:

Time-slicing rapidly switches between processes on the same GPU. Good for dev environments, but there's no isolation. One workload can hog all GPU memory and crash everyone else.

MPS is CUDA-level sharing. Better utilization than time-slicing, but memory isolation is soft limits only. Fine for training jobs that play nice together.

MIG partitions the GPU at the hardware level. Each partition has its own compute units, memory, and memory bandwidth. If one partition crashes, the others keep running. This is what you want for production and multi-tenant setups.

MIG: actual hardware partitioning

MIG (Multi-Instance GPU) carves supported GPUs into up to seven isolated instances. The isolation runs deep: each instance gets its own processors with separate paths through the entire memory system, including dedicated crossbar ports, L2 cache banks, memory controllers, and DRAM address buses. One workload can't impact another's scheduling or performance. This hardware-level isolation is what makes MIG suitable for multi-tenant environments where you need guaranteed QoS.

How MIG partitions the GPU

MIG works by slicing the GPU into two types of resources that get combined to create isolated instances:

GPU Instances (GI) combine memory slices with compute slices to create partitioned GPU resources. A GPU Instance includes a fraction of the GPU's memory, streaming multiprocessors (SMs), and other execution engines. Each GI provides memory Quality of Service with dedicated memory bandwidth and cache allocation.

Compute Instances (CI) can further subdivide a GPU Instance's SM resources. A Compute Instance contains a subset of the parent GPU instance's SMs while sharing memory and engines. For most inference workloads, you'll use one CI per GI.

Take the A100-40GB: it divides into 8 memory slices (5GB each) and 7 SM slices. You combine these to create profiles, but there's a constraint: valid combinations can't overlap vertically in NVIDIA's profile layout diagram.

The diagram above shows how the A100 hardware divides. Each vertical column represents a memory slice, and the horizontal bands represent SM slices. When you create a MIG profile, you're selecting which slices to group together.

Understanding MIG profile names

MIG profiles follow a naming convention: <compute>g.<memory>gb

• The first number indicates the fraction of streaming multiprocessors (SMs) allocated

• The memory amount follows in gigabytes

• For example, 1g.5gb means 1/7th of the SMs and 5GB of memory

Profile suffixes modify capabilities:

• +me profiles include at least one media engine (NVDEC, NVENC, NVJPG, or OFA)

• +gfx adds graphics API support (Blackwell architecture only)

• +me.all allocates all available media engines

• -me excludes media engines for compute-only workloads

For inference workloads, you typically use the base profiles without suffixes since you need compute and memory but not video encoding.

Available MIG profiles

For an A100 40GB GPU (used in p4d.24xlarge instances), the available profiles are:

For an A100 80GB GPU, you get larger memory profiles:

• 1g.10gb - 7 instances available

• 2g.20gb - 3 instances available

• 3g.40gb - 2 instances available

• 4g.40gb - 1 instance available

• 7g.80gb - 1 instance available (full GPU)

For inference, the 1g.5gb profile (A100 40GB) or 1g.10gb (A100 80GB) works well for small models. One A100 40GB can run seven inference services at once, each with hardware isolation.

The diagram shows all valid profile combinations. Notice how larger profiles (like 3g.20gb and 4g.20gb) can coexist because they don't vertically overlap, but some combinations aren't allowed due to how the memory and compute slices align.

MIG isolation advantages

MIG provides superior isolation compared to CUDA Streams and Multi-Process Service (MPS). Here's what you get that other GPU sharing methods can't provide:

Each partition has dedicated compute units that can't be preempted by other partitions. Memory boundaries are hardware-enforced, so one partition can't access another's memory. Dedicated memory controllers ensure predictable bandwidth per partition. And if one MIG instance crashes, the others keep running.

This is what makes MIG work for production multi-tenant environments where you need guaranteed QoS and security isolation between workloads.

GPU architecture support

Here's the catch: not all NVIDIA GPUs support MIG. MIG requires compute capability 8.0 or higher, which means Ampere architecture and later. But even within Ampere, not every GPU has it.

Ampere (different max instances):

• A100-SXM4/PCIe (40GB or 80GB): up to 7 instances

• A30 (24GB): up to 4 instances

Hopper:

• H100-SXM5/PCIe (80GB or 94GB): up to 7 instances

• H200-SXM5/NVL (141GB): up to 7 instances

• H20 (96GB): up to 7 instances

Blackwell:

• B200 (180GB): up to 7 instances

• GB200 (186GB): up to 7 instances

The A10G (g5 instances) is Ampere but doesn't support MIG. This trips people up. For MIG on AWS, you need p4d (A100) or p5 (H100) instances.

What you need

The complete code for this example is available on GitHub:

Prerequisites:

• EKS v1.34+ (DRA went GA in Kubernetes 1.34)

• Pulumi CLI and AWS credentials

• Pulumi EKS Provider v3+

• NVIDIA GPU Operator v25.10.0+

• NVIDIA DRA Driver v25.8.0+

[!WARNING]
p4d.24xlarge capacity is scarce in most AWS regions. Spot looks cheap ($2.48/hr vs $32/hr On-Demand in us-east-1a) but good luck actually getting it. On-Demand is more reliable but expensive.

Building the infrastructure

The stack has a VPC, an EKS cluster with GPU node groups, and the NVIDIA components for DRA. We're using Pulumi AWSX for the VPC and the Pulumi EKS provider for the cluster. Let's walk through it.

VPC and cluster setup

First, create the VPC and EKS cluster:

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";
import * as awsx from "@pulumi/awsx";
import * as eks from "@pulumi/eks";
import * as k8s from "@pulumi/kubernetes";

const config = new pulumi.Config();
const clusterName = config.get("clusterName") || "gpu-dra-cluster";

// VPC with required EKS subnet tags
const vpc = new awsx.ec2.Vpc("gpu-vpc", {
    enableDnsHostnames: true,
    cidrBlock: "10.0.0.0/16",
    subnetSpecs: [
        {
            type: awsx.ec2.SubnetType.Public,
            tags: {
                [`kubernetes.io/cluster/${clusterName}`]: "shared",
                "kubernetes.io/role/elb": "1",
            },
        },
        {
            type: awsx.ec2.SubnetType.Private,
            tags: {
                [`kubernetes.io/cluster/${clusterName}`]: "shared",
                "kubernetes.io/role/internal-elb": "1",
            },
        },
    ],
    subnetStrategy: "Auto",
});

// EKS cluster without Auto Mode
// We'll use managed node groups for system workloads and GPU workloads
const cluster = new eks.Cluster("gpu-cluster", {
    name: clusterName,
    vpcId: vpc.vpcId,
    publicSubnetIds: vpc.publicSubnetIds,
    privateSubnetIds: vpc.privateSubnetIds,
    authenticationMode: eks.AuthenticationMode.Api,
    skipDefaultNodeGroup: true,
    version: "1.34",
});

// Kubernetes provider for deploying resources
const k8sProvider = new k8s.Provider("k8s-provider", {
    kubeconfig: cluster.kubeconfigJson,
    enableServerSideApply: true,
});

export const kubeconfig = pulumi.secret(cluster.kubeconfig);

Creating node groups

We'll create two node groups: one for system workloads and one for GPU workloads.

// IAM role for managed node groups
const nodeRole = new aws.iam.Role("system-node-role", {
    assumeRolePolicy: JSON.stringify({
        Version: "2012-10-17",
        Statement: [{
            Action: "sts:AssumeRole",
            Effect: "Allow",
            Principal: { Service: "ec2.amazonaws.com" },
        }],
    }),
});

// Attach required policies for EKS worker nodes
const nodeRolePolicies = [
    "arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy",
    "arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy",
    "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly",
];

nodeRolePolicies.forEach((policyArn, index) => {
    new aws.iam.RolePolicyAttachment(`system-node-policy-${index}`, {
        role: nodeRole.name,
        policyArn: policyArn,
    });
});

// System node group for control plane workloads
const systemNodeGroup = new eks.ManagedNodeGroup("system-nodes", {
    cluster: cluster,
    nodeGroupName: "system-nodes",
    nodeRole: nodeRole,
    instanceTypes: ["m6i.large"],
    capacityType: "ON_DEMAND",
    scalingConfig: {
        desiredSize: 2,
        minSize: 1,
        maxSize: 4,
    },
    labels: { "node-role": "system" },
    taints: [],
}, { dependsOn: [cluster] });

// GPU node group with p4d.24xlarge (A100 40GB GPUs)
// Note: SPOT capacity for p4d.24xlarge is extremely limited and often unavailable
// Using ON_DEMAND for more reliable (though expensive) capacity
const gpuNodeGroup = new eks.ManagedNodeGroup("gpu-nodes", {
    cluster: cluster,
    nodeGroupName: "gpu-nodes",
    nodeRole: nodeRole,
    instanceTypes: ["p4d.24xlarge"],
    capacityType: "ON_DEMAND",  // SPOT often fails with UnfulfillableCapacity
    scalingConfig: {
        desiredSize: 1,
        minSize: 0,
        maxSize: 2,
    },
    diskSize: 100,
    amiType: "AL2023_x86_64_NVIDIA",  // EKS-optimized Amazon Linux 2023 with NVIDIA drivers
    labels: {
        "node-role": "gpu",
        "nvidia.com/gpu.present": "true",
        "nvidia.com/mig.config": "all-balanced",  // Configure all GPUs with balanced MIG profiles
    },
    taints: [{
        key: "nvidia.com/gpu",
        value: "true",
        effect: "NO_SCHEDULE",  // Prevent non-GPU workloads from scheduling
    }],
}, { dependsOn: [cluster] });

The GPU node group uses the AL2023_x86_64_NVIDIA AMI type, which includes pre-installed NVIDIA drivers and Container Toolkit. The nvidia.com/mig.config label tells the MIG Manager how to partition the GPUs when they come online.

A note on capacity: p4d.24xlarge instances are in high demand. Spot looks attractive on paper (92% savings) but frequently fails with UnfulfillableCapacity errors. On-Demand is more reliable but runs $32/hr per instance in us-east-1. For production, capacity reservations help. Checking multiple regions helps more.

Installing the NVIDIA GPU Operator

The GPU Operator manages NVIDIA software components on your cluster. When using the AL2023_x86_64_NVIDIA AMI, you must disable driver and toolkit installation since they're pre-installed:

// NVIDIA GPU Operator with DRA configuration
const gpuOperatorNamespace = new k8s.core.v1.Namespace("gpu-operator-ns", {
    metadata: { name: "gpu-operator" },
}, { provider: k8sProvider });

const gpuOperator = new k8s.helm.v3.Release("gpu-operator", {
    chart: "gpu-operator",
    namespace: gpuOperatorNamespace.metadata.name,
    repositoryOpts: {
        repo: "https://helm.ngc.nvidia.com/nvidia",
    },
    version: "v25.10.1",
    skipAwait: true,  // GPU Operator takes time to roll out across nodes
    values: {
        // AL2023_x86_64_NVIDIA AMI has pre-installed drivers
        driver: { enabled: false },
        // AL2023_x86_64_NVIDIA AMI has pre-installed toolkit
        toolkit: { enabled: false },
        // Disable traditional device plugin - DRA handles allocation
        devicePlugin: { enabled: false },
        // Set containerd as the runtime
        operator: {
            defaultRuntime: "containerd",
        },
        // Enable Node Feature Discovery for GPU detection
        nfd: { enabled: true },
        // Configure MIG strategy for heterogeneous profiles
        mig: {
            strategy: "mixed",
        },
        // Enable MIG Manager to apply MIG configurations
        migManager: {
            enabled: true,
            env: [{
                name: "WITH_REBOOT",
                value: "true",  // Allow node reboots if needed for MIG changes
            }],
        },
        // Enable DCGM Exporter for GPU metrics
        dcgmExporter: {
            enabled: true,
            serviceMonitor: { enabled: true },
        },
    },
}, { provider: k8sProvider, dependsOn: [gpuNodeGroup] });

A few things to watch here: you must set driver.enabled: false and toolkit.enabled: false when using the AL2023_x86_64_NVIDIA AMI, or you'll get conflicts with the pre-installed drivers. The mig.strategy: "mixed" is required because all-balanced creates heterogeneous MIG slice sizes. And operator.defaultRuntime: "containerd" is required for AL2023's cgroup v2 setup.

Understanding MIG configuration

That nvidia.com/mig.config: "all-balanced" label on line 435 does more than you might think. MIG Manager runs as a controller in your cluster, watching for this label on GPU nodes. When it spots it, the reconfiguration happens automatically: GPU workloads stop, nvidia-mig-parted applies the new geometry, the node reboots if necessary, then everything comes back up with the new partitioning.

The label value has to match a profile name in the default-mig-parted-config ConfigMap. The GPU Operator creates this ConfigMap with pre-defined configurations for different GPU models:

kubectl get configmap -n gpu-operator default-mig-parted-config -o yaml

Note: If you're using Pulumi ESC for cluster authentication, you can run this via pulumi env run <your-env> -- kubectl get configmap ... to inject credentials automatically.

For A100-40GB GPUs (p4d.24xlarge), you get these options:

• all-1g.5gb creates 7 identical 1g.5gb slices per GPU (56 total across 8 GPUs)

• all-2g.10gb creates 3 identical 2g.10gb slices per GPU

• all-3g.20gb creates 2 identical 3g.20gb slices per GPU

• all-balanced creates a mix: 2× 1g.5gb + 1× 2g.10gb + 1× 3g.20gb per GPU

The all-balanced profile is more useful than uniform slicing. You get small slices for inference, medium slices for training, and larger slices for bigger models, all on the same GPU with hardware isolation. Real workloads aren't all the same size, so why partition like they are?

Understanding MIG strategy: single vs mixed

The mig.strategy setting controls how GPU Feature Discovery (GFD) advertises MIG devices to Kubernetes. Get this wrong and your GPU nodes show nvidia.com/gpu.product=NVIDIA-A100-SXM4-40GB-MIG-INVALID with zero resources available. We spent an hour debugging this one.

Here's the thing that tripped us up: all-balanced creates different MIG slice sizes (1g.5gb, 2g.10gb, 3g.20gb) on the same GPU. If you use strategy: "single" with all-balanced, GFD sees multiple MIG types and marks everything invalid:

Invalid configuration detected for mig-strategy=single: more than one MIG device type present on node

Use mixed whenever your MIG configuration creates different slice sizes. Use single only with uniform profiles like all-1g.5gb where every slice is identical.

One clarification: single doesn't mean all GPUs must have the same profile. It means all MIG slices across the node must be the same type. GPU 0 and GPU 1 both running all-1g.5gb? That's fine with single because all slices are 1g.5gb. But all-balanced creates 1g.5gb, 2g.10gb, and 3g.20gb slices on every GPU, so you need mixed.

For the test workload we're building, all-balanced demonstrates the actual value of MIG with DRA: different workload sizes sharing the same GPU.

Installing the NVIDIA DRA Driver

The DRA driver publishes GPU resources to Kubernetes and handles the allocation lifecycle:

// NVIDIA DRA Driver for GPU allocation
const draDriverNamespace = new k8s.core.v1.Namespace("dra-driver-ns", {
    metadata: { name: "nvidia-dra-driver" },
}, { provider: k8sProvider });

const draDriver = new k8s.helm.v3.Release("nvidia-dra-driver", {
    chart: "nvidia-dra-driver-gpu",
    namespace: draDriverNamespace.metadata.name,
    repositoryOpts: {
        repo: "https://helm.ngc.nvidia.com/nvidia",
    },
    version: "v25.8.1",
    skipAwait: true,
    values: {
        // Set driver root for host-installed drivers (AL2023 AMI)
        nvidiaDriverRoot: "/",
        // Enable GPU resources override for proper DRA functionality
        gpuResourcesEnabledOverride: true,
        resources: {
            gpus: { enabled: true },
            computeDomains: { enabled: false },
        },
        // Configure kubelet plugin to tolerate GPU node taints
        kubeletPlugin: {
            tolerations: [{
                key: "nvidia.com/gpu",
                operator: "Exists",
                effect: "NoSchedule",
            }],
        },
        // Schedule DRA controller on system nodes (not GPU nodes)
        controller: {
            affinity: {
                nodeAffinity: {
                    requiredDuringSchedulingIgnoredDuringExecution: {
                        nodeSelectorTerms: [{
                            matchExpressions: [{
                                key: "node-role",
                                operator: "In",
                                values: ["system"],
                            }],
                        }],
                    },
                },
            },
        },
    },
}, { provider: k8sProvider, dependsOn: [gpuOperator] });

Put the DRA controller on system nodes, not GPU nodes. You don't want resource allocation conflicts. The kubelet plugin still runs on GPU nodes as a DaemonSet to handle device preparation.

The DRA driver creates two DeviceClasses automatically when resources.gpus.enabled: true is set:

• gpu.nvidia.com for full GPU allocations

• mig.nvidia.com for MIG partition allocations

Understanding ResourceClaimTemplates

ResourceClaimTemplates define how pods request specific GPU resources. The CEL expression in the selector determines which devices match. For a workload that needs a small MIG partition:

apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
  name: mig-small-template
spec:
  spec:
    devices:
      requests:
      - name: mig-gpu
        exactly:
          deviceClassName: mig.nvidia.com
          count: 1
          selectors:
          - cel:
              expression: |
                device.attributes["gpu.nvidia.com"].type == "mig" &&
                device.attributes["gpu.nvidia.com"].profile == "1g.5gb"

The CEL expression filters for MIG devices with the 1g.5gb profile. For different profiles, change the profile value to 2g.10gb, 3g.20gb, etc.

Real-world workload demonstration: Fashion-MNIST training and inference

To demonstrate MIG running different workload types on the same GPU, we deployed three PyTorch workloads using the Fashion-MNIST dataset: a large training job, a medium training job, and a small inference service, all sharing a single A100 GPU.

Fashion-MNIST is a dataset of 70,000 grayscale images (60,000 training, 10,000 test) showing clothing items across 10 categories: t-shirts, trousers, pullovers, dresses, coats, sandals, shirts, sneakers, bags, and ankle boots. Created by Zalando Research as a drop-in replacement for the classic MNIST handwritten digits dataset, it provides a more challenging classification task while maintaining the same 28x28 pixel format. It's commonly used for benchmarking ML frameworks and testing GPU workloads because it's large enough to stress hardware but small enough to iterate quickly.

The workloads

With the all-balanced MIG configuration, each A100 40GB GPU provides:

• 2× 1g.5gb partitions (small inference workloads)

• 1× 2g.10gb partition (medium training)

• 1× 3g.20gb partition (large training)

Here's what we deployed:

Large training workload (3g.20gb): ResNet-18 training on Fashion-MNIST with 256-batch size, using ~11GB GPU memory.

Medium training workload (2g.10gb): Custom CNN training with 128-batch size, using ~5GB GPU memory.

Small inference workload (1g.5gb): Continuous inference loop using ~0.01GB memory.

ConfigMap with training scripts

First, create a ConfigMap containing the Python scripts for each workload:

const fashionMnistScripts = new k8s.core.v1.ConfigMap("fashion-mnist-scripts", {
    metadata: {
        name: "fashion-mnist-scripts",
        namespace: migTestNamespace.metadata.name,
    },
    data: {
        "large-training-script.py": `import torch
import torch.nn as nn
import torch.optim as optim
from torchvision import datasets, transforms, models

print("Starting Large Training Workload (3g.20gb MIG)")
print(f"PyTorch version: {torch.__version__}")
print(f"CUDA available: {torch.cuda.is_available()}")
print(f"CUDA device: {torch.cuda.get_device_name(0) if torch.cuda.is_available() else 'N/A'}")

transform = transforms.Compose([
    transforms.Resize(224),
    transforms.Grayscale(3),
    transforms.ToTensor(),
    transforms.Normalize((0.5,), (0.5,))
])

train_dataset = datasets.FashionMNIST('./data', train=True, download=True, transform=transform)
train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=256, shuffle=True)

model = models.resnet18(pretrained=False, num_classes=10).cuda()
optimizer = optim.Adam(model.parameters(), lr=0.001)
criterion = nn.CrossEntropyLoss()

print("Starting training loop...")
for epoch in range(20):
    model.train()
    total_loss = 0
    correct = 0
    total = 0

    for batch_idx, (data, target) in enumerate(train_loader):
        data, target = data.cuda(), target.cuda()
        optimizer.zero_grad()
        output = model(data)
        loss = criterion(output, target)
        loss.backward()
        optimizer.step()

        total_loss += loss.item()
        pred = output.argmax(dim=1)
        correct += pred.eq(target).sum().item()
        total += target.size(0)

    accuracy = 100. * correct / total
    avg_loss = total_loss / len(train_loader)
    print(f"Epoch {epoch+1}/20 | Loss: {avg_loss:.4f} | Accuracy: {accuracy:.2f}% | GPU Mem: {torch.cuda.memory_allocated()/1e9:.2f}GB")

print("Training complete!")`,

        "medium-training-script.py": `import torch
import torch.nn as nn
import torch.optim as optim
from torchvision import datasets, transforms

print("Starting Medium Training Workload (2g.10gb MIG)")
print(f"PyTorch version: {torch.__version__}")
print(f"CUDA available: {torch.cuda.is_available()}")
print(f"CUDA device: {torch.cuda.get_device_name(0) if torch.cuda.is_available() else 'N/A'}")

transform = transforms.Compose([
    transforms.Resize(32),
    transforms.ToTensor(),
    transforms.Normalize((0.5,), (0.5,))
])

train_dataset = datasets.FashionMNIST('./data', train=True, download=True, transform=transform)
train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=128, shuffle=True)

# Custom CNN with 4 conv layers + 2 FC
model = nn.Sequential(
    nn.Conv2d(1, 64, kernel_size=3, padding=1),
    nn.ReLU(),
    nn.MaxPool2d(2),
    nn.Conv2d(64, 128, kernel_size=3, padding=1),
    nn.ReLU(),
    nn.MaxPool2d(2),
    nn.Conv2d(128, 256, kernel_size=3, padding=1),
    nn.ReLU(),
    nn.MaxPool2d(2),
    nn.Conv2d(256, 512, kernel_size=3, padding=1),
    nn.ReLU(),
    nn.AdaptiveAvgPool2d(1),
    nn.Flatten(),
    nn.Linear(512, 256),
    nn.ReLU(),
    nn.Dropout(0.5),
    nn.Linear(256, 10)
).cuda()

optimizer = optim.Adam(model.parameters(), lr=0.001)
criterion = nn.CrossEntropyLoss()

print("Starting training loop...")
for epoch in range(15):
    model.train()
    total_loss = 0
    correct = 0
    total = 0

    for batch_idx, (data, target) in enumerate(train_loader):
        data, target = data.cuda(), target.cuda()
        optimizer.zero_grad()
        output = model(data)
        loss = criterion(output, target)
        loss.backward()
        optimizer.step()

        total_loss += loss.item()
        pred = output.argmax(dim=1)
        correct += pred.eq(target).sum().item()
        total += target.size(0)

    accuracy = 100. * correct / total
    avg_loss = total_loss / len(train_loader)
    print(f"Epoch {epoch+1}/15 | Loss: {avg_loss:.4f} | Accuracy: {accuracy:.2f}% | GPU Mem: {torch.cuda.memory_allocated()/1e9:.2f}GB")

print("Training complete!")`,

        "small-inference-script.py": `import torch
import torch.nn as nn
from torchvision import datasets, transforms

print("Starting Small Inference Workload (1g.5gb MIG)")
print(f"PyTorch version: {torch.__version__}")
print(f"CUDA available: {torch.cuda.is_available()}")
print(f"CUDA device: {torch.cuda.get_device_name(0) if torch.cuda.is_available() else 'N/A'}")

transform = transforms.Compose([
    transforms.ToTensor(),
    transforms.Normalize((0.5,), (0.5,))
])

test_dataset = datasets.FashionMNIST('./data', train=False, download=True, transform=transform)
test_loader = torch.utils.data.DataLoader(test_dataset, batch_size=32, shuffle=False)

model = nn.Sequential(
    nn.Flatten(),
    nn.Linear(28*28, 512),
    nn.ReLU(),
    nn.Linear(512, 256),
    nn.ReLU(),
    nn.Linear(256, 10)
).cuda()

model.eval()

print("Starting continuous inference loop...")
iteration = 0
while True:
    for data, target in test_loader:
        data, target = data.cuda(), target.cuda()
        with torch.no_grad():
            output = model(data)
            pred = output.argmax(dim=1)

        if iteration % 50 == 0:
            print(f"Iteration {iteration} | GPU Mem: {torch.cuda.memory_allocated()/1e9:.2f}GB")

        iteration += 1`,
    },
}, { provider: k8sProvider });

ResourceClaimTemplates for different MIG profiles

Create three ResourceClaimTemplates, one for each MIG profile:

// Large training: 3g.20gb MIG profile
const migLargeClaimTemplate = new k8s.apiextensions.CustomResource("mig-large-template", {
    apiVersion: "resource.k8s.io/v1",
    kind: "ResourceClaimTemplate",
    metadata: {
        name: "mig-large-template",
        namespace: migTestNamespace.metadata.name,
    },
    spec: {
        spec: {
            devices: {
                requests: [{
                    name: "mig-large",
                    exactly: {
                        deviceClassName: "mig.nvidia.com",
                        count: 1,
                        selectors: [{
                            cel: {
                                expression: 'device.attributes["gpu.nvidia.com"].type == "mig" && device.attributes["gpu.nvidia.com"].profile == "3g.20gb"',
                            },
                        }],
                    },
                }],
            },
        },
    },
}, { provider: k8sProvider, dependsOn: [draDriver] });

// Medium training: 2g.10gb MIG profile
const migMediumClaimTemplate = new k8s.apiextensions.CustomResource("mig-medium-template", {
    apiVersion: "resource.k8s.io/v1",
    kind: "ResourceClaimTemplate",
    metadata: {
        name: "mig-medium-template",
        namespace: migTestNamespace.metadata.name,
    },
    spec: {
        spec: {
            devices: {
                requests: [{
                    name: "mig-medium",
                    exactly: {
                        deviceClassName: "mig.nvidia.com",
                        count: 1,
                        selectors: [{
                            cel: {
                                expression: 'device.attributes["gpu.nvidia.com"].type == "mig" && device.attributes["gpu.nvidia.com"].profile == "2g.10gb"',
                            },
                        }],
                    },
                }],
            },
        },
    },
}, { provider: k8sProvider, dependsOn: [draDriver] });

// Small inference: 1g.5gb MIG profile
const migSmallClaimTemplate = new k8s.apiextensions.CustomResource("mig-small-template", {
    apiVersion: "resource.k8s.io/v1",
    kind: "ResourceClaimTemplate",
    metadata: {
        name: "mig-small-template",
        namespace: migTestNamespace.metadata.name,
    },
    spec: {
        spec: {
            devices: {
                requests: [{
                    name: "mig-small",
                    exactly: {
                        deviceClassName: "mig.nvidia.com",
                        count: 1,
                        selectors: [{
                            cel: {
                                expression: 'device.attributes["gpu.nvidia.com"].type == "mig" && device.attributes["gpu.nvidia.com"].profile == "1g.5gb"',
                            },
                        }],
                    },
                }],
            },
        },
    },
}, { provider: k8sProvider, dependsOn: [draDriver] });

Deploying the workload pods

Finally, create three pods that run the workloads:

// Large training pod (3g.20gb)
const migLargeTrainingPod = new k8s.core.v1.Pod("mig-large-training", {
    metadata: {
        name: "mig-large-training-pod",
        namespace: migTestNamespace.metadata.name,
    },
    spec: {
        restartPolicy: "Never",
        tolerations: [{
            key: "nvidia.com/gpu",
            operator: "Exists",
            effect: "NoSchedule",
        }],
        nodeSelector: {
            "node-role": "gpu",
            "nvidia.com/gpu.present": "true",
        },
        containers: [{
            name: "training",
            image: "nvcr.io/nvidia/pytorch:25.12-py3",
            command: ["python", "/scripts/large-training-script.py"],
            volumeMounts: [{
                name: "scripts",
                mountPath: "/scripts",
                readOnly: true,
            }],
            resources: {
                claims: [{ name: "mig-large" }],
            },
        }],
        resourceClaims: [{
            name: "mig-large",
            resourceClaimTemplateName: "mig-large-template",
        }],
        volumes: [{
            name: "scripts",
            configMap: {
                name: fashionMnistScripts.metadata.name,
            },
        }],
    },
}, { provider: k8sProvider, dependsOn: [migLargeClaimTemplate, fashionMnistScripts] });

// Medium training pod (2g.10gb)
const migMediumTrainingPod = new k8s.core.v1.Pod("mig-medium-training", {
    metadata: {
        name: "mig-medium-training-pod",
        namespace: migTestNamespace.metadata.name,
    },
    spec: {
        restartPolicy: "Never",
        tolerations: [{
            key: "nvidia.com/gpu",
            operator: "Exists",
            effect: "NoSchedule",
        }],
        nodeSelector: {
            "node-role": "gpu",
            "nvidia.com/gpu.present": "true",
        },
        containers: [{
            name: "training",
            image: "nvcr.io/nvidia/pytorch:25.12-py3",
            command: ["python", "/scripts/medium-training-script.py"],
            volumeMounts: [{
                name: "scripts",
                mountPath: "/scripts",
                readOnly: true,
            }],
            resources: {
                claims: [{ name: "mig-medium" }],
            },
        }],
        resourceClaims: [{
            name: "mig-medium",
            resourceClaimTemplateName: "mig-medium-template",
        }],
        volumes: [{
            name: "scripts",
            configMap: {
                name: fashionMnistScripts.metadata.name,
            },
        }],
    },
}, { provider: k8sProvider, dependsOn: [migMediumClaimTemplate, fashionMnistScripts] });

// Small inference pod (1g.5gb)
const migSmallInferencePod = new k8s.core.v1.Pod("mig-small-inference", {
    metadata: {
        name: "mig-small-inference-pod",
        namespace: migTestNamespace.metadata.name,
    },
    spec: {
        restartPolicy: "Never",
        tolerations: [{
            key: "nvidia.com/gpu",
            operator: "Exists",
            effect: "NoSchedule",
        }],
        nodeSelector: {
            "node-role": "gpu",
            "nvidia.com/gpu.present": "true",
        },
        containers: [{
            name: "inference",
            image: "nvcr.io/nvidia/pytorch:25.12-py3",
            command: ["python", "/scripts/small-inference-script.py"],
            volumeMounts: [{
                name: "scripts",
                mountPath: "/scripts",
                readOnly: true,
            }],
            resources: {
                claims: [{ name: "mig-small" }],
            },
        }],
        resourceClaims: [{
            name: "mig-small",
            resourceClaimTemplateName: "mig-small-template",
        }],
        volumes: [{
            name: "scripts",
            configMap: {
                name: fashionMnistScripts.metadata.name,
            },
        }],
    },
}, { provider: k8sProvider, dependsOn: [migSmallClaimTemplate, fashionMnistScripts] });

Deployment results

After deploying with pulumi up, the three workloads scheduled onto the same physical GPU:

NAME                      READY   STATUS      RESTARTS   AGE
mig-large-training-pod    1/1     Running     0          21m
mig-medium-training-pod   0/1     Completed   0          21m
mig-small-inference-pod   1/1     Running     0          20m

The logs confirm each workload got its allocated MIG partition:

Medium training (completed 15 epochs):

Starting Medium Training Workload (2g.10gb MIG)
CUDA device: NVIDIA A100-SXM4-40GB MIG 2g.10gb
Epoch 1/15 | Loss: 0.7649 | Accuracy: 71.21% | GPU Mem: 0.05GB
...
Epoch 15/15 | Loss: 0.0827 | Accuracy: 96.95% | GPU Mem: 0.05GB
Training complete!

Large training (20 epochs with ResNet-18):

Starting Large Training Workload (3g.20gb MIG)
CUDA device: NVIDIA A100-SXM4-40GB MIG 3g.20gb
Epoch 1/20 | Loss: 0.4476 | Accuracy: 83.35% | GPU Mem: 0.26GB
...
Epoch 20/20 | Loss: 0.0183 | Accuracy: 99.36% | GPU Mem: 0.26GB
Training complete!

Small inference (continuous operation):

Starting Small Inference Workload (1g.5gb MIG)
CUDA device: NVIDIA A100-SXM4-40GB MIG 1g.5gb
Iteration 260000 | GPU Mem: 0.01GB

The medium training finished in about 21 minutes with 96.95% accuracy. The large training job completed 20 epochs, reaching 99.36% accuracy. The inference pod continues running, now past 576,000 iterations.

Monitoring MIG with Grafana

With DCGM Exporter enabled in the GPU Operator, you can visualize MIG partition metrics in Grafana. The screenshot below shows an NVIDIA MIG dashboard displaying metrics for multiple MIG instances on an A100 GPU:

The dashboard breaks down metrics by MIG partition: GPU utilization, memory consumption, SM clock speeds, and temperature. Useful for spotting underutilized partitions or memory pressure before they become problems.

Why platform engineers should care

If you're running a shared GPU cluster, you've probably dealt with the politics of GPU allocation. Team A wants dedicated hardware. Team B complains their jobs keep getting killed. Nobody trusts the "fair share" scheduler. MIG with DRA sidesteps a lot of this because the isolation is in the hardware, not a software policy that can be argued with.

The cost math is also hard to ignore. Seven 1g.5gb inference services on one A100 versus seven separate GPU instances? That's not a small difference on the AWS bill.

And there's the self-service angle. Once you've set up DeviceClasses and ResourceClaimTemplates, ML teams can request what they need without filing tickets. They pick a claim template, and the cluster figures out where to put the workload. You define the guardrails once; they stay out of your queue.

If you want to go further, Pulumi CrossGuard can enforce policies on GPU allocation. Here's a policy that blocks full GPU profiles while allowing the balanced MIG partitions:

import * as policy from "@pulumi/policy";

const allowedMigProfiles = ["1g.5gb", "2g.10gb", "3g.20gb"];
const blockedMigProfiles = ["4g.20gb", "7g.40gb", "7g.80gb"];

new policy.PolicyPack("mig-policy", {
    policies: [
        {
            name: "enforce-balanced-mig-profiles",
            description: "Prevents full GPU allocations while allowing balanced MIG profiles (1g.5gb, 2g.10gb, 3g.20gb)",
            enforcementLevel: "mandatory",
            validateResource: (args, reportViolation) => {
                const resource = args.props as any;

                if (args.type !== "kubernetes:resource.k8s.io/v1:ResourceClaimTemplate") {
                    return;
                }

                const spec = resource.spec;
                if (!spec?.spec?.devices?.requests) return;

                for (const request of spec.spec.devices.requests) {
                    const selectors = request.exactly?.selectors || [];
                    for (const selector of selectors) {
                        const expression = selector.cel?.expression;
                        if (!expression) continue;

                        for (const blocked of blockedMigProfiles) {
                            if (expression.includes(`"${blocked}"`)) {
                                reportViolation(
                                    `ResourceClaimTemplate '${resource.metadata?.name}' requests ` +
                                    `blocked profile '${blocked}'. Allowed: ${allowedMigProfiles.join(", ")}`
                                );
                            }
                        }
                    }
                }
            },
        },
    ],
});

Run it with pulumi preview --policy-pack ./mig-policy. If someone tries to request a 7g.80gb profile (the full GPU), the preview fails:

Policies:
    ❌ mig-policy@v1.0.0 (local: mig-policy)
        - [mandatory]  enforce-small-mig-profiles  (kubernetes:resource.k8s.io/v1:ResourceClaimTemplate: mig-invalid-template)
          Prevents using large MIG profiles (4g.20gb and above) to maximize GPU utilization for inference workloads
          ResourceClaimTemplate 'mig-invalid-template' requests blocked MIG profile '7g.80gb'. Only small profiles are allowed: 1g.5gb, 2g.10gb, 3g.20gb. Large profiles waste GPU resources for inference workloads.

The usual policy-as-code workflow, but applied to hardware allocation.

Capacity planning

The cluster is running. Someone asks: "Can I get three more MIG slices for my inference service?" You should be able to answer that without guessing.

Checking capacity with kubectl

The quickest way is kubectl against ResourceSlices and ResourceClaims:

# See all available MIG devices across the cluster
kubectl get resourceslices -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{range .spec.devices[*]}  {.name}: {.attributes.profile.string}{"\n"}{end}{end}'

# Count total vs allocated MIG slices
kubectl get resourceslices -o json | jq '[.items[].spec.devices[]] | length'
kubectl get resourceclaims -A -o json | jq '[.items[] | select(.status.allocation != null)] | length'

# See which claims are pending (waiting for capacity)
kubectl get resourceclaims -A -o json | jq '.items[] | select(.status.allocation == null) | .metadata.name'

On our test cluster with one p4d.24xlarge (8x A100 40GB GPUs, each with the all-balanced profile creating 4 slices: 2× 1g.5gb, 1× 2g.10gb, 1× 3g.20gb), the output shows:

Total MIG slices: 32
Allocated: 3
Available: 29

Monitoring with Prometheus and Grafana

DCGM Exporter (enabled in our GPU Operator config) pushes metrics to Prometheus. Some queries worth having:

# Available GPU memory per MIG instance
DCGM_FI_DEV_FB_FREE{gpu=~".*MIG.*"}

# GPU utilization per MIG partition
DCGM_FI_DEV_GPU_UTIL{gpu=~".*MIG.*"}

# Count of active MIG devices
count(DCGM_FI_DEV_GPU_UTIL{gpu=~".*MIG.*"})

Wire these into a Grafana dashboard. You'll want to know when you're running low before someone's deployment gets stuck pending.

Direct node inspection

When kubectl and Prometheus disagree, SSH into the GPU node and check with nvidia-smi:

# List all GPU instances and their status
nvidia-smi mig -lgi

# Show compute instances within each GPU instance
nvidia-smi mig -lci

# Detailed view with memory and compute info
nvidia-smi -L

This is the ground truth. If the hardware says seven slices but Kubernetes only sees five, you know where to start debugging.

What we learned the hard way

Here's what bit us during implementation:

GPU compatibility (this one got us)

Not all NVIDIA GPUs support MIG. We initially tried g5 instances with A10G GPUs. They're Ampere architecture, so MIG should work, right? Wrong. MIG requires compute capability 8.0+, but that's necessary, not sufficient. The A10G doesn't have the hardware partitioning silicon. Check the supported GPUs list before picking instance types. On AWS, p4d (A100) and p5 (H100) work. g5 (A10G) doesn't.

Spot capacity is a mirage

On paper, Spot pricing for p4d.24xlarge looks amazing: $2.48/hr versus $32/hr for On-Demand. That's 92% savings. In practice? We spent hours watching UnfulfillableCapacity errors scroll by.

The p4d.24xlarge packs 8x A100 40GB GPUs, 1.1TB RAM, and 400Gbps networking. Everyone wants these. Spot pools are perpetually exhausted.

What actually works: start with On-Demand (yes, it's expensive), check capacity across regions (us-west-2, eu-west-1, and ap-southeast-1 tend to have better availability), and use capacity reservations for anything critical. Don't plan to scale from zero to ten GPU nodes quickly. The capacity probably isn't there.

If you do get Spot capacity, spread instances across availability zones, implement graceful shutdown handlers for the 2-minute warning, and keep a small On-Demand node group as fallback. EventBridge can give you advance notice of interruptions.

AL2023 AMI quirks

The AL2023_x86_64_NVIDIA AMI comes with NVIDIA drivers and Container Toolkit pre-installed. This is convenient until you realize the GPU Operator will try to install them again. Disable both (driver.enabled: false, toolkit.enabled: false) or watch things break mysteriously. Also set operator.defaultRuntime: "containerd" for cgroup v2 support.

MIG strategy mismatch

This one had us scratching our heads. The GPU node came up, MIG Manager reported success, but the node label showed NVIDIA-A100-SXM4-40GB-MIG-INVALID and Kubernetes saw zero GPU resources. The culprit: we had mig.strategy: "single" but were using all-balanced, which creates multiple slice types. GFD expects uniform slices with single and marks anything else as invalid. Switch to strategy: "mixed" for heterogeneous profiles. Check with kubectl logs -n gpu-operator -l app=gpu-feature-discovery if you hit this.

DRA API changes from beta

Kubernetes 1.34 promoted DRA to GA, which means breaking changes from v1beta1. ResourceClaimTemplate now requires an exactly block with a count field. DeviceClass and ResourceClaim moved from v1beta1 to v1. If you're migrating from an earlier setup, your manifests need updating.

Where to go from here

DRA with MIG solves a problem that's been frustrating platform teams for years: how do you share expensive GPU hardware without losing isolation? The answer turns out to be hardware partitioning plus a scheduler that actually understands what it's allocating.

The setup isn't trivial. Between GPU compatibility gotchas, Spot capacity constraints, AL2023 AMI configuration, and MIG strategy mismatches, there's real complexity here. But once it's running, your ML teams can request GPU resources without filing tickets, and you can stop watching $32/hr hardware sit 90% idle.

The code in this post is available to adapt for your own infrastructure. For more details, see the Kubernetes DRA documentation, the AWS Labs DRA guide, and the NVIDIA GPU Operator docs. The NVIDIA DRA Driver repo has more CEL expression examples.

Nothing is more controversial in the Kubernetes community than whether to use Helm or Kustomize.

I always advocate the philosophy of using the right tool for the right job. It avoids the problem of everything looks like a nail when you have a hammer.

Yet, sometimes you need to combine similar tools to get the best results. Helm and Kustomize are two tools, when combined smartly, they can boost productivity and flexibility.

But, before we jump into the details, let's first understand what Helm and Kustomize are and what they do.

Helm

Helm is the de facto default package manager for Kubernetes applications. It provides an efficient way to package, share, and deploy your Kubernetes applications. It can be used via the CLI or through first-class support in CD tools like Argo CD or Flux. One of the key features I like about Helm is its powerful templating engine. It lets you create reusable templates for your Kubernetes resources. This keeps your YAML files DRY (Don't Repeat Yourself). It makes it easier to manage and maintain your Kubernetes resources.

Example of the template in Helm I took from the kube-prometheus-stack

{{- if and .Values.prometheus.enabled .Values.prometheus.serviceMonitor.selfMonitor}}
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: {{template "kube-prometheus-stack.fullname" .}}-prometheus
  namespace: {{template "kube-prometheus-stack.namespace" .}}
  labels:
    app: {{template "kube-prometheus-stack.name" .}}-prometheus
  {{include "kube-prometheus-stack.labels" . | indent 4}}
  {{- with .Values.prometheus.serviceMonitor.additionalLabels}}
  {{- toYaml . | nindent 4}}
  {{- end}}
spec: {{- include "servicemonitor.scrapeLimits" .Values.prometheus.serviceMonitor | nindent 2}}
  selector:
    matchLabels:
      app: {{template "kube-prometheus-stack.name" .}}-prometheus
      release: {{$.Release.Name | quote}}
      self-monitor: "true"
  namespaceSelector:
    matchNames:
    - {{printf "%s" (include "kube-prometheus-stack.namespace" .) | quote}}
  endpoints:
  - port: {{.Values.prometheus.prometheusSpec.portName}}
    {{- if .Values.prometheus.serviceMonitor.interval}}
    interval: {{.Values.prometheus.serviceMonitor.interval}}
    {{- end}}
    {{- if .Values.prometheus.serviceMonitor.scheme}}
    scheme: {{.Values.prometheus.serviceMonitor.scheme}}
    {{- end}}
    {{- if .Values.prometheus.serviceMonitor.tlsConfig}}
    tlsConfig: {{- toYaml .Values.prometheus.serviceMonitor.tlsConfig | nindent 6}}
    {{- end}}
    {{- if .Values.prometheus.serviceMonitor.bearerTokenFile}}
    bearerTokenFile: {{.Values.prometheus.serviceMonitor.bearerTokenFile}}
    {{- end}}
    path: "{{ trimSuffix "/" .Values.prometheus.prometheusSpec.routePrefix }}/metrics"
    {{- if .Values.prometheus.serviceMonitor.metricRelabelings}}
    metricRelabelings: {{- tpl (toYaml .Values.prometheus.serviceMonitor.metricRelabelings | nindent 6) .}}
    {{- end}}
    {{- if .Values.prometheus.serviceMonitor.relabelings}}
    relabelings: {{- toYaml .Values.prometheus.serviceMonitor.relabelings | nindent 6}}
    {{- end}}
  - port: reloader-web
    {{- if .Values.prometheus.serviceMonitor.interval}}
    interval: {{.Values.prometheus.serviceMonitor.interval}}
    {{- end}}
    {{- if .Values.prometheus.serviceMonitor.scheme}}
    scheme: {{.Values.prometheus.serviceMonitor.scheme}}
    {{- end}}
    {{- if .Values.prometheus.serviceMonitor.tlsConfig}}
    tlsConfig: {{- toYaml .Values.prometheus.serviceMonitor.tlsConfig | nindent 6}}
    {{- end}}
    path: "/metrics"
    {{- if .Values.prometheus.serviceMonitor.metricRelabelings}}
    metricRelabelings: {{- tpl (toYaml .Values.prometheus.serviceMonitor.metricRelabelings | nindent 6) .}}
    {{- end}}
    {{- if .Values.prometheus.serviceMonitor.relabelings}}
    relabelings: {{- toYaml .Values.prometheus.serviceMonitor.relabelings | nindent 6}}
    {{- end}}
  {{- range .Values.prometheus.serviceMonitor.additionalEndpoints}}
  - port: {{.port}}
    {{- if or $.Values.prometheus.serviceMonitor.interval .interval}}
    interval: {{default $.Values.prometheus.serviceMonitor.interval .interval}}
    {{- end}}
    {{- if or $.Values.prometheus.serviceMonitor.proxyUrl .proxyUrl}}
    proxyUrl: {{default $.Values.prometheus.serviceMonitor.proxyUrl .proxyUrl}}
    {{- end}}
    {{- if or $.Values.prometheus.serviceMonitor.scheme .scheme}}
    scheme: {{default $.Values.prometheus.serviceMonitor.scheme .scheme}}
    {{- end}}
    {{- if or $.Values.prometheus.serviceMonitor.bearerTokenFile .bearerTokenFile}}
    bearerTokenFile: {{default $.Values.prometheus.serviceMonitor.bearerTokenFile .bearerTokenFile}}
    {{- end}}
    {{- if or $.Values.prometheus.serviceMonitor.tlsConfig .tlsConfig}}
    tlsConfig: {{- default $.Values.prometheus.serviceMonitor.tlsConfig .tlsConfig | toYaml | nindent 6}}
    {{- end}}
    path: {{.path}}
    {{- if or $.Values.prometheus.serviceMonitor.metricRelabelings .metricRelabelings}}
    metricRelabelings: {{- tpl (default $.Values.prometheus.serviceMonitor.metricRelabelings .metricRelabelings | toYaml | nindent 6) .}}
    {{- end}}
    {{- if or $.Values.prometheus.serviceMonitor.relabelings .relabelings}}
    relabelings: {{- default $.Values.prometheus.serviceMonitor.relabelings .relabelings | toYaml | nindent 6}}
    {{- end}}
  {{- end}}
  {{- end}}

As powerful as Helm is, it has its limitations. One of them is the customisation of third-party Helm charts. And this is where Kustomize comes in. But more on that later.

Kustomize

Kustomize lets you, as the name suggests, customise your raw, template-free YAML files. It does not alter the original YAML files but instead applies patches to them. Like Helm, Kustomize has a stand-alone CLI tool. It also has first-class integrations in kubectl and CD tools like Argo CD and Flux. There are different ways to use Kustomize. The most common variation is to use Kustomize with a base and overlay structure. The base has the raw, template-free YAML files. The overlay has the patches to apply to the base.

Example file structure:

~/someApp
├── base
│   ├── deployment.yaml
│   ├── kustomization.yaml
│   └── service.yaml
└── overlays
    ├── development
    │   ├── cpu_count.yaml
    │   ├── kustomization.yaml
    │   └── replica_count.yaml
    └── production
        ├── cpu_count.yaml
        ├── kustomization.yaml
        └── replica_count.yaml

Keep in mind, that you need to have the kustomization.yaml file in each directory to tell Kustomize which resources to apply and which patches to use.

Example of a kustomization.yaml file:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

labels:
- pairs:
    app: someApp

resources:
- deployment.yaml
- service.yaml

This example will apply the deployment.yaml and service.yaml files in the base directory and add the label app: someApp to each resource.

Combining Helm and Kustomize

Now that we know the basics of Helm and Kustomize, let's combine them to get the best of both worlds. Before we use Argo CD to deploy our application, let's create a demo use case.

Use Case

Think about a scenario where you would like to use the new Gateway API from Kubernetes. Unfortunately, most of the Helm charts do not support the Gateway API yet.

Now this is where Kustomize comes in. As an example of this use case, I am going to deploy a Minecraft server.

Create the Kustomize Project

First, let's create a new Kustomize project. We will use the kustomize create command to create a new project.

kustomize init

Now open the kustomization.yaml file and add the following content:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
- base/gateway.yaml

helmCharts:
- name: minecraft
  releaseName: my
  repo: https://itzg.github.io/minecraft-server-charts/
  version: 4.23.2
  valuesInline:
    minecraftServer:
      eula: "true"

Create the base directory and add the gateway.yaml file with the following content:

kind: GatewayClass
apiVersion: gateway.networking.k8s.io/v1
metadata:
  name: eg
spec:
  controllerName: gateway.envoyproxy.io/gatewayclass-controller
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: minecraft-gateway
spec:
  gatewayClassName: eg
  listeners:
  - name: minecraft
    protocol: TCP
    port: 8088
    allowedRoutes:
      kinds:
      - kind: TCPRoute
---
apiVersion: gateway.networking.k8s.io/v1alpha2
kind: TCPRoute
metadata:
  name: minecraft-route
spec:
  parentRefs:
  - name: minecraft-gateway
    sectionName: minecraft
  rules:
  - backendRefs:
    - name: my-minecraft
      port: 25565

Now let's apply the Kustomize project:

kustomize build --enable-helm | kubectl apply -f -

Important to note is the --enable-helm flag. This flag tells Kustomize to enable Helm support.

So far, so good. We have deployed a Minecraft server using Helm. We added the missing Gateway API support using Kustomize.

The question now is, how can we deploy this application using Argo CD?

Deploy the Application using Argo CD

First we need to deploy Argo CD using Helm and following content in the values.yaml file:

configs:
  cm:
    kustomize.buildOptions: "--enable-helm"
  # the rest of the values are only for demonstration purposes
  secret:
    argocdServerAdminPassword: "$2a$10$RjjTokiJSaTQt8jAMOUTK.O0VIZ3.0AEs3/JxtaFKGZir93yFPEOG"
    argocdServerAdminPasswordMtime: "2023-11-13T09:23:16Z"
  params:
    "server.insecure": true

The most important part is the kustomize.buildOptions: "--enable-helm" configuration. This tells Argo CD to enable Helm support in Kustomize.

Now we can deploy Argo CD and the Envoy Gateway using Helm:

helm repo add argo https://argoproj.github.io/argo-helm
helm repo update
helm upgrade -i my-argo-cd argo/argo-cd -f values.yaml --namespace argocd --create-namespace
helm install eg oci://docker.io/envoyproxy/gateway-helm --version v1.2.1 -n envoy-gateway-system --create-namespace

After the deployment is finished we can create a new Argo CD application using the following content:

kubectl apply -f minecraft-application.yaml

With the following content in the minecraft-application.yaml file:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: minecraft-application
  namespace: argocd
spec:
  destination:
    namespace: minecraft
    server: https://kubernetes.default.svc
  project: default
  source:
    repoURL: https://github.com/dirien/quick-bites.git
    targetRevision: main
    path: argocd-kustomize-helms-support
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
    - ServerSideApply=true
    - CreateNamespace=true

With this setup, we have successfully deployed a Minecraft server using Helm and added the missing Gateway API support. You can check the TCPRoute and Gateway resources by running the following command:

kubectl get gateway,tcproute -n minecraft

NAME                                                  CLASS   ADDRESS   PROGRAMMED   AGE
gateway.gateway.networking.k8s.io/minecraft-gateway   eg                False        5m19s

NAME                                                 AGE
tcproute.gateway.networking.k8s.io/minecraft-route   5m19s

Start yor Minecraft client and connect to the server using the IP address of the minecraft-gateway resource and the port. In this case it would be 8088.

Note: Enabling Helm support in every app has a slight performance cost. If you want more control over which Applications to use, create an Argo CD ConfigurationManagementPlugin.

Conclusion

Combining Helm and Kustomize can be a powerful way to deploy your app. Use it when you need to customize third-party Helm charts. Or, if you prefer Kustomize and don't want to write Helm charts for your own deployments.

Introduction

This article is part three of my series on secret management on Kubernetes with the help of Pulumi. In my first article, we talked about the Sealed Secrets controller. The second article, we talked about the Secrets Store CSI Driver and how it compares to the Sealed Secrets controller.

If you haven't read those articles yet, I recommend you do so before continuing with this one:

And as we saw in the previous articles, managing secrets in Kubernetes can be a pain in the neck. But we all agree. Proper secret management is vital for our apps and infrastructure security. Especially, and now comes the important part, if we store our secrets in a Git repository. This is also an essential part of the GitOps workflow. We saw that the Sealed Secrets controller could encrypt our secrets and store them in Git. Yet, we soon recognized its flaws. Think about the nightmare of managing keys for many projects, clusters, and secrets used by various teams. And, rotating them.

The second solution we saw was the Secrets Store CSI Driver. This solution is great. But you must build your app with the Secrets Store CSI Driver in mind to retrieve the secrets from a file. You could argue this is manageable.

But what about external tools that don't support this driver?

What is the External Secrets Operator?

Here comes the External Secrets Operator (ESO) to the rescue. The ESO is build following the Kubernetes operator pattern and manages secrets in a highly secure and scalable way. ESO syncs secrets in external systems, like Pulumi ESC, HashiCorp Vault, AWS Secrets Manager, and Azure Key Vault, into Kubernetes secrets. This lets us tame secret sprawl. It centralizes secret management in one place. It also provides a secure, controlled way to access them.

ESO offers us the following benefits:

• By leveraging external secret management systems, ESO acts as a bridge between Kubernetes and external secret management.

• With ESO, we don't have any manual intervention to manage secrets in Kubernetes. Everything is automated and managed by ESO.

• ESO providers support secret rotation when the external secret management system supports it.

• Through the unified interface ESO provides, handling cross-cluster and cross-cloud secret management becomes straightforward and independent of the underlying infrastructure.

The Architecture of the External Secrets Operator

ESO is built on top of the Kubernetes operator pattern and extends the Kubernetes API with several different custom resource definitions (CRDs) to manage secrets. ESO is watching for these CRDs, and when secrets at the external secret management system match the defined CRDs, ESO will synchronize the secrets into Kubernetes secrets. Now we have the secret as part of the Kubernetes reconciliation loop. This means that the secret is always up to date and in sync with the external secret management system.

Custom Resource Definitions (CRDs) of the External Secrets Operator

Following CRDs, you should know when working with ESO:

• (Cluster)SecretStore: This CRD defines the external secret management system that ESO should use to manage secrets. It contains the configuration to connect to the external secret management system. SecretStore is namespaced, while the ClusterSecretStore is cluster-scoped, which means that it can be used across many namespaces.

• (Cluster)ExternalSecret: This CRD defines the secret that should be synchronized into Kubernetes. It contains the reference to the SecretStore and the path to the secret in the external secret management system. ExternalSecret is namespaced, while the ClusterExternalSecret is cluster-scoped, which means that it can be used across many namespaces.

Setting Up the External Secrets Operator

Like the other two articles, we will use Pulumi to deploy the External Secrets Operator to our Kubernetes cluster. This time, I am going to use DigitalOcean as the cloud provider and Pulumi ESC as our external secret management system. But you can use any other cloud provider and external secret management system you like.

What is Pulumi ESC?

Pulumi ESC provides a comprehensive solution for managing secrets, environments, and configurations. While it integrates seamlessly with Pulumi IaC projects through pulumiConfig, exposing stored values to your Pulumi stacks, its functionality extends beyond this use case.

As a standalone service, Pulumi ESC supports diverse applications with dedicated SDKs for various programming languages. Additionally, it offers a CLI for command-line management of secrets and configurations, enabling context creation for CLI tools like Terraform.

This new secrets management and orchestration service can be employed both within and outside of Pulumi's Infrastructure as Code ecosystem. To explore Pulumi ESC's capabilities further, consult the official documentation.

Prerequisites

To follow this article, you will need the following:

• Pulumi CLI is installed.

• kubectl installed.

• optional K9s, if you want to quickly interact with your cluster.

• DigitalOcean account.

I will skip the whole Pulumi installation and config part. I explained this in my previous posts.

Create a new Pulumi project

In the previous articles, I used TypeScript and Go, but to show you that you can use a lot of different languages with Pulumi, it's time for Python.

We start by creating a new Pulumi project:

mkdir pulumi-external-secret-operator
cd pulumi-external-secret-operator
pulumi new python --force

Leave the default values and at the end you should see something like this:

pulumi new digitalocean-python --force
This command will walk you through creating a new Pulumi project.

Enter a value or leave blank to accept the (default), and press <ENTER>.
Press ^C at any time to quit.

Project name (pulumi-external-secret-operator):
Project description (A minimal DigitalOcean Python Pulumi program):
Created project 'pulumi-external-secret-operator'

Please enter your desired stack name.
To create a stack in an organization, use the format <org-name>/<stack-name> (e.g. `acmecorp/dev`).
Stack name (dev):
Created stack 'dev'

The toolchain to use for installing dependencies and running the program pip
Installing dependencies...

Creating virtual environment...
Finished creating virtual environment
Updating pip, setuptools, and wheel in virtual environment...
Requirement already satisfied: pip in ./venv/lib/python3.12/site-packages (24.2)
Collecting setuptools
  Using cached setuptools-75.1.0-py3-none-any.whl.metadata (6.9 kB)
Collecting wheel
  Using cached wheel-0.44.0-py3-none-any.whl.metadata (2.3 kB)
Using cached setuptools-75.1.0-py3-none-any.whl (1.2 MB)
Using cached wheel-0.44.0-py3-none-any.whl (67 kB)
Installing collected packages: wheel, setuptools
Successfully installed setuptools-75.1.0 wheel-0.44.0
Finished updating
Installing dependencies in virtual environment...
Collecting pulumi<4.0.0,>=3.0.0 (from -r requirements.txt (line 1))
Downloading pulumi-3.135.1-py3-none-any.whl (273 kB)
Downloading pulumi_digitalocean-4.33.0-py3-none-any.whl (377 kB)
Installing collected packages: arpeggio, six, semver, pyyaml, protobuf, grpcio, dill, debugpy, attrs, pulumi, parver, pulumi-digitalocean
Successfully installed arpeggio-2.0.2 attrs-24.2.0 debugpy-1.8.6 dill-0.3.9 grpcio-1.60.2 parver-0.5 protobuf-4.25.5 pulumi-3.135.1 pulumi-digitalocean-4.33.0 pyyaml-6.0.2 semver-2.13.0 six-1.16.0
Finished installing dependencies
Finished installing dependencies

Your new project is ready to go! ✨

To perform an initial deployment, run `pulumi up`

Writing the Pulumi code to deploy the External Secrets Operator

After the project is created, we can start writing our code. You should see the following files in your project:

tree -L 1
.
├── Pulumi.yaml
├── __main__.py
├── requirements.txt
└── venv

2 directories, 3 files

Add the pulumi-kubernetes dependencies to your requirements.txt file:

pulumi>=3.0.0,<4.0.0
pulumi-digitalocean>=4.0.0,<5.0.0
pulumi-kubernetes==4.18.1

Install the dependencies:

./venv/bin/pip install -r requirements.txt

Replace the content of the __main__.py file with the following code:

"""A DigitalOcean Python Pulumi program"""

import pulumi
import pulumi_digitalocean as digitalocean
import pulumi_kubernetes as kubernetes

do_cluster = digitalocean.KubernetesCluster(
    "do_cluster",
    name="esc-cluster",
    region="nyc1",
    version="1.31.1-do.1",
    destroy_all_associated_resources=True,
    node_pool=digitalocean.KubernetesClusterNodePoolArgs(
        name="default", size="s-2vcpu-2gb", node_count=1
    ),
)

do_k8s_provider = kubernetes.Provider(
    "do_k8s_provider",
    enable_server_side_apply=True,
    kubeconfig=do_cluster.kube_configs[0].apply(lambda config: config.raw_config),
)

namespace = kubernetes.core.v1.Namespace(
    "external-secrets",
    metadata={
        "name": "external-secrets",
    },
    opts=pulumi.ResourceOptions(provider=do_k8s_provider),
)

# Deploy a Helm release into the namespace
external_secrets = kubernetes.helm.v3.Release(
    "external-secrets",
    chart="external-secrets",
    version="0.10.4",  # Specify the version of the chart
    namespace=namespace.metadata["name"],
    repository_opts={
        "repo": "https://charts.external-secrets.io",
    },
    opts=pulumi.ResourceOptions(provider=do_k8s_provider),
)

# Deploy a secret into the namespace
pulumi_access_token = pulumi.Config().require("pulumi-pat")

my_secret = kubernetes.core.v1.Secret(
    "my-secret",
    metadata={
        "namespace": namespace.metadata["name"],
        "name": "pulumi-access-token",
    },
    string_data={
        "PULUMI_ACCESS_TOKEN": pulumi_access_token,
    },
    type="Opaque",
    opts=pulumi.ResourceOptions(provider=do_k8s_provider),
)

pulumi.export("kubeconfig", do_k8s_provider.kubeconfig)

This code creates a new DigitalOcean Kubernetes cluster. It has one node and a size of s-2vcpu-2gb. We also create a dedicated Kubernetes provider for this cluster as we need it later to deploy the External Secrets Operator, the ClusterSecretStore, and ExternalSecret.

Before we can deploy and create the cluster, we need to provide the DigitalOcean token. There are different ways to achieve this. You can set the token as an environment variable or use the Pulumi configuration.

I am going to use the most secure way and use Pulumi ESC to store the token and provide it to the Pulumi stack. To do this, you need to create a new Pulumi ESC environment. You can do this by running the following command:

pulumi env init <your-org>/eso-do-cluster/eso-dev

We define the DigitalOcean token inside the Pulumi ESC environment using following sytnax:

values:
  pulumiConfig:
    digitalocean:token:
      fn::secret: <your-do-token>
    pulumi-pat:
      fn::secret: <your-pulumi-pat>

Replace <your-do-pat> with your DigitalOcean token. You can find the token in your DigitalOcean account settings. And replace <your-pulumi-pat> with your Pulumi Personal Access Token.

To apply the configuration, run the pulumi env edit command and copy the above YAML into the editor:

pulumi env edit <your-org>/eso-do-cluster/eso-dev

Last step is to link the Pulumi ESC environment to the Pulumi stack by creating a new Pulumi.dev.yaml file:

cat <<EOF > Pulumi.dev.yaml
environment:
- eso-do-cluster/eso-dev
EOF

Now we can deploy the stack:

pulumi up

You can check that everything is running by running the following command:

pulumi stack output kubeconfig --show-secrets > kubeconfig.yaml
kubectl --kubeconfig kubeconfig.yaml get secret -n external-secrets pulumi-access-token -o jsonpath='{.data.PULUMI_ACCESS_TOKEN}' | base64 -d

You should see your Pulumi Personal Access Token printed to the console.

Create an external secret and use it in a deployment.

Now that the External Secrets Operator is running, we can create an ExternalSecret and use it in a deployment.

For this, we are going to create a new ESC project called eso-to-esc-app and a development environment:

pulumi env init <your-org>/eso-to-esc-app/dev

Add the following values to the Pulumi ESC environment:

values:
  app:
    hello: world
    hello-secret:
      fn::secret: world

Then run this command:

pulumi env edit <your-org>/eso-to-esc-app/dev

Now we can create the ClusterSecretStore:

"""A DigitalOcean Python Pulumi program"""

import pulumi
import pulumi_digitalocean as digitalocean
import pulumi_kubernetes as kubernetes

# Cut for brevity

cluster_secret_store = kubernetes.apiextensions.CustomResource(
    "cluster-secret-store",
    api_version="external-secrets.io/v1beta1",
    kind="ClusterSecretStore",
    metadata=kubernetes.meta.v1.ObjectMetaArgs(
        name="secret-store",
    ),
    spec={
        "provider": {
            "pulumi": {
                "organization": pulumi.get_organization(),
                "project": "eso-to-esc-app",
                "environment": "dev",
                "accessToken": {
                    "secretRef": {
                        "name": my_secret.metadata.name,
                        "key": "PULUMI_ACCESS_TOKEN",
                        "namespace": my_secret.metadata.namespace,
                    },
                },
            },
        },
    },
    opts=pulumi.ResourceOptions(provider=do_k8s_provider),
)

Finally we can create the ExternalSecret and wire it up to our demo application:

"""A DigitalOcean Python Pulumi program"""

import pulumi
import pulumi_digitalocean as digitalocean
import pulumi_kubernetes as kubernetes

# Cut for brevity

external_secret = kubernetes.apiextensions.CustomResource(
    "external-secret",
    api_version="external-secrets.io/v1beta1",
    kind="ExternalSecret",
    metadata=kubernetes.meta.v1.ObjectMetaArgs(
        name="esc-secret-store",
    ),
    spec={
        "dataFrom": [
            {
                "extract": {
                    "conversionStrategy": "Default",
                    "key": "app",
                }
            }
        ],
        "refreshInterval": "10s",
        "secretStoreRef": {
            "kind": cluster_secret_store.kind,
            "name": cluster_secret_store.metadata["name"],
        },
    },
    opts=pulumi.ResourceOptions(provider=do_k8s_provider),
)

hello_server_deployment = kubernetes.apps.v1.Deployment(
    "hello-server-deployment",
    metadata=kubernetes.meta.v1.ObjectMetaArgs(
        name="hello",
        labels={"app": "hello"},
    ),
    spec=kubernetes.apps.v1.DeploymentSpecArgs(
        replicas=1,
        selector=kubernetes.meta.v1.LabelSelectorArgs(
            match_labels={"app": "hello"},
        ),
        template=kubernetes.core.v1.PodTemplateSpecArgs(
            metadata=kubernetes.meta.v1.ObjectMetaArgs(
                labels={"app": "hello"},
            ),
            spec=kubernetes.core.v1.PodSpecArgs(
                containers=[
                    kubernetes.core.v1.ContainerArgs(
                        name="hello-server",
                        image="ghcr.io/dirien/hello-server/hello-server:latest",
                        env_from=[
                            kubernetes.core.v1.EnvFromSourceArgs(
                                secret_ref=kubernetes.core.v1.SecretEnvSourceArgs(
                                    name=external_secret.metadata["name"]
                                )
                            )
                        ],
                        ports=[
                            kubernetes.core.v1.ContainerPortArgs(
                                container_port=8080,
                            )
                        ],
                        resources=kubernetes.core.v1.ResourceRequirementsArgs(
                            limits=None,
                            requests=None,
                        ),
                    )
                ],
            ),
        ),
    ),
    opts=pulumi.ResourceOptions(provider=do_k8s_provider),
)

Deploy all the changes by running:

pulumi up

After the deployment is finished, you can check the logs of the hello-server pod by running:

pod_name=$(kubectl --kubeconfig kubeconfig.yaml get pods -l app=hello -o jsonpath='{.items[0].metadata.name}')
kubectl port-forward --kubeconfig kubeconfig.yaml $pod_name 8080:8080

Now you can access the hello-server by running:

curl http://localhost:8080/env/hello
curl http://localhost:8080/env/hello-secret

hello=world
hello-secret=world%

Bingo! We successfully retrieved the secret from the External Secrets Operator and used it in our application.

Housekeeping

Don't forget to clean up your resources by running:

pulumi destroy

Conclusion

The External Secrets Operator is a great tool to manage secrets in Kubernetes. It provides a secure and efficient way to manage secrets in a cloud-native environment, improving security, efficiency, and compliance when consuming secrets in your Kubernetes cluster. All this is done by not adding any more complexity to your applications and operations. Especially from an operational perspective, the External Secrets Operator is great as it follows the Kubernetes way of providing a declarative API to manage secrets and how it plays well with the GitOps workflow.

All in all, I highly recommend you give the External Secrets Operator a try and see how it can help your organization manage secrets in a secure and efficient manner.

Resources

• External Secrets Operator

• Pulumi ESC

Introduction

This blog post is all about running Rust-based functions on AWS Lambda. Will see, how much effort is required to get started and what the pros and cons are. If you are already familiar with AWS Lambdas, you may know that AWS Lambdas offers a variety of runtimes. A runtime provides a language-specific environment to run your function in. You can use this AWS provided runtimes, or you can build your own.

Now comes the interesting part for us: Because Rust is compiled to native code, we do not need a dedicated runtime to run our Rust code. We will use instead the Rust runtime client to build locally (or in your CI system) and then upload the binary using the AWS CLI by defining the AWS Lambda function. As runtime, we will use provided.al2 which is a runtime that does not include any language-specific dependencies.

The demo code? A simple function that returns as AI-generated Dad Joke. The function will give us some, hopefully, funny dad jokes.

Additionally, we cross-compile our lambda function to arm64 architecture to profit from the AWS Graviton processor. Graviton-based instances offer the best bang for your buck and are up to 40% cheaper than Intel-based solutions.

You are new to Rust and stumbled over this blog article? I have a whole series of blog posts about rust, if you are interested in more, check out my Rust 🦀 series:

Why AWS CLI?

You may ask yourself: Why we are not using an infrastructure as code tool like to define and deploy our AWS Lambda function?

The reason is simple: I want to keep the blog post as simple as possible. We will use Pulumi later in some more sophisticated blog posts. For now, we will use the AWS CLI to define our AWS Lambda function. Stay tuned for more to come.

Prerequisites

To follow this blog post, you should have a basic understanding of Rust and the Cargo build tool. If you are new to Rust check out my blog post:

Before we start, we need to make sure we have the following tools installed:

• Rust

• An IDE or text editor of your choice

• AWS account (free tier is enough)

• AWS CLI

• ChatGPT API key (No need for ChatGPT Plus)

Create a new Rust project

Before we can start with the rust code, we need to install the cargo-lambda Cargo plugin. As I am using a Mac, I will use brew to install the plugin.

If you are using a different OS, check out the Getting started section of the official documentation.

brew tap cargo-lambda/cargo-lambda
brew install cargo-lambda

Now we can create our function by running the cargo-lambda with the subcommand new.

cargo lambda new dadjoke

You will be asked a few questions. I have answered them as follows:

cargo lambda new dadjoke   
> Is this function an HTTP function? Yes

This will generate a new Rust project inside the folder named dadjoke.

Change into the newly created folder and add our chatgpt_rs crate using the cargo add command.

cd dadjoke
cargo add chatgpt_rs

Open the src/main.rs file and replace the existing content with the following code, as we want to access the ChatGPT API to generate AI-generated Dad Jokes for us.

use chatgpt::prelude::ChatGPT;
use chatgpt::types::CompletionResponse;
use lambda_http::{run, service_fn, Error, Request, Response};


async fn function_handler(_: Request) -> Result<Response<String>, Error> {
    let chatgpt_api_key = std::env::var("CHATGPT_API_KEY").expect("env variable `CHATGPT_API_KEY` should be set");

    let client = ChatGPT::new(chatgpt_api_key)?;

    let response: CompletionResponse = client
        .send_message("Imagine you are a dad and tell a good dad joke, tell only the joke without any other text, all in one line without line breaks:")
        .await.unwrap();

    let mut reponse_string = response.message().content.to_string();
    reponse_string = reponse_string.replace("\n", "");

    println!("{}", reponse_string);

    let resp = Response::builder()
        .status(200)
        .header("content-type", "text/html")
        .body(reponse_string)
        .map_err(Box::new)?;
    Ok(resp)
}

#[tokio::main]
async fn main() -> Result<(), Error> {
    tracing_subscriber::fmt()
        .with_max_level(tracing::Level::INFO)
        .with_target(false)
        .without_time()
        .init();

    run(service_fn(function_handler)).await
}

The main action happens in the function_handler function. The API key is provided via the environment variable CHATGPT_API_KEY. We will set this variable later when we deploy our function to AWS Lambda. We then make a non-streaming call to ChatGPT to give us a dad joke back. I use the following phrase to give the AI the right context from the start:

Imagine you are a dad and tell a good dad joke, tell only the joke without any other text, all in one line without line breaks:

Build and Test Our Function Locally

Before we deploy our function to AWS, we want to test it locally to be sure it works as expected. To do so, the cargo lambda subcommand watch offers us a very easy way to build our function and start a local HTTP server on the port 9000.

export CHATGPT_API_KEY=<YOUR_CHATGPT_API_KEY>
cargo lambda watch -a 0.0.0.0 -p 9000

Now we can invoke our function using cargo lambda invoke. This will send an HTTP request to our local server and print the response with handling all the event parameters for us.

cargo lambda invoke dadjoke --data-ascii '{}' -a 0.0.0.0

You should see something like this:

{"statusCode":200,"headers":{"content-type":"text/html"},"multiValueHeaders":{"content-type":["text/html"]},"body":"Response: Why don't skeletons fight each other? They don't have the guts!","isBase64Encoded":false}

Deploy our function to AWS Lambda

We are ready to deploy our function to AWS!

First, we need to build and bundle our function. We can do this, again, by using the cargo lambda with the subcommand build.

cargo lambda build --release --arm64 --output-format zip

This will create a zip file inside the target/lambda/release folder.

Now we can use the following AWS CLI command to create the execution role for our function and then create the function itself.

aws iam create-role --role-name rust-in-the-cloud-role --assume-role-policy-document '{"Version": "2012-10-17","Statement": [{ "Effect": "Allow", "Principal": {"Service": "lambda.amazonaws.com"}, "Action": "sts:AssumeRole"}]}'

After the role is created, we need to attach the AWSLambdaBasicExecutionRole policy to the role.

aws iam attach-role-policy --role-name rust-in-the-cloud-role --policy-arn arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole

With the role in place, we can now create our function. We will use the aws lambda create-function command to do so.

aws lambda create-function --function-name rust-in-the-cloud \
--handler bootstrap \
--zip-file fileb://./target/lambda/dadjoke/bootstrap.zip \
--runtime provided.al2 \
--role arn:aws:iam::<YOUR_AWS_ACCOUNT_ID>:role/rust-in-the-cloud-role \
--environment Variables={CHATGPT_API_KEY=<YOUR_CHATGPT_API_KEY>} \
--tracing-config Mode=Active

To get the role arn, you can use the aws iam list-roles command.

aws iam list-roles | grep rust-in-the-cloud-role

Now we can use the aws lambda invoke command to synchronously invoke our function.

aws lambda invoke --cli-binary-format raw-in-base64-out  \
--function-name rust-in-the-cloud \
--payload '{}' response.json

Now we can use the jq command to extract the response from the response.json file.

cat response.json | jq '.body'

You should see a dad joke like this:

"Why don't eggs tell jokes? Because they might crack up!"

Conclusion

We have seen how easy it is to run Rust-based functions in AWS Lambda. With the help of the cargo-lambda plugin, we can build and bundle our function and then use for example the AWS CLI to deploy it to AWS Lambda.

Let's have a look at the pros and cons of this approach:

Pros

• Easy to get started

• Lightning-fast cold start times and low memory footprint

• No need to build a custom runtime

But there are also some cons:

Cons

• I have the feeling that AWS is not supporting Rust as a first-class citizen.

• The documentation is not that great, and you need to figure some stuff out by yourself.

Overall, I am still very happy using Rust in AWS Lambda and I could see adding Rust-based functions to a polyglot microservice architecture in the future.

If you want to learn more, check out the official documentation here.

Housekeeping

To clean up, we can delete the function and the role.

aws lambda delete-function --function-name rust-in-the-cloud
aws iam detach-role-policy --role-name rust-in-the-cloud-role --policy-arn arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
aws iam delete-role --role-name rust-in-the-cloud-role

Introduction

My motivation to write this article comes from the recent blog post of Lily Cohen who had a severe data loss on her Kubernetes cluster.

During a routine GitOps cleanup, YAML manifests responsible for creating Kubernetes namespaces were moved to a directory not tracked by ArgoCD, leading to unintentional data deletion. Although backups were taken using Velero every six hours, the Persistent Volume Claim (PVC) data was not included due to a missing Restic flag, rendering the backups incomplete and the data irretrievable.

Let's have a look at Velero and Pulumi to back up and restore an EKS cluster including the persistent volumes.

What is Velero?

Velero is an open-source tool to backup and restore, perform disaster recovery, and migrate Kubernetes cluster resources and persistent volumes.

Velero gives us the following benefits:

• Disaster Recovery: Reduces time to recovery in case of infrastructure loss, data corruption, and/or service outages

• Data Migration: Enables cluster portability by easily migrating Kubernetes resources from one cluster to another and integrates with DevOps workflows to create ephemeral clones of Kubernetes namespaces

• Ephemeral Clusters: Provides a reliable tool to unlock new approaches to cluster lifecycle management treating clusters as "cattle"

Using Velero With Managed Kubernetes Services

Managed Kubernetes services like EKS, AKS, GKE, etc. are great as they take away the burden of managing the control plane. The etcd key-value store is managed by the cloud provider and therefore only accessible through the Kubernetes API Server. Here comes Velero into play.

Velero retrieves etcd data via the Kubernetes API Server, offering significant flexibility in backup options. You can filter which resources to back up based on criteria like namespace or label, and even choose to exclude certain resources from the backup.

Velero manages backup and restore tasks using Custom Resources (CRs) within the Kubernetes cluster. The Velero controller monitors these CRs to execute backup and restore procedures.

This Kubernetes-native methodology offers excellent opportunities to harness the broader Kubernetes ecosystem, including GitOps for delivery or admission controllers for implementing policy-as-code to avoid misconfigurations. Additionally, Velero's CLI can be integrated into existing CI/CD systems, allowing pipelines to trigger backup and restore actions on demand.

Velero Workflow in a Nutshell

Velero has two main components:

• A server that runs in your Kubernetes cluster

• A command-line client that runs locally

With the command-line client, we can create backups and restore them by creating CRs in the Kubernetes cluster.

Backup

1. The user initiates a call to the Kubernetes API server using the Velero CLI.

2. The API server generates a Custom Resource (CR) of kind backups.velero.io.

3. The Velero Controller then takes the following steps:

   • Verifies the presence of the CR objects.

   • Requests the relevant resources from the API server.

   • Compresses these resources and stores them in an S3 bucket.

Restore

1. The user initiates a restore request to the Kubernetes API server via the Velero CLI.

2. In response, the API server creates a Custom Resource (CR) for the restore operation, of kind restores.velero.io.

3. The Velero Controller then performs the following actions:

   • Checks for the existence of the specified Restore CR objects.

   • Fetches the corresponding compressed resources from the S3 bucket.

   • Decompress these resources and calls the API server to restore them into the Kubernetes cluster.

Prerequisites

Having covered the theoretical aspects of Velero, let's now dive into the practical portion of this blog article.

• Pulumi

• Pulumi Account - this is optional but convenient for handling the state of stack.

• AWS Account

• kubectl - Required to interact with the Kubernetes cluster. You can install it by following the instructions here.

• Go

• Velero CLI

I will use Golang as my programming language of choice. You can use of course any other language supported by Pulumi.

To create a Pulumi project, run the following commands:

mkdir pulumi-eks-velero
cd pulumi-eks-velero
pulumi new go --force

I am using the --force flag as I already created the directory beforehand.

Define the EKS Cluster

To deploy an EKS cluster, we will use the Pulumi EKS package. The package is available on the Pulumi Registry and is called pulumi-eks.

I am not going into much detail here on how to deploy an EKS cluster with Pulumi. Check the demo code for more details. The only thing I want to mention is that we need to activate the OpenID Connect (OIDC) provider for the cluster and install the AWS EBS CSI driver to support the dynamic provisioning of EBS volumes.

You can do this by setting the CreateOidcProvider flag to true when creating the cluster. We need this as we're going to use IRSA (IAM Roles for Service Accounts) to grant Velero access to the S3 bucket.

The EBS CSI driver installation consists of several steps:

• Create the IAM assume role for the EBS CSI driver

• Create the EBS IAM policy

• Deploy the EBS CSI driver via Helm

Here is the code for the EBS CSI driver installation:

package main

// omitted code for brevity

func main() {
    pulumi.Run(func(ctx *pulumi.Context) error {
        // omitted code for brevity
        ebsRole, err := iam.NewRole(ctx, "velero-ebs-role", &iam.RoleArgs{
            AssumeRolePolicy: pulumi.All(cluster.Core.OidcProvider().Arn(), cluster.Core.OidcProvider().Url()).ApplyT(func(args []interface{}) string {
                arn := args[0].(string)
                url := args[1].(string)
                assumeRolePolicy, _ := iam.GetPolicyDocument(ctx, &iam.GetPolicyDocumentArgs{
                    Statements: []iam.GetPolicyDocumentStatement{
                        {
                            Effect: pulumi.StringRef("Allow"),
                            Actions: []string{
                                "sts:AssumeRoleWithWebIdentity",
                            },
                            Principals: []iam.GetPolicyDocumentStatementPrincipal{
                                {
                                    Type: "Federated",
                                    Identifiers: []string{
                                        arn,
                                    },
                                },
                            },
                            Conditions: []iam.GetPolicyDocumentStatementCondition{
                                {
                                    Test: "StringEquals",
                                    Values: []string{
                                        ebsServiceAccount,
                                    },
                                    Variable: fmt.Sprintf("%s:sub", url),
                                },
                            },
                        },
                    },
                })
                return assumeRolePolicy.Json
            }).(pulumi.StringOutput),
        })
        if err != nil {
            return err
        }

        ebsPolicyFile, _ := os.ReadFile("./policies/ebs-iam-policy.json")

        ebsIAMPolicy, err := iam.NewPolicy(ctx, "velero-ebs-policy", &iam.PolicyArgs{
            Policy: pulumi.String(ebsPolicyFile),
        }, pulumi.DependsOn([]pulumi.Resource{ebsRole}))
        if err != nil {
            return err
        }

        ebsRolePolicyAttachment, err := iam.NewRolePolicyAttachment(ctx, "velero-esb-role-attachment", &iam.RolePolicyAttachmentArgs{
            PolicyArn: ebsIAMPolicy.Arn,
            Role:      ebsRole.Name,
        }, pulumi.DependsOn([]pulumi.Resource{ebsIAMPolicy}))
        if err != nil {
            return err
        }
        _, err = helm.NewRelease(ctx, "aws-ebs-csi-driver", &helm.ReleaseArgs{
            Chart:       pulumi.String("aws-ebs-csi-driver"),
            Version:     pulumi.String("2.22.0"),
            Namespace:   pulumi.String("kube-system"),
            ForceUpdate: pulumi.Bool(true),
            RepositoryOpts: helm.RepositoryOptsArgs{
                Repo: pulumi.String("https://kubernetes-sigs.github.io/aws-ebs-csi-driver"),
            },
            Values: pulumi.Map{
                "controller": pulumi.Map{
                    "serviceAccount": pulumi.Map{
                        "annotations": pulumi.Map{
                            "eks.amazonaws.com/role-arn": ebsRole.Arn,
                        },
                    },
                },
                "storageClasses": pulumi.Array{
                    pulumi.Map{
                        "name":              pulumi.String("ebs-sc"),
                        "volumeBindingMode": pulumi.String("WaitForFirstConsumer"),
                    },
                },
                "volumeSnapshotClasses": pulumi.Array{
                    pulumi.Map{
                        "name":           pulumi.String("ebs-vsc"),
                        "deletionPolicy": pulumi.String("Delete"),
                    },
                },
            },
        }, pulumi.Provider(provider), pulumi.DependsOn([]pulumi.Resource{ebsRolePolicyAttachment}))
        if err != nil {
            return err
        }
        return nil
    })
}

Important to note here that we create the service account for the EBS CSI driver with the eks.amazonaws.com/role-arn annotation. This annotation is required by the EBS CSI driver to assume the IAM role we created before. Additionally, we create the storage class and volume snapshot class for the EBS CSI driver as well. You can decide if you want to keep this or deploy the storage class and volume snapshot class as separate Kubernetes resources.

Define the S3 Bucket

Next, we need to create the S3 bucket where we will store the backups. We will use the Pulumi AWS package to create the S3 bucket and the corresponding bucket public access block.

package main

// omitted code for brevity

func main() {
    pulumi.Run(func(ctx *pulumi.Context) error {
        // omitted code for brevity
        bucket, err := s3.NewBucket(ctx, "velero-bucket", &s3.BucketArgs{
            Bucket: pulumi.String("velero-eks-bucket-dirien"),
            Tags: pulumi.StringMap{
                "Name": pulumi.String("velero-eks-bucket-dirien"),
            },
        })
        if err != nil {
            return err
        }
        _, err = s3.NewBucketPublicAccessBlock(ctx, "velero-bucket-public-access-block", &s3.BucketPublicAccessBlockArgs{
            Bucket:                bucket.ID(),
            BlockPublicAcls:       pulumi.Bool(true),
            BlockPublicPolicy:     pulumi.Bool(true),
            IgnorePublicAcls:      pulumi.Bool(true),
            RestrictPublicBuckets: pulumi.Bool(true),
        })
        if err != nil {
            return err
        }
        return nil
    })
}

One thing to note here is that we set the BlockPublicAcls, BlockPublicPolicy, IgnorePublicAcls, and the RestrictPublicBuckets flags to true. This is a best practice to prevent the S3 bucket from being publicly accessible.

Define the Velero Installation

At last, we're ready to deploy Velero. The first step is setting up the IRSA to give Velero the required permissions to access the S3 bucket. This process is similar to installing the EBS CSI driver, albeit with different policies. Once that's done, we'll proceed to deploy Velero using its Helm chart.

package main

// omitted code for brevity

func main() {
    pulumi.Run(func(ctx *pulumi.Context) error {
        // omitted code for brevity
        saRole, err := iam.NewRole(ctx, "velero-sa-role", &iam.RoleArgs{
            AssumeRolePolicy: pulumi.All(cluster.Core.OidcProvider().Arn(), cluster.Core.OidcProvider().Url()).ApplyT(func(args []interface{}) string {
                arn := args[0].(string)
                url := args[1].(string)
                assumeRolePolicy, _ := iam.GetPolicyDocument(ctx, &iam.GetPolicyDocumentArgs{
                    Statements: []iam.GetPolicyDocumentStatement{
                        {
                            Actions: []string{
                                "sts:AssumeRoleWithWebIdentity",
                            },
                            Conditions: []iam.GetPolicyDocumentStatementCondition{
                                {
                                    Test: "StringEquals",
                                    Values: []string{
                                        veleroServiceAccount,
                                    },
                                    Variable: fmt.Sprintf("%s:sub", url),
                                },
                            },
                            Principals: []iam.GetPolicyDocumentStatementPrincipal{
                                {
                                    Type: "Federated",
                                    Identifiers: []string{
                                        arn,
                                    },
                                },
                            },
                            Effect: pulumi.StringRef("Allow"),
                        },
                    },
                })
                return assumeRolePolicy.Json
            }).(pulumi.StringOutput),
        })
        if err != nil {
            return err
        }

        veleroIAMPolicy, err := iam.NewPolicy(ctx, "velero-iam-policy", &iam.PolicyArgs{
            Policy: pulumi.All(bucket.Bucket).ApplyT(func(args []interface{}) string {
                name := args[0].(string)
                veleroIAMPolicy, _ := iam.GetPolicyDocument(ctx, &iam.GetPolicyDocumentArgs{
                    Statements: []iam.GetPolicyDocumentStatement{
                        {
                            Effect: pulumi.StringRef("Allow"),
                            Actions: []string{
                                "ec2:DescribeVolumes",
                                "ec2:DescribeSnapshots",
                                "ec2:CreateTags",
                                "ec2:CreateVolume",
                                "ec2:CreateSnapshot",
                                "ec2:DeleteSnapshot",
                            },
                            Resources: []string{
                                "*",
                            },
                        },
                        {
                            Effect: pulumi.StringRef("Allow"),
                            Actions: []string{
                                "s3:GetObject",
                                "s3:DeleteObject",
                                "s3:PutObject",
                                "s3:AbortMultipartUpload",
                                "s3:ListMultipartUploadParts",
                            },
                            Resources: []string{
                                fmt.Sprintf("arn:aws:s3:::%s/*", name),
                            },
                        },
                        {
                            Effect: pulumi.StringRef("Allow"),
                            Actions: []string{
                                "s3:ListBucket",
                            },
                            Resources: []string{
                                fmt.Sprintf("arn:aws:s3:::%s", name),
                            },
                        },
                    },
                })
                return veleroIAMPolicy.Json
            }).(pulumi.StringOutput),
        })
        if err != nil {
            return err
        }
        _, err = iam.NewRolePolicyAttachment(ctx, "velero-iam-role-attachment", &iam.RolePolicyAttachmentArgs{
            PolicyArn: veleroIAMPolicy.Arn,
            Role:      saRole.Name,
        })

        _, err = helm.NewRelease(ctx, "velero", &helm.ReleaseArgs{
            Chart:   pulumi.String("velero"),
            Version: pulumi.String("5.0.2"),
            RepositoryOpts: helm.RepositoryOptsArgs{
                Repo: pulumi.String("https://vmware-tanzu.github.io/helm-charts"),
            },
            Namespace:       pulumi.String(veleroNamespace),
            CreateNamespace: pulumi.Bool(true),
            Values: pulumi.Map{
                "serviceAccount": pulumi.Map{
                    "server": pulumi.Map{
                        "name": pulumi.String("velero"),
                        "annotations": pulumi.StringMap{
                            "eks.amazonaws.com/role-arn": saRole.Arn,
                        },
                    },
                },
                "credentials": pulumi.Map{
                    "useSecret": pulumi.Bool(false),
                },
                "podSecurityContext": pulumi.Map{
                    "fsGroup": pulumi.Int(65534),
                },
                "initContainers": pulumi.Array{
                    pulumi.Map{
                        "name":            pulumi.String("velero-plugin-for-aws"),
                        "image":           pulumi.String("velero/velero-plugin-for-aws:v1.7.1"),
                        "imagePullPolicy": pulumi.String("IfNotPresent"),
                        "volumeMounts": pulumi.Array{
                            pulumi.Map{
                                "name":      pulumi.String("plugins"),
                                "mountPath": pulumi.String("/target"),
                            },
                        },
                    },
                },
                "configuration": pulumi.Map{
                    "backupStorageLocation": pulumi.Array{
                        pulumi.Map{
                            "name":     pulumi.String("velero-k8s"),
                            "provider": pulumi.String("aws"),
                            "bucket":   bucket.Bucket,
                            "prefix":   pulumi.String("velero"),
                            "config": pulumi.Map{
                                "region": pulumi.String("eu-central-1"),
                            },
                            "default": pulumi.Bool(true),
                        },
                    },
                    "volumeSnapshotLocation": pulumi.Array{
                        pulumi.Map{
                            "name":     pulumi.String("velero-k8s-snapshots"),
                            "provider": pulumi.String("aws"),
                            "config": pulumi.Map{
                                "region": pulumi.String("eu-central-1"),
                            },
                        },
                    },
                },
            },
        }, pulumi.Provider(provider))
        if err != nil {
            return err
        }
        return nil
    })
}

Similar to the EBS CSI driver setup, we first create a service account for Velero, annotating it with eks.amazonaws.com/role-arn. Thanks to the IRS approach, there's no need to create a separate AWS secret; you can simply set credentials.useSecret to false in the Helm chart values.

To enable backup and restore functionalities for EBS volumes, it's crucial to install the AWS plugin for Velero. This can be achieved using initContainers in the Helm chart.

The final step involves establishing the backup storage location and the volume snapshot location. We will use the previously created S3 bucket for storing backups and set the default AWS volume snapshot location, all of which can be configured in the Helm chart values (configurations).

Deploy the stack

Now we can deploy the stack. Run the following commands to deploy the stack:

pulumi up

Make sure you have the AWS CLI configured with the correct credentials and region before you run the command.

Testing Velero backup and restore

With the stack now deployed, it's time to test the backup and restore functionalities. Ensure that you have the Velero CLI installed, as we'll be using it to execute the backup and restore processes.

Before proceeding with creating a backup, you'll first need to set up a namespace and a simple pod that includes a persistent volume claim (PVC).

apiVersion: v1
kind: Namespace
metadata:
  name: velero-test
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: ebs-claim
  namespace: velero-test
spec:
  accessModes:
  - ReadWriteOnce
  storageClassName: ebs-sc
  resources:
    requests:
      storage: 1Gi
---
apiVersion: v1
kind: Pod
metadata:
  name: app
  namespace: velero-test
spec:
  containers:
  - name: app
    image: centos
    command: ["/bin/sh"]
    args: ["-c", "while true; do sleep 5; done"]
    volumeMounts:
    - name: persistent-storage
      mountPath: /data
  volumes:
  - name: persistent-storage
    persistentVolumeClaim:
      claimName: ebs-claim

Save the code above in a file called test.yaml and run the following command to create the Pod. You need to get the kubeconfig for the cluster first using the following command:

pulumi stack output kubeconfig --show-secrets  > kubeconfig.yaml

Now you can create the pod using the following command:

kubectl --kubeconfig=kubeconfig.yaml apply -f test.yaml

This will create a Pod in the velero-test namespace with a persistent volume claim. To create some data in the volume, we can exec into the Pod and create a file in the volume:

kubectl exec -it app -n velero-test -- sh -c 'echo "Hi, today is $(date)" > /data/test.txt'

Check if the file is created:

kubectl exec -it app -n velero-test -- sh -c 'cat /data/test.txt'

You should see the following output:

Hi, today is Tue Sep  5 09:21:53 UTC 2023

This should be enough to simulate to show the backup and restore functionality of Velero.

Create a backup

To create a backup, we need to create a backup CR in the Kubernetes cluster. We can do this by running the following command:

velero backup create velero-test-backup --include-namespaces velero-test --wait

This will create a backup of the velero-test namespace and wait until the backup is completed.

Backup request "velero-test-backup" submitted successfully.
Waiting for backup to complete. You may safely press ctrl-c to stop waiting - your backup will continue in the background.
..
Backup completed with status: Completed. You may check for more information using the commands `velero backup describe velero-test-backup` and `velero backup logs velero-test-backup`.

To display details about the backup, run the following command with the --details flag.

velero backup describe velero-test-backup --details

You should see similar output like this.

Name:         velero-test-backup
Namespace:    velero
Labels:       velero.io/storage-location=velero-k8s
Annotations:  velero.io/source-cluster-k8s-gitversion=v1.27.4-eks-2d98532
              velero.io/source-cluster-k8s-major-version=1
              velero.io/source-cluster-k8s-minor-version=27+

Phase:  Completed


Namespaces:
  Included:  velero-test
  Excluded:  <none>

Resources:
  Included:        *
  Excluded:        <none>
  Cluster-scoped:  auto

Label selector:  <none>

Storage Location:  velero-k8s

Velero-Native Snapshot PVs:  auto

TTL:  720h0m0s

CSISnapshotTimeout:    10m0s
ItemOperationTimeout:  1h0m0s

Hooks:  <none>

Backup Format Version:  1.1.0

Started:    2023-09-03 15:45:28 +0200 CEST
Completed:  2023-09-03 15:45:30 +0200 CEST

Expiration:  2023-10-03 15:45:28 +0200 CEST

Total items to be backed up:  6
Items backed up:              6

Resource List:
  v1/ConfigMap:
    - velero-test/kube-root-ca.crt
  v1/Namespace:
    - velero-test
  v1/PersistentVolume:
    - pvc-1395e1fd-eae6-4bb2-922a-e5dd5839a696
  v1/PersistentVolumeClaim:
    - velero-test/ebs-claim
  v1/Pod:
    - velero-test/app
  v1/ServiceAccount:
    - velero-test/default

Velero-Native Snapshots:
  pvc-1395e1fd-eae6-4bb2-922a-e5dd5839a696:
    Snapshot ID:        snap-0595966a00a41783b
    Type:               gp3
    Availability Zone:  eu-central-1b
    IOPS:               <N/A>

Navigate to the AWS console and inspect the S3 bucket. Inside, you'll likely find a folder named velero containing the backup files.

Simulate a Data Loss

To simulate a data loss, we can simply delete the velero-test namespace:

kubectl delete ns velero-test

Check if the namespace is deleted:

kubectl get ns

Oh no, the namespace is gone. We need to restore it from the backup. Urgently! Production is down!

Restore a backup

To restore the backup, we need to create a restore CR in the Kubernetes cluster. We can do this by running the following command:

velero restore create velero-test-restored --from-backup velero-test-backup --wait

This will restore the backup and wait until the restore is completed.

Restore request "velero-test-restored" submitted successfully.
Waiting for restore to complete. You may safely press ctrl-c to stop waiting - your restore will continue in the background.
.
Restore completed with status: Completed. You may check for more information using the commands `velero restore describe velero-test-restored` and `velero restore logs velero-test-restored`.

We can also check the details of the restore similar to the backup:

velero restore describe velero-test-restored --details

velero restore describe velero-test-restored --details
Name:         velero-test-restored
Namespace:    velero
Labels:       <none>
Annotations:  <none>

Phase:                       Completed
Total items to be restored:  6
Items restored:              6

Started:    2023-09-03 18:18:26 +0200 CEST
Completed:  2023-09-03 18:18:27 +0200 CEST

... omitted for brevity

Now we can check if the everything is restored, including the data in the volume:

kubectl exec -it app -n velero-test -- sh -c 'cat /data/test.txt'

And yes, the data is back, the timestamp is the same as before:

kubectl exec -it app -n velero-test -- sh -c 'cat /data/test.txt'
Hi, today is Tue Sep  5 09:21:53 UTC 2023

Housekeeping

Since the stack is no longer needed, you can proceed to delete it. Execute the following command to remove the stack:

pulumi destroy

This will delete all the resources created by the stack.

Conclusion

In this blog post, we've explored how to deploy Velero using Pulumi and how to carry out backup and restore operations on an EKS cluster with EBS volumes. Velero is a great tool to handle disaster recovery and data migration scenarios.

In further blog posts, I will show some more advanced use cases of Velero including the integration into GitOps engines like ArgoCD or Flux.

Stay tuned for more!

Resources

Introduction

Thanks to the posts of Aurélie Vache, I discovered the OVH Managed Kubernetes Service.

I had already heard about it, but I had never tried it. So I decided to give it a try and use Pulumi to deploy a Kubernetes cluster in the OVH data center in Strasbourg.

What could be better than a Kubernetes cluster in Strasbourg, one of the European Union capitals, to deploy a Minecraft server?

Prerequisites

• Pulumi

• Pulumi Account - this is optional, but convenient to handle the state of stack.

• OVH Account - this is required to use the OVH Managed Kubernetes Service.

• kubectl - this is required to interact with the Kubernetes cluster. You can install it by following the instructions here.

• Go

• Minecraft client - this is required to connect to the Minecraft server. You can download it here.

Create a Pulumi Project

Everything starts with a Pulumi project. Let's create one and we will use the Golang as our programming language of choice.

mkdir pulumi-ovh-kube
cd pulumi-ovh-kube
pulumi new go --force

For the sake of simplicity, we can keep all the default values in the wizard.

Now, we need to install the OVH provider for Pulumi. We can do that by running the following command:

go get github.com/dirien/pulumi-ovh/sdk

Now, we can start writing our code. Open the main.go file and replace the content with the following code:

package main

import (
    "github.com/dirien/pulumi-ovh/sdk/go/ovh/cloudproject"
    "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
    "github.com/pulumi/pulumi/sdk/v3/go/pulumi/config"
)

func main() {
    pulumi.Run(func(ctx *pulumi.Context) error {

        serviceName := config.Require(ctx, "serviceName")

        mykube, err := cloudproject.NewKube(ctx, "mykube", &cloudproject.KubeArgs{
            ServiceName: pulumi.String(serviceName),
            Name:        pulumi.String("myFirstOVHKubernetesCluster"),
            Region:      pulumi.String("SBG5"),
        })
        if err != nil {
            return err
        }
        ctx.Export("kubeconfig", mykube.Kubeconfig)

        nodePool, err := cloudproject.NewKubeNodePool(ctx, "mykubenodepool", &cloudproject.KubeNodePoolArgs{
            ServiceName:   pulumi.String(serviceName),
            KubeId:        mykube.ID(),
            Name:          pulumi.String("default"),
            Autoscale:     pulumi.BoolPtr(true),
            DesiredNodes:  pulumi.Int(1),
            MinNodes:      pulumi.Int(1),
            MaxNodes:      pulumi.Int(2),
            FlavorName:    pulumi.String("d2-8"),
            MonthlyBilled: pulumi.BoolPtr(false),
        })
        if err != nil {
            return err
        }
        ctx.Export("nodePoolId", nodePool.ID())
        return nil
    })
}

This code will create a Kubernetes cluster in the OVH data center in Strasbourg (SBG5). Additionally, it will create a node pool with 2 nodes. The node pool will be configured to scale between 1 and 2 nodes and I want that it starts with 1 (desiredNodes).

The node pool will use the flavor d2-8 which is a flavor with 4 vCPUs and 8GB of RAM from the so-called Discover range.

These instances are perfect for test, development and sandbox environments and Minecraft is a sandbox game, so it fits perfectly!

Add the Kubernetes provider

Before deploying applications in our Kubernetes clusters, we need to add the Kubernetes provider to our project as we are going to run the Minecraft server as a container.

I wrote a dedicated blog post about the usage of Helm charts in Pulumi, go check it out:

We need to add the Kubernetes provider like this:

go get github.com/pulumi/pulumi-kubernetes/sdk/v4

And after we successfully added the Kubernetes provider, we can add the following code to the main.go file:

package main

import (
    "github.com/dirien/pulumi-ovh/sdk/go/ovh/cloudproject"
    k8s "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes"
    "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/helm/v3"
    "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
    "github.com/pulumi/pulumi/sdk/v3/go/pulumi/config"
)

func main() {
    pulumi.Run(func(ctx *pulumi.Context) error {
        // truncated for brevity

        k8sProvider, err := k8s.NewProvider(ctx, "k8s", &k8s.ProviderArgs{
            Kubeconfig: mykube.Kubeconfig,
        }, pulumi.DependsOn([]pulumi.Resource{nodePool, mykube}))
        if err != nil {
            return err
        }
        _, err = helm.NewRelease(ctx, "minecraft", &helm.ReleaseArgs{
            Chart:   pulumi.String("minecraft"),
            Version: pulumi.String("4.9.3"),
            RepositoryOpts: &helm.RepositoryOptsArgs{
                Repo: pulumi.String("https://itzg.github.io/minecraft-server-charts"),
            },
            CreateNamespace: pulumi.Bool(true),
            Namespace:       pulumi.String("minecraft"),
            Values: pulumi.Map{
                "minecraftServer": pulumi.Map{
                    "eula":        pulumi.Bool(true),
                    "motd":        pulumi.String("OVH Strasbourg - Minecraft Server"),
                    "serviceType": pulumi.String("LoadBalancer"),
                },
            },
        }, pulumi.Provider(k8sProvider))
        if err != nil {
            return err
        }
        return nil
    })
}

This code will programmatically create a Kubernetes provider using the kubeconfig of the Kubernetes cluster we just created.

One thing to note here is the pulumi.DependsOn([]pulumi.Resource{nodePool, mykube}) part. This will make sure that the Kubernetes provider is created after the Kubernetes cluster AND the node pool are created. Because we need the be able to schedule pods on the Kubernetes cluster. Only the Kubernetes API up and running is not enough.

Secondly, it will create a Helm release using the itzg/minecraft-server-charts chart. This chart will create a vanilla Java Minecraft server with a load balancer service, so that we can connect to the Minecraft server from the internet.

Deploy the stack

Don't forget to set the serviceName configuration value to the name of your OVH Public Cloud project and add this to the Pulumi.yaml file.

config:
serviceName: <your-service-name>

Also, don't forget to set the OVH_APPLICATION_KEY, OVH_APPLICATION_SECRET, OVH_CONSUMER_KEY and OVH_ENDPOINT environment variables.

Now, we can deploy the stack by running the following command:

go mod tidy
pulumi up

This will take a few minutes, because it needs to create the Kubernetes cluster and the node pool.

Connect to the Minecraft server

To connect to the Minecraft server, we need to get the IP address of the load balancer service. We can do that by running the following command:

pulumi stack output kubeconfig --show-secrets  > kubeconfig.yaml
kubectl --kubeconfig=kubeconfig.yaml get svc -n minecraft -o jsonpath='{.items[0].status.loadBalancer.ingress[0].ip}'

This will give us the kubeconfig file we defined as output in the main.go file, and the kubectl command will be used to get the IP address of the load balancer service.

Now, we can connect to the Minecraft server using the IP address we just got:

Housekeeping

To clean up the resources we created, we can run the following command:

pulumi destroy

This will destroy all the resources we created.

Conclusion

We saw how easy it is to use Pulumi to create infrastructure as code.

This time, we used the OVH Public Cloud and it was a breeze to create a Kubernetes cluster and deploy a Minecraft server on it.

Introduction

In this blog post, we will have a look at how to cross-compile your Rust applications using cross-rs and GitHub Actions. We will also have a look at how to use the cross-rs Docker image to cross-compile your Rust applications locally.

But before we dig into the details, let's have a look at multi-platform support in Rust.

Motivation for multi-platform support

Nowadays, it is common to use different operating systems and architectures. We have IoT devices that run on ARM processors and servers which run on x86 processors. Or Apple computers that run on Apple Silicon processors. And then we have Windows, Linux and macOS. Enough reasons to support multiple platforms when developing software.

But now comes the downside: Most OS APIs are not compatible with each other. This difference in APIs is the reason why have to create platform-dependent code.

How Rust supports multi-platform

The good news is that Rust makes it easy to write multi-platform code. Rust has a built-in macro called cfg which enables us the conditional compilation of code. It supports a lot of options so we can easily write platform-dependent parts of our code depending on the target platform.

https://doc.rust-lang.org/reference/conditional-compilation.html

Some examples of cfg:

#[cfg(target_os = "linux")]
fn main() {
    println!("This is Linux");
}

#[cfg(target_os = "macos")]
fn main() {
    println!("This is macOS");
}

The cfg macro also supports any, all and not:

• any - If any of the given predicates is true, the code is included.

• all - If all of the given predicates are true, the code is included.

• not - If the given predicate is false, the code is included.

#[cfg(any(target_os = "linux", target_os = "macos"))]
fn main() {
    println!("This is Linux or macOS");
}

#[cfg(all(target_os = "linux", target_arch = "x86_64"))]
fn main() {
    println!("This is Linux on x86_64");
}

#[cfg(not(target_os = "windows"))]
fn main() {
    println!("This is not Windows");
}

Rust also supports platform-dependent dependencies. We can use the target attribute to specify dependencies for a specific platform. The cfg syntax is also supported.

[dependencies]
# This dependency is only used on Linux
[target.'cfg(unix)'.dependencies]
openssl = "1.0.1"

Same for dev-dependencies and build-dependencies:

[dev-dependencies]
# This dev-dependency is only used on Linux
[target.'cfg(unix)'.dev-dependencies]
openssl = "1.0.1"

[build-dependencies]
# This build-dependency is only used on Linux
[target.'cfg(unix)'.build-dependencies]
openssl = "1.0.1"

Rust support tiers

Rust is organized in support tiers when it comes to multi-platform support. The Rust team provides different levels of support for different platforms. The support tiers are:

• Tier 1 - Tier 1 platforms are "guaranteed to work", this is the highest level of support

• Tier 2 - Tier 2 platforms are "guaranteed to build", but not necessarily to pass all tests

• Tier 3 - Tier 3 platforms are those for which the Rust code has support, but which are not built or tested automatically. So there is no guarantee that they work.

According to the Rust team, the following platforms are Tier 1:

• aarch64-unknown-linux-gnu

• i686-pc-windows-gnu

• i686-pc-windows-msvc

• i686-unknown-linux-gnu

• x86_64-apple-darwin

• x86_64-pc-windows-gnu

• x86_64-pc-windows-msvc

• x86_64-unknown-linux-gnu

cross-rs to the rescue

With cross-rs we have an easy way to cross-compile our Rust applications. cross-rs works by using pre-made Dockerfiles to build and run your application inside a Docker container. The list of supported platforms is quite long, have a look at the following link for more information:

Here is a short selection of supported platforms:

1. Dockerfile.x86_64-unknown-linux-gnu

2. Dockerfile.x86_64-pc-windows-gnu

3. Dockerfile.aarch64-unknown-linux-gnu

4. Dockerfile.i686-unknown-linux-gnu

5. Dockerfile.i686-pc-windows-gnu

6. Dockerfile.armv7-unknown-linux-gnueabihf

7. Dockerfile.riscv64gc-unknown-linux-gnu

8. Dockerfile.mips64-unknown-linux-gnuabi64

9. Dockerfile.powerpc64le-unknown-linux-gnu

10. Dockerfile.x86_64-unknown-linux-musl

Pretty impressive, right?

Let us create a simple demo project to see cross-rs in action.

Prerequisites

• Rust

• An IDE or text editor of your choice

• Docker

• GitHub account

• GitHub CLI

Initialize the demo project

The demo project is a simple Rust application that prints a FIGlet text to the console. We use clap to parse the command line arguments. That's it. Nothing fancy!

cargo init --bin figctl

Then we add the clap dependency:

cargo add clap --features derive

And then we add a figlet-rs dependency:

cargo add figlet-rs

Now we can add the following code to src/main.rs:

use clap::{Parser, Args};
use figlet_rs::FIGfont;

#[derive(Parser, Debug)]
struct FigletCtl {
    message: String,
}

fn main() {
    let args = FigletCtl::parse();
    let standard_font = FIGfont::standard().unwrap();
    let figure = standard_font.convert(args.message.as_str());
    println!("{}", figure.unwrap());
}

If you run the application now, you should see the following output:

➜ cargo run -q -- 'Hello World!'
  _   _          _   _            __        __                 _       _   _ 
 | | | |   ___  | | | |   ___     \ \      / /   ___    _ __  | |   __| | | |
 | |_| |  / _ \ | | | |  / _ \     \ \ /\ / /   / _ \  | '__| | |  / _` | | |
 |  _  | |  __/ | | | | | (_) |     \ V  V /   | (_) | | |    | | | (_| | |_|
 |_| |_|  \___| |_| |_|  \___/       \_/\_/     \___/  |_|    |_|  \__,_| (_)

Cross-compile the demo project

You can install cross-rs with the following command:

cargo install cross --git https://github.com/cross-rs/cross

And let test it with a simple example.

cross build --target aarch64-unknown-linux-gnu

Now you should have a target/aarch64-unknown-linux-gnu/debug/figctl binary. Let's run it and see what happens.

➜ ./target/aarch64-unknown-linux-gnu/debug/figctl "Hello World!"
zsh: exec format error: ./target/aarch64-unknown-linux-gnu/debug/figctl

The binary is not executable on our host system. But we can run it inside a Docker container.

docker run --rm -it -v $(pwd):/app  --platform=linux/arm64 -w /app rust ./target/aarch64-unknown-linux-gnu/debug/figctl 'Hello World!'
  _   _          _   _            __        __                 _       _   _ 
 | | | |   ___  | | | |   ___     \ \      / /   ___    _ __  | |   __| | | |
 | |_| |  / _ \ | | | |  / _ \     \ \ /\ / /   / _ \  | '__| | |  / _` | | |
 |  _  | |  __/ | | | | | (_) |     \ V  V /   | (_) | | |    | | | (_| | |_|
 |_| |_|  \___| |_| |_|  \___/       \_/\_/     \___/  |_|    |_|  \__,_| (_)

It works! So let's see how we can use cross-rs to build our application for multiple platforms with GitHub Actions.

GitHub Actions

We need to create a file called .github/workflows/ci.yml and add the following content:

name: build and release

on:
  workflow_dispatch:
  release:
    types: [ created ]

permissions:
  contents: write

jobs:
  build:
    name: ${{ matrix.platform.os_name }} with rust ${{ matrix.toolchain }}
    runs-on: ${{ matrix.platform.os }}
    strategy:
      fail-fast: false
      matrix:
        platform:
          - os_name: Linux-aarch64
            os: ubuntu-20.04
            target: aarch64-unknown-linux-musl
            bin: figctl-linux-arm64
          - os_name: Linux-x86_64
            os: ubuntu-20.04
            target: x86_64-unknown-linux-gnu
            bin: figctl-linux-amd64
          - os_name: Windows-x86_64
            os: windows-latest
            target: x86_64-pc-windows-msvc
            bin: figctl-amd64.exe
          - os_name: macOS-x86_64
            os: macOS-latest
            target: x86_64-apple-darwin
            bin: figctl-darwin-amd64
          - os_name: macOS-aarch64
            os: macOS-latest
            target: aarch64-apple-darwin
            bin: figctl-darwin-arm64
        toolchain:
          - stable
    steps:
      - uses: actions/checkout@v3
      - name: Build binary
        uses: houseabsolute/actions-rust-cross@v0
        with:
          command: "build"
          target: ${{ matrix.platform.target }}
          toolchain: ${{ matrix.toolchain }}
          args: "--locked --release"
          strip: true
      - name: Rename binary (linux and macos)
        run: mv target/${{ matrix.platform.target }}/release/figctl target/${{ matrix.platform.target }}/release/${{ matrix.platform.bin }}
        if: matrix.platform.os_name != 'Windows-x86_64'
      - name: Rename binary (windows)
        run: mv target/${{ matrix.platform.target }}/release/figctl.exe target/${{ matrix.platform.target }}/release/${{ matrix.platform.bin }}
        if: matrix.platform.os_name == 'Windows-x86_64'
      - name: Generate SHA-256
        run: shasum -a 256 target/${{ matrix.platform.target }}/release/${{ matrix.platform.bin }} | cut -d ' ' -f 1 > target/${{ matrix.platform.target }}/release/${{ matrix.platform.bin }}.sha256
      - name: Release binary and SHA-256 checksum to GitHub
        uses: softprops/action-gh-release@v1
        with:
          files: |
            target/${{ matrix.platform.target }}/release/${{ matrix.platform.bin }}
            target/${{ matrix.platform.target }}/release/${{ matrix.platform.bin }}.sha256

This workflow will build the application for the following platforms:

• Linux-aarch64

• Linux-x86_64

• Windows-x86_64

• macOS-x86_64

• macOS-aarch64

It will also create a GitHub release for each platform.

Let's push the changes to GitHub and create a new release.

git add .
git commit -m "Add all the things"
git push

Now go to the GitHub repository and create a new release or use the GitHub CLI.

➜ gh release create v0.1.0 
? Title (optional) My figctl cli
? Release notes Leave blank
? Is this a prerelease? No
? Submit? Publish release
https://github.com/dirien/rust-cross-compile/releases/tag/v0.1.0

This will trigger the GitHub Actions workflow and build the application for all the platforms.

Conclusion

Creating a cross-platform application with Rust is easy as Rust has a great toolchain and ecosystem. But it's not always easy to build the application for multiple platforms. With cross-rs and GitHub Actions, we can build our application for multiple platforms and with GitHub Releases, we can distribute the application to our users.

Links

• https://github.com/cross-rs/cross

• https://doc.rust-lang.org/nightly/rustc/platform-support.html

• https://doc.rust-lang.org/cargo/reference/specifying-dependencies.html

As usual, the link to the repo:

Introduction

In this blog article, we will discover how we can leverage Pulumi and the kubernetes provider to write and deploy a validating admission policy in Kubernetes. And all of these during the creation of the Kubernetes cluster.

This is interesting for a variety of reasons. It enables the immediate installation of essential policies right after the initiation of the cluster. Not only ensures this alignment with your set of company rules from the outset but also guarantees some kind of peace of mind. You can rest assured that any tools or services deployed on the cluster subsequently will abide by these policies.

The application of policies takes precedence over any service deployment, including potential policy tools like Kyverno or Gatekeeper (OPA). This is crucial because the sequence of subsequent tool deployments can't always be relied upon, thus these policies must be established before anything else is deployed.

This feature itself is currently in alpha and only available in Kubernetes starting from version 1.26 and above and requires that the ValidatingAdmissionPolicy feature gate is enabled.

Depending on the way you deploy your cluster, this might be the case already. For example, if you use the kubeadm tool to bootstrap your cluster, you can enable the feature gate by adding the following to your kubeadm-config.yaml file:

apiVersion: kubeadm.k8s.io/v1beta3
kind: ClusterConfiguration
featureGates:
  ValidatingAdmissionPolicy: true

But this is not the topic of this article. We will focus on the provision of a managed Kubernetes cluster with the possibility to enable the feature gate during the creation of the cluster. Recently Scaleway added the ValidatingAdmissionPolicy to their list of supported features.

To see all the features supported by Scaleway, you can run the following command using their scw CLI:

scw k8s version get 1.26.0
Name    1.26.0
Label   Kubernetes 1.26.0
Region  fr-par

Available Kubelet Arguments:
map[containerLogMaxFiles:uint16 containerLogMaxSize:quantity cpuCFSQuota:bool cpuCFSQuotaPeriod:duration cpuManagerPolicy:enum:none|static enableDebuggingHandlers:bool imageGCHighThresholdPercent:uint32 imageGCLowThresholdPercent:uint32 maxPods:uint16]

Available CNIs:
[cilium calico kilo]

Available Ingresses:
[none]

Available Container Runtimes:
[containerd]

Available Feature Gates:
[HPAScaleToZero GRPCContainerProbe ReadWriteOncePod ValidatingAdmissionPolicy CSINodeExpandSecret]

Available Admission Plugins:
[PodNodeSelector AlwaysPullImages PodTolerationRestriction]

As you can see, the ValidatingAdmissionPolicy feature gate is available, we just have to keep it in mind when creating our Pulumi code.

It is also worth mentioning that the validating admission policies use CEL (Common Expression Language) to declare its rules. So what is CEL?

CEL?

The Common Expression Language (CEL) implements common semantics for expression evaluation, enabling different applications to more easily interoperate.

In short, CEL is a language that allows you to write expressions that can be evaluated. It is capable of creating complex policies that can be used in a variety of use cases with dealing with Webhooks at all.

The full spec is available on GitHub:

In my former blog article, I wrote how to write a DIY policy engine using webhooks and a server. Feel free to check this article, it is very interesting.

Prerequisites

To follow along with this tutorial, you will need the following:

• A Scaleway account

• Pulumi CLI installed on your machine (installation instructions)

• Free Pulumi account

• node.js

Optionally, you can also install the following tools:

• kubectl

• The Scaleway CLI

Create a new Pulumi project

You may spot it already, that we will use TypeScript as the programming language for this project. But feel free to use any of the Pulumi-supported languages.

To create a new Pulumi project, run the following command in your terminal:

mkdir pulumi-validating-admission-policy
cd pulumi-validating-admission-policy
pulumi new typescript --force

You will be prompted with a wizard to create a new Pulumi project. You can keep the default values for all questions except the last one.

Note: You may have to run pulumi login before you can create a new project, depending if you already have a Pulumi account or not.

pulumi new typescript --force
This command will walk you through creating a new Pulumi project.

Enter a value or leave blank to accept the (default), and press <ENTER>.
Press ^C at any time to quit.

project name: (pulumi-validating-admission-policy) 
project description: (A minimal TypeScript Pulumi program) 
Created project 'pulumi-validating-admission-policy'

Please enter your desired stack name.
To create a stack in an organization, use the format <org-name>/<stack-name> (e.g. `acmecorp/dev`).
stack name: (dev) 
Created stack 'dev'

Installing dependencies...


added 193 packages, and audited 194 packages in 14s

65 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities
Finished installing dependencies

Your new project is ready to go! ✨

To perform an initial deployment, run `pulumi up`

Now we can add the scaleway provider to our project. To do so, run the following command:

npm install @ediri/scaleway@2.25.1 --save-exact

And we need also the kubernetes provider:

npm install @pulumi/kubernetes@4.0.3 --save-exac

I like to use the --save-exact flag to ensure that the exact version of the provider is used. Reduce the risk of potential breaking changes/bugs, if a new version of the provider is released. But feel free to omit this flag.

Create a new Kubernetes cluster

After having the libraries ready, we can add some configuratrion to your Pulumi.yaml file. We will add the following configuration:

name: pulumi-validating-admission-policy
...
config:
  scaleway:region: "fr-par"
  scaleway:zone: "fr-par-1"
  cluster:version: "1.27"
  cluster:auto_upgrade: true
  node:node_type: "PLAY2-NANO"
  node:auto_scale: false
  node:node_count: 1
  node:auto_heal: true

This set some default values for our cluster, like the region, zone, version, node type, node count and so on. You can keep the default values or change them to your needs. Every Pulumi stack can have its configuration values and override the default values.

After having the configuration in place, we can add the following code to our index.ts file:

import * as pulumi from "@pulumi/pulumi";
import * as scaleway from "@ediri/scaleway";

const clusterConfig = new pulumi.Config("cluster")

const kapsule = new scaleway.K8sCluster("pulumi-validating-admission-policy-cluster", {
    version: clusterConfig.require("version"),
    cni: "cilium",
    deleteAdditionalResources: true,
    featureGates: [
        "ValidatingAdmissionPolicy",
    ],
    autoUpgrade: {
        enable: clusterConfig.requireBoolean("auto_upgrade"),
        maintenanceWindowStartHour: 3,
        maintenanceWindowDay: "monday"
    },
});

const nodeConfig = new pulumi.Config("node")

new scaleway.K8sPool("pulumi-validating-admission-policy-node-pool", {
    nodeType: nodeConfig.require("node_type"),
    size: nodeConfig.requireNumber("node_count"),
    autoscaling: nodeConfig.requireBoolean("auto_scale"),
    autohealing: nodeConfig.requireBoolean("auto_heal"),
    clusterId: kapsule.id,
});

export const kapsuleName = kapsule.name;
export const kubeconfig = pulumi.secret(kapsule.kubeconfigs[0].configFile);

Before we start to deploy our validating admission policy, let's have a look at the code.

The first resource we create is of type K8sCluster. This resource creates a new Kubernetes cluster on Scaleway. We use the configuration values to set the properties of the cluster.

The featureGates property is the one we are interested in. Here we activate the ValidatingAdmissionPolicy plugin.

The second resource we create is a K8sPool. This resource creates a new node pool for our cluster. Next to the definition of the node type, aka the instance type, we also set the size property to our node_count configuration value.

The last two lines of the code are used to export the name of the cluster and the kubeconfig.

Writing Validating Admission Policies

Now that we have our cluster defined, we can head over to create some examples of validating admission policies.

First, we create a Pulumi Kubernetes provider. This provider gets the kubeconfig from our cluster and uses it to connect to the cluster.

import * as k8s from "@pulumi/kubernetes";

// omitted code for brevity

const provider = new k8s.Provider("k8s-provider", {
    kubeconfig: kubeconfig,
}, {dependsOn: [kapsule, nodePool]});

As the next step, we can finally start to write our validating admission policy. The policy itself will do a basic check if a team label is set on the Deployment and StatefulSet metadata tag. If not, we will reject the resource creation and return an error message.

const teamLabel = new k8s.admissionregistration.v1alpha1.ValidatingAdmissionPolicy("pulumi-validating-admission-policy-0", {
    metadata: {
        name: "team-label",
    },
    spec: {
        failurePolicy: "Fail",
        matchConstraints: {
            resourceRules: [
                {
                    apiGroups: ["apps"],
                    apiVersions: ["v1"],
                    operations: ["CREATE", "UPDATE"],
                    resources: ["deployments"],
                },
                {
                    apiGroups: ["apps"],
                    apiVersions: ["v1"],
                    operations: ["CREATE", "UPDATE"],
                    resources: ["statefulsets"],
                }
            ]
        },
        matchConditions: [
            {
                name: "team-label",
                expression: `has(object.metadata.namespace) && !(object.metadata.namespace.startsWith("kube-"))`
            }
        ],
        validations: [
            {
                expression: `has(object.metadata.labels.team)`,
                message: "Team label is missing.",
                reason: "Invalid",
            },
            {
                expression: `has(object.spec.template.metadata.labels.team)`,
                message: "Team label is missing from pod template.",
                reason: "Invalid",
            }
        ]
    }
}, {provider: provider});

There is a lot of code in this example, so let's have a more in-depth look at it:

• We set the failurePolicy to Fail. This means that the resource creation will fail if the validation fails. The other option is Ignore, which will ignore the validation and create the resource.

• The matchConstraints property defines the resources, which should be validated. In our case, we want to validate Deployment and StatefulSet resources.

• The matchConditions property defines the conditions, which should be met, to run the validation. In our case, we want to validate only resources in namespaces, which are not kube-*. This helps us to even add more fine-grained control over the resources, which should be validated.

• Last but not least, the validations property defines the actual validation. In our case, we check if the team label is set on the resource and the pod template.

Now we need to connect the validating admission policy to a specific scope or context. We can do this by creating a ValidatingAdmissionPolicyBinding resource.

Note: You can bind one validating admission policy to multiple scopes or contexts.

const teamLabelBinding = new k8s.admissionregistration.v1alpha1.ValidatingAdmissionPolicyBinding("pulumi-validating-admission-policy-1", {
    metadata: {
        name: "team-label-binding",
    },
    spec: {
        policyName: teamLabel.metadata.name,
        validationActions: ["Deny"],
        matchResources: {}
    }
}, {provider: provider});

Here we set the policyName to the name of the validating admission policy we created before. One interesting property is the validationActions property. Here we can define, how validations of a policy should be enforced. We have the following options:

• Audit: This will only audit the validation and not enforce it and will be added to the audit event.

• Deny: This will deny the request and return an error message.

• Warn: This will warn the user, but will not deny the request.

The last property we need to set is the matchResources property. This property defines the resources, in our case the value and this means all resources, which should be validated by the validating admission policy.

Deploy the Cluster and the Validating Admission Policies

Now that we have our cluster and our validating admission policies defined, we can deploy them. To do so, run the following command:

pulumi up
Previewing update (dev)

     Type                                                                                  Name                                                   Plan       
 +   pulumi:pulumi:Stack                                                                   pulumi-validating-admission-policy-dev                 create     
 +   ├─ scaleway:index:K8sCluster                                                          pulumi-validating-admission-policy-cluster             create     
 +   ├─ scaleway:index:K8sPool                                                             pulumi-validating-admission-policy-node-pool           create     
 +   ├─ pulumi:providers:kubernetes                                                        k8s-provider                                           create     
 +   ├─ kubernetes:admissionregistration.k8s.io/v1alpha1:ValidatingAdmissionPolicy         pulumi-validating-admission-policy-team-label          create     
 +   └─ kubernetes:admissionregistration.k8s.io/v1alpha1:ValidatingAdmissionPolicyBinding  pulumi-validating-admission-policy-binding-team-label  create     


Outputs:
    kapsuleName: "pulumi-validating-admission-policy-cluster-d786e2b"
    kubeconfig : output<string>

Resources:
    + 6 to create

Do you want to perform this update?  [Use arrows to move, type to filter]
  yes
> no
  details

And confirm the deployment with yes. After a few minutes, the deployment should be finished and you should see similar output:

Do you want to perform this update? yes
Updating (dev)

...

Resources:
    + 6 created

Duration: 5m39s

Test the Validating Admission Policy

Now that we have our cluster and our validating admission policies deployed, we can test them. To do so, we will create a Deployment resource without the team label. First, we need to get the kubeconfig from the cluster. We can do this by running this command:

pulumi stack output kubeconfig --show-secrets -s dev > kubeconfig

Create a new file called deployment.yaml and add this as content:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: frontend
  labels:
    environment: production
spec:
  selector:
    matchLabels:
      app: guestbook
      tier: frontend
  replicas: 2
  template:
    metadata:
      labels:
        app: guestbook
        tier: frontend
    spec:
      containers:
        - name: php-redis
          image: gcr.io/google-samples/gb-frontend:v4
          resources:
            requests:
              cpu: 100m
              memory: 100Mi
            limits:
              cpu: "3"
              memory: 4Gi

Create the Deployment resource by executing this command:

kubectl apply -f deployment.yaml --kubeconfig kubeconfig

This should fail with a similar error message:

The deployments "frontend" is invalid: : ValidatingAdmissionPolicy 'team-label' with binding 'team-label-binding' denied request: Team label is missing.

Create More Complex Validating Admission Policy

In the previous example, we created a simple validating admission policy, which checks if the team label is set. Useful but still pretty basic.

Let us build a more complex validating admission policy. We will check in the next policy several different rules for the Deployment resource using multiple expression in the validations field.

const prodReadyPolicy = new k8s.admissionregistration.v1alpha1.ValidatingAdmissionPolicy("pulumi-validating-admission-policy-prod-ready", {
    metadata: {
        name: "prod-ready-policy",
    },
    spec: {
        failurePolicy: "Fail",
        matchConstraints: {
            resourceRules: [
                {
                    apiGroups: ["apps"],
                    apiVersions: ["v1"],
                    operations: ["CREATE", "UPDATE"],
                    resources: ["deployments"],
                }]
        },
        validations: [
            {
                expression: `object.spec.template.spec.containers.all(
    c, has(c.resources) && has(c.resources.limits) && has(c.resources.limits.cpu)
)`,
                message: "No CPU resource limits specified for any container.",
                reason: "Invalid",
            },
            {
                expression: `object.spec.template.spec.containers.all(
    c, has(c.resources) && has(c.resources.limits) && has(c.resources.limits.memory)
)`,
                message: "No memory resource limits specified for any container.",
                reason: "Invalid",
            },
            {
                expression: `object.spec.template.spec.containers.all(
    c, !c.image.endsWith(':latest')
)`,
                message: "Image tag must not be latest.",
                reason: "Invalid",
            },
            {
                expression: `object.spec.template.spec.containers.all(
    c, c.image.startsWith('myregistry.azurecr.io')
)`,
                message: "Image must be pulled from myregistry.azurecr.io.",
            },
            {
                expression: `object.spec.template.spec.containers.all(
    c, has(c.securityContext)
)`,
                message: "Security context is missing.",
            },
            {
                expression: `object.spec.template.spec.containers.all(
c, has(c.readinessProbe) && has(c.livenessProbe)
)`,
                message: "No health checks configured.",
            },
            {
                expression: `object.spec.template.spec.containers.all(
    c, !('securityContext' in c) || !('privileged' in c.securityContext) || c.securityContext.privileged == false
)`,
                message: "Privileged containers are not allowed.",
            },
            {
                expression: `object.spec.template.spec.initContainers.all(
    c, !('securityContext' in c) || !('privileged' in c.securityContext) || c.securityContext.privileged == false
)`,
                message: "Privileged init containers are not allowed.",
            }]
    }
}, {provider: provider});

const prodReadyPolicyBinding = new k8s.admissionregistration.v1alpha1.ValidatingAdmissionPolicyBinding("pulumi-validating-admission-policy-prod-ready-binding", {
    metadata: {
        name: "prod-ready-policy-binding",
    },
    spec: {
        policyName: prodReadyPolicy.metadata.name,
        validationActions: ["Deny"],
        matchResources: {
            objectSelector: {
                matchLabels: {
                    "environment": "production",
                }
            }
        }
    }
}, {provider: provider});

Let us go through this monster step by step and we focus only on the validations property as the rest is similar to the previous example:

• object.spec.template.spec.containers.all(c, has(c.resources) && has(c.resources.limits) && has(c.resources.limits.cpu)): This expression checks if all containers have a CPU resource limit set.

• object.spec.template.spec.containers.all(c, has(c.resources) && has(c.resources.limits) && has(c.resources.limits.memory)): This expression checks if all containers have a memory resource limit set.

• object.spec.template.spec.containers.all(c, !c.image.endsWith(':latest')): This expression checks if the image tag is not latest. We enforce this to make sure that we do not use the latest tag in production.

• object.spec.template.spec.containers.all(c, c.image.startsWith('myregistry.azurecr.io')): This expression checks if the image is pulled from myregistry.azurecr.io. This can be useful if you want to enforce that only images from a specific registry are used.

• object.spec.template.spec.containers.all(c, has(c.securityContext)): This expression checks if all containers have a security context set.

• object.spec.template.spec.containers.all(c, has(c.readinessProbe) && has(c.livenessProbe)): This expression checks if all containers have a readiness and liveness probe configured.

• object.spec.template.spec.containers.all(c, !('securityContext' in c) || !('privileged' in c.securityContext) || c.securityContext.privileged == false): This expression checks that no container is privileged.

• object.spec.template.spec.initContainers.all(c, !('securityContext' in c) || !('privileged' in c.securityContext) || c.securityContext.privileged == false): This expression checks that no init container is privileged.

Parameter resources

Now we come to an even more advanced topic. In the previous example, we hard-coded the policy configuration within the definition. With the property paramKind we can define a custom Kubernetes resource that contains the policy configuration.

But we need to create a custom resource definition (CRD) for our parameter resource before.

const prodReadyCRD = new k8s.apiextensions.v1.CustomResourceDefinition("pulumi-validating-admission-policy-prod-ready-crd", {
    metadata: {
        name: "prodreadychecks.pulumi.com",
    },
    spec: {
        group: "pulumi.com",
        versions: [
            {
                name: "v1",
                served: true,
                storage: true,
                schema: {
                    openAPIV3Schema: {
                        type: "object",
                        properties: {
                            spec: {
                                type: "object",
                                properties: {
                                    registry: {
                                        type: "string",
                                    },
                                    version: {
                                        type: "string",
                                    },
                                    privileged: {
                                        type: "boolean",
                                    }
                                }
                            }
                        }
                    }
                }
            }
        ],
        scope: "Namespaced",
        names: {
            plural: "prodreadychecks",
            singular: "prodreadycheck",
            kind: "ProdReadyCheck",
            shortNames: ["prc"],
        },
    },
}, {provider: provider});

const prodReadyCR = new k8s.apiextensions.CustomResource("pulumi-validating-admission-policy-prod-ready-crd-validation", {
    apiVersion: "pulumi.com/v1",
    kind: "ProdReadyCheck",
    metadata: {
        name: "prodreadycheck-validation",
    },
    spec: {
        registry: "myregistry.azurecr.io/",
        version: "latest",
        privileged: false,
    }

}, {provider: provider, dependsOn: [prodReadyCRD]});

The CRD defines the schema of the parameter resource. In our case, we define a schema with three properties:

• registry: The registry from which the image must be pulled.

• version: The image version to block.

• privileged: If the privileged flag is allowed.

And then we create a custom resource (CR) out of this.

We can now use this CR in our policy, for this example I will create a new policy and binding:

const prodReadyPolicyParam = new k8s.admissionregistration.v1alpha1.ValidatingAdmissionPolicy("pulumi-validating-admission-policy-prod-ready-param", {
    metadata: {
        name: "prod-ready-policy-param",
    },
    spec: {
        failurePolicy: "Fail",
        paramKind: {
            apiVersion: prodReadyCR.apiVersion,
            kind: prodReadyCR.kind,
        },
        matchConstraints: {
            resourceRules: [
                {
                    apiGroups: ["apps"],
                    apiVersions: ["v1"],
                    operations: ["CREATE", "UPDATE"],
                    resources: ["deployments"],
                }]
        },
        validations: [
            {
                expression: `object.spec.template.spec.containers.all(
    c, has(c.resources) && has(c.resources.limits) && has(c.resources.limits.cpu)
)`,
                message: "No CPU resource limits specified for any container.",
                reason: "Invalid",
            },
            {
                expression: `object.spec.template.spec.containers.all(
    c, has(c.resources) && has(c.resources.limits) && has(c.resources.limits.memory)
)`,
                message: "No memory resource limits specified for any container.",
                reason: "Invalid",
            },
            {
                expression: `object.spec.template.spec.containers.all(
    c, !c.image.endsWith(params.spec.version)
)`,
                messageExpression: "'Image tag must not be ' + params.spec.version",
                reason: "Invalid",
            },
            {
                expression: `object.spec.template.spec.containers.all(
    c, c.image.startsWith(params.spec.registry)
)`,
                reason: "Invalid",
                messageExpression: "'Registry only allowed from: ' + params.spec.registry",
            },
            {
                expression: `object.spec.template.spec.containers.all(
    c, has(c.securityContext)
)`,
                message: "Security context is missing.",
            },
            {
                expression: `object.spec.template.spec.containers.all(
c, has(c.readinessProbe) && has(c.livenessProbe)
)`,
                message: "No health checks configured.",
            },
            {
                expression: `object.spec.template.spec.containers.all(
    c, !('securityContext' in c) || !('privileged' in c.securityContext) || c.securityContext.privileged == params.spec.privileged
)`,
                message: "Privileged containers are not allowed.",
            },
            {
                expression: `object.spec.template.spec.initContainers.all(
    c, !('securityContext' in c) || !('privileged' in c.securityContext) || c.securityContext.privileged == params.spec.privileged
)`,
                message: "Privileged init containers are not allowed.",
            }]
    }
}, {provider: provider});

const prodReadyPolicyBindingParam = new k8s.admissionregistration.v1alpha1.ValidatingAdmissionPolicyBinding("pulumi-validating-admission-policy-prod-ready-binding-param", {
    metadata: {
        name: "prod-ready-policy-binding-param",
    },
    spec: {
        policyName: prodReadyPolicyParam.metadata.name,
        validationActions: ["Deny"],
        paramRef: {
            name: prodReadyCR.metadata.name,
            namespace: prodReadyCR.metadata.namespace,
        },
        matchResources: {
            objectSelector: {
                matchLabels: {
                    "environment": "production",
                }
            }
        }
    }
}, {provider: provider});

The policy is the same as in the previous example, but now we use the parameter resource in the policy and in the binding resource. To access the value we use the params.spec.xxx CEL path, where xxx stands for the property we defined in our CR.

Note: To get better error messages, we use the messageExpression property instead of the message property.

Now we can test the new policy by creating a deployment with a container that violates the policy and we should see a message similar to this:

The deployments "frontend" is invalid: : ValidatingAdmissionPolicy 'prod-ready-policy-param' with binding 'prod-ready-policy-binding-param' denied request: Image tag must not be latest

Housekeeping

To clean up all resources created by this example, run the following command:

pulumi destroy

Conclusion

We have seen how to use the ValidatingAdmissionPolicy resource to create a policy that validates the resources using CEL expressions. We have also seen how to use parameter resources to make the policy more flexible and reusable.

It is very nice to have a policy functionality inbuilt in Kubernetes without the need to install a separate policy engine. Especially when you want to ensure that the policy right from the start of the cluster is available.

And: You don't need to handle Webhooks, which is a big plus for me. You can easily render your cluster useless when your webhook is not available or not working correctly.

Links

• https://kubernetes.io/docs/reference/access-authn-authz/validating-admission-policy/#getting-started-with-validating-admission-policy

• https://minikube.sigs.k8s.io/docs/handbook/config/#:~:text=in%20constants.go-,Enabling%20feature%20gates,is%20the%20status%20of%20it

• https://www.pulumi.com/registry/packages/kubernetes/api-docs/admissionregistration/v1alpha1/

In this blog post, we're going to explore how to use Testcontainers as part of our integration testing strategy in Rust. To have hands-on experience, we're going to build a simple web application that exposes a REST API to manage cars. The cars are stored in a MongoDB database and as a web framework, we're going to use Actix-Web.

I highly recommend my previous blog post about the actix-web framework

Or check my whole Rust learning journey

What is Testcontainers?

Testcontainers is an open-source framework to provide throwaway instances of dependencies such as databases, message brokers, or any other service that can be started in a Docker container. It's available for many programming languages such as Java, Python, Go, and Rust and allows us to write test code that allows the user to start and stop containers.

This has several advantages:

• We can run tests against real components like PostgreSQL instead of an H2 in-memory database. This allows us to use PostgreSQL-specific features like JSONB columns or full-text search.

• We can mock AWS services with Localstack. This allows us to test our code against AWS services without the need to create real resources in the cloud.

• We can run our tests also in an offline environment.

• We can also test better some edge cases like network failures or slow responses from the database.

In this blog post, we will use the Rust version of Testcontainers, which is called rust-testcontainers.

Testcontainers offers also preconfigured implementations called modules for different databases and services. As they are not available for Rust, I skipped them in this blog post.

The Testing Pyramid

Before we head over to the implementation, let's talk about the different testing approaches. There are three different approaches to testing strategies:

• The Test Ice Cream Cone

• The Test Pyramid

• The Practical Test Pyramid

The Test Ice Cream Cone

The Test Ice Cream Cone is a testing strategy that is often used by companies that are new to testing. The problem with this approach is that we have a lot of manual tests that are expensive to maintain and slow to execute.

The Test Ice Cream Cone is an anti-pattern and should be avoided at all costs.

The Test Pyramid

The Test Pyramid is a testing strategy that was introduced by Mike Cohn in his book "Succeeding with Agile". The pyramid shows three levels of tests: small, medium, and large.

Unit tests are the primary level of the pyramid. They are small, fast, and cheap to maintain. They are focused on testing the logic in the code.

Service tests are the second level of the pyramid. They are medium-sized, slower, and more expensive to maintain. They are not as productive as unit tests.

And the third level of the pyramid is UI tests. They are large, slow, and expensive to maintain as they are very fragile since every change in the UI can break the tests.

One of the problems with the Test Pyramid is that it's not clear what service tests are which leads to the situation that developers jump directly to UI tests.

The Practical Test Pyramid

The Practical Test Pyramid is an improved version of the Test Pyramid. It was introduced by Alister Scott and emphasizes more on medium-level tests and manual exploratory testing.

The Services Tests from the Test Pyramid are replaced by Component Tests, Integration Tests, and Contract Tests.

The next improvement of the Test Pyramid is that is now more clear that manual testing is also part of the testing strategy. As we can never be 100% sure that our tests are covering all edge cases, we need to add Manual Exploratory Testing on top of our automated tests.

If you find a bug with manual testing, you should write a new test to cover this case.

Add Testcontainers to our Project

I will not go into the details of the existing code for setting up the project. If you want to know more about this, feel free to browse the code on GitHub.

To add Testcontainers to our project, we need to add the following dependency to our project:

cargo add testcontainers

Now open the main.rs file and add following code at the end of the file:

#[cfg(test)]
mod tests {
    use std::env;
    use std::io::Read;
    use std::thread::sleep;
    use super::*;
    use actix_web::http::StatusCode;
    use actix_web::test;
    use actix_web::test::TestRequest;
    use testcontainers::{clients, Image};
    use testcontainers::core::{ExecCommand, WaitFor};
    use testcontainers::images::generic::GenericImage;
    use crate::model::CarType;
    use crate::model::Car;

    #[actix_web::test]
    async fn test_index() {
        let app = test::init_service(App::new().service(index)).await;
        let req = TestRequest::default().to_request();
        let resp = test::call_service(&app, req).await;
        assert_eq!(StatusCode::OK, resp.status());
    }

    #[actix_web::test]
    async fn test_healthcheck() {
        let app = test::init_service(App::new().service(healthcheck)).await;
        let req = TestRequest::default().uri("/health").to_request();
        let resp = test::call_service(&app, req).await;
        assert_eq!(StatusCode::OK, resp.status());
    }

    #[actix_web::test]
    async fn test_get_cars() {
        let docker = clients::Cli::default();
        let msg = WaitFor::message_on_stdout("server is ready");
        let generic = GenericImage::new("mongo", "6.0.7").with_wait_for(msg.clone())
            .with_env_var("MONGO_INITDB_DATABASE", "cars_info")
            .with_env_var("MONGO_INITDB_ROOT_USERNAME", "root")
            .with_env_var("MONGO_INITDB_ROOT_PASSWORD", "root")
            .with_exposed_port(27017);

        let node = docker.run(generic);
        let port = node.get_host_port_ipv4(27017);
        println!("Port: {}", port);

        let data = setup(Config::new_mongodb_uri(format!("mongodb://root:root@localhost:{}", port))).await;
        let app = test::init_service(App::new().app_data(data.clone()).service(get_cars).service(create_car)).await;
        let req = TestRequest::default().uri("/cars").to_request();
        let resp = test::call_service(&app, req).await;
        assert_eq!(StatusCode::OK, resp.status());
        let result: Vec<Car> = test::read_body_json(resp).await;
        assert_eq!(result.len(), 0);

        let post = create_one_test_car();
        let resp = test::call_service(&app, post.to_request()).await;
        assert_eq!(StatusCode::OK, resp.status());
        let result: Car = test::read_body_json(resp).await;
        assert_eq!(result.name, "Test");

        let req = TestRequest::default().uri("/cars").to_request();
        let resp = test::call_service(&app, req).await;
        assert_eq!(StatusCode::OK, resp.status());
        let result: Vec<Car> = test::read_body_json(resp).await;
        assert_eq!(result.len(), 1);
        assert_eq!(result[0].name, "Test");
    }

    #[actix_web::test]
    async fn test_get_car() {
        let docker = clients::Cli::default();
        let msg = WaitFor::message_on_stdout("server is ready");
        let generic = GenericImage::new("mongo", "6.0.7").with_wait_for(msg.clone())
            .with_env_var("MONGO_INITDB_DATABASE", "cars_info")
            .with_env_var("MONGO_INITDB_ROOT_USERNAME", "root")
            .with_env_var("MONGO_INITDB_ROOT_PASSWORD", "root");

        let node = docker.run(generic);
        let port = node.get_host_port_ipv4(27017);

        let data = setup(Config::new_mongodb_uri(format!("mongodb://root:root@localhost:{}", port))).await;
        let app = test::init_service(App::new().app_data(data.clone()).service(get_cars).service(create_car).service(get_car)).await;


        let create_car_req = create_one_test_car();
        let resp = test::call_service(&app, create_car_req.to_request()).await;
        assert_eq!(StatusCode::OK, resp.status());
        let new_car: CarDto = test::read_body_json(resp).await;
        assert_eq!(new_car.name, "Test");

        let get_car_req = TestRequest::get().uri(format!("/cars/{}", new_car.id.unwrap()).as_str()).to_request();
        let resp = test::call_service(&app, get_car_req).await;
        assert_eq!(StatusCode::OK, resp.status());
        let result: CarDto = test::read_body_json(resp).await;
        assert_eq!(result.name, new_car.name);
    }

    #[actix_web::test]
    async fn test_delete_car() {
        let docker = clients::Cli::default();
        let msg = WaitFor::message_on_stdout("server is ready");
        let generic = GenericImage::new("mongo", "6.0.7").with_wait_for(msg.clone())
            .with_env_var("MONGO_INITDB_DATABASE", "cars_info")
            .with_env_var("MONGO_INITDB_ROOT_USERNAME", "root")
            .with_env_var("MONGO_INITDB_ROOT_PASSWORD", "root");

        let node = docker.run(generic);
        let port = node.get_host_port_ipv4(27017);

        let data = setup(Config::new_mongodb_uri(format!("mongodb://root:root@localhost:{}", port))).await;
        let app = test::init_service(App::new().app_data(data.clone())
            .service(get_cars).service(create_car).service(get_car)
            .service(delete_car)).await;

        let create_car_req = create_one_test_car();
        let resp = test::call_service(&app, create_car_req.to_request()).await;
        assert_eq!(StatusCode::OK, resp.status());
        let new_car: CarDto = test::read_body_json(resp).await;
        assert_eq!(new_car.name, "Test");

        let new_car_id = new_car.id.unwrap();
        let get_car_req = TestRequest::get().uri(format!("/cars/{}", new_car_id).as_str()).to_request();
        let resp = test::call_service(&app, get_car_req).await;
        assert_eq!(StatusCode::OK, resp.status());
        let result: CarDto = test::read_body_json(resp).await;
        assert_eq!(result.name, new_car.name);

        let delete_car_req = TestRequest::delete().uri(format!("/cars/{}", new_car_id).as_str()).to_request();
        let resp = test::call_service(&app, delete_car_req).await;
        assert_eq!(StatusCode::OK, resp.status());

        let get_car_req = TestRequest::get().uri(format!("/cars/{}", new_car_id).as_str()).to_request();
        let resp = test::call_service(&app, get_car_req).await;
        assert_eq!(StatusCode::NOT_FOUND, resp.status());
    }

    fn create_one_test_car() -> TestRequest {
        let post = TestRequest::post().uri("/cars").set_json(&dto::CarDto {
            id: None,
            name: "Test".to_string(),
            brand: "Test".to_string(),
            year: 2021,
            r#type: CarType::Other,
        });
        post
    }
}

Let's describe what is happening here:

• The module defines a set of tests and imports the required dependencies.

• Each test is marked with the #[actix_web::test] attribute for recognition by the testing framework.

• The initial two tests (test_index and test_healthcheck) validate the response of index and healthcheck endpoints, respectively. These tests simply call the service and assert that the response is as expected.

• The test_get_cars function introduces the use of Testcontainers. It creates an instance of a Testcontainers client, which allows the use of Docker containers during tests. This function specifically employs MongoDB as the test container:

  • A new container is instantiated from the "mongo" image (version "6.0.7").

  • The container is prepared with specific environment variables for MongoDB's initial database setup.

  • An exposed port, 27017, is specified for network communications.

  • A MongoDB URI is assembled, pointing towards the Docker-hosted MongoDB instance.

  • After the setup, test requests are sent and their responses are validated.

• The test_get_car and test_delete_car functions employ a similar testing approach as test_get_cars. These functions create a MongoDB container using the Testcontainers client and then execute tests that are supposed to interact with the MongoDB database.

Testing

To run the tests, execute the following command:

cargo test

Because the tests are using Docker containers, the first run will take a while to download the required images. And, of course, you need Docker installed on your machine.

If everything goes well, you should see the following output:

running 5 tests
test tests::test_healthcheck ... ok
test tests::test_index ... ok
test tests::test_get_car ... ok
test tests::test_delete_car ... ok
test tests::test_get_cars ... ok

test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 6.44s

Conclusion

Testcontainers undoubtedly offer a powerful framework to improve the fidelity of your application testing through the use of Docker containers. However, they are not without limitations. Here are some notable considerations based on my personal experience:

• The Rust iteration of Testcontainers is not on par with its Java or Go counterparts, resulting in the absence of certain features.

• As of July 5th, 2023, no Rust Testcontainer Modules exist, which means that you must use the GenericImage and configure the container yourself.

• Testcontainers introduce another dependency to manage in your project.

• Containers can be resource-intensive, requiring careful management to avoid overconsumption.

• Your CI/CD pipeline must not only support containers but also possess the capacity to run them efficiently.

• These requirements are equally valid if you're running tests locally on your own machine.

In conclusion, the decision to adopt Testcontainers hinges on individual project needs. I found the tool valuable and plan to use it in my non-Rust projects.

Resources

As usual, here the link to the code:

Introduction

In this tutorial, we will build a simple SSH client using Rust 🦀. Having a way to connect to a remote server is often a requirement for DevOps tools.

If you already know Go, you might have used the ssh package. Here is a refresh of how to use it, with the password authentication method:

package main

import (
    "fmt"
    "log"
    "os"

    "golang.org/x/crypto/ssh"
)

func main() {
    // SSH connection parameters
    host := "192.168.64.5"
    port := 22
    user := "steve"
    password := "password"

    // Create SSH client configuration
    config := &ssh.ClientConfig{
        User: user,
        Auth: []ssh.AuthMethod{
            ssh.Password(password),
        },
        HostKeyCallback: ssh.InsecureIgnoreHostKey(),
    }

    // Connect to the remote server
    conn, err := ssh.Dial("tcp", fmt.Sprintf("%s:%d", host, port), config)
    if err != nil {
        log.Fatalf("Failed to connect to SSH server: %v", err)
    }
    defer conn.Close()

    // Create a new SSH session
    session, err := conn.NewSession()
    if err != nil {
        log.Fatalf("Failed to create SSH session: %v", err)
    }
    defer session.Close()

    // Set the output writer for session's output
    session.Stdout = os.Stdout

    // Run a command on the remote server
    err = session.Run("ls -l")
    if err != nil {
        log.Fatalf("Failed to run command on remote server: %v", err)
    }
}

Easy, right? Now let's see how to do the same thing in Rust.

Prerequisites

To follow this blog post, you should have a basic understanding of Rust 🦀 and the Cargo build tool. If you are new to Rust 🦀 check out my blog post "Learn Rust in under 10 mins":

Before we start, we need to make sure we have the following tools installed:

• Rust

• An IDE or text editor of your choice

Initialize the demo project

cargo init

This will create a new Rust 🦀 project in the current directory. Now, we need to add the dependencies we need for our demo application.

Now add can add the ssh2 crate to the Cargo.toml file using the following command:

cargo add ssh2 --features vendored-openssl

Create the SSH client

With the dependencies in place, we can now head over to the src/main.rs file and start writing our SSH client.

use std::io::Read;
use std::net::TcpStream;
use ssh2::Session;

fn main() {
    let stream = TcpStream::connect(format!("{}:22", "192.168.64.5"));
    match stream {
        Ok(stream) => {
            println!("Connected to the server!");
            let session = Session::new();
            match session {
                Ok(mut session) => {
                    session.set_tcp_stream(stream);
                    session.handshake().unwrap();
                    let auth = session.userauth_password("steve", "password");
                    match auth {
                        Ok(_) => {
                            println!("Authenticated!");
                            let channel = session.channel_session();
                            match channel {
                                Ok(mut channel) => {
                                    channel.exec("whoami").unwrap();
                                    let mut s = String::new();
                                    channel.read_to_string(&mut s).unwrap();
                                    println!("{}", s);
                                    channel.wait_close().unwrap();
                                    let exit_status = channel.exit_status().unwrap();
                                    if exit_status != 0 {
                                        eprint!("Exited with status {}", exit_status);
                                    }
                                }
                                Err(e) => {
                                    eprint!("Failed to create channel: {}", e);
                                }
                            }
                        }
                        Err(e) => {
                            eprint!("Failed to authenticate: {:?}", e);
                        }
                    }
                }
                Err(e) => {
                    eprint!("Failed to create session: {}", e);
                }
            }
        }
        Err(e) => {
            eprint!("Failed to connect: {}", e);
        }
    }
}

As you can see from the code, we try(!) to not use the unwrap() method as much as possible. Instead, we handle the errors explicitly. This is a good practice to follow in your code, to avoid unwrap() calls all over the place, as it can mask errors and make your code less readable.

But let's go through the code step by step:

• Initially, a new TCP stream is established to the remote server via the TcpStream::connect() method. Upon successful connection, a new Session object is created.

• Subsequently, this TCP stream is set on the Session object utilizing Session::set_tcp_stream().

• We then carry out the SSH handshake through Session::handshake().

• After the handshake, an attempt at authentication is made using Session::userauth_password(). Alternatively, authentication could be done using a private key with Session::userauth_pubkey_file().

• Finally, a new channel is formed with Session::channel_session(), followed by the execution of the whoami command on the remote server using Channel::exec(). The output of this command is read into a string via Channel:: read_to_string() and subsequently printed to the console.

• The channel is then closed with Channel::wait_close() and the exit status of the command is printed to the console using Channel::exit_status().

• If the exit status is not 0, an error message is printed to the console.

Run the demo

Now that we have our SSH client ready, we can run it using the following command:

cargo run

If everything went well, you should see the following output:

Connected to the server!
Authenticated!
steve

Conclusion

In this tutorial, we have seen how quickly we can create an SSH client in Rust 🦀 using the ssh2 crate. Now we can start building our own SSH client applications in Rust 🦀!

Introduction

Recently, I got a request from one of our customers asking how to upgrade their AKS cluster using only Pulumi and not the Azure CLI. The customer also mentioned that they did not find a solution to this problem. So I took up this challenge, hoping that I can find a solution to this problem.

First I asked in my tech bubble for help, and as always Richard jumped in to give input and a way to achieve this in Bicep.

So, I knew it must be possible! Also with Pulumi! BTW: the link to Richards's repo for the Bicep way is listed under Additional Resources

Prerequisites

Before we start, you need to have the following prerequisites in place:

• Pulumi

• Optional: Pulumi Account (for state storage), see Pulumi Cloud Account

• Node.js

• Azure CLI

• kubectl

In the folder infrastructure you will find the Pulumi code to create the AKS cluster. This will be the starting point for this blog post.

Create the AKS Cluster

Note: You need to be logged in to Azure. You can see all the different ways to log into Azure in the Pulumi provider documentation for Azure Native.

To create the AKS cluster, you need to run the following commands:

cd infrastructure
pulumi up

The code is written in TypeScript, because why not! It may take a while to create the AKS cluster. So go and grab yourself a coffee.

Upgrade Your AKS Cluster, the Azure CLI Way

Before I start with the Pulumi solution, I want to show you how to upgrade your AKS cluster using the Azure CLI. No, I am not showing you how to upgrade your AKS cluster using the Azure Portal. I am sure you will figure that out yourself, I am not going to support ClickOps here!

Before we start our upgrade, let's check the available Kubernetes versions for our AKS cluster:

az aks get-upgrades --name my-cluster --resource-group my-resource-group0012049e --output table

The output should look like this:

Name     ResourceGroup              MasterVersion    Upgrades
-------  -------------------------  ---------------  -----------------------
default  my-resource-group0012049e  1.24.9           1.24.10, 1.25.5, 1.25.6

Nice, we pick the latest version, which is 1.25.6 and upgrade our AKS cluster:

az aks upgrade --name my-cluster --resource-group my-resource-group0012049e --kubernetes-version 1.25.6

With the flag --control-plane-only you can upgrade only the control plane and not the nodes.

Note: Keep in mind, that you can only upgrade one minor version at a time. So if you are on version 1.24.9 you can't upgrade to 1.26.x directly. You need to upgrade to 1.25.x first and then to 1.26.x.

You should see the following output:

ubernetes may be unavailable during cluster upgrades.
 Are you sure you want to perform this operation? (y/N): y
Since control-plane-only argument is not specified, this will upgrade the control plane AND all nodepools to version 1.25.6. Continue? (y/N): y
{
...
}

And we check also via the kubectl command, if the upgrade was successful:

kubectl get nodes

aks-agentpool-32360557-vmss000000   Ready    agent   12m     v1.25.6
aks-agentpool-32360557-vmss000001   Ready    agent   11m     v1.25.6
aks-agentpool-32360557-vmss000002   Ready    agent   9m53s   v1.25.6
aks-workload1-32360557-vmss000000   Ready    agent   15m     v1.25.6
aks-workload1-32360557-vmss000001   Ready    agent   14m     v1.25.6
aks-workload1-32360557-vmss000002   Ready    agent   12m     v1.25.6

And the control plane:

kubectl version --short

...
Server Version: v1.25.6

Looks good, right? Before we move on to the Pulumi solution, let me give you some background information on the choreography Azure is doing in the background to ensure minimal disruption to your running workloads.

How Does the Upgrade Work?

As mentioned before, Azure is doing certain steps in the background to ensure minimal disruption to your running.

• Adds a new, so-called buffer node to the cluster with the specified Kubernetes version.

• Cordons and drains one of the old nodes.

• When the node is drained, it's re-imaged with the new Kubernetes version and becomes a new buffer node.

• Rinse and repeat until all nodes are upgraded.

• In the end, the buffer node is removed to make sure you have the same number of nodes as before.

Upgrade Your AKS Cluster, the Pulumi Way!

With the knowledge we have now, we can start to write our Pulumi code to upgrade our AKS cluster. First of all, we reset our AKS cluster to the original state

Note: You can't downgrade your AKS cluster. So if you want to downgrade your AKS cluster, you need to delete it and create a new one.

pulumi destroy
pulumi up

The Problem

If you look at the Pulumi code, you will see that there are two variables defined for the Kubernetes version.

const kubernetesVersion = "1.24.9"
const agentKubernetesVersion = "1.24.9"

So naturally, you would think that you can just change the value to the new version and run pulumi up again. But if you do that, you will see the following error:

 error: Code="NotAllAgentPoolOrchestratorVersionSpecifiedAndUnchanged" Message="Using managed cluster api, all Agent pools' OrchestratorVersion must be all specified or all unspecified. If all specified, they must be stay unchanged or the same with control plane. For agent pool specific change, please use per agent pool operations: https://aka.ms/agent-pool-rest-api"

To safely upgrade your AKS cluster, we create a second Pulumi program, which we can execute when we want to upgrade our AKS cluster.

To do that, we create a new folder upgrade-aks and initialize a new Pulumi project:

mkdir upgrade-aks
cd upgrade-aks
pulumi new azure-typescript -n upgrade-aks -d "upgrade aks cluster" -s dev

And the magic sauce is the usage of the pulumi-azapi provider. In particular, the UpdateResource resource. This resource is used to add or modify properties on an existing resource. The good thing is, we can delete this Pulumi project afterward, because when UpdateResource is deleted, no operation will be performed in Azure.

To get information about the AKS cluster, we will also use the Pulumi concept of StackReference. This allows us to access the information from the other Pulumi project.

If you peek into the index.ts file, in the infrastructure folder, you will see that we are exporting the following values

export const resourceGroupName = resourceGroup.name;
export const managedClusterName = managedCluster.name;

export const managedClusterId = managedCluster.id;
export const resourceGroupId = resourceGroup.id;

The code for the upgrade-aks project looks like this:

import * as pulumi from "@pulumi/pulumi";
import * as cluster from "@pulumi/azure-native/containerservice/v20230301";

const stackReference = new pulumi.StackReference("<org/projectname/stack>");

import * as azapi from "@ediri/azapi";

const newKubernetesVersion = "1.24.10"
const newAgentKubernetesVersion = "1.24.10"

const clusterUpdate = new azapi.UpdateResource("cluster", {
    type: "Microsoft.ContainerService/managedClusters@2023-03-01",
    parentId: stackReference.requireOutput("resourceGroupId"),
    name: stackReference.requireOutput("managedClusterName"),
    body: pulumi.jsonStringify({
        properties: {
            id: stackReference.requireOutput("managedClusterId"),
            kubernetesVersion: newKubernetesVersion
        }
    })
})

const aks = cluster.getManagedClusterOutput({
    resourceGroupName: stackReference.requireOutput("resourceGroupName"),
    resourceName: stackReference.requireOutput("managedClusterName"),
})

aks.agentPoolProfiles?.apply(agentPoolProfiles => {
    for (const agentPoolProfile of agentPoolProfiles ?? []) {
        const agentPoolUpdate = new azapi.UpdateResource(`agentpool-${agentPoolProfile.name}`, {
            type: "Microsoft.ContainerService/managedClusters/agentPools@2023-03-01",
            parentId: stackReference.requireOutput("managedClusterId"),
            name: agentPoolProfile.name,
            body: pulumi.jsonStringify({
                properties: {
                    id: stackReference.requireOutput("managedClusterId"),
                    orchestratorVersion: newAgentKubernetesVersion
                }
            })
        }, {
            dependsOn: clusterUpdate
        })
    }
})

And we add the pulumi-azapi provider to the package.json

{
  "name": "upgrade-aks",
  "main": "index.ts",
  "devDependencies": {
    "@types/node": "^16"
  },
  "dependencies": {
    "@pulumi/pulumi": "^3.0.0",
    "@pulumi/azure-native": "2.0.0-beta.1",
    "@ediri/azapi": "1.2.9"
  }
}

Now we can change the values of the variables in the index.ts file and run pulumi up.

const newKubernetesVersion = "1.24.10"
const newAgentKubernetesVersion = "1.24.10"

Note: Keep in mind to have not a resource quota limit in your subscription. Otherwise, the update will fail, with an error message like this: Operation could not be completed as it results in exceeding approved standardBSFamily Cores quota. In this case, you need to request a quota increase.

This will first update the AKS cluster and then update the agent pools. This can take a while, so be patient.

Updating (dev)

View in Browser (Ctrl+O): https://app.pulumi.com/dirien/agent-pool/dev/updates/14

     Type                           Name                 Status              
 +   pulumi:pulumi:Stack            agent-pool-dev       created (0.85s)     
 +   ├─ azapi:index:UpdateResource  cluster              created (134s)      
 +   ├─ azapi:index:UpdateResource  agentpool-workload2  created (197s)      
 +   ├─ azapi:index:UpdateResource  agentpool-workload1  created (177s)      
 +   └─ azapi:index:UpdateResource  agentpool-agentpool  created (474s)      


Resources:
    + 5 created

Duration: 10m16s

How To Speed Up the Update Process With the Help of surge?

If you have a lot of agent pools, the update process can take a while. To speed up the process, you can tweak the max surge settings of your AKS cluster. Per default, the max surge setting is set to 1. This means that one extra node will be created and only one old node cordoned and drained.

So if you set the surge to 100% will cause all old nodes to be cordoned off and drained at the same time. While this speeds up your update process, it can also cause some disruption to your workload. So be careful with this setting!

A rule of thumb is to set the surge for production clusters to 33%.

Wrap Up

In this blog post, I showed you an easy way to upgrade your AKS cluster with the help of Pulumi and the help of an extra update Pulumi project. This allows you to upgrade your AKS cluster in a controlled way without using the Azure Portal or the Azure CLI.

Additional Resources

• The Bicep version from Richard -> https://github.com/PixelRobots/AKS-Bicep

• The Azure CLI way -> https://learn.microsoft.com/en-us/azure/aks/tutorial-kubernetes-upgrade-cluster?

• Surge -> https://learn.microsoft.com/en-us/azure/aks/upgrade-cluster?tabs=azure-cli#customize-node-surge-upgrade

Introduction

This article is part three of my series on how to manage secrets on Kubernetes by using Pulumi. If you haven't read the previous articles, here are the links to them:

In this article, we will be looking at how to use Doppler to manage secrets on Kubernetes. But before we get into the details of how to set up the integration, let's take a look at what Doppler is what it can do for us.

What is Doppler?

Doppler defines itself as a "SecretOps Platform". Currently, it is a cloud-based service that allows teams to securely manage and distribute secrets across their applications and infrastructure.

The Doppler platform is built with collaboration in mind. It supports out-of-the-box concepts like users, groups, and roles. So far it's like any other tool that supports role-based access control (RBAC). But with Doppler, there is also the concept of projects. A project is a collection of secrets that are scoped to a specific application or service. This allows you also to have project-based access control. Next to projects, Doppler also supports environments, which allows you to have different values for the same secret in different environments. For example, you can have a DB_URL secret has a different value in your dev environment than in your prod environment.

What I like the most about Doppler is a very huge list of integrations that it supports. And of all the integrations, the Kubernetes integration is the one that I like the most. And that's what we will be looking at in this article.

The Doppler Secrets Operator is an automated service that operates within a Kubernetes cluster, ensuring that secrets are continuously synchronized and deployments are updated accordingly.

Operating from within its unique namespace, doppler-operator-system, the Operator maintains strict access control through RBAC policies. It utilizes DopplerSecret custom resources, which specify the Doppler configuration to synchronize, the name of the Kubernetes secret under its management, and the namespace where it will be created.

The Doppler Secrets Operator is exclusively responsible for the constant synchronization of secret updates from Doppler to the Kubernetes secrets it manages. It can also optionally reload deployments that reference a managed secret, assuring your applications always have access to the most recent version of secrets.

Prerequisites

To follow along with this article, you will need the following:

• Pulumi CLI installed.

• kubectl installed.

• optional K9s, if you want to quickly interact with your cluster.

• Doppler Account.

• DigitalOcean Account.

• vcluster cli installed.

The Demo Setup

In this demo, we will add a little twist to the setup! We will deploy two DigitalOcean Kubernetes (DOKS) clusters, one for development and staging and one for production. The twist is, that the development and staging cluster will be deployed as vcluster on top of the DOKS cluster. The production cluster will be deployed as a standalone cluster.

You don't know, what vcluster is? Spend three minutes and watch this.

Create a New Pulumi Project

Let's start by creating a new Pulumi project. We will call it pulumi-doppler and we will use golang as the language

mkdir pulumi-doppler
cd pulumi-doppler
pulumi new digitalocean-go --force

You can leave the default values in the prompts.

This command will walk you through creating a new Pulumi project.

Enter a value or leave blank to accept the (default), and press <ENTER>.
Press ^C at any time to quit.

project name: (pulumi-doppler) 
project description: (A minimal DigitalOcean Go Pulumi program) 
Created project 'pulumi-doppler'

Please enter your desired stack name.
To create a stack in an organization, use the format <org-name>/<stack-name> (e.g. `acmecorp/dev`).
stack name: (dev)   
Created stack 'dev'

Installing dependencies...

go: downloading github.com/pulumi/pulumi-digitalocean/sdk/v4 v4.19.1
Finished installing dependencies

Your new project is ready to go! ✨

To perform an initial deployment, run `pulumi up`

We can then delete the contents of the main.go file and replace it with the following code:

package main

import (
    "github.com/pulumi/pulumi-digitalocean/sdk/v4/go/digitalocean"
    "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

type NodePoolArgs struct {
    Name      string
    Size      string
    NodeCount int
    Label     string
}

type ClusterArgs struct {
    Name     string
    Region   string
    Version  string
    NodePool *[]NodePoolArgs
}

const (
    DefaultNodePoolName = "default"
    DefaultNodePoolSize = "s-2vcpu-4gb"
)

func createDOKSCLuster(ctx *pulumi.Context, args *ClusterArgs) (pulumi.StringOutput, error) {
    // Create a new DOKS cluster
    cluster, err := digitalocean.NewKubernetesCluster(ctx, args.Name, &digitalocean.KubernetesClusterArgs{
        Region:  pulumi.String(args.Region),
        Version: pulumi.String(args.Version),
        NodePool: &digitalocean.KubernetesClusterNodePoolArgs{
            Name:      pulumi.String(DefaultNodePoolName),
            Size:      pulumi.String(DefaultNodePoolSize),
            NodeCount: pulumi.Int(1),
            AutoScale: pulumi.Bool(false),
        },
    })

    if err != nil {
        return pulumi.StringOutput{}, err
    }
    if args.NodePool != nil {
        for _, nodePool := range *args.NodePool {
            _, err := digitalocean.NewKubernetesNodePool(ctx, nodePool.Name, &digitalocean.KubernetesNodePoolArgs{
                ClusterId: cluster.ID(),
                Name:      pulumi.String(nodePool.Name),
                Size:      pulumi.String(nodePool.Size),
                NodeCount: pulumi.Int(nodePool.NodeCount),
                Labels:    pulumi.StringMap{"env": pulumi.String(nodePool.Label)},
            })
            if err != nil {
                return pulumi.StringOutput{}, err
            }
        }
    }

    output, _ := cluster.KubeConfigs.ApplyT(func(kcs []digitalocean.KubernetesClusterKubeConfig) (string, error) {
        if len(kcs) == 0 {
            return "", nil
        }
        return *kcs[0].RawConfig, nil
    }).(pulumi.StringOutput)

    return output, nil
}

func main() {
    pulumi.Run(func(ctx *pulumi.Context) error {
        // Create the DOKS cluster for development and staging
        preProd, err := createDOKSCLuster(ctx, &ClusterArgs{
            Name:    "pre-prod-cluster",
            Region:  "fra1",
            Version: "1.26.3-do.0",
            NodePool: &[]NodePoolArgs{
                {
                    Name:      "development",
                    Size:      "s-2vcpu-4gb",
                    NodeCount: 1,
                    Label:     "development",
                },
                {
                    Name:      "staging",
                    Size:      "s-2vcpu-4gb",
                    NodeCount: 1,
                    Label:     "staging",
                },
            },
        })
        if err != nil {
            return err
        }
        ctx.Export("preProdKubeConfig", pulumi.ToSecret(preProd))

        // Create the DOKS cluster for production
        prod, err := createDOKSCLuster(ctx, &ClusterArgs{
            Name:    "prod-cluster",
            Region:  "fra1",
            Version: "1.25.8-do.0",
        })
        if err != nil {
            return err
        }
        ctx.Export("prodKubeConfig", pulumi.ToSecret(prod))
        return nil
    })
}

Let's break down the code into smaller pieces to understand what's going on here:

• Two structures, NodePoolArgs and ClusterArgs, are declared. These structures are used to encapsulate information necessary for the creation of a DOKS cluster.

  • NodePoolArgs encapsulates details about a node pool, which includes its Name, Size, NodeCount (the number of nodes in the pool), and Label (a string to label the node pool).

  • ClusterArgs encapsulates information about a DOKS cluster. This includes its Name, Region, Version, and NodePool configuration.

• The createDOKSCLuster function is declared. This function is responsible for creating a DOKS cluster using the provided arguments, which are encapsulated in the ClusterArgs structure.

  • The function uses the NewKubernetesCluster method from the DigitalOcean Pulumi provider to create a new DOKS cluster.

  • If the cluster creation is successful and additional NodePoolArgs are provided, the function iterates over the slice of NodePoolArgs to create additional node pools using the NewKubernetesNodePool method.

  • If the creation of additional node pools is successful, the function retrieves the raw Kubernetes configuration of the newly created cluster. This configuration is essential for interacting with the cluster.

• The main function serves as the entry point of the program and two DOKS clusters are created – one for pre-production and another for production. Each cluster has its unique Name, Region, Version, and NodePool configuration.

  • The pre-production cluster has a default node pool and two additional node pools labeled "development" and " staging". Each of these additional node pools has 1 node of size "s-2vcpu-4gb".

  • The production cluster is created with a default node pool.

Deploy the DOKS Clusters

With the infrastructure code in place, we can now deploy the DOKS clusters. Don't worry, we will add the vcluster and Doppler components later.

Note: Set the DIGITALOCEAN_TOKEN environment variable to your DigitalOcean API token before running the Pulumi command.

export DIGITALOCEAN_TOKEN=<your-digitalocean-api-token>
pulumi up

Deploying the DOKS clusters will take a few minutes. In the process, Pulumi will ask you to confirm the deployment and give you a summary of the resources that will be created.

Note: You can use the --yes and --skip-preview flags to skip the confirmation prompt and preview, respectively.

 pulumi up
Previewing update (dev)

View in Browser (Ctrl+O): https://app.pulumi.com/dirien/pulumi-doppler/dev/previews/758944fa-5643-410a-b11a-ee4a3a95a227

     Type                                     Name                Plan       
 +   pulumi:pulumi:Stack                      pulumi-doppler-dev  create     
 +   ├─ digitalocean:index:KubernetesCluster  prod-cluster        create     
 +   └─ digitalocean:index:KubernetesCluster  pre-prod-cluster    create     


Outputs:
    preProdKubeConfig: [secret]
    prodKubeConfig   : [secret]

Resources:
    + 3 to create

Do you want to perform this update? yes
Updating (dev)

View in Browser (Ctrl+O): https://app.pulumi.com/dirien/pulumi-doppler/dev/updates/1

     Type                                     Name                Status             
 +   pulumi:pulumi:Stack                      pulumi-doppler-dev  created (415s)     
 +   ├─ digitalocean:index:KubernetesCluster  prod-cluster        created (413s)     
 +   └─ digitalocean:index:KubernetesCluster  pre-prod-cluster    created (302s)     


Outputs:
    preProdKubeConfig: [secret]
    prodKubeConfig   : [secret]

Resources:
    + 3 created

Duration: 6m58s

Configure kubectl to Interact with the DOKS Clusters

If you have the kubectl CLI installed, you can use the pulumi stack output command to retrieve the raw Kubernetes configuration of the clusters.

This is helpful in case you want to debug the clusters or interact with them using the kubectl CLI.

pulumi stack output preProdKubeConfig --show-secrets > pre-prod-cluster.yaml
export KUBECONFIG=pre-prod-cluster.yaml
kubectl get nodes -o wide

You should see an output similar to the following:

pulumi stack output preProdKubeConfig --show-secrets > pre-prod-cluster.yaml
export KUBECONFIG=pre-prod-cluster.yaml
kubectl get nodes -o wide
NAME                STATUS   ROLES    AGE     VERSION   INTERNAL-IP   EXTERNAL-IP       OS-IMAGE                         KERNEL-VERSION          CONTAINER-RUNTIME
default-fx9bk       Ready    <none>   4m30s   v1.26.3   10.135.0.5    142.93.162.26     Debian GNU/Linux 11 (bullseye)   6.0.0-0.deb11.6-amd64   containerd://1.6.14
development-fx9gg   Ready    <none>   61s     v1.26.3   10.135.0.6    134.209.245.105   Debian GNU/Linux 11 (bullseye)   6.0.0-0.deb11.6-amd64   containerd://1.6.14
staging-fx9gw       Ready    <none>   26s     v1.26.3   10.135.0.7    207.154.215.11    Debian GNU/Linux 11 (bullseye)   6.0.0-0.deb11.6-amd64   containerd://1.6.14

Add the vcluster and Doppler Components

Now that we have the DOKS clusters up and running, we can add the vcluster and Doppler components to them.

First, we need to add the pulumi-kubernetes package to our project:

go get  github.com/pulumi/pulumi-kubernetes/sdk/go/kubernetes

Then head over to our main.go file and add the following code:

package main

// omitted for brevity

func deployProd(ctx *pulumi.Context, cluster pulumi.StringOutput) error {
    productionKubernetesProvider, err := kubernetes.NewProvider(ctx, "production-k8s", &kubernetes.ProviderArgs{
        Kubeconfig:            cluster,
        EnableServerSideApply: pulumi.Bool(true),
    })
    if err != nil {
        return err
    }

    dopplerOperatorHelm, err := helm.NewRelease(ctx, "doppler-operator", &helm.ReleaseArgs{
        Name:            pulumi.String("doppler-operator"),
        Chart:           pulumi.String("doppler-kubernetes-operator"),
        Namespace:       pulumi.String("doppler-operator-system"),
        CreateNamespace: pulumi.Bool(false),
        RepositoryOpts: &helm.RepositoryOptsArgs{
            Repo: pulumi.String("https://helm.doppler.com"),
        },
        Version: pulumi.String("1.2.5"),
    }, pulumi.Provider(productionKubernetesProvider))
    if err != nil {
        return err
    }

    secret, err := v1.NewSecret(ctx, "doppler-token-secret", &v1.SecretArgs{
        Metadata: &metav1.ObjectMetaArgs{
            Name:      pulumi.String("doppler-token-secret"),
            Namespace: pulumi.String("doppler-operator-system"),
        },
        Type: pulumi.String("Opaque"),
        Data: pulumi.StringMap{
            "serviceToken": config.GetSecret(ctx, "doks-production").ApplyT(func(s string) (string, error) {
                return base64.StdEncoding.EncodeToString([]byte(s)), nil
            }).(pulumi.StringOutput),
        },
    }, pulumi.Provider(productionKubernetesProvider), pulumi.DependsOn([]pulumi.Resource{dopplerOperatorHelm}))
    if err != nil {
        return err
    }

    _, err = apiextensions.NewCustomResource(ctx, "dopplersecrets", &apiextensions.CustomResourceArgs{
        ApiVersion: pulumi.String("secrets.doppler.com/v1alpha1"),
        Kind:       pulumi.String("DopplerSecret"),
        Metadata: &metav1.ObjectMetaArgs{
            Name:      pulumi.String("doppler-token-secret"),
            Namespace: secret.Metadata.Namespace().Elem(),
        },
        OtherFields: kubernetes.UntypedArgs{
            "spec": pulumi.Map{
                "tokenSecret": pulumi.Map{
                    "name":      secret.Metadata.Name().Elem(),
                    "namespace": secret.Metadata.Namespace().Elem(),
                },
                "managedSecret": pulumi.Map{
                    "name":      pulumi.String("doppler-secret"),
                    "namespace": pulumi.String("default"),
                },
            },
        },
    }, pulumi.Provider(productionKubernetesProvider), pulumi.DependsOn([]pulumi.Resource{dopplerOperatorHelm}))
    if err != nil {
        return err
    }
    _, err = appsv1.NewDeployment(ctx, "hello-server", &appsv1.DeploymentArgs{
        Metadata: &metav1.ObjectMetaArgs{
            Name: pulumi.String("hello-server"),
            Annotations: pulumi.StringMap{
                "secrets.doppler.com/reload": pulumi.String("true"),
            },
        },
        Spec: &appsv1.DeploymentSpecArgs{
            Replicas: pulumi.Int(1),
            Selector: &metav1.LabelSelectorArgs{
                MatchLabels: pulumi.StringMap{
                    "app": pulumi.String("hello-server"),
                },
            },
            Template: &v1.PodTemplateSpecArgs{
                Metadata: &metav1.ObjectMetaArgs{
                    Labels: pulumi.StringMap{
                        "app": pulumi.String("hello-server"),
                    },
                },
                Spec: &v1.PodSpecArgs{
                    Containers: v1.ContainerArray{
                        v1.ContainerArgs{
                            Name:  pulumi.String("hello-server"),
                            Image: pulumi.String("ghcr.io/dirien/hello-server/hello-server:latest"),
                            Ports: v1.ContainerPortArray{
                                v1.ContainerPortArgs{
                                    ContainerPort: pulumi.Int(8080),
                                },
                            },
                            EnvFrom: v1.EnvFromSourceArray{
                                v1.EnvFromSourceArgs{
                                    SecretRef: &v1.SecretEnvSourceArgs{
                                        Name: pulumi.String("doppler-secret"),
                                    },
                                },
                            },
                        },
                    },
                },
            },
        },
    }, pulumi.Provider(productionKubernetesProvider))
    if err != nil {
        return err
    }
    return nil
}

func deployPreProd(ctx *pulumi.Context, cluster pulumi.StringOutput, stage string) error {
    preProdKubernetesProvider, err := kubernetes.NewProvider(ctx, fmt.Sprintf("%s-k8s", stage), &kubernetes.ProviderArgs{
        Kubeconfig:            cluster,
        EnableServerSideApply: pulumi.Bool(true),
    })
    if err != nil {
        return err
    }
    _, err = helm.NewRelease(ctx, fmt.Sprintf("%s-vcluster", stage), &helm.ReleaseArgs{
        Name:  pulumi.String(fmt.Sprintf("%s-vcluster", stage)),
        Chart: pulumi.String("vcluster"),
        RepositoryOpts: &helm.RepositoryOptsArgs{
            Repo: pulumi.String("https://charts.loft.sh"),
        },
        Values: pulumi.Map{
            "sync": pulumi.Map{
                "nodes": pulumi.Map{
                    "enabled":         pulumi.Bool(true),
                    "enableScheduler": pulumi.Bool(true),
                    "nodeSelector":    pulumi.Sprintf("env=%s", stage),
                },
            },
            "nodeSelector": pulumi.Map{
                "env": pulumi.String(stage),
            },
            "init": pulumi.Map{
                "helm": pulumi.Array{
                    pulumi.Map{
                        "chart": pulumi.Map{
                            "name":    pulumi.String("doppler-kubernetes-operator"),
                            "repo":    pulumi.String("https://helm.doppler.com"),
                            "version": pulumi.String("1.2.5"),
                        },
                        "release": pulumi.Map{
                            "name":      pulumi.String("doppler-operator"),
                            "namespace": pulumi.String("doppler-operator-system"),
                        },
                    },
                },
            },
        },
    }, pulumi.Provider(preProdKubernetesProvider))
    if err != nil {
        return err
    }
    return nil
}
func main() {
    pulumi.Run(func(ctx *pulumi.Context) error {
        // omitted for brevity

        // Deploy the vcluster to development and staging
        err = deployPreProd(ctx, preProd, "development")
        if err != nil {
            return err
        }
        err = deployPreProd(ctx, preProd, "staging")
        if err != nil {
            return err
        }

        // omitted for brevity
        err = deployProd(ctx, prod)
        if err != nil {
            return err
        }
        return nil
    })
}

A lot is going on here, so let's break it down again.

• The deployPreProd function is responsible for deploying a vcluster to the pre-production environments, including development and staging. This function employs the Helm chart for deploying the vcluster and each vcluster is configured to use specific node pools - namely development and staging.

  • The preProdKubernetesProvider is created with the given cluster configuration, which is identified by the stage parameter. This provider enables Pulumi to manage resources in a Kubernetes cluster.

  • The sync.nodes.nodeSelector field is used to specify the node pool for the vcluster synchronization feature, whereas the nodeSelector field determines the node pool for the vcluster itself. This dual configuration is implemented to ensure the correct allocation and utilization of resources within the respective environments.

  • In addition to this, the init.helm field is leveraged to deploy the Doppler Secrets Operator within the vcluster. This is achieved by specifying the name, repository, and version of the Helm chart.

• The function deployProd is used to deploy the Doppler Secrets Operator without the usage of vcluster. Remember this was our production cluster.

  • A Kubernetes provider productionKubernetesProvider, which is a representation of the Kubernetes cluster where the resources will be deployed. It uses the provided cluster string output (which is presumably a kubeconfig) to connect to the Kubernetes cluster.

  • A Kubernetes secret, named doppler-token-secret, is created in the doppler-operator-system namespace. This secret is created with data retrieved from the Pulumi configuration using config.GetSecret(ctx, " doks-production"). This data is then base64 encoded before being added to the secret.
    See below for how to set the config on Pulumi!

  • A custom resource, of kind DopplerSecret, which includes references to the doppler-token-secret and a managed secret named doppler-secret in the default namespace.

  • A Kubernetes deployment, named hello-server, which creates a pod running the ghcr.io/dirien/hello-server/hello-server:latest Docker image and exposes port 8080. This deployment includes an environment variable sourced from the doppler-secret.

• Both functions will be called in the main function, after the creation of the DOKS cluster.

To generate a service token, head over to the Doppler dashboard, and select your project and environment. Click the Access tab and generate the token there

Now you can use the Pulumi config command to set the service token:

pulumi config set doks-production <toke> --secret

Deploying the vcluster and Doppler Secrets Operator

Now with all set, we can call our Pulumi function to deploy the vclusters and Doppler Secrets Operator to all of our environments.

pulumi up

If you love the thrill, set --yes to skip the confirmation prompt and --skip-preview to skip the preview step.

After a couple of minutes, everything should be deployed and ready to go. We can now try to test everything out.

Testing the Doppler Secrets Operator in all environments

I left the demo example at Doppler, as it is.

The easiest way to test the Doppler Secrets Operator is in the production environment. This is because I already have deployed a demo application that uses the secrets the Doppler Secrets Operator retrieves from Doppler and created a Kubernetes secret for it.

Again, this is a short demo. I did not create any Ingress or Service to expose the application. Instead, we will use kube port forwarding to access the application.

pulumi stack output prodKubeConfig --show-secrets > prod.yaml
export KUBECONFIG=prod.yaml
kubectl port-forward deployment/hello-server 8080:8080

And then use curl to access the application and retrieve the secret. The env endpoint will return the value of any environment variable set in the pod. In this case, the DB_URL environment variable.

curl http://localhost:8080/env/DB_URL
DB_URL=psql://autopilot@10.127.172.12/modelX%

That looks good, let's try if the pod gets restarted when we update the secret in Doppler. To achieve this, you need to add "secrets.doppler.com/reload": "true" to the annotations of the deployment.

We keep an eye on the pod using kubectl get pods -w and update the secret in Doppler. After a couple of seconds, the pod should be restarted.

k get pods -o wide -w
NAME                            READY   STATUS    RESTARTS   AGE   IP             NODE            NOMINATED NODE   READINESS GATES
hello-server-66bd4dd5f5-swktv   1/1     Running   0          19m   10.244.0.124   default-fx9cn   <none>           <none>
hello-server-55bbf4b78c-b49xq   0/1     Pending   0          0s    <none>         <none>          <none>           <none>
hello-server-55bbf4b78c-b49xq   0/1     Pending   0          0s    <none>         default-fx9cn   <none>           <none>
hello-server-55bbf4b78c-b49xq   0/1     ContainerCreating   0          0s    <none>         default-fx9cn   <none>           <none>
hello-server-55bbf4b78c-b49xq   1/1     Running             0          2s    10.244.0.73    default-fx9cn   <none>           <none>
hello-server-66bd4dd5f5-swktv   1/1     Terminating         0          20m   10.244.0.124   default-fx9cn   <none>           <none>
hello-server-66bd4dd5f5-swktv   0/1     Terminating         0          20m   10.244.0.124   default-fx9cn   <none>           <none>
hello-server-66bd4dd5f5-swktv   0/1     Terminating         0          20m   10.244.0.124   default-fx9cn   <none>           <none>
hello-server-66bd4dd5f5-swktv   0/1     Terminating         0          20m   10.244.0.124   default-fx9cn   <none>           <none>

Recreating the port-forward and calling the env endpoint again should show the updated secret.

curl http://localhost:8080/env/DB_URL
DB_URL=psql://me@127.0.0.1/reload

Full success! Now we can test the same in the development and staging environments. For this, we need to use the kubeconfig of the pre-prod-cluster

pulumi stack output preProdKubeConfig --show-secrets > pre-prod.yaml
export KUBECONFIG=pre-prod.yaml

And then we use the vcluster context to access the vcluster in the development and staging environments.

vcluster connect development-vcluster -n default
done √ Switched active kube context to vcluster_development-vcluster_default_do-fra1-pre-prod-cluster-e058708
warn   Since you are using port-forwarding to connect, you will need to leave this terminal open
- Use CTRL+C to return to your previous kube context
- Use `kubectl get namespaces` in another terminal to access the vcluster
Forwarding from 127.0.0.1:10511 -> 8443
Forwarding from [::1]:10511 -> 8443

Check that the vcluster has the development node pool assigned to it!

kubectl get nodes
NAME STATUS ROLES AGE VERSION
development-fx9gg Ready    <none>   25h v1.26.3

That looks very good, last check is to see if the Doppler Secrets Operator is running in the vcluster.

kubectl get pods -n doppler-operator-system
NAME                                                   READY   STATUS    RESTARTS   AGE
doppler-operator-controller-manager-57b55f6fdf-qmh26   2/2     Running   0          24h

Everything looks good, we can now test the Doppler Secrets Operator in the development environment.

We create a service token as Kubernetes secret in the doppler-operator-system namespace.

kubectl create secret generic doppler-token-secret \
  --namespace doppler-operator-system \
  --from-literal=serviceToken=<DOPPLER_SERVICE_TOKEN_FOR_DEVELOPMENT_ENVIRONMENT>

secret/doppler-token-secret created

Create the DopperSecret in the default namespace.

cat <<EOF > manifest.yaml
apiVersion: secrets.doppler.com/v1alpha1
kind: DopplerSecret
metadata:
  name: doppler-secret-test
  namespace: doppler-operator-system
spec:
  tokenSecret:
    name: doppler-token-secret
  managedSecret:
    name: doppler-test-secret
    namespace: default
EOF
kubectl apply -f manifest.yaml

dopplersecret.secrets.doppler.com/doppler-secret-test created

Deploy the application in the default namespace.

cat <<EOF > deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello-server
  annotations:
    secrets.doppler.com/reload: "true"
spec:
  replicas: 1
  selector:
    matchLabels:
      app: hello-server
  template:
    metadata:
      labels:
        app: hello-server
    spec:
      containers:
        - name: hello-server
          image: ghcr.io/dirien/hello-server/hello-server:latest
          ports:
            - containerPort: 8080
          envFrom:
            - secretRef:
                name: doppler-test-secret
EOF
kubectl apply -f deployment.yaml

deployment.apps/hello-server created

And now port-forward the pod and check if the secret is available.

kubectl port-forward deployments/hello-server 8080:8080

In another terminal, we can check the env endpoint like before.

curl http://localhost:8080/env/DB_URL
DB_URL=psql://elon@localhost/modelX%

Success! The same steps can be done for the staging environment. I will leave that as an exercise for you.

Cleanup

The cleanup is very easy, we just need to call the destroy command.

pulumi destroy

Conclusion

In my opinion, the Doppler SecretOps Platform is a very promising way to approach the problem of managing secrets for a nearly infinite number of integration possibilities. In this example, I showed how to use the Doppler Secrets Operator as all things Kubernetes is what I am most familiar with. But the Doppler SecretOps Platform can be used for much more!

I particularly liked the Dashboard the most! I can see all my projects and environments in one place and can easily change the secrets for each environment. I can also see the history of changes and who changed what and when and of course I can create different teams and assign them to different projects and environments.

But I have to mention that the Doppler Secrets Operator is still in beta and there are some things that I did not like much and tried to avoid in this example. For example: currently, you can not create the namespace for the operator in advance. The namespace is created by the operator itself. That broke for me the automation of the vcluster! The init values in vcluster are not allowing to run the installation of Helm charts before the manifests are.

This is an issue for me when I want to use a vanilla GitOps approach with Flux or ArgoCD. There are issues open for both, the Doppler Secrets Operator and the vcluster project. I hope that this will be fixed soon.

I have to say, that I could maybe go around this by using more glue code but this is a demo and I wanted to keep it as simple as possible.

On the other hand, the production cluster without the vcluster is working very well and everything works as expected.

I hope you enjoyed this demo and I hope that you will try out the Doppler SecretOps Platform yourself!

Resources

Nothing is more annoying than having a project that has outdated dependencies. If you are not using tools like Dependabot or Mend(Renovate) the process of updating dependencies can be a tedious and error-prone task. This is especially true if you have a project which has a large number of dependencies.

Outdated dependencies are a security risk and could cause bugs or performance impacts. So it is important to keep them up to date. But how can we do this in a safe and efficient way?

In this short blog post, I will show you how an easy way to update your dependencies using only the command line and npm.

Preferred Way (as GitHub user)

I always recommend using dependabot or Mend(Renovate) to check for outdated dependencies and create pull requests for you. This is the most convenient way to keep your project up to date without any manual work on your side. Both tools are free and support a huge variety of package systems and languages.

Alternative Way (without GitHub)

If you don't use GitHub, or can't use one of the tools mentioned above, you can use the following approach to update your dependencies.

NPM ships with a command called npm outdated. This command will list all outdated dependencies of your project.

npm outdated
Package              Current   Wanted   Latest  Location                         Depended by
@pulumi/pulumi        3.50.0   3.50.0   3.65.1  node_modules/@pulumi/pulumi      purrl-ts
@pulumiverse/purrl     0.3.1    0.3.1    0.4.0  node_modules/@pulumiverse/purrl  purrl-ts
@types/node         18.11.17  18.16.2  18.16.2  node_modules/@types/node         purrl-ts

As you can see, the command lists all outdated dependencies. The Current column shows the version of the dependency you are currently using. The Wanted column shows the version you want to use according to your semver rules. The Latest column shows the latest version of the dependency available in the registry.

Now you can use the npm update command to update your dependencies.

npm update (-S)

This command will update all dependencies to the latest version according to your semver rules and will always use the Wanted version.

Reference Implementation in Azure DevOps and MS Teams

Let's take a look at how we can use this approach in Azure DevOps Pipelines and connect it to MS Teams to get a little ChatOps feeling.

First of all, you need to create a webhook in MS Teams. You can find a detailed description of how to do this in the official Microsoft documentation.

Next, you create a new pipeline in Azure DevOps. I will use a simple pipeline with a single stage and a single job for the sake of simplicity.

trigger:
  - main

schedules:
  - cron: '0 14 * * *'
    branches:
      include:
        - main

parameters:
  - name: webhookUrl

pool:
  vmImage: ubuntu-latest

steps:
  - task: NodeTool@0
    inputs:
      versionSource: 'spec'
      versionSpec: '18.x'
    displayName: '🧑‍🔧 Install Node.js'

  - script: |
      npm install
      npm outdated
    displayName: '🔎 Check for outdated dependencies'
    workingDirectory: 'examples/purrl-ts'

  - bash: |
      curl -v -X POST ${{ parameters.webhookUrl }} \
        -H 'Content-Type: application/json; charset=utf-8' \
        --data-binary @- << EOF
        {
          "type":"message",
          "attachments":[
            {
              "contentType":"application/vnd.microsoft.card.adaptive",
              "contentUrl":null,
              "content":{
                "$schema":"http://adaptivecards.io/schemas/adaptive-card.json",
                "type":"AdaptiveCard",
                "version":"1.2",
                "body":[
                  {
                    "type": "TextBlock",
                    "wrap": true,
                    "text": "NPN detected outdated packages at $(Build.Repository.Name) in the $(Build.SourceBranchName), more details can be found in the build log. \r\r $(Build.Repository.Uri)"
                  }
                ]
              }
            }
          ]
        }
      EOF      
    displayName: '🛫Invoke webhook'
    condition: failed()

The pipeline is triggered on a schedule every day at 2pm. When the npm outdated command detects outdated dependencies the pipeline will fail. In this case the Invoke webhook task will be executed and send a message to MS Teams.

Conclusion

Keeping your project's dependencies up to date is essential for maintaining security, fixing bugs, and improving performance. Tools like Dependabot and Mend(Renovate) can help automate the process, making it convenient and effortless. However, if you don't use GitHub or can't use these tools, you can still manage your dependencies efficiently using the command line and NPM with commands like npm outdated and npm update.

By adopting these practices, you can ensure that your project stays current, secure, and performs at its best while minimizing the manual work involved in dependency management.

Resources

ChatGPT is a large language model developed by OpenAI based on the GPT-3.5 architecture. It is designed to generate human-like responses to text inputs, making it an ideal tool for natural language processing tasks such as language translation, text summarization, and question answering. ChatGPT has been trained on a vast amount of data, allowing it to understand and generate responses to a wide range of topics.

Getting Started

To begin using ChatGPT, you'll need an internet connection and a compatible web browser. While you don't have to install any software, additional steps like creating an account or obtaining an API key may be necessary, depending on the specific version of ChatGPT you want to access. For more information on getting started with an OpenAI-powered ChatGPT, visit their website for detailed instructions.

What is ChatGPT?

ChatGPT is an AI language model that generates human-like responses to text inputs. It uses a deep learning architecture called GPT-3.5, which has been trained on a large amount of text data. ChatGPT can understand and generate responses to a wide range of topics, making it an ideal tool for natural language processing tasks. With ChatGPT, you can have a conversation with an AI that feels like talking to a real person.

Rust

According to its official website, "Rust is a systems programming language that runs blazingly fast, prevents segfaults, and guarantees thread safety." Rust is a statically typed, compiled programming language that was designed with a focus on system-level programming, but it can also be used for web development, game development, and other application domains.

Rust's syntax is similar to that of C and C++, but it is a more modern language that provides features like memory safety, zero-cost abstractions, and guaranteed thread safety. Rust is a systems programming language that is memory-safe and thread-safe by default, making it a popular choice for building high-performance and reliable software.

Rust is a multi-paradigm language that supports both functional and imperative programming styles. It is designed to be easy to learn and use, with a clear and concise syntax that emphasizes code readability. Rust also has a growing community of developers who contribute to its ecosystem of libraries, frameworks, and tools.

Overall, Rust is a powerful and versatile programming language that is well-suited for building reliable and high-performance systems. Its emphasis on safety and concurrency make it a popular choice for developers who value those qualities in their software.

Let's get Rust-y with it!

Okay, that was a terrible joke, but don't let that discourage you from learning Rust. Rust is a seriously cool language that offers a lot of benefits over other programming languages. It may take a bit of effort to get started, but trust me, it's worth it. So let's dive in and start learning Rust!

Keep in mind that the response you receive could be completely different from the ones I got!

We've got a solid training plan so far. Let's ask our teacher to begin with the first chapters.

Moving on to the next chapter about Basic Rust Programming.

This is excellent output; we can delve deeper if necessary. Let's switch to the next topic, Rust Functions and Modules.

I noticed a new thing called Cargo.toml. Let's ask our teacher about it.

That's pretty good, to be honest. Now we're getting to the big topic, Rust Ownership and Borrowing, and I want to compare it to Go.

Next, I want to learn more about Rust Collections and Iterators.

Error handling is always important in a new programming language, so let's ask our teacher about Rust Error Handling and how it compares to Golang.

It's time to apply what we've learned so far. Let's ask our teacher to show and explain a basic CLI for calculating Fibonacci numbers.

That worked very well! It's compiling and providing the numbers!

➜ cargo run 10
   Compiling fibonacci_cli v0.1.0 (/Users/dirien/trash-stacks/rust/fibonacci_cli)
    Finished dev [unoptimized + debuginfo] target(s) in 0.76s
     Running `target/debug/fibonacci_cli 10`
The 10th Fibonacci number is 55

Let's move on to the next topic, Rust Concurrency, and compare it to Golang once again.

Let's ask for a working example here too:

cargo run  
   Compiling fibonacci_cli v0.1.0 (/Users/dirien/trash-stacks/rust/cli)
    Finished dev [unoptimized + debuginfo] target(s) in 0.74s
     Running `target/debug/cli`
Result: 0
Result: 2
Result: 4
Result: 8
Result: 6

That's not bad. Let's see how our teacher handles the next topic about Rust Web Development.

The teacher talks about dependencies! We should ask about this and how to add them to our project.

Oh, I got an error message! Let's ask the teacher for help.

Great, this answer fixed the error messages, and the program is working fine!

curl 127.0.0.1:8080/hello
{"greeting":"Hello, World!"}%

Let's ask how we can create a second endpoint and post a JSON with our name to it! The new endpoint should then greet us with our name.

curl -X POST -H "Content-Type: application/json" -d '{"name": "Alice"}' http://localhost:8080/greet

{"greeting":"Hello, Alice!"}%

Everything worked fine so far. It's time to finish the session for today! Let's ask our teacher for some further resource recommendations.

Conclusion

Using ChatGPT to learn Rust can be a helpful tool for getting started with the language. Its ability to generate human-like responses to text inputs can provide a conversational approach to learning that can be engaging and intuitive. However, it is important to keep in mind that ChatGPT is an AI language model and it may generate responses that are incorrect, misleading, or outdated.

Verify the Information

As a learner, it is essential to verify the information provided by ChatGPT by consulting official documentation or reliable sources. While ChatGPT can provide useful insights and guidance, it is not a substitute for learning from reputable sources and practicing programming on your own.

Stay Informed

Additionally, it is important to note that ChatGPT may not always have the most up-to-date information on Rust or its libraries. As Rust is a fast-evolving language with new features and updates being released frequently, it is essential to stay informed on the latest developments and to use current resources when learning the language.

In summary, while ChatGPT can be a useful tool for learning Rust, it is important to approach it with caution and to supplement its responses with official documentation and reliable sources. By doing so, learners can maximize the benefits of ChatGPT while ensuring that they are learning accurate and up-to-date information about Rust.

Disclaimer

All content in this blog was generated by ChatGPT, an AI language model. While the information provided is generally accurate, it may not be 100% correct or up-to-date. Always consult official documentation or seek assistance from the Rust community when in doubt.

When I saw this tweet on my "For You" page, I knew that I needed to submit a CfP.

There's no shortage of meetups, conferences, and user groups out there, and most of them are top-notch and definitely worth attending. If only I could visit them all, with an endless budget! Conference in Japan? Konnichi wa! How about India? Namaste, here I come!

But alas, that's not my reality. So, every year, I start my search for the most exciting conferences to add to my agenda. One of my go-to sources for this is Aurélie Vache's fantastic repository:

And, for real, KCD Israel is one of these highlights! I had to submit a CfP. After quickly confirming that English presentations were welcome, I got to work.

Time to craft the perfect CfP!

I'm currently employed by Pulumi, and my position in a cloud-native startup provides endless inspiration for talk ideas! Plus, I genuinely enjoy sharing my knowledge and engaging with others.

When Things Don't Go as Planned

Well, this is what can happen:

Yes, sometimes CfPs get declined. There are countless reasons for this, and it used to frustrate me to no end.

But that's just the name of the game! You win some, you lose some. As the French say, "c'est la vie"!

Despite the rejection, my manager at Pulumi and I agreed that it still made sense for me to head to Tel Aviv. We have clients there, and it's a great opportunity to engage with the local community and get a feel for the cloud-native landscape.

So Shalom Aleichem KCD Israel! I'm coming for you!

Day -1: Flight and the first evening

Bright and early, I drove to Frankfurt and boarded my flight to Tel Aviv.

It was a 5 hours flight out there, very calm and relaxed. Most of my fellow passengers were Germans embarking on a journey to explore the Holy Land! When they asked me where I was going, they were surprised that I go to a tech conference!

Check out this stunning shot of our approach to Tel Aviv

After I arrived, I went out to grab something to eat. Luckily I meet with Viktor Farcic and had a very good lunch that evening. And as you can see from the picture, a very intense first talk about GitOps, Security and Infrastructure as Code!

The conference day!

Finally, the day of the conference! Yeah! I put on my best Pulumi swag shirt and head off from my hotel to the venue center.

Quickly, went to collect my badge to then head over to the main stage to wait for the KCD to start!

Important: Find a spot in the middle! It is really like the cinema! You don't want to sit too far away from the action!

So, I think I get one of the best spots in the main stage place! Ready to enjoy a day of cloud native!

I am still very shy, when it comes to taking selfies! I need to learn how to smile!

The organization team

Big thanks and shoutout to the organization team (Shimon Tolts, Dana Fine, Arthur Berezin, Dotan Horovits, Itay Shakury and Ronen Levinson) of KCD Israel and their well-deserved big round of applause.

Shimon Tolts spend some time explaining the values of this event, unfortunatly it was on Hebrew and I was not able to understand everything but the slides transported the message very well.

The first talk of the conference was from Lori Lorusso called The CNCF & You - Growing Community in the CloudNative Landscape

She gave a very good talk about the growing community and the role of the CNCF! Very interesting to hear, as I did not understand before all the parts of the CNCF.

The next speaker was Andrew Martin and his talk: The Irresistible Rise of Cloud Native: What the Future Means for YOU

First time I listen to a talk from Andrew and I thoroughly enjoyed his prediction on what the cloud-native future will be. Of course, he mentioned things like WASM and platform engineering!

Next Daniel Gur about Prometheus That Scales. This talk was very interesting, as Daniel shared his experiences and lessons learned from running Prometheus at scale in his company Outbrain!

Next was from Lior Yantovski and Chen Fleisher about GitOps with Argo CD and Helm: Managing Your Kubernetes Repository. This talk was in Hebrew so I had to rely on the slides and the people in the audience!

Even though I did not understand the talk, I loved on memes on their slides!

Yepp, you should never

kubectl apply ...

in production! Use GitOps!

Funny is also the amount of Kelsey Hightower quotes on the slides! Here is number one!

Now we had three lighting talks going on. I could not take a picture of all of them, as I wanted to listen to them and they were really quick with their slides:

• Udi Rot with Caretta: Mapping Services With eBPF

  

• Benny Rochwerger and Never Go Limitless – On the Importance of ResourceQuotas at the Edge
  Very important talk and a call for all Kubernetes ops teams: Set your resource quotes on edge Kubernetes clusters!

• Anton Weiss: The Promise of WASM
  No further explanation needed! WASM so hot right now!

Time for the lunch break, and I had the time to stroll a little bit through the booths. I spotted this nice booth from groundcover! I love this kind of statements!

Monitoring is worthless

Next to visiting all the booths and eating delicious hummus, I had the opportunity to talk to different persons at the venue about all things cloud and infrastructure as code. Interesting to see how people think and knew about Pulumi.

After the lunch break, we continue the talks with Knative: Is it for me? from Elad Hirsch and Kornel Chlebovics.

I love event-driven architecture and Knative is one of my preferred projects. I worked with it since nearly day one on our Openshift Clusters at my former workplace. I organized a meetup around this too. Check out the recording, if you are interested into!

And it is time for the second Kelsey quote of the day! And tbh, you can't repeat the great quotes of Kelsey enough!

The next talk is Kubernetes Reloaded: AKA How to Avoid the Dev Plateau with CRDs by Rona Hirsch and Guy Menahem!

CRDs are crucial to extending Kubernetes to your needs and Rona and Guy did a great work to explain what CRDs are under the hood!

And the last Kelsey quote of the day!

A quick switch of the speakers on stage and we are already in the next talk The Power of OPA Gatekeeper: Enforcing Kubernetes Access Control to secure your runtime from Batel Zohar Tova and Dana Rozen

They talked about how to secure your Kubernetes environment using the OPA Gatekeeper while using ArgoCD! Very powerful combo and a good talk!

Eran Bibi from Firefly gave the next talk Enter the Machines: Reducing Friction in Cloud Native using AI

He talked about using AI for code and config generation! A very good talk and fits very well the current trend on LLM and GPT!

The talk from Boris Cherkasky about Stop The Kuberspendes - Getting Your K8s Cost Under Control I did not visit, as it was in Hebrew. I used the time with talking with people in the exhibition area! Was also very good and fruitful for me!

And now the last talk of the event comes from Viktor Farcic called Shifting Left Stateful Applications In Kubernetes!

Always enjoy the talks of Viktor and this one was no exception. He explained on how to run stateful apps in Kubernetes in a self-service approach without the need to open a JIRA ticket and wait that others to perform the task for you!

And with this talk, the KCD Israel 2023 ends. Last time I hit the exhibition area and experienced this unique atmosphere!

Conclusion and flight back home

KCD Israel's inaugural edition was a fantastic experience, and I thoroughly enjoyed myself! I heard that tech workers in Tel Aviv work hard and play hard, and this event seemed to embody that spirit!

You could feel how much the people enjoyed the event. Everyone had a smile on their face, and the talks were always full. Unlike some other events I've attended, people were genuinely present, not working on their laptops during presentations.

KCD Israel is definitely an event I'll revisit!

Before I knew it, time had flown by, and I found myself waiting for the train to whisk me back to the airport.

Links

In this article, I want to compare the performance of two different web frameworks for Rust and Go. Both frameworks are very similar in their design (all are inspired by Express.js) and both claim to be the fastest web framework (blazing fast). Both frameworks are easy to use, which is a big plus for me (I am not the smartest guy in the world, so easy is good).

The Competitors in detail

Nickele.rs (Rust)

For Rust, I have chosen the Nickel.rs framework. It is a minimal and lightweight framework for web apps in Rust. It is inspired by Express.js and provides a lot of features like flexible routing, middleware, JSON handling, and more. And it's blazingly fast!

Fiber (Go)

As a contender for Go, I have chosen the Fiber framework. This framework is also inspired by Express.js and is built on top of Fasthttp. It has a lot of features like middleware, routing, WebSockets, and more. And it claims to have extreme performance and a small memory footprint.

The Battle 🥊

The specs of my machine are Apple M1 Max (10 Core CPU) with 32GB of RAM.

The Tests will be written in bombardier and will be executed for 50, 100 and 500 concurrent users with executing 5M requests.

I use the following versions:

• Go: go1.20.3 darwin/arm64

• Rust: rustc 1.65.0 (897e37553 2022-11-02)

The Test Code

Nickel.rs (Rust)

#[macro_use]
extern crate nickel;

use nickel::Nickel;

fn main() {
    let mut server = Nickel::new();

    server.utilize(router! {
        get "**" => |_req, _res| {
            "Hello world!"
        }
    });

    server.listen("127.0.0.1:6767").unwrap();
}

Fiber (Go)

package main

import "github.com/gofiber/fiber/v2"

func main() {
    app := fiber.New()

    app.Get("/", func(c *fiber.Ctx) error {
        return c.SendString("Hello, World 🐹!")
    })

    app.Listen(":3000")
}

The Results

50 concurrent users

100 concurrent users

500 concurrent users

Conclusion 🎉

Based on the data provided, Nickel.rs (Rust) is the winner. There are several reasons for this:

1. 🚀 Faster response times: Nickel.rs (Rust) has lower mean, median, and 90th percentile response times across all levels of concurrent users compared to Fiber (Go).

2. ⚡ Higher request per second: Nickel.rs (Rust) can handle more requests per second than Fiber (Go) in each test.

3. 🌡️ Lower CPU usage: Nickel.rs (Rust) uses less CPU (about half) compared to Fiber (Go) in all tests.

4. 🧠 Lower memory usage: Nickel.rs (Rust) uses significantly less memory (about 3 to 7 times less) compared to Fiber ( Go) in all tests.

In conclusion, 🏆 Rust (specifically Nickel.rs) outperforms Go (Fiber) in terms of response times, request handling, CPU usage, and memory consumption, making it the winner in this comparison.

Resources

• Gopher from cover: https://dribbble.com/shots/14749679-Golang-gopher-master-karate-ninja-cool-sticker-shirt-print

• Rust crab from cover: https://ih1.redbubble.net/image.1001647428.6797/st,small,507x507-pad,600x600,f8f8f8.jpg

Introduction

In this blog post, I want to show you how to create and use AWS Cognito as an OAuth2 provider for ArgoCD. And this will be all done by using Pulumi. This is very convenient, as you can do not need any manual steps to configure the ArgCD deployment with properties of the Cognito service. The Pulumi code will do all the work for you.

The demo infrastructure will be deployed to AWS and will look like this:

• AWS EKS Cluster

• AWS Cognito User Pool with a Client and Cognito User.

• AWS Load Balancer Controller

• External DNS (as I host my domains on DigitalOcean)

• ArgoCD

So let's get started!

Prerequisites

To follow this article, you will need the following:

• Pulumi CLI installed.

• kubectl installed.

• optional K9s, if you want to quickly interact with your cluster.

• AWS: AWS account

I will not go into detail on how to install the prerequisites. If you need help, please refer to the links above.

Creating the infrastructure with Pulumi

Create your Pulumi project

Pulumi is a multi-language infrastructure as Code tool using imperative languages to create a declarative infrastructure description.

You have a wide range of programming languages available, and you can use the one you and your team are the most comfortable with. Currently, (11/2022) Pulumi supports the following languages:

• Node.js (JavaScript / TypeScript)

• Python

• Go

• Java

• .NET (C#, VB, F#)

• YAML

In this article, we will use Go as our programming language. You can of course use any other language supported by Pulumi.

Create a project folder (for example pulumi-cognito-argocd) and navigate into the newly created directory:

mkdir pulumi-cognito-argocd
cd pulumi-cognito-argocd

Now, we need to initialize our project. We will use the pulumi new command to do this. This command will create a new Pulumi project in the current directory. We will use the aws-go template, which will create a new project with a Go template for AWS.

pulumi new aws-go --force

You can leave the default values in the prompt but maybe adjust the AWS region to your preference. I chose eu-central-1 as my region for this demo.

As we're going to deploy several Helm charts, we need to install the pulumi-kubernetes provider. To install providers, we need to add the following go libarie to our go.mod file:

go get github.com/pulumi/pulumi-kubernetes/sdk/v3

And, as I don't want to create an EKS cluster from scratch, we will use the pulumi-eks provider to create the EKS cluster for us. We can configure the EKS cluster with some convenient options, like the Kubernetes version, OIDC support and more.

go get github.com/pulumi/pulumi-eks/sdk

Creating the network infrastructure

With all libraries installed, we can start to create our infrastructure. First, we need to create the network infrastructure. To do this, head over to the main.go file and add the following code:

package main

// omitted for brevity

func main() {
    pulumi.Run(func(ctx *pulumi.Context) error {
        vpc, err := ec2.NewVpc(ctx, "cognito-argocd-vpc", &ec2.VpcArgs{
            CidrBlock: pulumi.String("172.31.0.0/16"),
        })
        if err != nil {
            return err
        }
        igw, err := ec2.NewInternetGateway(ctx, "cognito-argocd-igw", &ec2.InternetGatewayArgs{
            VpcId: vpc.ID(),
        })
        if err != nil {
            return err
        }
        rt, err := ec2.NewRouteTable(ctx, "cognito-argocd-rt", &ec2.RouteTableArgs{
            VpcId: vpc.ID(),
            Routes: ec2.RouteTableRouteArray{
                &ec2.RouteTableRouteArgs{
                    CidrBlock: pulumi.String("0.0.0.0/0"),
                    GatewayId: igw.ID(),
                },
            },
        })
        if err != nil {
            return err
        }

        var publicSubnetIDs pulumi.StringArray

        // Create a subnet for each availability zone
        for i, az := range availabilityZones {
            publicSubnet, err := ec2.NewSubnet(ctx, fmt.Sprintf("cognito-argocd-subnet-%d", i), &ec2.SubnetArgs{
                VpcId:                       vpc.ID(),
                CidrBlock:                   pulumi.String(publicSubnetCidrs[i]),
                MapPublicIpOnLaunch:         pulumi.Bool(true),
                AssignIpv6AddressOnCreation: pulumi.Bool(false),
                AvailabilityZone:            pulumi.String(az),
                Tags: pulumi.StringMap{
                    "Name":                   pulumi.Sprintf("eks-public-subnet-%d", az),
                    clusterTag:               pulumi.String("owned"),
                    "kubernetes.io/role/elb": pulumi.String("1"),
                },
            })
            if err != nil {
                return err
            }
            _, err = ec2.NewRouteTableAssociation(ctx, fmt.Sprintf("cognito-argocd-rt-association-%s", az), &ec2.RouteTableAssociationArgs{
                RouteTableId: rt.ID(),
                SubnetId:     publicSubnet.ID(),
            })
            if err != nil {
                return err
            }
            publicSubnetIDs = append(publicSubnetIDs, publicSubnet.ID())
        }

        return nil
    })
}

This code will create a new VPC with a public subnet in each availability zone. An internet gateway and a route table will be created as well. The route table will be associated with the public subnets.

Creating the AWS Cognito infrastructure

With the network infrastructure created, we can work on the creation of the AWS Cognito infrastructure. Add the following code to the main.go file:

package main

// omitted for brevity

func main() {
    pulumi.Run(func(ctx *pulumi.Context) error {
        // omitted for brevity
        userPool, err := cognito.NewUserPool(ctx, "cognito-argocd-user-pool", &cognito.UserPoolArgs{
            AliasAttributes: pulumi.StringArray{
                pulumi.String("email"),
                pulumi.String("preferred_username"),
            },
            AutoVerifiedAttributes: pulumi.StringArray{
                pulumi.String("email"),
            },
            Schemas: cognito.UserPoolSchemaArray{
                &cognito.UserPoolSchemaArgs{
                    AttributeDataType:      pulumi.String("String"),
                    DeveloperOnlyAttribute: pulumi.Bool(false),
                    Mutable:                pulumi.Bool(true),
                    Name:                   pulumi.String("name"),
                    Required:               pulumi.Bool(true),
                    StringAttributeConstraints: &cognito.UserPoolSchemaStringAttributeConstraintsArgs{
                        MinLength: pulumi.String("3"),
                        MaxLength: pulumi.String("70"),
                    },
                },
                &cognito.UserPoolSchemaArgs{
                    AttributeDataType:      pulumi.String("String"),
                    DeveloperOnlyAttribute: pulumi.Bool(false),
                    Mutable:                pulumi.Bool(true),
                    Name:                   pulumi.String("email"),
                    Required:               pulumi.Bool(true),
                    StringAttributeConstraints: &cognito.UserPoolSchemaStringAttributeConstraintsArgs{
                        MinLength: pulumi.String("3"),
                        MaxLength: pulumi.String("70"),
                    },
                },
            },
            AdminCreateUserConfig: &cognito.UserPoolAdminCreateUserConfigArgs{
                AllowAdminCreateUserOnly: pulumi.Bool(true),
            },
        })
        if err != nil {
            return err
        }

        _, err = cognito.NewUser(ctx, "cognito-argocd-admin", &cognito.UserArgs{
            UserPoolId:        userPool.ID(),
            Username:          pulumi.String("admin"),
            TemporaryPassword: pulumi.String("Admin123!"),
            Attributes: pulumi.StringMap{
                "email":          pulumi.String(adminUserEmail),
                "email_verified": pulumi.String("true"),
            },
        })
        if err != nil {
            return err
        }

        _, err = cognito.NewUserPoolDomain(ctx, "cognito-argocd-user-pool-domain", &cognito.UserPoolDomainArgs{
            Domain:     pulumi.String("argocd-ui"),
            UserPoolId: userPool.ID(),
        })
        if err != nil {
            return err
        }

        userPoolClient, err := cognito.NewUserPoolClient(ctx, "cognito-argocd-user-pool-client", &cognito.UserPoolClientArgs{
            UserPoolId: userPool.ID(),
            AllowedOauthFlows: pulumi.StringArray{
                pulumi.String("code"),
                pulumi.String("implicit"),
            },
            AllowedOauthFlowsUserPoolClient: pulumi.Bool(true),
            AllowedOauthScopes: pulumi.StringArray{
                pulumi.String("openid"),
                pulumi.String("email"),
                pulumi.String("profile"),
            },
            SupportedIdentityProviders: pulumi.StringArray{
                pulumi.String("COGNITO"),
            },
            GenerateSecret: pulumi.Bool(true),
            CallbackUrls: pulumi.StringArray{
                pulumi.String("https://argocd.ediri.online/auth/callback"),
            },
        })
        if err != nil {
            return err
        }

        return nil
    })
}

This code will create the AWS Cognito user pool, an admin user, and a user pool client. Some important things to note here:

• The adminUser resource is created with a temporary password. This will be used to log in for the first time to the Argo CD UI. You will need to change the password after the first login.

• The domain name for the Argo CD UI is set to argocd-ui. This is a must to get OAuth2 working with Argo CD. You will need to change this to match your domain name or you can use one that is provided by AWS Cognito, which will be in the format https://<domain_name>.auth.<region>.amazoncognito.com.

• The userPoolClient resource is created with a callback URL. This is the URL that the user will be redirected to after they have authenticated with AWS Cognito. This URL will be used by the Argo CD UI to authenticate the user. The URL is set to https://argocd.ediri.online/auth/callback in this example, but you will need to change it to match your domain name.

When we later create the whole infrastructure, you will end up with a whole lot of endpoints:

• The AWS Cognito Auth endpoint: https://<domain_name>.auth.<region>.amazoncognito.com/oauth2/authorize

• The AWS Cognito Token endpoint: https://<domain_name>.auth.<region>.amazoncognito.com/oauth2/token

• The AWS Cognito User info endpoint: https://<domain_name>.auth.<region>.amazoncognito.com/oauth2/userInfo

• The AWS Cognito End session endpoint: https://<domain_name>.auth.<region>.amazoncognito.com/logout

• The AWS Cognito JWKS endpoint: https://<domain_name>.auth.<region>.amazoncognito.com/.well-known/jwks.json

• The AWS Cognito Issuer: https://cognito-idp.<region>.amazonaws.com/<user_pool_id>

The issuer, we will use later when we create the Argo CD configuration.

Create the EKS cluster

After creating the network infrastructure and the AWS Cognito infrastructure, we can now create the EKS cluster. Add the following code to the main.go file:

package main

// omitted for brevity

func main() {
    pulumi.Run(func(ctx *pulumi.Context) error {
        // omitted for brevity
        cluster, err := eks.NewCluster(ctx, clusterName, &eks.ClusterArgs{
            Name:            pulumi.String(clusterName),
            VpcId:           vpc.ID(),
            SubnetIds:       publicSubnetIDs,
            InstanceType:    pulumi.String("t3.medium"),
            DesiredCapacity: pulumi.Int(2),
            MinSize:         pulumi.Int(1),
            MaxSize:         pulumi.Int(3),
            ProviderCredentialOpts: eks.KubeconfigOptionsArgs{
                ProfileName: pulumi.String("default"),
            },
            CreateOidcProvider: pulumi.Bool(true),
        })
        if err != nil {
            return err
        }

        ctx.Export("kubeconfig", pulumi.ToSecret(cluster.Kubeconfig))

        return nil
    })
}

That's it! This very short code will create a full-blown EKS cluster with 2 worker nodes. The kubeconfig output will be made available as a secret in the Pulumi stack. We will use this later to connect to the cluster via kubectl or k9s.

But now comes the tricky part! We need to deploy several services to the cluster, to get Argo CD up and running. These services are:

• AWS Load Balancer Controller

• External DNS

• Argo CD

For the TLS certificate, we will use the AWS Certificate Manager. I created everything beforehand, and this will be not covered in this post. In short, I created a CNAME record and my DNS provider, and the AWS Certificate Manager created a certificate for me. I use the arn of the certificate in the annotations of the Argo CD ingress.

Deploy Argo CD

We reach the final part of this blog post! Add this code to the main.go file:

package main

// omitted for brevity

func main() {
    pulumi.Run(func(ctx *pulumi.Context) error {
        // omitted for brevity
        albRole, err := iam.NewRole(ctx, "cognito-argocd-alb-role", &iam.RoleArgs{
            AssumeRolePolicy: pulumi.All(cluster.Core.OidcProvider().Arn(), cluster.Core.OidcProvider().Url()).ApplyT(func(args []interface{}) (string, error) {
                arn := args[0].(string)
                url := args[1].(string)
                return fmt.Sprintf(`{
                        "Version": "2012-10-17",
                        "Statement": [
                            {
                                "Effect": "Allow",
                                "Principal": {
                                    "Federated": "%s"
                                },
                                "Action": "sts:AssumeRoleWithWebIdentity",
                                "Condition": {
                                    "StringEquals": {
                                        "%s:sub": "%s"
                                    }
                                }
                            }
                        ]
                    }`, arn, url, albServiceAccount), nil
            }).(pulumi.StringOutput),
        })
        if err != nil {
            return err
        }

        albPolicyFile, _ := os.ReadFile("../policies/alb-iam-policy.json")
        albIAMPolicy, err := iam.NewPolicy(ctx, "cognito-argocd-alb-policy", &iam.PolicyArgs{
            Policy: pulumi.String(string(albPolicyFile)),
        }, pulumi.DependsOn([]pulumi.Resource{albRole}))
        if err != nil {
            return err
        }
        _, err = iam.NewRolePolicyAttachment(ctx, "cognito-argocd-alb-role-attachment", &iam.RolePolicyAttachmentArgs{
            PolicyArn: albIAMPolicy.Arn,
            Role:      albRole.Name,
        }, pulumi.DependsOn([]pulumi.Resource{albIAMPolicy}))
        if err != nil {
            return err
        }

        provider, err := kubernetes.NewProvider(ctx, "my-provider", &kubernetes.ProviderArgs{
            Kubeconfig:            cluster.KubeconfigJson,
            EnableServerSideApply: pulumi.Bool(true),
        })
        if err != nil {
            return err
        }

        ns, err := corev1.NewNamespace(ctx, albNamespace, &corev1.NamespaceArgs{
            Metadata: &metav1.ObjectMetaArgs{
                Name: pulumi.String(albNamespace),
                Labels: pulumi.StringMap{
                    "app.kubernetes.io/name": pulumi.String("aws-load-balancer-controller"),
                },
            },
        }, pulumi.Provider(provider))
        if err != nil {
            return err
        }
        sa, err := corev1.NewServiceAccount(ctx, "aws-lb-controller-sa", &corev1.ServiceAccountArgs{
            Metadata: &metav1.ObjectMetaArgs{
                Name:      pulumi.String("aws-lb-controller-serviceaccount"),
                Namespace: ns.Metadata.Name(),
                Annotations: pulumi.StringMap{
                    "eks.amazonaws.com/role-arn": albRole.Arn,
                },
            },
        }, pulumi.Provider(provider), pulumi.DependsOn([]pulumi.Resource{ns}))
        if err != nil {
            return err
        }

        _, err = helm.NewRelease(ctx, "aws-load-balancer-controller", &helm.ReleaseArgs{
            Chart:     pulumi.String("aws-load-balancer-controller"),
            Version:   pulumi.String("1.4.8"),
            Namespace: ns.Metadata.Name(),
            RepositoryOpts: helm.RepositoryOptsArgs{
                Repo: pulumi.String("https://aws.github.io/eks-charts"),
            },
            Values: pulumi.Map{
                "clusterName": cluster.EksCluster.ToClusterOutput().Name(),
                "region":      pulumi.String("eu-central-1"),
                "serviceAccount": pulumi.Map{
                    "create": pulumi.Bool(false),
                    "name":   sa.Metadata.Name(),
                },
                "vpcId": cluster.EksCluster.VpcConfig().VpcId(),
                "podLabels": pulumi.Map{
                    "stack": pulumi.String("eks"),
                    "app":   pulumi.String("aws-lb-controller"),
                },
            },
        }, pulumi.Provider(provider), pulumi.DependsOn([]pulumi.Resource{ns, sa}))
        if err != nil {
            return err
        }

        externalDNS, err := helm.NewRelease(ctx, "external-dns", &helm.ReleaseArgs{
            Chart:           pulumi.String("external-dns"),
            Version:         pulumi.String("1.12.2"),
            Namespace:       pulumi.String("external-dns"),
            CreateNamespace: pulumi.Bool(true),
            RepositoryOpts: helm.RepositoryOptsArgs{
                Repo: pulumi.String("https://kubernetes-sigs.github.io/external-dns/"),
            },
            Values: pulumi.Map{
                "provider": pulumi.String("digitalocean"),
                "sources": pulumi.Array{
                    pulumi.String("ingress"),
                },
                "env": pulumi.Array{
                    pulumi.Map{
                        "name":  pulumi.String("DO_TOKEN"),
                        "value": config.GetSecret(ctx, "do"),
                    },
                },
            },
        }, pulumi.Provider(provider))

        _, err = helm.NewRelease(ctx, "argocd", &helm.ReleaseArgs{
            Chart:           pulumi.String("argo-cd"),
            Version:         pulumi.String("5.28.0"),
            Namespace:       pulumi.String("argocd"),
            CreateNamespace: pulumi.Bool(true),
            RepositoryOpts: helm.RepositoryOptsArgs{
                Repo: pulumi.String("https://argoproj.github.io/argo-helm"),
            },
            Values: pulumi.Map{
                "dex": pulumi.Map{
                    "enabled": pulumi.Bool(false),
                },
                "server": pulumi.Map{
                    "extraArgs": pulumi.Array{
                        pulumi.String("--insecure"),
                        pulumi.String("--enable-gzip"),
                    },
                    "ingress": pulumi.Map{
                        "enabled":          pulumi.Bool(true),
                        "hosts":            pulumi.Array{pulumi.String("argocd.ediri.online")},
                        "ingressClassName": pulumi.String("alb"),
                        "annotations": pulumi.Map{
                            "alb.ingress.kubernetes.io/target-type":        pulumi.String("ip"),
                            "alb.ingress.kubernetes.io/scheme":             pulumi.String("internet-facing"),
                            "alb.ingress.kubernetes.io/load-balancer-name": pulumi.String("argocd"),
                            "alb.ingress.kubernetes.io/certificate-arn":    config.GetSecret(ctx, "cert_arn"),
                        },
                    },
                },
                "configs": pulumi.Map{
                    "rbac": pulumi.Map{
                        "policy.default": pulumi.String("role:readonly"),
                        "policy.csv":     pulumi.Sprintf(`g, %s, role:admin`, adminUserEmail),
                        "scopes":         pulumi.String(`[email]`),
                    },
                    "cm": pulumi.Map{
                        "admin.enabled": pulumi.Bool(false),
                        "url":           pulumi.String("https://argocd.ediri.online"),
                        "oidc.config": pulumi.Sprintf(`name: Cognito
issuer: https://cognito-idp.eu-central-1.amazonaws.com/%s
clientID: %s
clientSecret: %s
requestedScopes: ["openid", "profile", "email"]
requestedIDTokenClaims: {"email": {"essential": true}}`, userPool.ID(), userPoolClient.ID(), userPoolClient.ClientSecret),
                    },
                },
            },
        }, pulumi.Provider(provider), pulumi.DependsOn([]pulumi.Resource{ns, externalDNS}))
        if err != nil {
            return err
        }

        return nil
    })
}

There is a lot, and I mean really a lot going on here. I will try to explain it as best as I can. Let's start with the ALB Controller part:

package main

// omitted for brevity

func main() {
    albRole, err := iam.NewRole(ctx, "cognito-argocd-alb-role", &iam.RoleArgs{
        AssumeRolePolicy: pulumi.All(cluster.Core.OidcProvider().Arn(), cluster.Core.OidcProvider().Url()).ApplyT(func(args []interface{}) (string, error) {
            arn := args[0].(string)
            url := args[1].(string)
            return fmt.Sprintf(`{
                        "Version": "2012-10-17",
                        "Statement": [
                            {
                                "Effect": "Allow",
                                "Principal": {
                                    "Federated": "%s"
                                },
                                "Action": "sts:AssumeRoleWithWebIdentity",
                                "Condition": {
                                    "StringEquals": {
                                        "%s:sub": "%s"
                                    }
                                }
                            }
                        ]
                    }`, arn, url, albServiceAccount), nil
        }).(pulumi.StringOutput),
    })
    if err != nil {
        return err
    }

    albPolicyFile, _ := os.ReadFile("../policies/alb-iam-policy.json")
    albIAMPolicy, err := iam.NewPolicy(ctx, "cognito-argocd-alb-policy", &iam.PolicyArgs{
        Policy: pulumi.String(string(albPolicyFile)),
    }, pulumi.DependsOn([]pulumi.Resource{albRole}))
    if err != nil {
        return err
    }
    _, err = iam.NewRolePolicyAttachment(ctx, "cognito-argocd-alb-role-attachment", &iam.RolePolicyAttachmentArgs{
        PolicyArn: albIAMPolicy.Arn,
        Role:      albRole.Name,
    }, pulumi.DependsOn([]pulumi.Resource{albIAMPolicy}))
    if err != nil {
        return err
    }
}

We define the role that will be assumed by the ALB Controller. The AssumeRolePolicy field specifies the trust policy for the role, which allows the ALB to assume the role using the OpenID Connect (OIDC) provider for the EKS cluster.

The code then creates an AWS IAM policy for the ALB to use with the iam.NewPolicy function. The Policy field specifies the contents of the policy, which is read from a JSON file using the os.ReadFile function. The role is then attached to the policy using the iam.NewRolePolicyAttachment function.

Next, we create the ALB Controller itself:

package main

// omitted for brevity

func main() {
    // omitted for brevity
    provider, err := kubernetes.NewProvider(ctx, "my-provider", &kubernetes.ProviderArgs{
        Kubeconfig:            cluster.KubeconfigJson,
        EnableServerSideApply: pulumi.Bool(true),
    })
    if err != nil {
        return err
    }
    ns, err := corev1.NewNamespace(ctx, albNamespace, &corev1.NamespaceArgs{
        Metadata: &metav1.ObjectMetaArgs{
            Name: pulumi.String(albNamespace),
            Labels: pulumi.StringMap{
                "app.kubernetes.io/name": pulumi.String("aws-load-balancer-controller"),
            },
        },
    }, pulumi.Provider(provider))
    if err != nil {
        return err
    }
    sa, err := corev1.NewServiceAccount(ctx, "aws-lb-controller-sa", &corev1.ServiceAccountArgs{
        Metadata: &metav1.ObjectMetaArgs{
            Name:      pulumi.String("aws-lb-controller-serviceaccount"),
            Namespace: ns.Metadata.Name(),
            Annotations: pulumi.StringMap{
                "eks.amazonaws.com/role-arn": albRole.Arn,
            },
        },
    }, pulumi.Provider(provider), pulumi.DependsOn([]pulumi.Resource{ns}))
    if err != nil {
        return err
    }

    _, err = helm.NewRelease(ctx, "aws-load-balancer-controller", &helm.ReleaseArgs{
        Chart:     pulumi.String("aws-load-balancer-controller"),
        Version:   pulumi.String("1.4.8"),
        Namespace: ns.Metadata.Name(),
        RepositoryOpts: helm.RepositoryOptsArgs{
            Repo: pulumi.String("https://aws.github.io/eks-charts"),
        },
        Values: pulumi.Map{
            "clusterName": cluster.EksCluster.ToClusterOutput().Name(),
            "region":      pulumi.String("eu-central-1"),
            "serviceAccount": pulumi.Map{
                "create": pulumi.Bool(false),
                "name":   sa.Metadata.Name(),
            },
            "vpcId": cluster.EksCluster.VpcConfig().VpcId(),
            "podLabels": pulumi.Map{
                "stack": pulumi.String("eks"),
                "app":   pulumi.String("aws-lb-controller"),
            },
        },
    }, pulumi.Provider(provider), pulumi.DependsOn([]pulumi.Resource{ns, sa}))
    if err != nil {
        return err
    }
    externalDNS, err := helm.NewRelease(ctx, "external-dns", &helm.ReleaseArgs{
        Chart:           pulumi.String("external-dns"),
        Version:         pulumi.String("1.12.2"),
        Namespace:       pulumi.String("external-dns"),
        CreateNamespace: pulumi.Bool(true),
        RepositoryOpts: helm.RepositoryOptsArgs{
            Repo: pulumi.String("https://kubernetes-sigs.github.io/external-dns/"),
        },
        Values: pulumi.Map{
            "provider": pulumi.String("digitalocean"),
            "sources": pulumi.Array{
                pulumi.String("ingress"),
            },
            "env": pulumi.Array{
                pulumi.Map{
                    "name":  pulumi.String("DO_TOKEN"),
                    "value": config.GetSecret(ctx, "do"),
                },
            },
        },
    }, pulumi.Provider(provider))
}

The kubernetes.NewProvider() function is creating a new provider that will manage Kubernetes resources, we are using the server-side apply feature.

We then create a new namespace for the ALB Controller using the corev1.NewNamespace function. Next, we create a new service account for the ALB Controller using the corev1.NewServiceAccount function. The Annotations field specifies the ARN of the role that the ALB Controller will assume.

Finally, we create the ALB Controller using the helm.NewRelease function. Important to note here is that we are passing the already created service account as a value to the serviceAccount.name field. This is important because the ALB Controller will use the service account to assume the role that we created earlier.

The helm.NewRelease function is also used to create the External DNS controller. This is really because of me, as I want that the address of the ingress will be added as an A record in my DigitalOcean DNS. You can skip this part if you do use for example AWS Route53.

package main

// omitted for brevity

func main() {
    // omitted for brevity
    _, err = helm.NewRelease(ctx, "argocd", &helm.ReleaseArgs{
        Chart:           pulumi.String("argo-cd"),
        Version:         pulumi.String("5.28.0"),
        Namespace:       pulumi.String("argocd"),
        CreateNamespace: pulumi.Bool(true),
        RepositoryOpts: helm.RepositoryOptsArgs{
            Repo: pulumi.String("https://argoproj.github.io/argo-helm"),
        },
        Values: pulumi.Map{
            "dex": pulumi.Map{
                "enabled": pulumi.Bool(false),
            },
            "server": pulumi.Map{
                "extraArgs": pulumi.Array{
                    pulumi.String("--insecure"),
                    pulumi.String("--enable-gzip"),
                },
                "ingress": pulumi.Map{
                    "enabled":          pulumi.Bool(true),
                    "hosts":            pulumi.Array{pulumi.String("argocd.ediri.online")},
                    "ingressClassName": pulumi.String("alb"),
                    "annotations": pulumi.Map{
                        "alb.ingress.kubernetes.io/target-type":        pulumi.String("ip"),
                        "alb.ingress.kubernetes.io/scheme":             pulumi.String("internet-facing"),
                        "alb.ingress.kubernetes.io/load-balancer-name": pulumi.String("argocd"),
                        "alb.ingress.kubernetes.io/certificate-arn":    config.GetSecret(ctx, "cert_arn"),
                    },
                },
            },
            "configs": pulumi.Map{
                "rbac": pulumi.Map{
                    "policy.default": pulumi.String("role:readonly"),
                    "policy.csv":     pulumi.Sprintf(`g, %s, role:admin`, adminUserEmail),
                    "scopes":         pulumi.String(`[email]`),
                },
                "cm": pulumi.Map{
                    "admin.enabled": pulumi.Bool(false),
                    "url":           pulumi.String("https://argocd.ediri.online"),
                    "oidc.config": pulumi.Sprintf(`name: Cognito
issuer: https://cognito-idp.eu-central-1.amazonaws.com/%s
clientID: %s
clientSecret: %s
requestedScopes: ["openid", "profile", "email"]
requestedIDTokenClaims: {"email": {"essential": true}}`, userPool.ID(), userPoolClient.ID(), userPoolClient.ClientSecret),
                },
            },
        },
    }, pulumi.Provider(provider), pulumi.DependsOn([]pulumi.Resource{ns, externalDNS}))
    if err != nil {
        return err
    }
}

The last part of the code is creating the ArgoCD deployment. We are using the helm.NewRelease function again to create the deployment. The Values field contains the configuration for the deployment. The server.ingress field contains the now some interesting parts. We need to set the ingressClassName to alb so that the ingress will be managed by the ALB Controller. We also need to set the alb.ingress.kubernetes.io/certificate-arn annotation to the ARN of the certificate in our AWS Certificate Manager. We also need to set the alb.ingress.kubernetes.io/scheme and the alb.ingress.kubernetes.io/target-type annotations to internet-facing and ip respectively.

The configs.cm.oidc.config field contains the configuration for the OIDC provider. We are using Cognito as the OIDC provider set the issuer field to the URL of the Cognito user pool. The clientID and clientSecret fields are the ID and secret of the Cognito user pool client. The requestedScopes and requestedIDTokenClaims fields are the scopes and claims that we want to request from the OIDC provider.

Don't forget to set the url field to your domain name, in my case argocd.ediri.online.

Deploying the stack

Finally, we can deploy the stack. We can do this by running the following command:

pulumi up

This will create the stack and deploy the resources. This will take a few minutes and after that, you should be able to access the ArgoCD dashboard at the URL that you specified! In my case, this is https://argocd.ediri.online.

Housekeeping

After you are done with the stack, you can destroy it by running the following command:

pulumi destroy

This will destroy all the resources that were created by the stack.

Conclusion

Using AWS Cognito as an OIDC provider for ArgoCD is a great way to secure your ArgoCD deployment. If you already run your infrastructure on AWS it is very easy to set Cognito up and configure it to use it as an OAuth2 provider for ArgoCD.

Resources

• ArgoCD

• AWS Load Balancer Controller

• AWS Cognito

• AWS Certificate Manager

• AWS IAM

Introduction

We keep reading about the importance of Observability in our applications. Charity Majors CEO of Honeycomb made a very good tweet thread in 2018(!!)

And this great blog post from Dotan Horovits:

Or this page created by Michael Friedrich

Aurélie Vache made a good blog article for Golang!

There are numerous tools and frameworks that we can use during development to offer an Observability layer. In this article, we will add OpenTelemetry to our REST API with Actix Web application we developed in the previous article. If you haven't read the previous article:

Let's talk first about Observability and OpenTelemetry to build a basic understanding before we head over to the code part.

Observability?

When we talk about Observability, we are referring to the ability to monitor, understand, and troubleshoot the internal state of a system based on its external outputs. It is a crucial aspect of modern software development and operations (DevOps) that helps ensure the reliability, stability, and performance of applications. Observability is achieved through the collection and analysis of various types of telemetry data, such as logs, metrics, and traces.

Why is Observability important?

Complexity

Modern applications are often complex, distributed, and composed of multiple microservices. This makes it difficult to pinpoint the root cause of issues when they arise. Observability helps developers and operators gain insights into the application's behavior, making it easier to identify and resolve problems.

Continuous deployment

In a fast-paced development environment, frequent changes are made to applications. Observability ensures that issues introduced by these changes are quickly detected and addressed, reducing downtime and ensuring a smooth user experience.

Scalability

As applications grow and scale, so do the number of components and services that need to be monitored. Observability enables the tracking of these components, ensuring that performance remains consistent even as the system expands.

Is Observability not the same as Monitoring?

The fact is, monitoring (or application performance monitoring) is a subset of Observability and a step towards achieving it. Observability itself uses different types of telemetry data to provide insights into the state of the system and to understand the reason for any issues that may arise.

Here's a simpler version of the difference between Observability and monitoring:

• Monitoring uses set data, dashboards, and alerts to check how well applications and systems work.

• Observability lets you understand what's happening inside complex systems that change over time by looking at all the information available right away.

Three Pillars of Observability

The three components of Observability (logs, metrics, and traces) are often referred to as the Three Pillars of Observability:

• Metrics: Metrics are numerical values that are used to measure the health and performance of a system.

• Logs: A timestamped sequence of events that provide insight into the behavior of a system. While metrics show the first signs of a problem, logs provide you with the context to understand the root cause.

• Traces: Traces are a set of events that occur in a distributed system, and are used to understand the flow of requests through the system. When a request moves through a distributed system, it is called a span.

The Benefits of Observability:

Faster issue resolution

Observability helps to quickly identify the root cause of issues, reducing the mean time to resolution (MTTR) and minimizing the impact on users.

Proactive problem detection

Observability allows for the early detection of potential problems, enabling teams to take preventive measures before they escalate and affect users.

Improved performance

By providing visibility into how applications are performing, Observability allows teams to optimize and fine-tune their systems, resulting in better performance and a higher-quality user experience.

Data-driven decision-making

Observability provides valuable insights into application behavior, which can be used to make informed decisions about system architecture, design, and resource allocation.

Enhanced collaboration

Observability promotes a shared understanding of the system's state, fostering better collaboration between development and operations teams, and improving overall software delivery efficiency.

Prerequisites

• Rust

• An IDE or text editor of your choice

• Docker and docker-compose installed

• optional: If you want to interact with the PostgreSQL database, you can install psql with brew install postgresql

• The code from the last blog article, which you can find here

  %[https://github.com/dirien/quick-bites/tree/main/rust-actix-web-rest-api-diesel]

Setting up the Database

We're going to use the same way to set up the database as we did in the previous article. Simply run the following command

docker-compose -f postgres.yml up -d

And run the diesel cli to create the database and the tables

diesel setup
diesel migration run

As part of the up.sql migration, we create two tables. The first one is the todos table and the second one is the categories table. The todos table has a foreign key category_id to the categories table.

We also pre-populate the categories table with some categories.

CREATE SEQUENCE categories_id_seq;

CREATE TABLE categories
(
    id          INTEGER PRIMARY KEY DEFAULT nextval('categories_id_seq'),
    name        VARCHAR(255) NOT NULL,
    description TEXT
);

CREATE TABLE todos
(
    id          VARCHAR(255) PRIMARY KEY,
    title       VARCHAR(255) NOT NULL,
    description TEXT,
    created_at  TIMESTAMP,
    updated_at  TIMESTAMP,
    category_id INTEGER,
    FOREIGN KEY (category_id) REFERENCES categories (id)
);

INSERT INTO categories (name, description)
VALUES ('Work', 'Tasks related to work or job responsibilities'),
       ('Personal', 'Personal tasks and errands'),
       ('Health', 'Health and fitness related tasks'),
       ('Hobbies', 'Tasks related to hobbies and interests'),
       ('Education', 'Tasks related to learning and education');

In the src/repository/database.rs file, I created two new functions to get all categories and all todos with joined categories. I use a new struct called TodoItemData to return all the data I need for API responses rather than calling two separate functions to get the todos and the categories.

In a real-world application, I would probably create a backend for frontend (BFF) service that would call the backend API and return the data in a format that is easier to work with on the frontend.

pub fn get_categories(&self) -> Vec<Category> {
    categories
        .load::<Category>(&mut self.pool.get().unwrap())
        .expect("Error loading all categories")
}

pub fn get_todos_with_category(&self) -> Vec<TodoItemData> {
    let mut empty_todo_item_data_list: Vec<TodoItemData> = Vec::new();
    todos
        .inner_join(categories)
        .load::<(Todo, Category)>(&mut self.pool.get().unwrap())
        .expect("Error loading all todos")
        .into_iter()
        .for_each(|(todo, category)| {
            println!("todo: {:?}, category: {:?}", todo, category);
            let todo_item_data = TodoItemData {
                id: todo.id,
                title: todo.title,
                description: todo.description,
                created_at: todo.created_at,
                updated_at: todo.updated_at,
                category: Some(CategoryData {
                    id: category.id,
                    name: category.name,
                    description: category.description,
                }),
            };
            empty_todo_item_data_list.push(todo_item_data);
        });
    empty_todo_item_data_list
}

Why OpenTelemetry?

Numerous Observability platforms currently exist, offering in-depth insights into your code and displaying traces, such as Dynatrace, NewRelic, DataDog, etc. So, what makes OpenTelemetry a desirable choice?

OpenTelemetry addresses a significant challenge: establishing a standard for reporting and transmitting measurements.

By using OpenTelemetry with Solution A, you can effortlessly switch to Solution B as your Observability platform without losing any trace history.

As a result, OpenTelemetry has emerged as the go-to standard for many organizations implementing Observability in their systems.

Setting up OpenTelemetry

Before we can start to add OpenTelemetry code to our application, we need to import some necessary crates.

cargo add actix_web_opentelemetry - - features opentelemetry-prometheus,metrics,metrics-prometheus,prometheus
cargo add opentelemetry - - features=rt-tokio,trace
cargo add opentelemetry-jaeger - - features=collector_client,rt-tokio,isahc_collector_client,isahc_collector_client
cargo add tracing-opentelemetry
cargo add tracing
cargo add tracing-bunyan-formatter
cargo add tracing-subscriber - - features=env-filter,registry

If you're not going to use jaeger as your tracing backend, you can remove the opentelemetry-jaeger crate and change it to opentelemetry-otlp or opentelemetry-zipkin crate. See the OpenTelemetry Rust Crates list for more information.

To set up the tracing layers and metrics exporter, we need to create a new file called tracing.rs in the src folder.

use actix_web_opentelemetry::{PrometheusMetricsHandler, RequestMetricsBuilder};
use dotenv::dotenv;
use opentelemetry::sdk::export::metrics::aggregation;
use opentelemetry::sdk::metrics::{controllers, processors};
use opentelemetry::{global, sdk};
use tracing_bunyan_formatter::{BunyanFormattingLayer, JsonStorageLayer};
use tracing_subscriber::{EnvFilter, Registry};
use tracing_subscriber::{prelude::*};

#[derive(Debug, Clone)]
pub struct OpenTelemetryStack {
    request_metrics: actix_web_opentelemetry::RequestMetrics,
    metrics_handler: PrometheusMetricsHandler,
}

impl OpenTelemetryStack {
    pub fn new() -> Self {
        dotenv().ok();
        let app_name = std::env::var("CARGO_BIN_NAME").unwrap_or("demo".to_string());

        global::set_text_map_propagator(opentelemetry_jaeger::Propagator::new());
        let tracer = opentelemetry_jaeger::new_agent_pipeline()
            .with_endpoint(std::env::var("JAEGER_ENDPOINT").unwrap_or("localhost:6831".to_string()))
            .with_service_name(app_name.clone())
            .install_batch(opentelemetry::runtime::Tokio)
            .expect("Failed to install OpenTelemetry tracer.");

        let telemetry = tracing_opentelemetry::layer().with_tracer(tracer);
        let env_filter = EnvFilter::try_from_default_env().unwrap_or(EnvFilter::new("INFO"));
        let formatting_layer = BunyanFormattingLayer::new(app_name.clone().into(), std::io::stdout);
        let subscriber = Registry::default()
            .with(telemetry)
            .with(JsonStorageLayer)
            .with(formatting_layer)
            .with(env_filter);
        tracing::subscriber::set_global_default(subscriber)
            .expect("Failed to install `tracing` subscriber.");

        let controller = controllers::basic(processors::factory(
            sdk::metrics::selectors::simple::histogram([0.1, 0.5, 1.0, 2.0, 5.0, 10.0]),
            aggregation::cumulative_temporality_selector(),
        )).build();
        let prometheus_exporter = opentelemetry_prometheus::exporter(controller).init();
        let meter = global::meter("global");
        let request_metrics = RequestMetricsBuilder::new().build(meter);
        let metrics_handler = PrometheusMetricsHandler::new(prometheus_exporter.clone());
        Self {
            request_metrics,
            metrics_handler
        }
    }

    pub fn metrics(&self) -> actix_web_opentelemetry::RequestMetrics {
        self.request_metrics.clone()
    }

    pub fn metrics_handler(&self) -> PrometheusMetricsHandler {
        self.metrics_handler.clone()
    }
}

In the OpenTelemetryStack struct, we create a new RequestMetrics and PrometheusMetricsHandler instance. In the implementation block of the OpenTelemetryStack struct, we initialize the tracing subscriber and the metrics exporter in the new function.

The endpoint of the opentelemetry_jaeger crate is set to localhost:6831 by default. If you're using a different endpoint, you can change it by setting the JAEGER_ENDPOINT environment variable. In the next step, we initialize an environment filter (EnvFilter) that determines which trace events to collect based on their log levels. It attempts to create the filter using the default environment configuration. If it fails, it falls back to creating a filter with the INFO log level.

The BunyanFormattingLayer is used to format the logs in a way that is compatible with the Bunyan log format, which is a popular JSON-based logging format. Then we add all these layers to the tracing subscriber.

The opentelemetry_prometheus crate is used to export metrics to Prometheus. The controllers::basic function is used to initialize the metrics controller using a basic controller provided by the OpenTelemetry SDK. sdk::metrics:: selectors::simple::histogram([0.1, 0.5, 1.0, 2.0, 5.0, 10.0]), sets up a simple histogram selector with the specified boundaries for aggregating metric data. The boundaries are [0.1, 0.5, 1.0, 2.0, 5.0, 10.0]. The aggregation::cumulative_temporality_selector(), line configures the aggregation to use cumulative temporality, meaning the metric values will be aggregated over time, rather than being reset periodically.

let meter = global::meter("global"); retrieves a global meter instance with the name global (creative I know!). Meters are responsible for creating and recording metric instruments.

The next two lines create a new RequestMetrics and PrometheusMetricsHandler instance with the previously created meter and exporter.

Wiring up OpenTelemetry in Actix

With the tracing.rs file in place and the heavy lifting done, we can now wire up OpenTelemetry in our Actix application.

Head over to the main.rs file and change the main function to the following:

#[actix_web::main]
async fn main() -> std::io::Result<()> {
    let todo_db = repository::database::Database::new();
    let app_data = web::Data::new(todo_db);

    let telemetry = telemetry::OpenTelemetryStack::new();
    let telemetry_data = web::Data::new(telemetry.clone());

    HttpServer::new(move || {
        App::new()
            .app_data(app_data.clone())
            .app_data(telemetry_data.clone())
            .configure(api::api::config)
            .service(healthcheck)
            .service(metrics)
            .default_service(web::route().to(not_found))
            .wrap(actix_web::middleware::Logger::default())
            .wrap(RequestTracing::new())
            .wrap(telemetry.metrics())
    })
        .bind(("127.0.0.1", 8080))?
        .run()
        .await
}

The telemetry variable is initialized with the OpenTelemetryStack struct. The telemetry_data variable is used to pass the telemetry variable to the App instance. The telemetry.metrics() function is used to create a new RequestMetrics instance that is used to collect metrics for the Actix application.

Adding the #[tracing::instrument] attribute

The #[tracing::instrument] attribute is used to instrument the code. It adds spans to the code and adds the tracing context to the logs. The #[tracing::instrument] attribute is added to all the functions in the api module.

Here's an example of the create_todo function in the todos module:

#[post("/todos")]
#[tracing::instrument]
pub async fn create_todo(db: web::Data<Database>, new_todo: web::Json<Todo>) -> HttpResponse {
    let todo = db.create_todo(new_todo.into_inner());
    match todo {
        Ok(todo) => HttpResponse::Ok().json(todo),
        Err(err) => HttpResponse::InternalServerError().body(err.to_string()),
    }
}

// rest of the code is omitted for brevity

The metrics endpoint

The /metrics endpoint is used to expose the metrics to Prometheus. The metrics endpoint is added to the Actix and looks like this:

#[get("/metrics")]
async fn metrics(telemetry: web::Data<telemetry::OpenTelemetryStack>, db: web::Data<Database>, request: HttpRequest) -> impl Responder {
    let categories = db.get_categories();
    let todos = db.get_todos();

    let meter = global::meter("global");
    let todo_count = meter.i64_observable_gauge("todo_count").with_description("Number of todos").init();
    let category_count = meter.i64_observable_gauge("category_count").with_description("Number of categories").init();

    let cx = Context::current();
    todo_count.observe(&cx, todos.len() as i64, &[]);
    category_count.observe(&cx, categories.len() as i64, &[]);
    telemetry.metrics_handler().call(request).await
}

The telemetry variable is used to retrieve the PrometheusMetricsHandler instance. The db variable is used to get access to the Database instance.

I created two new metrics, todo_count and category_count, that are used to count the number of todos and categories as a gauge.

A gauge is a metric that represents a single numerical value that can increase or decrease.

In the end, the PrometheusMetricsHandler instance is called to handle exposing the additional metrics to Prometheus.

Testing the application

Now with everything in place, we can start the application and test it. But before we do that, let us take care that we have a Jaeger and Prometheus instance running. For this, I created a new docker-compose file called telemetry.yaml:

version: '3.8'

services:
  jaeger:
    image: jaegertracing/all-in-one:1.43.0
    ports:
      - "16686:16686"
      - "14268:14268"
      - "9411:9411"
      - "5778:5778"
      - "6831:6831/udp"
      - "6832:6832/udp"
      - "5775:5775/udp"
    environment:
      - COLLECTOR_ZIPKIN_HTTP_PORT=9411

  prometheus:
    image: prom/prometheus:v2.43.0
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yaml:/etc/prometheus/prometheus.yml

And the corresponding prometheus.yaml file, so that Prometheus can scrape the metrics from the Actix application running currently on localhost:8080:

global:
  scrape_interval: 5s # How frequently to scrape targets by default
  evaluation_interval: 5s # How frequently to evaluate rules by default

scrape_configs:
  - job_name: 'demo'
    static_configs:
      - targets: [ 'host.docker.internal:8080' ] # The host.docker.internal is a special DNS name that resolves to the internal IP address used by the host.
        labels:
          group: 'demo'

Now we can start the application and Prometheus:

docker-compose -f telemetry.yaml up -d

And you can access the Jaeger UI at http://localhost:16686 and Prometheus at http://localhost:9090.

Finally, we can start the application:

cargo run

Check the /metrics endpoint:

curl http://localhost:8080/metrics

You should see the following output:

# HELP category_count Number of categories
# TYPE category_count gauge
category_count{service_name="unknown_service"} 5
# HELP http_server_active_requests HTTP concurrent in-flight requests per route
# TYPE http_server_active_requests gauge
http_server_active_requests{http_flavor="HTTP/1.1",http_host="host.docker.internal:8080",http_method="GET",http_scheme="http",http_server_name="127.0.0.1:8080",http_target="/metrics",net_host_port="8080",service_name="unknown_service"} 1
# HELP http_server_duration HTTP inbound request duration per route
# TYPE http_server_duration histogram
http_server_duration_bucket{http_flavor="HTTP/1.1",http_host="host.docker.internal:8080",http_method="GET",http_scheme="http",http_server_name="127.0.0.1:8080",http_status_code="200",http_target="/metrics",net_host_port="8080",service_name="unknown_service",le="0.1"} 0
http_server_duration_bucket{http_flavor="HTTP/1.1",http_host="host.docker.internal:8080",http_method="GET",http_scheme="http",http_server_name="127.0.0.1:8080",http_status_code="200",http_target="/metrics",net_host_port="8080",service_name="unknown_service",le="0.5"} 0
http_server_duration_bucket{http_flavor="HTTP/1.1",http_host="host.docker.internal:8080",http_method="GET",http_scheme="http",http_server_name="127.0.0.1:8080",http_status_code="200",http_target="/metrics",net_host_port="8080",service_name="unknown_service",le="1"} 0
http_server_duration_bucket{http_flavor="HTTP/1.1",http_host="host.docker.internal:8080",http_method="GET",http_scheme="http",http_server_name="127.0.0.1:8080",http_status_code="200",http_target="/metrics",net_host_port="8080",service_name="unknown_service",le="2"} 0
http_server_duration_bucket{http_flavor="HTTP/1.1",http_host="host.docker.internal:8080",http_method="GET",http_scheme="http",http_server_name="127.0.0.1:8080",http_status_code="200",http_target="/metrics",net_host_port="8080",service_name="unknown_service",le="5"} 3
http_server_duration_bucket{http_flavor="HTTP/1.1",http_host="host.docker.internal:8080",http_method="GET",http_scheme="http",http_server_name="127.0.0.1:8080",http_status_code="200",http_target="/metrics",net_host_port="8080",service_name="unknown_service",le="10"} 5
http_server_duration_bucket{http_flavor="HTTP/1.1",http_host="host.docker.internal:8080",http_method="GET",http_scheme="http",http_server_name="127.0.0.1:8080",http_status_code="200",http_target="/metrics",net_host_port="8080",service_name="unknown_service",le="+Inf"} 6
http_server_duration_sum{http_flavor="HTTP/1.1",http_host="host.docker.internal:8080",http_method="GET",http_scheme="http",http_server_name="127.0.0.1:8080",http_status_code="200",http_target="/metrics",net_host_port="8080",service_name="unknown_service"} 36.187
http_server_duration_count{http_flavor="HTTP/1.1",http_host="host.docker.internal:8080",http_method="GET",http_scheme="http",http_server_name="127.0.0.1:8080",http_status_code="200",http_target="/metrics",net_host_port="8080",service_name="unknown_service"} 6
# HELP todo_count Number of todos
# TYPE todo_count gauge
todo_count{service_name="unknown_service"} 0

As you can see, we have a few metrics that are being collected by default. We can also see that the category_count metric is being collected, and it has a value of 5. This is because we initialized the database with 5 categories.

the todo_count metric is being collected, and it has a value of 0. This is because we haven't created any todos yet.

Let us create some todos, one for each category:

curl -s -X POST -H "Content-Type: application/json" -d '{"title": "Prepare presentation slides", "description": "Create slides for the upcoming project meeting", "category_id": 1}' http://localhost:8080/api/todos | jq
curl -s -X POST -H "Content-Type: application/json" -d '{"title": "Buy groceries", "description": "Purchase groceries for the week", "category_id": 2}' http://localhost:8080/api/todos | jq
curl -s -X POST -H "Content-Type: application/json" -d '{"title": "Run 5 miles", "description": "Complete a 5-mile run for weekly exercise", "category_id": 3}' http://localhost:8080/api/todos | jq
curl -s -X POST -H "Content-Type: application/json" -d '{"title": "Practice guitar", "description": "Spend 30 minutes practicing guitar chords", "category_id": 4}' http://localhost:8080/api/todos | jq
curl -s -X POST -H "Content-Type: application/json" -d '{"title": "Study for the exam", "description": "Review course material for the upcoming exam", "category_id": 5}' http://localhost:8080/api/todos | jq

{
  "id": "55aff8a6-42c2-4d94-8872-b94a55beae08",
  "title": "Prepare presentation slides",
  "description": "Create slides for the upcoming project meeting",
  "created_at": "2023-04-04T09:40:07.581186",
  "updated_at": "2023-04-04T09:40:07.581199",
  "category_id": 1
}
{
  "id": "e10567cb-c1df-413a-8996-6c072e124a0c",
  "title": "Buy groceries",
  "description": "Purchase groceries for the week",
  "created_at": "2023-04-04T09:40:07.604805",
  "updated_at": "2023-04-04T09:40:07.604809",
  "category_id": 2
}
{
  "id": "4f0bb9d3-fd24-4a51-b11e-1a12231a4529",
  "title": "Run 5 miles",
  "description": "Complete a 5-mile run for weekly exercise",
  "created_at": "2023-04-04T09:40:07.619386",
  "updated_at": "2023-04-04T09:40:07.619389",
  "category_id": 3
}
{
  "id": "055b7ab2-53d9-468b-a4ac-82e70d190a33",
  "title": "Practice guitar",
  "description": "Spend 30 minutes practicing guitar chords",
  "created_at": "2023-04-04T09:40:07.633674",
  "updated_at": "2023-04-04T09:40:07.633676",
  "category_id": 4
}
{
  "id": "b422a294-57f7-4c0c-bd67-9d7babf1bd9c",
  "title": "Study for the exam",
  "description": "Review course material for the upcoming exam",
  "created_at": "2023-04-04T09:40:07.649398",
  "updated_at": "2023-04-04T09:40:07.649401",
  "category_id": 5
}

Now, let's get all todos with the /api/todos endpoint:

curl -s http://127.0.0.1:8080/api/todos | jq                                                                                                                                                                                             
[
  {
    "id": "55aff8a6-42c2-4d94-8872-b94a55beae08",
    "title": "Prepare presentation slides",
    "description": "Create slides for the upcoming project meeting",
    "created_at": "2023-04-04T09:40:07.581186",
    "updated_at": "2023-04-04T09:40:07.581199",
    "category": {
      "id": 1,
      "name": "Work",
      "description": "Tasks related to work or job responsibilities"
    }
  },
  {
    "id": "e10567cb-c1df-413a-8996-6c072e124a0c",
    "title": "Buy groceries",
    "description": "Purchase groceries for the week",
    "created_at": "2023-04-04T09:40:07.604805",
    "updated_at": "2023-04-04T09:40:07.604809",
    "category": {
      "id": 2,
      "name": "Personal",
      "description": "Personal tasks and errands"
    }
  },
  {
    "id": "4f0bb9d3-fd24-4a51-b11e-1a12231a4529",
    "title": "Run 5 miles",
    "description": "Complete a 5-mile run for weekly exercise",
    "created_at": "2023-04-04T09:40:07.619386",
    "updated_at": "2023-04-04T09:40:07.619389",
    "category": {
      "id": 3,
      "name": "Health",
      "description": "Health and fitness related tasks"
    }
  },
  {
    "id": "055b7ab2-53d9-468b-a4ac-82e70d190a33",
    "title": "Practice guitar",
    "description": "Spend 30 minutes practicing guitar chords",
    "created_at": "2023-04-04T09:40:07.633674",
    "updated_at": "2023-04-04T09:40:07.633676",
    "category": {
      "id": 4,
      "name": "Hobbies",
      "description": "Tasks related to hobbies and interests"
    }
  },
  {
    "id": "b422a294-57f7-4c0c-bd67-9d7babf1bd9c",
    "title": "Study for the exam",
    "description": "Review course material for the upcoming exam",
    "created_at": "2023-04-04T09:40:07.649398",
    "updated_at": "2023-04-04T09:40:07.649401",
    "category": {
      "id": 5,
      "name": "Education",
      "description": "Tasks related to learning and education"
    }
  }
]

And check the /metrics endpoint:

# HELP category_count Number of categories
# TYPE category_count gauge
category_count{service_name="unknown_service"} 5

# omitted for brevity

# HELP todo_count Number of todos
# TYPE todo_count gauge
todo_count{service_name="unknown_service"} 5

The todo_count metric is now showing the correct value of 5, as expected.

Head over to the UI of Prometheus and check the todo_count metric

And same for the traces in the Jaeger UI

Housekeeping

To stop and remove the postgres container and volume, run the following commands:

docker-compose -f postgres.yaml down
docker volume rm rust-actix-web-rest-api-diesel_postgres-data

To stop the telemetry containers, run the following commands:

docker-compose -f telemetry.yaml down

Conclusion

Congrats! We, again, successfully added telemetry to our Rust Actix Web REST API demo application. We also configured Prometheus and Jaeger to collect and visualize the metrics and traces. And everything on top of our existing application code.

Resources

• Jaeger

• Prometheus

• OpenTelemetry

• Actix Web OpenTelemetry

• postgres

• prom/prometheus

• jaegertracing/all-in-one

• Docker Compose

• actix.rs

• o11y.love

Introduction

In this blog post, I am going to show you how to run a Fermyon Spin WebAssembly (WASM) application on a WebAssembly System Interface (WASI) node pool.

Everything will be powered by using Pulumi.

Microsoft has revealed their decision to transition from krustlet and adopt containerd shims for operating WASM workloads in their WASI node pools within AKS.

The objective of containerd shims is to offer implementations capable of executing WASM/WASI workloads utilizing the runwasi library. By installing these shims on Kubernetes nodes, a runtime class can be added to Kubernetes, enabling the scheduling of Wasm workloads on the respective nodes. This allows your Wasm pods and deployments to function similarly to container workloads.

Runwasi is a project focused on operating wasm workloads on Wasmtime, a rapid and secure WebAssembly runtime managed by containerd.

We will focus this article on the Spin shim which is powered by the Fermyon Spin engine.

What is WASM?

WebAssembly (WASM) is a binary format that is designed for maximum execution speed and portability using a WASM runtime. The WASM runtime is designed to run on a target architecture and execute WebAssemblies in a sandboxed environment to ensure security at near-native speed.

The WebAssembly System Interface (WASI) establishes a standardized connection between the WASM runtime and the host system, granting access to system resources like the file system or network.

What is Fermyon Spin?

Fermyon Spin is a framework for building cloud-native applications with WebAssembly components. It is created by Fermyon and is fully open source. You can find the source code on GitHub.

Prerequisites

• Pulumi

• Azure CLI

• kubectl

• Rust installed

• Go installed

• IDE of your choice (VS Code, IntelliJ, etc.)

Activate Preview Features for AKS

To install the aks-preview extension for Azure CLI, run the following command:

az extension add --name aks-preview

or update your existing aks-preview extension to the latest version:

az extension update --name aks-preview

Register the WasmNodePoolPreview Feature

You may need to register the WasmNodePoolPreview feature for your subscription by simply running the following command:

az feature register --namespace "Microsoft.ContainerService" --name "WasmNodePoolPreview"

This will take a few minutes to complete. You can check the status of the feature registration by running the following command:

az feature show --namespace "Microsoft.ContainerService" --name "WasmNodePoolPreview"

Once the state property of the feature is Registered, you can create a WASI node pool for your AKS cluster.

Set Up Your AKS Cluster

Let's start a new Pulumi project with the following command, using the pulumi-azure-native provider and Go as the language of choice:

mkdir pulumi-aks-wasm-spin && cd pulumi-aks-wasm-spin
pulumi new azure-go --force

You will need to provide some details about your project. You can use the default values for all questions for this demo.

We can now generate our AKS cluster using the subsequent code:

package main

import (
    "encoding/base64"

    containerservice "github.com/pulumi/pulumi-azure-native-sdk/containerservice/v20230101"
    resources "github.com/pulumi/pulumi-azure-native-sdk/resources/v20220901"
    "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
    pulumi.Run(func(ctx *pulumi.Context) error {
        // Create an Azure Resource Group
        resourceGroup, err := resources.NewResourceGroup(ctx, "wasm-aks-rg", &resources.ResourceGroupArgs{
            ResourceGroupName: pulumi.String("wasm-aks-rg"),
        })
        if err != nil {
            return err
        }

        wasmCluster, err := containerservice.NewManagedCluster(ctx, "wasm-aks-cluster", &containerservice.ManagedClusterArgs{
            ResourceGroupName: resourceGroup.Name,
            KubernetesVersion: pulumi.String("1.25.5"),
            ResourceName:      pulumi.String("wasm-aks-cluster"),
            Identity: &containerservice.ManagedClusterIdentityArgs{
                Type: containerservice.ResourceIdentityTypeSystemAssigned,
            },
            DnsPrefix: pulumi.String("wasm-aks-cluster"),
            AgentPoolProfiles: containerservice.ManagedClusterAgentPoolProfileArray{
                &containerservice.ManagedClusterAgentPoolProfileArgs{
                    Name:         pulumi.String("agentpool"),
                    Mode:         pulumi.String("System"),
                    OsDiskSizeGB: pulumi.Int(30),
                    OsType:       pulumi.String("Linux"),
                    Count:        pulumi.Int(1),
                    VmSize:       pulumi.String("Standard_B4ms"),
                },
            },
        })
        if err != nil {
            return err
        }

        wasmPool, err := containerservice.NewAgentPool(ctx, "wasm-aks-agentpool", &containerservice.AgentPoolArgs{
            AgentPoolName:     pulumi.String("wasmpool"),
            ResourceGroupName: resourceGroup.Name,
            ResourceName:      wasmCluster.Name,
            WorkloadRuntime:   pulumi.String("WasmWasi"),
            Count:             pulumi.Int(1),
            VmSize:            pulumi.String("Standard_B4ms"),
            OsType:            pulumi.String("Linux"),
        })
        if err != nil {
            return err
        }

        kubeconfig := pulumi.All(wasmCluster.Name, resourceGroup.Name).ApplyT(func(args []interface{}) (*string, error) {
            clusterName := args[0].(string)
            resourceGroupName := args[1].(string)
            creds, err := containerservice.ListManagedClusterUserCredentials(ctx, &containerservice.ListManagedClusterUserCredentialsArgs{
                ResourceGroupName: resourceGroupName,
                ResourceName:      clusterName,
            })
            if err != nil {
                return nil, err
            }
            decoded, err := base64.StdEncoding.DecodeString(creds.Kubeconfigs[0].Value)
            if err != nil {
                return nil, err
            }
            return pulumi.StringRef(string(decoded)), nil
        }).(pulumi.StringPtrOutput)

        ctx.Export("resourceGroupName", resourceGroup.Name)
        ctx.Export("wasmClusterName", wasmCluster.Name)
        ctx.Export("wasmAgentPoolName", wasmPool.Name)
        ctx.Export("kubeconfig", pulumi.ToSecret(kubeconfig))

        return nil
    })
}