Molecule-Kubevirt Driver

Running Ansible unit tests in Kubernetes is painful. In this note, I detail existing solutions, expose my alternative proposal and explain how to use it.

Ansible is a well-known provisioning tool

Ansible, a project lead by Red Hat, is probably the most used tool for provisioning servers : creating users, installing system packages, rendering some config files, … The tool works on anything that could be defined as a server : Virtual Machines or bare-metal servers (i.e. physical servers).

When you want to test your ansible code, you need to ensure automation for creating anything that could be as close as possible to what a server is.

This means some systems where you can ssh, use sudo commands, have access to systemd daemons, …

Molecule is Ansible test framework

Molecule is a well known framework for unit testing Ansible roles. Its role is to automate this creation of target system where your Ansible code will be tested.

Many so-called drivers do exist for molecule, including the creation of real Virtual Machines on cloud-providers or on virtualization tools.

Molecule and Kubernetes

When you’re in the situation of running your Ansible tests inside a Kubernetes cluster you don’t have any real good solution.

You can use Docker or Podman drivers:

• They will start some fake servers running inside containers but : you’ll have to run them with high privileged and specific SYS_CAP_ADMIN capabilities and other horrible things for security.
• Also running in containers means you won’t have access to user privileges escalations : you cannot sudo.

Else you can use non-container solutions like Vagrant or libvirt. This means installing using a tool from outside the cluster. Not good what we’re looking for.

Starting VMs in a Kubernetes cluster

There is a way to start real Virtual Machines inside a Kubernetes cluster: it’s named Kubevirt. It’s a tool that can start Qemu(KVM) virtual images inside a Kubernetes cluster.

It’s a great tool. I use it daily to test my own Ansible code.

But you’ve just said you can’t test Ansible code in Kubernetes ?!?

Clearifying the point here : I can use it for testing a full provisionning of a server (a.k.a playbooks in Ansible).

But I can’t use it for unit testing roles I code (pieces of the playbooks).

Simply because there is no molecule (unit test tool) driver for Kubevirt (good VMs in Kubernetes).

Please welcome Molecule-Kubevirt

Ok so here we are. Since no good solution for unit testing Ansible roles in Kubernetes seems to be existing, let’s try to write one.

This brand new driver makes use of Kubevirt in order to:

1. create VM according to unit test platforms definition
2. ensure molecule gets access to the VM : create a ssh key, inject its public part in cloud-init config and create a Kubernetes Service for ssh
3. And… well.. that’s all 😊

Project repo is here but if you want to give a try, just read next section.

Showtime

Sample usage : migrating from Docker to Kubevirt

Let’s take a well known Ansible role from Jeff Geerling (a real Ansible guru FYI): ansible-role-nginx. This role is unit tested in Github Actions, using Docker privileged containers.

Let’s git clone this repo, install requirements, switch its molecule tests from Docker to new molecule-kubevirt driver and test.

Install Kubevirt

Take a look at the Kubevirt documentation to install Kubevirt in your Kubernetes cluster.

Change few lines of code

Change molecule driver to kubevirt and target platform image to one compatible with Kubevirt ContainerDisk format:

Also change converge playbook to execute tests as root user via become directive:

Few notes about those changes :

• You can see I use a self-built container, for convenience, containing the Virtual Disk of the Virtual Machine. Documentation explains how you can build your own,
• Switching to Kubevirt implies we now run tests on real Virtual Machines. The nginx role will install some system packages. On real VMs this can only be done by root user (or sudoer),
• Pro-tip: don’t be confused on the Virtual Disk. Tests will not run inside this container. Kubevirt will start a Pod with Qemu virtualization tool, Qemu will load the Virtual Disk from the container hosting the disk file. But this container hosting the disk file will run no process.

Build an image to run tests inside the cluster

To run tests inside the Kubernetes cluster, we need to build a Docker image and run it inside a Pod. The image must contian the source code plus required dependencies installed.

Create a Dockerfile:

Build the image and push it to a container registry (change with your own repository) :

# build
docker build . -t molecule_kubevirt_runner:latest
# push to any registry
docker tag molecule_kubevirt_runner:latest my-registry/molecule_kubevirt_runner:latest

Add authorization to the Pod

We’re running tests inside the cluster and we need to ensure the test Pod container can talk to the Kubernetes API and create new Virtual Machines and Services. The Pod also needs to be authorized to delete VMs and Services.

Create as molecule-kubevirt-sa.yaml file containing these lines :

Then apply the file:

kubectl apply -f molecule-kubevirt-sa.yaml

Start the test Pod

Create a file for the Pod as described in next lines (remember to change repository to yours):

Start the pod and look at its logs:

kubectl apply -f molecule-pod.yaml
# wait for Pod to start 
kubectl logs -l app=molecule -f 

We can see the VM being created with a valid ssh access.

And a bit later, molecule runs the tests inside the VM and tests the role, just as usual.

Voilà. We unit tested the role with a pure Kubernetes solution, without any compromise in security.

Pitfall

Main difficulty with Kubevirt, is to get enough information if something goes wrong. When running in any CI pipeline, this means collecting a lot of logs. Including Kubevirts Pods logs, Events, … but also the VM “serial” console via virtctl console.

Not only beautiful to look at, console log can give great help in case of trouble in your CI.

Take a look at my Git Actions workflow for more information.

Is this really working ?

Sure, you can see it in action here on Github Actions.

Only CentOS 7 was tested right now but this should work with any well built Kubevirt container image.

Next Steps ?

Some improvement that I have in mind:

• make use of Kubevirt’s Containerized Data Importer for another option than self-building container disk
• some kind of offload mode (launch molecule in your local env and starts VMs on remote Kubernetes cluster)
• any Pull Request you might submit :)

End of the story

Enjoyed reading this ? Please send me feedback in comments.

Please also give claps👏, github stars ⭐ or issues / Pull Requests.

Thanks

Thanks to hawk-eyed Leo Colombaro for his strict review of this note👍

Thanks to Blur rock band, especially for their album “Parklife”. Coding with cool songs gives me a sense of enormous well-being.

Great songs for great coding.