Build, Test, and Automate a Kubernetes Interfacing Application in Go | by Fernando Diaz | Apr, 2022

Photo by Clovis Wood Photography on Unsplash

Kubernetes’ Client-Go provides all sorts of packages for interacting with your cluster. These packages include the following:

  • The kubernetes package contains the clientset to access Kubernetes API.
  • The discovery package is used to discover APIs supported by a Kubernetes API server.
  • The dynamic package contains a dynamic client that can perform generic operations on arbitrary Kubernetes API objects.
  • The plugin/pkg/client/auth Packages contain optional authentication plugins for obtaining credentials from external sources.
  • The transport package is used to set up auth and start a connection.
  • The tools/cache package is useful for writing controllers.

Along with all the above packages, Client-Go also contains a fake client, which allows you to mockup the creation, reading, editing, and removal of a particular Kubernetes resource in order to easily increase unit-test coverage.

Back in KubeCon Europe 2019, I presented on performing unit tests using the Go-Client fake client:

Today, I’d like to go over creating a simple out-of-cluster application used to perform actions on your Kubernetes cluster. Then I’ll show you how to mock out the Kubernetes API calls in your unit tests, and automate running these tests and more with GitLab!

In order to get started, there are a few programs you must have installed:

  • Go— an open source programming language.
  • MiniKube— a tool that quickly sets up a local Kubernetes cluster on macOS, Linux, and Windows. You will also need a virtualization driver for MiniKube to run such as Docker, HyperV, Podman, etc. More information can be found here.

Note: You can use another cluster other than minikubeand it’ll work just fine based on the kubectl context which is set. This guide is written using minikubebecause it’s accessible to everyone.

You must also have knowledge of the following:

  • Go Basics — The code I have written and will go over is in Go. Make sure you have a basic understanding.
  • Kubernetes Basics — Understand how to work with Kubernetes clusters, and using kubectl.
  • Kubernetes Secrets — Understand what secrets are in Kubernetes.

It is good to know the following:

  • Git Basics — Good to know Git to create forks and make modifications to the code yourself.
  • GitLab CI— My code is housed in GitLab, and I am using the GitLab CI in order to run my unit tests and security scans.

I will be using my secreto-server project which performs the following functions:

  • Generates generic Kubernetes secrets
  • Lists the secrets per a given namespace
  • Obtains secret data which includes the payload
  • Delete secrets

To start off, we’ll create a cluster using MiniKube and make sure we can interact with Kubernetes secrets. Then we will launch the application locally and verify all the application’s functions.

Creating Kubernetes Cluster

  1. Install MiniKube.You’ll be able to download a version of Minikube based on your OS and Architecture on the Getting Started Page.

2. Run Minikube. It may take a few minutes to download the required packages, but running Minikube is straightforward as long as you have the required virtualization drivers. Docker Desktop works for most, but I’m using podman to try something different. See the Minikube documentation for more information on drivers.

$ minikube startminikube v1.25.2 on Darwin 12.3 (arm64)
Using the podman (experimental) driver based on existing profile
Starting control plane node minikube in cluster minikube
Pulling base image ...
E0321 11:05:07.616563 66007 cache.go:203] Error downloading kic artifacts: not yet implemented, see issue #8426
Restarting existing podman container for "minikube" ...
Preparing Kubernetes v1.23.3 on Docker 20.10.12 ...E0321 11:05:13.251398 66007 start.go:126] Unable to get host IP: RoutableHostIPFromInside is currently only implemented for linux
▪ kubelet.housekeeping-interval=5m
Verifying Kubernetes components...
▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
Enabled addons: storage-provisioner, default-storageclass
kubectl not found. If you need it, try: 'minikube kubectl -- get pods -A'
Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

3. Verify that Minikube is running correctly. We can do this by checking if the Minikube node has a STATUS of ready.

$ minikube kubectl get nodesNAME       STATUS   ROLES                  AGE     VERSION
minikube Ready control-plane,master 3m50s v1.23.3

4. Create a secret. We will do this through kubectl just to verify that we have access to the secrets API. I’m creating a generic secret with the literal[shhh]=supersecret .

$ minikube kubectl create secret generic my-secret -- --from-literal=shhh=supersecretsecret/my-secret created

5. Now, let’s verify that the secret was created successfully

We can grab it in YAML mode and see the base64 encoded supersecret (c3VwZXJzZWNyZXQ=).

$ minikube kubectl get secrets my-secret -- -o yamlminikube kubectl get secrets my-secret -- -o yaml
apiVersion: v1
data:
shhh: c3VwZXJzZWNyZXQ=
kind: Secret
metadata:
creationTimestamp: "2022-03-20T21:16:48Z"
name: my-secret
namespace: default
resourceVersion: "728"
uid: 9fcb7814-77f1-44dc-b476-066db11598bd
type: Opaque
  1. Clone the application to your GOPATH
$ git clone git@gitlab.com:k2511/secreto-server.gitgit clone git@gitlab.com:k2511/secreto-server.git
Cloning into 'secreto-server'...
remote: Enumerating objects: 235, done.
remote: Counting objects: 100% (232/232), done.
remote: Compressing objects: 100% (121/121), done.
remote: Total 235 (delta 97), reused 177 (delta 69), pack-reused 3
Receiving objects: 100% (235/235), 282.99 KiB | 3.11 MiB/s, done.
Resolving deltas: 100% (97/97), done.
$ cd secreto-server

2. Build the application executable. I created a Makefile which makes it easy. Once this command is run, there should be a new executable created named secreto-server.

$ make buildgo mod download
GOOS=darwin GOARCH=arm64 go build -o secreto-server .
chmod +x secreto-server

Note: You may need to change $GOOS and $GOARCH variables in the Makefile if you aren’t running on an M1 Mac. More details here.

3. Run the application locally. This is done by just passing the -local flag when running the executable. Running it without the -local flag, would require the application to be running in the Kubernetes cluster because it uses a different auth method.

$ ./secreto-server -local2022/03/20 16:18:30 KubeClient running with local configuration
2022/03/20 16:18:30 Starting server on the port 8080

You can also change the port by setting the SECRETO_PORT environment variable before executing the program.

Now the application is running. Let’s go ahead and confirm it’s working. We can do this by opening up another terminal and sending a request to the server to obtain its version.

$ curl http://localhost:8080/api/secreto/version{"version":1}

We now have a working application! Let’s verify several of the application’s functions.

Adding a secret

We can add a secret by passing a name and payload to the secreto API path. Make sure you also add the namespace to the end of the path as seen below.

$ curl -X POST http://localhost:8080/api/secreto/default -d '{"name": "my-secret2", "payload": "my-secret-yoo"}'{"Secret Created Successfully":{"name":"my-secret2","namespace":"default","date":"2022-03-20 17:39:41 -0500 CDT","payload":"my-secret-yoo"}}

Viewing secrets

We can list all of our secrets, sort by namespace, and even look up the payload. This is done by performing a GET on different paths:

  • /api/secreto: lists all secrets
  • /api/secreto/{namespace}: lists all secrets in {namespace}
  • /api/secreto/{namespace}/{name}: lists data for secret {name} in {namespace}
$ curl -X GET http://localhost:8080/api/secreto[{"name":"default-token-m9wsq","namespace":"default","date":"2022-03-20 16:10:27 -0500 CDT","payload":""},{"name":"my-secret","namespace":"default","date":"2022-03-20 16:16:48 -0500 CDT","payload":"supersecret"},{"name":"yeet","namespace":"default","date":"2022-03-20 16:56:24 -0500 CDT","payload":"my-secret-yoo"},{"name":"default-token-dq5wr","namespace":"kube-node-lease","date":"2022-03-20 16:10:26 -0500 CDT","payload":""},{"name":"default-token-nwbxx","namespace":"kube-public","date":"2022-03-20 16:10:26 -0500 CDT","payload":""},{"name":"attachdetach-controller-token-cdfl4","namespace":"kube-system","date":"2022-03-20 16:10:14 -0500 CDT","payload":""},{"name":"bootstrap-signer-token-ljx9n","namespace":"kube-system","date":"2022-03-20 16:10:14 -0500 CDT","payload":""},{"name":"bootstrap-token-81dbvo","namespace":"kube-system","date":"2022-03-20 16:10:13 -0500 CDT","payload":""},{"name":"certificate-controller-token-9nqdf","namespace":"kube-system","date":"2022-03-20 16:10:15 -0500 CDT","payload":""},{"name":"clusterrole-aggregation-controller-token-wb95r","namespace":"kube-system","date":"2022-03-20 16:10:13 -0500 CDT","payload":""},{"name":"coredns-token-7sldt","namespace":"kube-system","date":"2022-03-20 16:10:14 -0500 CDT","payload":""},{"name":"cronjob-controller-token-l5msx","namespace":"kube-system","date":"2022-03-20 16:10:16 -0500 CDT","payload":""},{"name":"daemon-set-controller-token-ppr8p","namespace":"kube-system","date":"2022-03-20 16:10:16 -0500 CDT","payload":""},{"name":"default-token-mxzhs","namespace":"kube-system","date":"2022-03-20 16:10:26 -0500 CDT","payload":""},{"name":"deployment-controller-token-ctsrt","namespace":"kube-system","date":"2022-03-20 16:10:13 -0500 CDT","payload":""},{"name":"disruption-controller-token-kf9qs","namespace":"kube-system","date":"2022-03-20 16:10:14 -0500 CDT","payload":""},{"name":"endpoint-controller-token-kkp5b","namespace":"kube-system","date":"2022-03-20 16:10:13 -0500 CDT","payload":""},{"name":"endpointslice-controller-token-b4bwk","namespace":"kube-system","date":"2022-03-20 16:10:13 -0500 CDT","payload":""},{"name":"endpointslicemirroring-controller-token-g7bqq","namespace":"kube-system","date":"2022-03-20 16:10:15 -0500 CDT","payload":""},{"name":"ephemeral-volume-controller-token-t7s6h","namespace":"kube-system","date":"2022-03-20 16:10:14 -0500 CDT","payload":""},{"name":"expand-controller-token-wvhn8","namespace":"kube-system","date":"2022-03-20 16:10:26 -0500 CDT","payload":""},{"name":"generic-garbage-collector-token-q62cw","namespace":"kube-system","date":"2022-03-20 16:10:26 -0500 CDT","payload":""},{"name":"horizontal-pod-autoscaler-token-wmkcc","namespace":"kube-system","date":"2022-03-20 16:10:13 -0500 CDT","payload":""},{"name":"job-controller-token-9492p","namespace":"kube-system","date":"2022-03-20 16:10:26 -0500 CDT","payload":""},{"name":"kube-proxy-token-6z9ht","namespace":"kube-system","date":"2022-03-20 16:10:14 -0500 CDT","payload":""},{"name":"namespace-controller-token-lrwx8","namespace":"kube-system","date":"2022-03-20 16:10:15 -0500 CDT","payload":""},{"name":"node-controller-token-x9vwn","namespace":"kube-system","date":"2022-03-20 16:10:16 -0500 CDT","payload":""},{"name":"persistent-volume-binder-token-vdw68","namespace":"kube-system","date":"2022-03-20 16:10:26 -0500 CDT","payload":""},{"name":"pod-garbage-collector-token-jl9z2","namespace":"kube-system","date":"2022-03-20 16:10:16 -0500 CDT","payload":""},{"name":"pv-protection-controller-token-jv9d8","namespace":"kube-system","date":"2022-03-20 16:10:13 -0500 CDT","payload":""},{"name":"pvc-protection-controller-token-d4ccm","namespace":"kube-system","date":"2022-03-20 16:10:13 -0500 CDT","payload":""},{"name":"replicaset-controller-token-hbdj6","namespace":"kube-system","date":"2022-03-20 16:10:15 -0500 CDT","payload":""},{"name":"replication-controller-token-74kl8","namespace":"kube-system","date":"2022-03-20 16:10:13 -0500 CDT","payload":""},{"name":"resourcequota-controller-token-767r2","namespace":"kube-system","date":"2022-03-20 16:10:13 -0500 CDT","payload":""},{"name":"root-ca-cert-publisher-token-7zbhn","namespace":"kube-system","date":"2022-03-20 16:10:14 -0500 CDT","payload":""},{"name":"service-account-controller-token-vdxgt","namespace":"kube-system","date":"2022-03-20 16:10:26 -0500 CDT","payload":""},{"name":"service-controller-token-nvt8n","namespace":"kube-system","date":"2022-03-20 16:10:15 -0500 CDT","payload":""},{"name":"statefulset-controller-token-97d8r","namespace":"kube-system","date":"2022-03-20 16:10:26 -0500 CDT","payload":""},{"name":"storage-provisioner-token-nsblb","namespace":"kube-system","date":"2022-03-20 16:10:16 -0500 CDT","payload":""},{"name":"token-cleaner-token-wdbdn","namespace":"kube-system","date":"2022-03-20 16:10:15 -0500 CDT","payload":""},{"name":"ttl-after-finished-controller-token-rgjt4","namespace":"kube-system","date":"2022-03-20 16:10:16 -0500 CDT","payload":""},{"name":"ttl-controller-token-tzjfc","namespace":"kube-system","date":"2022-03-20 16:10:14 -0500 CDT","payload":""}]$ curl -X GET http://localhost:8080/api/secreto/default[{"name":"default-token-m9wsq","namespace":"default","date":"2022-03-20 16:10:27 -0500 CDT","payload":""},{"name":"my-secret","namespace":"default","date":"2022-03-20 16:16:48 -0500 CDT","payload":"supersecret"},{"name":"my-secret2","namespace":"default","date":"2022-03-20 16:56:24 -0500 CDT","payload":"my-secret-yoo"}]$ curl -X GET http://localhost:8080/api/secreto/default/my-secret{"name":"my-secret","namespace":"default","date":"2022-03-20 16:16:48 -0500 CDT","payload":"supersecret"}

Deleting secrets

Now, let’s go ahead and delete a secret we created earlier. This is done by performing a DELETE on the full path of a secret.

$ curl -X DELETE http://localhost:8080/api/secreto/default/my-secret{"Secret Deleted Successfully":"my-secret"}

Now let’s dive into the application code. I’m primarily going to go over the parts which interact with the Kubernetes API. The application is split up in the following way:

  • middleware: Contains all the application logic for processing a request and generating a response. This includes communicating with the Kubernetes API in order to perform different functions on secrets.
  • router: Routes calls from particular URIs to the application logic in the middleware.
  • model: Contains structs that are used in the application to display and create secrets.
  • main.go: Starts the application, loading the web-server.

Now let’s take a dive into how the source code works.

Authentication and setup

Authenticating with a Kubernetes client can be seen in middleware.go. In this article, we will be going over out-of-cluster authentication.

This code was taken from the out-of-cluster config example of Client-Go. The main parts to notice are:

  • The Kubernetes config is set by looking at what’s active in the ~/.kube/config folder
  • If no home directory exists, then we must pass -kubeconfig along with the path of our Kubernetes configuration in order for it to load properly
  • There is a global variable ClientSet=client set so that we don’t need to keep loading the kubeconfig and can just run commands with the ClientSet

Adding a secret

In order to add a secret, the code below accepts a request, calls the private createSecret function, and then returns the response to the user. The main parts to notice are:

  • Sets headers for CORS, for example, Access-Control-Allow-Methods which will allow browsers to use different types of methods
  • Grabs parameters for the namespace from the URI route {namespace} defined in router.go
  • Generates a Kubernetes API call based on the items in the request body
  • Encodes either the actual secret and returns it, or returns an error

The code below is a private function that uses Client-Go in order to create a secret. The main parts to notice are:

  • metav1.ObjectMeta is setup along with the secret payload which is encapsulated in the maps secretData and secretDataBytes
  • A Kubernetes v1.Secret based on the data sent in the function is passed to the Kubernetes API for the creation of the secret
  • If the generation of the secret is successful then we generate a Secreto object and fill it with data from the secret before returning it

Viewing a secret

We have several functions used to view secrets. They grab Kubernetes’ secrets and their info using Client-Go. I’m just going to go over obtaining the secret details since most functions are similar. The main parts to notice are:

  • Sets headers for CORS, for example, Access-Control-Allow-Methods which will allow browsers to use different types of methods
  • Grabs parameters for the namespace and name from the URI route {namespace}/{name} defined in router.go, and generates a call to the Kubernetes API
  • Encodes either the actual secret and returns it, or returns an error

The code below is a private function that uses Client-Go in order to describe a secret obtaining its details with Client-Go.

func getSecretDetails(namespace string, name string) (*v1.Secret, error) {
secret, err := ClientSet.CoreV1().Secrets(namespace).Get(context.TODO(), name, metav1.GetOptions{})

if err != nil {
return nil, err
}

return secret, nil
}

Deleting a secret

Deleting a secret is pretty straightforward. The main parts to notice are:

  • Sets headers for CORS, for example, Access-Control-Allow-Methods which will allow browsers to use different types of methods
  • Grabs parameters for the namespace and name from the URI route {namespace}/{name} defined in router.go, and generates a call to the Kubernetes API
  • Returns a message saying that the secret was successfully deleted, or returns an error

The code below is a private function that uses Client-Go in order to delete a secret:

func deleteSecret(name string, namespace string) error {
err := ClientSet.CoreV1().Secrets(namespace).Delete(context.TODO(), name, metav1.DeleteOptions{})
if err != nil {
return err
}

return nil
}

Now that we have seen all the different functions within the application, we are gonna go ahead and write some unit tests. Unit tests are individual tests on different parts of our application, which are important for verifying our logic and making sure our application is doing what it should.

All the unit tests I have written are located in middleware_test.go.

General setup

I created a function that just sets up test values ​​for the different unit tests. You can see the setupSecrets() function which will generate different secrets as well as expected values ​​to look for.

Mocking Client-Go

ClientSet is a global variable defined in middleware.go. Within the tests, we overwrite it, allowing all the requests to return fake values ​​without communicating to our Kubernetes cluster. The fake client makes requests returning mock objects and values.

ClientSet = fake.NewSimpleClientset()

Mocking requests

Requests can be mocked using httptest, which provides utilities for HTTP testing and allows us to “record” requests. A few things to notice are:

  • A request is created and test variables are passed in the requestBody
  • The CreateSecret function is called with the fake w http.ResponseWriter, r *http.Request that we generated

Running tests

Now that we’ve gone over the tests, we can go ahead and run them. This can be done by running the following command:

$ make testgo test -v ./...
? gitlab.com/k2511/kube-secreto [no test files]
=== RUN TestProcessSecrets
--- PASS: TestProcessSecrets (0.00s)
=== RUN TestGetSecretsClient
--- PASS: TestGetSecretsClient (0.00s)
=== RUN TestGetSecretDetailsClient
--- PASS: TestGetSecretDetailsClient (0.00s)
=== RUN TestCreateSecretClient
--- PASS: TestCreateSecretClient (0.00s)
=== RUN TestDeleteSecretClient
--- PASS: TestDeleteSecretClient (0.00s)
=== RUN TestGetSecretsByNamespace
--- PASS: TestGetSecretsByNamespace (0.00s)
=== RUN TestGetSecretDetails
--- PASS: TestGetSecretDetails (0.00s)
=== RUN TestCreateSecret
--- PASS: TestCreateSecret (0.00s)
=== RUN TestDeleteSecret
--- PASS: TestDeleteSecret (0.00s)
=== RUN TestGetVersion
--- PASS: TestGetVersion (0.00s)
PASS
ok gitlab.com/k2511/kube-secreto/internal/server/middleware 1.406s
? gitlab.com/k2511/kube-secreto/internal/server/models [no test files]
? gitlab.com/k2511/kube-secreto/internal/server/router [no test files]

I am using GitLab for CI to automate building, testing, and pushing the application container to my registry. This makes it so that I don’t have to manually perform these functions each time I push code.

I can configure GitLab so that my application is automatically tested so I can verify my application logic. My application is also built, containerized, and pushed to my container registry. I can also check if my source code is secure using GitLab SAST.

My GitLab Yaml looks as follows:

Now, if we look at the newest running GitLab Pipeline, we can see the following:

  • Build stage: runs build which builds the application and docker-build which builds the application container and pushes it to my container registry
  • Test stage: unit runs unit tests and generates a coverage report, gosec-sast and semgrep-sast uses gosec and semgrep respectively to scan the application source code for vulnerabilities

When clicking on the Security tab, we can see a few vulnerabilities we should resolve, sorted by incision.

Clicking on one provides a description, location, CVE, and solution. You can also dismiss the vulnerability or create a confidential issue to work on remediation with others without alerting those without permission.

There’s also more security scanners you can look into as well as other cool CICD tools at GitLab.

Leave a Comment