Part 2: Let’s Kustomize, install the MongoDB operator, and Ondat in our CI/CD pipeline
This is the second article in a series. The first one is available here, where we present the tools allowing us to create a Kubernetes-friendly and automated environment for developing a web application based on Flask, MongoDB, Pymongo, and the Marvel API. In this second part, we deploy and configure Kustomize, MongoDB, and Ondat.
As mentioned in the previous article, there are 2 ways to use Kustomize. You can originally use
kubectl -k or install the binary. As for the latter, you can use the following command that automatically detects your OS and install Kustomize:
curl -s "https://raw.githubusercontent.com/kubernetes-sigs/kustomize/master/hack/install_kustomize.sh" | bash
However, this script doesn’t work for ARM-based architectures. Alternatively, navigate to the release page, pick the correct binary for your distribution, and move it into your
To test that Kustomize is working as expected, you can download the application manifests from https://github.com/vfiftyfive/CFD12-Demo-Manifests and generate the dev manifests. For this, run the following commands:
$ git clone https://github.com/vfiftyfive/CFD12-Demo-Manifests && cd CFD12-Demo-Manifests
Cloning into 'CFD12-Demo-Manifests'...
remote: Enumerating objects: 164, done.
remote: Counting objects: 100% (164/164), done.
remote: Compressing objects: 100% (96/96), done.
remote: Total 164 (delta 80), reused 136 (delta 52), pack-reused 0
Receiving objects: 100% (164/164), 15.02 KiB | 3.75 MiB/s, done.
Resolving deltas: 100% (80/80), done.$ kustomize build overlay/dev
We truncated the output, but the command produces all the manifests required to deploy the application into the dev cluster. You can also notice there’s an additional file named
kustomization.yaml within the
dev folders. These files are required for Kustomize to know which manifests to render and how. The customization file located in the
base folder contains the following code:
Once Kustomize looks into the
base folder, it applies customization to all its children YAML resources described in that file under resources.
The dev folder also contains a
kustomization.yaml file, which details the specific changes needed to render the final version of the manifests.
- name: mongo-config
- name: admin-password
Let’s take a look at the different sections:
pacthesStrategicMergelists the files Kustomize should look at when building the target manifests. These files are located under the dev folder and describe the amendments applied to the manifests located in the base folder. These files are accessible within the git repo, and here is a summary of the changes they describe:
ondat.sc.yaml: Create 1 replica for all volumes using that
StorageClassand enable encryption.
marvel_deploy.yaml: Set the number of replicas to 2, inject the MongoDB password from the Kustomize
Secretgenerator, and add the environment variables from the
mongodbcommunity_cr.yaml: Set the MongoDB version to 5.0.5, set up the admin user, configure the Kubernetes
volumeClaimTemplateusing the Ondat
StorageClassand set the data and logs volume size.
job.yaml: Inject the MongoDB password from the Kustomize
Secretgenerator and environment variables from the
In the remaining sections, we have:
resourceswhich defines the location of the base manifests.
configMapGeneratorwhich generates a
configMapwith a unique name every time Customize runs. You can use literals or files to define your variables.
Secretswith a unique name every time Customize runs. You can use literals or files to define your secrets.
configurations, which specifies custom objects path that Customize uses to modify or insert particular values. For example, in our scenario, Customize dynamically configures the reference to a secret in the MongoDB custom resource. As it is a custom resource, Kustomize doesn’t know where to find the equivalent of a secret name parameter. Instead, we define it in the file
- kind: Secret
- kind: MongoDBCommunity
As a result, a unique
Secret name composed of the initial
Secret name and a random string will be added at this location every time Kustomize is invoked. Kustomize will also replace all secrets within native objects YAML definition in relevant resources. Those are the resources you have defined in the
For this article, we have used the MongoDB Community Operator since it is free and easy to use. However, for production environments, we recommend using an Enterprise version of the Operator or being ready to support it yourself and get help from the community. There are multiple options available, such as the MongoDB Enterprise Operator, the Percona Operator, or the KubeDB Operator.
Understanding the Operator
Some of you may not be familiar with Kubernetes Operators or why we need one to install a database cluster altogether. So let’s focus a little bit on this aspect. A Kubernetes Operator is fundamentally made up of 2 parts:
- A piece of code provided as a container that continuously monitors specific objects in the Kubernetes API and performs actions as a result of this active monitoring. It is called a custom controller.
- The monitored custom resources. A custom resource is a Kubernetes object that is not native. The custom resource schema is defined within the Custom Resource Definiton (CRD), provided as a YAML file by the user, and sent to the Kubernetes API. Every object created with that schema is further saved in the Kubernetes etcd store. You can compare the CRD to a class in OOP and the custom resource to an instance of that class. It extends the existing Kubernetes API.
The Kubernetes Operator’s job is to perform automated actions and interact with the cluster or systems outside the cluster like a human operator. It is not limited to deploying components only. It can perform CRUD operations in reaction to Kubernetes events, which are dynamic by nature. It utilizes events data as input to workflows. A common use case for Operators is to automate the deployment of software solutions within Kubernetes. So in the case of a database, the Kubernetes Operator will install the database (cluster or standalone) once the corresponding database resource has been ingested by the Kubernetes API, typically in the form of a YAML manifest that describes the configuration of the database as a custom resource.
Also, remember the database is as a
StatefulSet. When you scale the
StatefulSetKubernetes doesn’t automagically scale the database cluster. It just deploys new containers with the database image. There is some extra work needed to configure it. It is performed by the Kubernetes Operator. The same is true when you scale down the
In the case of the MongoDB Operator, the added CRD produces an object of kind
mongodbcommunity . The custom resource encapsulates all the information required to deploy and maintain a MongoDB database. First, you need to install the Operator and create the custom resource. It can be achieved during the automated deployment of your Kubernetes cluster or manually once the cluster has been installed. You can find all the required files in this repo. We’re going to perform a couple of steps to install the Operator:
- Install the CRD
- Create the Operator configuration manifest. You can refer to the MongoDB Community Operator documentation and examples. You can also find the configuration we have used in the git repo. The Operator configuration manifest is
You need to configure the scope of the Operator first. It defines which namespaces the Operator monitors: the
Namespace where the Operator is installed, a specific namespace, or all namespaces. In our example, the
manager.yaml file has the following configuration:
- name: WATCH_NAMESPACE
It tells the Operator to monitor all namespaces for MongoDB custom resources operations.
- Create cluster-wide roles and role-bindings. The
ClusterRoleBindingservice account namespace must be modified with the name of the namespace you want to use (in bold in the text below). It is located in the file
- kind: ServiceAccount
- For each namespace that you wish the Operator to watch, you need to deploy a
ServiceAccountin that namespace
- The last step is creating and deploying the database configuration manifest (the custom resource), which is detailed later.
Deploy the Operator
Execute the following commands to install and configure the Operator:
#Clone the git repository
$ git clone https://github.com/vfiftyfive/mongodb-community-operator-manifests && cd mongodb-community-operator-manifests
Cloning into 'mongodb-community-operator-manifests'...
remote: Enumerating objects: 17, done.
remote: Counting objects: 100% (17/17), done.
remote: Compressing objects: 100% (14/14), done.
remote: Total 17 (delta 5), reused 15 (delta 3), pack-reused 0
Receiving objects: 100% (17/17), done.
Resolving deltas: 100% (5/5), done.#Install the CRD
$ kubectl apply -f mongo-crd.yaml#Deploy the clusterwide RBAC resources
$ kubectl apply -f clusterwide/
clusterrolebinding.rbac.authorization.k8s.io/mongodb-kubernetes-operator created#Create a namespace for the operator
$ kubectl create ns mongo-operator
namespace/mongo-operator created#Deploy namespace RBAC resources in the operator namespace.
$ kubectl apply -k rbac/ -n mongo-operator
rolebinding.rbac.authorization.k8s.io/mongodb-kubernetes-operator created#Deploy namespace RBAC resources in the default namespace (where the app will be deployed)
$ kubectl apply -k rbac/
rolebinding.rbac.authorization.k8s.io/mongodb-kubernetes-operator created#Deploy the Operator
$ k apply -f manager.yaml -n mongo-operator
deployment.apps/mongodb-kubernetes-operator created#Check the Operator has correctly been deployed
$ kubectl get po -n mongo-operator
NAME READY STATUS ...
mongodb-kubernetes-operator-6d46dd4b74-ztx9c 1/1 Running ...
The Operator is now ready to deploy a new MongoDB database as soon as it detects that a new custom resource has been added into the Kubernetes API.
The next step is to install the distributed storage layer, Ondat (formerly StorageOS).
Ondat provides a data-mesh acting as a distributed persistent storage layer that provides all sorts of premium features that are not included by default in Kubernetes. This includes replication, encryption, performance optimization, intelligent volume placement, etc, all managed as part of native Kubernetes labels and annotations.
It can be with a single line through a
kubectl plugin (or alternatively using a gel chart available here):
kubectl storageos install --include-etcd
But first, let’s install the plugin by executing the following command:
curl -sSLo kubectl-storageos.tar.gz
&& tar -xf kubectl-storageos.tar.gz
&& chmod +x kubectl-storageos
&& sudo mv kubectl-storageos /usr/local/bin/
&& rm kubectl-storageos.tar.gz
As there are some prereqs for your Kubernetes hosts’ kernel modules, you should first run the preflight checks to see if your cluster is compatible by running the following command:
kubectl storageos preflight
(Quick tip: if you are using GKE, use the ubuntu_containerd image, it already includes the linux-modules-extra-xyz package)
The command will output a comprehensive report like the one below. The screenshot depicts an example with no NVMe drives present in our 3-node cluster. You can safely ignore this warning if you’re not running performance-intensive workloads.
You can then use the plugin with the install sub-command:
kubeclt storageos install --include-etcd
There are many options to this command. If you consider a production deployment, you should take a look at them and get familiar with best practices. As this is not the topic of this article, I’m just going to point you in the right direction here!
To check the deployment is complete, make sure the
DaemonSet is up and running by running the following command:
$ kubectl get pods -n storageos | grep node
storageos-node-8chxx 3/3 Running
storageos-node-gsdzt 3/3 Running
storageos-node-sq2ds 3/3 Running
All containers must be up and running. You should also notice a new
StorageClass named “storageos”. But remember that Kustomize is going to create a new
StorageClassso we’re not going to use that one.