Deploy and Configure Kustomize and MongoDB in Kubernetes Stateful Applications | by Nic Vermande | Feb, 2022

Part 2: Let’s Kustomize, install the MongoDB operator, and Ondat in our CI/CD pipeline

Nick Vermande
Photo by Vlad Hilitanu on Unsplash

This is the second article in a series. The first one is available here, where we present the tools allowing us to create a Kubernetes-friendly and automated environment for developing a web application based on Flask, MongoDB, Pymongo, and the Marvel API. In this second part, we deploy and configure Kustomize, MongoDB, and Ondat.

As mentioned in the previous article, there are 2 ways to use Kustomize. You can originally use kubectl -k or install the binary. As for the latter, you can use the following command that automatically detects your OS and install Kustomize:

curl -s "https://raw.githubusercontent.com/kubernetes-sigs/kustomize/master/hack/install_kustomize.sh"  | bash

However, this script doesn’t work for ARM-based architectures. Alternatively, navigate to the release page, pick the correct binary for your distribution, and move it into your $PATH.

To test that Kustomize is working as expected, you can download the application manifests from https://github.com/vfiftyfive/CFD12-Demo-Manifests and generate the dev manifests. For this, run the following commands:

$ git clone https://github.com/vfiftyfive/CFD12-Demo-Manifests && cd CFD12-Demo-Manifests
Cloning into 'CFD12-Demo-Manifests'...
remote: Enumerating objects: 164, done.
remote: Counting objects: 100% (164/164), done.
remote: Compressing objects: 100% (96/96), done.
remote: Total 164 (delta 80), reused 136 (delta 52), pack-reused 0
Receiving objects: 100% (164/164), 15.02 KiB | 3.75 MiB/s, done.
Resolving deltas: 100% (80/80), done.
$ kustomize build overlay/dev
allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: ondat
parameters:
csi.storage.k8s.io/controller-expand-secret-name: csi-controller-expand-secret
csi.storage.k8s.io/controller-expand-secret-namespace: kube-system
csi.storage.k8s.io/controller-publish-secret-name: csi-controller-publish-secret
csi.storage.k8s.io/controller-publish-secret-namespace: kube-system
csi.storage.k8s.io/fstype: ext4
csi.storage.k8s.io/node-publish-secret-name: csi-node-publish-secret
csi.storage.k8s.io/node-publish-secret-namespace: kube-system
csi.storage.k8s.io/provisioner-secret-name: csi-provisioner-secret
csi.storage.k8s.io/provisioner-secret-namespace: kube-system
storageos.com/replicas: "1"
provisioner: csi.storageos.com
...

We truncated the output, but the command produces all the manifests required to deploy the application into the dev cluster. You can also notice there’s an additional file named kustomization.yaml within the base and dev folders. These files are required for Kustomize to know which manifests to render and how. The customization file located in the base folder contains the following code:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ondat_sc.yaml
- mongo_sts.yaml
- marvel_deploy.yaml
- job.yaml
- marvel_svc.yaml

Once Kustomize looks into the base folder, it applies customization to all its children YAML resources described in that file under resources.

The dev folder also contains a kustomization.yaml file, which details the specific changes needed to render the final version of the manifests.

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
patchesStrategicMerge:
- ondat_sc.yaml
- marvel_deploy.yaml
- mongodbcommunity_cr.yaml
- job.yaml
resources:
- ../../base
configMapGenerator:
- name: mongo-config
literals:
- MONGO_SEED0=mongodb-0.mongodb.default.svc.cluster.local
- MONGO_SEED1=mongodb-1.mongodb.default.svc.cluster.local
- MONGO_SEED2=mongodb-2.mongodb.default.svc.cluster.local
- OFFSET=600
- MONGO_USERNAME=admin
secretGenerator:
- name: admin-password
literals:
- password=mongo
configurations:
- name_reference.yaml

Let’s take a look at the different sections:

  • pacthesStrategicMerge lists the files Kustomize should look at when building the target manifests. These files are located under the dev folder and describe the amendments applied to the manifests located in the base folder. These files are accessible within the git repo, and here is a summary of the changes they describe:
  • ondat.sc.yaml: Create 1 replica for all volumes using that StorageClassand enable encryption.
  • marvel_deploy.yaml: Set the number of replicas to 2, inject the MongoDB password from the Kustomize Secretgenerator, and add the environment variables from the configMap generator.
  • mongodbcommunity_cr.yaml: Set the MongoDB version to 5.0.5, set up the admin user, configure the Kubernetes StatefulSet with a volumeClaimTemplate using the Ondat StorageClassand set the data and logs volume size.
  • job.yaml: Inject the MongoDB password from the Kustomize Secretgenerator and environment variables from the configMap generator.

In the remaining sections, we have:

  • resources which defines the location of the base manifests.
  • configMapGenerator which generates a configMap with a unique name every time Customize runs. You can use literals or files to define your variables.
  • secretGenerator which generates Secrets with a unique name every time Customize runs. You can use literals or files to define your secrets.
  • configurations , which specifies custom objects path that Customize uses to modify or insert particular values. For example, in our scenario, Customize dynamically configures the reference to a secret in the MongoDB custom resource. As it is a custom resource, Kustomize doesn’t know where to find the equivalent of a secret name parameter. Instead, we define it in the file name_reference.yaml:
nameReference:
- kind: Secret
fieldSpecs:
- kind: MongoDBCommunity
path: spec/users/passwordSecretRef/name

As a result, a unique Secret name composed of the initial Secret name and a random string will be added at this location every time Kustomize is invoked. Kustomize will also replace all secrets within native objects YAML definition in relevant resources. Those are the resources you have defined in the kustomization.yaml file.

For this article, we have used the MongoDB Community Operator since it is free and easy to use. However, for production environments, we recommend using an Enterprise version of the Operator or being ready to support it yourself and get help from the community. There are multiple options available, such as the MongoDB Enterprise Operator, the Percona Operator, or the KubeDB Operator.

Understanding the Operator

Some of you may not be familiar with Kubernetes Operators or why we need one to install a database cluster altogether. So let’s focus a little bit on this aspect. A Kubernetes Operator is fundamentally made up of 2 parts:

  • A piece of code provided as a container that continuously monitors specific objects in the Kubernetes API and performs actions as a result of this active monitoring. It is called a custom controller.
  • The monitored custom resources. A custom resource is a Kubernetes object that is not native. The custom resource schema is defined within the Custom Resource Definiton (CRD), provided as a YAML file by the user, and sent to the Kubernetes API. Every object created with that schema is further saved in the Kubernetes etcd store. You can compare the CRD to a class in OOP and the custom resource to an instance of that class. It extends the existing Kubernetes API.

The Kubernetes Operator’s job is to perform automated actions and interact with the cluster or systems outside the cluster like a human operator. It is not limited to deploying components only. It can perform CRUD operations in reaction to Kubernetes events, which are dynamic by nature. It utilizes events data as input to workflows. A common use case for Operators is to automate the deployment of software solutions within Kubernetes. So in the case of a database, the Kubernetes Operator will install the database (cluster or standalone) once the corresponding database resource has been ingested by the Kubernetes API, typically in the form of a YAML manifest that describes the configuration of the database as a custom resource.

Also, remember the database is as a StatefulSet. When you scale the StatefulSetKubernetes doesn’t automagically scale the database cluster. It just deploys new containers with the database image. There is some extra work needed to configure it. It is performed by the Kubernetes Operator. The same is true when you scale down the StatefulSet.

In the case of the MongoDB Operator, the added CRD produces an object of kind mongodbcommunity . The custom resource encapsulates all the information required to deploy and maintain a MongoDB database. First, you need to install the Operator and create the custom resource. It can be achieved during the automated deployment of your Kubernetes cluster or manually once the cluster has been installed. You can find all the required files in this repo. We’re going to perform a couple of steps to install the Operator:

  • Install the CRD
  • Create the Operator configuration manifest. You can refer to the MongoDB Community Operator documentation and examples. You can also find the configuration we have used in the git repo. The Operator configuration manifest is manager.yaml.

You need to configure the scope of the Operator first. It defines which namespaces the Operator monitors: the Namespace where the Operator is installed, a specific namespace, or all namespaces. In our example, the manager.yaml file has the following configuration:

env:
- name: WATCH_NAMESPACE
value: "*"

It tells the Operator to monitor all namespaces for MongoDB custom resources operations.

  • Create cluster-wide roles and role-bindings. The ClusterRoleBindingservice account namespace must be modified with the name of the namespace you want to use (in bold in the text below). It is located in the file clusterwide/role_binding.yaml
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: mongodb-kubernetes-operator
subjects:
- kind: ServiceAccount
namespace: mongo-operator
name: mongodb-kubernetes-operator
roleRef:
kind: ClusterRole
name: mongodb-kubernetes-operator
apiGroup: rbac.authorization.k8s.io
  • For each namespace that you wish the Operator to watch, you need to deploy a Role, RoleBindingand ServiceAccount in that namespace
  • The last step is creating and deploying the database configuration manifest (the custom resource), which is detailed later.

Deploy the Operator

Execute the following commands to install and configure the Operator:

#Clone the git repository
$ git clone https://github.com/vfiftyfive/mongodb-community-operator-manifests && cd mongodb-community-operator-manifests
Cloning into 'mongodb-community-operator-manifests'...
remote: Enumerating objects: 17, done.
remote: Counting objects: 100% (17/17), done.
remote: Compressing objects: 100% (14/14), done.
remote: Total 17 (delta 5), reused 15 (delta 3), pack-reused 0
Receiving objects: 100% (17/17), done.
Resolving deltas: 100% (5/5), done.
#Install the CRD
$ kubectl apply -f mongo-crd.yaml
#Deploy the clusterwide RBAC resources
$ kubectl apply -f clusterwide/
clusterrole.rbac.authorization.k8s.io/mongodb-kubernetes-operator created
clusterrolebinding.rbac.authorization.k8s.io/mongodb-kubernetes-operator created
#Create a namespace for the operator
$ kubectl create ns mongo-operator
namespace/mongo-operator created
#Deploy namespace RBAC resources in the operator namespace.
$ kubectl apply -k rbac/ -n mongo-operator
serviceaccount/mongodb-database created
serviceaccount/mongodb-kubernetes-operator created
role.rbac.authorization.k8s.io/mongodb-database created
role.rbac.authorization.k8s.io/mongodb-kubernetes-operator created
rolebinding.rbac.authorization.k8s.io/mongodb-database created
rolebinding.rbac.authorization.k8s.io/mongodb-kubernetes-operator created
#Deploy namespace RBAC resources in the default namespace (where the app will be deployed)
$ kubectl apply -k rbac/
serviceaccount/mongodb-database created
serviceaccount/mongodb-kubernetes-operator created
role.rbac.authorization.k8s.io/mongodb-database created
role.rbac.authorization.k8s.io/mongodb-kubernetes-operator created
rolebinding.rbac.authorization.k8s.io/mongodb-database created
rolebinding.rbac.authorization.k8s.io/mongodb-kubernetes-operator created
#Deploy the Operator
$ k apply -f manager.yaml -n mongo-operator
deployment.apps/mongodb-kubernetes-operator created
#Check the Operator has correctly been deployed
$ kubectl get po -n mongo-operator
NAME READY STATUS ...
mongodb-kubernetes-operator-6d46dd4b74-ztx9c 1/1 Running ...

The Operator is now ready to deploy a new MongoDB database as soon as it detects that a new custom resource has been added into the Kubernetes API.

The next step is to install the distributed storage layer, Ondat (formerly StorageOS).

Ondat provides a data-mesh acting as a distributed persistent storage layer that provides all sorts of premium features that are not included by default in Kubernetes. This includes replication, encryption, performance optimization, intelligent volume placement, etc, all managed as part of native Kubernetes labels and annotations.

It can be with a single line through a kubectl plugin (or alternatively using a gel chart available here):

kubectl storageos install --include-etcd

But first, let’s install the plugin by executing the following command:

curl -sSLo kubectl-storageos.tar.gz 
https://github.com/storageos/kubectl-storageos/releases/download/v1.0.0/kubectl-storageos_1.0.0_linux_amd64.tar.gz
&& tar -xf kubectl-storageos.tar.gz
&& chmod +x kubectl-storageos
&& sudo mv kubectl-storageos /usr/local/bin/
&& rm kubectl-storageos.tar.gz

As there are some prereqs for your Kubernetes hosts’ kernel modules, you should first run the preflight checks to see if your cluster is compatible by running the following command:

kubectl storageos preflight

(Quick tip: if you are using GKE, use the ubuntu_containerd image, it already includes the linux-modules-extra-xyz package)

The command will output a comprehensive report like the one below. The screenshot depicts an example with no NVMe drives present in our 3-node cluster. You can safely ignore this warning if you’re not running performance-intensive workloads.

You can then use the plugin with the install sub-command:

kubeclt storageos install --include-etcd

There are many options to this command. If you consider a production deployment, you should take a look at them and get familiar with best practices. As this is not the topic of this article, I’m just going to point you in the right direction here!

To check the deployment is complete, make sure the DaemonSet is up and running by running the following command:

$ kubectl get pods -n storageos | grep node                                                                                                                                                 
storageos-node-8chxx 3/3 Running
storageos-node-gsdzt 3/3 Running
storageos-node-sq2ds 3/3 Running

All containers must be up and running. You should also notice a new StorageClass named “storageos”. But remember that Kustomize is going to create a new StorageClassso we’re not going to use that one.

Leave a Comment