All wrapped in an easy, repeatable, and maintainable process
The standard implementation of Netflix Conductor is configured to use Elasticsearch 6. This article covers the changes that need to be made to use Elasticsearch 7 and PostgreSQL, as well as providing several scripts to automate building and running Conductor, both locally and in a cloud architecture .
At the time of writing, the latest released version of Netflix Conductor is 3.5.3. The information in this article is based on version 3.5.1 but should be applicable (and easily modified) to the latest version. If you run into any issues, please let me know, and I’ll do my best to help. Another great source of help is the Netflix Conductor discussion board.
I am running all the supplied code on a Windows 10 machine, from Git Bash, which comes included with Git for Windows. Our cloud architecture runs on CentOS.
Netflix Conductor is a workflow orchestration engine that was built by Netflix to “orchestrate microservices-based process flows” (source: Netflix Conductor documentation). The full feature list of Netflix Conductor is located here, but a few key features are:
- Workflow and task definitions are implemented in JSON format. This allows for versioning and easy support for custom tools to create and manage workflows and tasks.
- Tasks and Workers, ie, the building blocks that make up the workflows and the microservices that host them, respectively, are completely language agnostic, which allows the implementation to be done in the language best suited for the tasks. Libraries exist for Java, Python, .NET (shameless plug: this one is developed by me), and others.
- The database architecture is pluggable, which means we can choose what database we want to use for Conductor. Conductor comes with numerous ready-made plugins, supporting (amongst others) Dynomite (also by Netflix), MySQL, and PostgreSQL.
- Netflix Conductor comes with an (optional) UI, which allows insight and control over workflows, like (re)starting, pausing, stopping.
- The entire project is open source and is being actively developed and supported.
In short, Netflix Conductor allows you to automate and manage processes that consist of several tasks (which might be processes in themselves), that may need to be executed in sequence, in a reliable, consistent, and scalable manner.
At my current project, we use Netflix Conductor and PostgreSQL to run scheduled ETL processes, create rapports, download and upload files, and for monitoring and alerting. We have also investigated using a conductor to automate the provisioning of user accounts and other infrastructure.
As mentioned in the introduction, the standard implementation of Netflix Conductor is configured to use Elasticsearch 6. Support for Elasticsearch 7 is built-in, but several changes need to be made to enable it. There is documentation available on GitHub to make these changes, but it does not seem entirely complete. I needed the changes I describe below to build and run against Elasticsearch 7.
At the time of writing, Elasticsearch 7.17.1 is the latest version of Elasticsearch 7.
Here’s a quick summary of the required changes (all file references are relative to the root folder):
ext['elasticsearch.version'] = revElasticSearch6
ext['elasticsearch.version'] = revElasticSearch7
Remove the line
The standard way to apply these changes is to create a fork of the Conductor repository and do your work from there.
However, I find copying over an entire repository to change a handful of files to be a waste of resources, so I am following a different approach.
On build, I
clone the Conductor repository, then apply several
patch files to make the changes I need. While these steps increase the build time slightly, I feel this is a good compromise between speed and keeping my own repository clean and maintainable.
Here’s how that works:
- Clone the Conductor repo inside your project directory.
- Make the above changes.
- In the Conductor folder, run
git diff > v3.5.1-update-to-es7.patchto create a
v3.5.1is for convenience, to indicate this patch only works against that version.
- Copy the
patchfile to the root directory, so you can safely remove the conductor repo and apply the changes whenever you clone the repo again.
You also need to remove any
dependencies.lock files before building. This is as simple as running
rm -f **/dependencies.lock.
To automate removing the
lock files and applying the patches, I created the
You can see I am actually applying a few more patches. These are not required to run against Elasticsearch 7.
At this point, you can build Conductor. I created
02.build-conductor-server.sh for this purpose:
In the section, “Putting it all together,” I’ll provide several additional scripts to automate the clone, patch, and build the process further, including how to use environment variables to configure Elasticsearch.
Running Conductor against PostgreSQL is described in the README file, as well. Running
docker-compose -f docker-compose.yaml -f docker-compose-postgres.yaml upwill start an additional container instance for PostgreSQL and configure Conductor to use it.
But in our case, we wanted to run against an existing installation of PostgreSQL. Besides that, we did not want the Conductor database objects to be created in the
public schema, as is the default.
To achieve this, we need to change the
config-postgres.properties file. This lives in the
conductor/docker/server/config folder. But since we’re getting the Conductor repo on each build, we cannot change it there.
So, I created a
docker folder in my project root that will contain all customizations to the files in the
conductor/docker folder. On build, I simply copy the contents of this folder.
02.build-conductor-server.sh script now becomes:
config-postgres.properties file is simply a copy of the original, with the section on PostgreSQL modified, as follows:
currentSchema to the datasource URL, we make sure the Conductor database objects are created in the specified schema. The
applicationName property helps us with debugging database sessions if needed.
Except that it doesn’t work. When we run Conductor using just these changes, the
currentSchema property is ignored and everything is created in the
public schema. Not cool. I filed a bug against the PR that caused it.
I’ll spare you the several hours of searching I did to find the cause and just present the “solution.” Including air quotes, because it feels more like a hack. But it works.
In the file
postgres-persistence/src/main/java/com/netflix/conductor/postgres/config/PostgresConfiguration.javaFlyway (which handles database migrations) is configured with the line
.schemas("public"), which causes any attempt to change the default schema to be ignored. Removing this line solves the issue.
v3.5.1-fix-currentSchema-override.patch for this and included it in the
If you’re wondering about the
## placeholders there, these will be replaced by the values from the matching environment variables. I’ll get to that in the “Putting it all together” section. In the meantime, just replace these with your values.
As a small but useful bonus, we enabled completed and terminated workflows to be archived after 60 seconds (the default), by configuring the workflow status listener.
conductor.workflow-status-listener.type=archive to the
This will keep your database from overflowing with workflows you’ll never want to look at again, anyway. And even if you do, the data will still be stored in Elasticsearch, and accessible on demand.
In order to make the above work locally as well as in our cloud environment, we want to use environment variables to configure Elasticsearch and PostgreSQL. In order to do this, we turn to the
startup.sh file in the
conductor/docker/server/bin folder. We copy this file to our local
docker folder so we can make some changes.
We add the following lines right after the
## placeholders you saw earlier? We use the tool
sed to replace them with the values configured in the matching environment variables.
To run locally, we add the environment variables and their values to the
docker-compose-postgres.yaml files, which we copied to our local
docker folder. As an example, see the
We use the magic
host.docker.internal value to point to our local PostgreSQL instance, since
localhost won’t work from within a Docker container.
02.build-conductor-server.sh script copies
Finally, there’s only a few more scripts left.
03.build-conductor-ui.sh takes care of building the Conductor UI image:
We can run Conductor using
04.run-local.shwhich really doesn’t do more than call
docker compose up:
Or, we can run
04.run-local-postgres.sh to run against our PostgreSQL database:
There you have it. Using the above scripts, you should be able to build and run a local instance of Netflix Conductor in a few minutes. And hooking up the scripts to your favorite CI/CD tool should be perfectly doable, as well.
Running Netflix Conductor with Elasticsearch 7 and PostgreSQL takes a bit of work, but by using the power of
git diff and bash scripting, we can make the process easy, repeatable, and useable in both a local and a cloud context.
All the above source code is available on GitHub.
I hope this article helped you get started with Netflix Conductor using Elasticsearch 7 and PostgreSQL.