Running Netflix Conductor 3 in Docker Using Elasticsearch 7 and PostgreSQL | by Erik Burger | Mar, 2022

All wrapped in an easy, repeatable, and maintainable process

The standard implementation of Netflix Conductor is configured to use Elasticsearch 6. This article covers the changes that need to be made to use Elasticsearch 7 and PostgreSQL, as well as providing several scripts to automate building and running Conductor, both locally and in a cloud architecture .

At the time of writing, the latest released version of Netflix Conductor is 3.5.3. The information in this article is based on version 3.5.1 but should be applicable (and easily modified) to the latest version. If you run into any issues, please let me know, and I’ll do my best to help. Another great source of help is the Netflix Conductor discussion board.

I am running all the supplied code on a Windows 10 machine, from Git Bash, which comes included with Git for Windows. Our cloud architecture runs on CentOS.

Netflix Conductor is a workflow orchestration engine that was built by Netflix to “orchestrate microservices-based process flows” (source: Netflix Conductor documentation). The full feature list of Netflix Conductor is located here, but a few key features are:

  • Workflow and task definitions are implemented in JSON format. This allows for versioning and easy support for custom tools to create and manage workflows and tasks.
  • Tasks and Workers, ie, the building blocks that make up the workflows and the microservices that host them, respectively, are completely language agnostic, which allows the implementation to be done in the language best suited for the tasks. Libraries exist for Java, Python, .NET (shameless plug: this one is developed by me), and others.
  • The database architecture is pluggable, which means we can choose what database we want to use for Conductor. Conductor comes with numerous ready-made plugins, supporting (amongst others) Dynomite (also by Netflix), MySQL, and PostgreSQL.
  • Netflix Conductor comes with an (optional) UI, which allows insight and control over workflows, like (re)starting, pausing, stopping.
  • The entire project is open source and is being actively developed and supported.

In short, Netflix Conductor allows you to automate and manage processes that consist of several tasks (which might be processes in themselves), that may need to be executed in sequence, in a reliable, consistent, and scalable manner.

At my current project, we use Netflix Conductor and PostgreSQL to run scheduled ETL processes, create rapports, download and upload files, and for monitoring and alerting. We have also investigated using a conductor to automate the provisioning of user accounts and other infrastructure.

As mentioned in the introduction, the standard implementation of Netflix Conductor is configured to use Elasticsearch 6. Support for Elasticsearch 7 is built-in, but several changes need to be made to enable it. There is documentation available on GitHub to make these changes, but it does not seem entirely complete. I needed the changes I describe below to build and run against Elasticsearch 7.

At the time of writing, Elasticsearch 7.17.1 is the latest version of Elasticsearch 7.

Here’s a quick summary of the required changes (all file references are relative to the root folder):

Change ext['elasticsearch.version'] = revElasticSearch6
to ext['elasticsearch.version'] = revElasticSearch7

Remove the line include 'es6-persistence'

Change implementation project(':conductor-es6-persistence')
to implementation project(':conductor-es7-persistence')

  • server/src/main/resources/application.properties

Change conductor.elasticsearch.version=6
to conductor.elasticsearch.version=7

  • docker/docker-compose.yaml

Change image: elasticsearch:6.8.15
to image: docker.elastic.co/elasticsearch/elasticsearch:7.17.1

  • test-harness/build.gradle

Change testImplementation project(‘:conductor-es6-persistence’)
to testImplementation project(‘:conductor-es7-persistence’)

  • /test-harness/src/test/java/com/netflix/conductor/test/integration/AbstractEndToEndTest.java

Change conductor.elasticsearch.version=6
to conductor.elasticsearch.version=7

Change DockerImageName.parse("docker.elastic.co/elasticsearch/elasticsearch-oss").withTag("6.8.12"));
to DockerImageName.parse("docker.elastic.co/elasticsearch/elasticsearch").withTag("7.17.1"));

The standard way to apply these changes is to create a fork of the Conductor repository and do your work from there.

However, I find copying over an entire repository to change a handful of files to be a waste of resources, so I am following a different approach.

On build, I clone the Conductor repository, then apply several patch files to make the changes I need. While these steps increase the build time slightly, I feel this is a good compromise between speed and keeping my own repository clean and maintainable.

Here’s how that works:

  1. Clone the Conductor repo inside your project directory.
  2. Make the above changes.
  3. In the Conductor folder, run git diff > v3.5.1-update-to-es7.patch to create a patch file. The v3.5.1 is for convenience, to indicate this patch only works against that version.
  4. Copy the patch file to the root directory, so you can safely remove the conductor repo and apply the changes whenever you clone the repo again.

You also need to remove any dependencies.lock files before building. This is as simple as running rm -f **/dependencies.lock.

To automate removing the lock files and applying the patches, I created the 01.get-conductor.sh script:

You can see I am actually applying a few more patches. These are not required to run against Elasticsearch 7.

At this point, you can build Conductor. I created 02.build-conductor-server.sh for this purpose:

In the section, “Putting it all together,” I’ll provide several additional scripts to automate the clone, patch, and build the process further, including how to use environment variables to configure Elasticsearch.

Running Conductor against PostgreSQL is described in the README file, as well. Running docker-compose -f docker-compose.yaml -f docker-compose-postgres.yaml upwill start an additional container instance for PostgreSQL and configure Conductor to use it.

But in our case, we wanted to run against an existing installation of PostgreSQL. Besides that, we did not want the Conductor database objects to be created in the public schema, as is the default.

To achieve this, we need to change the config-postgres.properties file. This lives in the conductor/docker/server/config folder. But since we’re getting the Conductor repo on each build, we cannot change it there.

So, I created a docker folder in my project root that will contain all customizations to the files in the conductor/docker folder. On build, I simply copy the contents of this folder.

The 02.build-conductor-server.sh script now becomes:

Our local config-postgres.properties file is simply a copy of the original, with the section on PostgreSQL modified, as follows:

By adding currentSchema to the datasource URL, we make sure the Conductor database objects are created in the specified schema. The applicationName property helps us with debugging database sessions if needed.

Except that it doesn’t work. When we run Conductor using just these changes, the currentSchema property is ignored and everything is created in the public schema. Not cool. I filed a bug against the PR that caused it.

I’ll spare you the several hours of searching I did to find the cause and just present the “solution.” Including air quotes, because it feels more like a hack. But it works.

In the file postgres-persistence/src/main/java/com/netflix/conductor/postgres/config/PostgresConfiguration.javaFlyway (which handles database migrations) is configured with the line .schemas("public"), which causes any attempt to change the default schema to be ignored. Removing this line solves the issue.

I created v3.5.1-fix-currentSchema-override.patch for this and included it in the 01.get-conductor.sh script.

If you’re wondering about the ## placeholders there, these will be replaced by the values ​​from the matching environment variables. I’ll get to that in the “Putting it all together” section. In the meantime, just replace these with your values.

As a small but useful bonus, we enabled completed and terminated workflows to be archived after 60 seconds (the default), by configuring the workflow status listener.

Just add conductor.workflow-status-listener.type=archive to the config-postgres.properties file.

This will keep your database from overflowing with workflows you’ll never want to look at again, anyway. And even if you do, the data will still be stored in Elasticsearch, and accessible on demand.

In order to make the above work locally as well as in our cloud environment, we want to use environment variables to configure Elasticsearch and PostgreSQL. In order to do this, we turn to the startup.sh file in the conductor/docker/server/bin folder. We copy this file to our local docker folder so we can make some changes.

We add the following lines right after the if..fi block:

Remember those ## placeholders you saw earlier? We use the tool sed to replace them with the values ​​configured in the matching environment variables.

To run locally, we add the environment variables and their values ​​to the docker-compose.yaml and docker-compose-postgres.yaml files, which we copied to our local docker folder. As an example, see the docker-compose-postgres.yaml file:

We use the magic host.docker.internal value to point to our local PostgreSQL instance, since localhost won’t work from within a Docker container.

The 02.build-conductor-server.sh script copies startup.shas well.

Finally, there’s only a few more scripts left.

03.build-conductor-ui.sh takes care of building the Conductor UI image:

We can run Conductor using 04.run-local.shwhich really doesn’t do more than call docker compose up:

Or, we can run 04.run-local-postgres.sh to run against our PostgreSQL database:

There you have it. Using the above scripts, you should be able to build and run a local instance of Netflix Conductor in a few minutes. And hooking up the scripts to your favorite CI/CD tool should be perfectly doable, as well.

Running Netflix Conductor with Elasticsearch 7 and PostgreSQL takes a bit of work, but by using the power of git diff and bash scripting, we can make the process easy, repeatable, and useable in both a local and a cloud context.

All the above source code is available on GitHub.

I hope this article helped you get started with Netflix Conductor using Elasticsearch 7 and PostgreSQL.

Happy coding!

Leave a Comment