A step-by-step guide to running your first workflow in Netflix Conductor.
Two of the key concepts of Netflix Conductor are Workers and Tasks. Workers are the microservices that contain the implementation of Tasks, which are executed by Conductor as part of a workflow. Tasks are the building blocks of workflows, ie the steps executed by the process that is represented by a workflow.
Libraries to implement Workers and Tasks are available for various programming languages, most notably Go, Java and Python. In this article, I will introduce a library for .NET called SuperSimpleConductor, which I developed and maintain.
In this article, I will use the SuperSimpleConductor library to show how easy it is to get started using Netflix Conductor using .NET.
Netflix Conductor is a workflow orchestration engine that was built by Netflix to “orchestrate microservices-based process flows” (source: Netflix Conductor documentation). The full feature list of Netflix Conductor is located here, but a few key features are:
- Workflow and task definitions are implemented in JSON format. This allows for versioning and easy support for custom tools to create and manage workflows and tasks.
- Tasks and Workers, ie, the building blocks that make up the workflows and the microservices that host them, respectively, are completely language agnostic, which allows the implementation to be done in the language best suited for the tasks.
- The database architecture is pluggable, which means we can choose what database we want to use for Conductor. Conductor comes with numerous ready-made plugins, supporting (amongst others) Dynomite (also by Netflix), MySQL, and PostgreSQL.
- Netflix Conductor comes with an (optional) UI, which allows insight and control over workflows, like (re)starting, pausing, stopping.
- The entire project is open source and is being actively developed and supported.
In short, Netflix Conductor allows you to automate and manage processes that consist of several tasks (which might be processes in themselves), that may need to be executed in sequence, in a reliable, consistent, and scalable manner.
At my current project, we use Netflix Conductor and PostgreSQL to run scheduled ETL processes, create rapports, download and upload files, and for monitoring and alerting. We have also investigated using a conductor to automate the provisioning of user accounts and other infrastructure.
In order to code along with this article, you’ll need a running instance of Conductor. The easiest way to do this is to run Conductor locally using Docker. Using the article I wrote below, you should be up and running in a couple of minutes.
If you have followed the steps in the article, you should be able to access the Conductor UI at http://localhost:5000/, and the Conductor Server API at http://localhost:8080/swagger-ui/index.html .
Now that we’ve got Conductor up and running, let’s get started using it!
Our first step is to register a Task with Conductor. Our task definition looks like this:
Here we’re creating the
I’ve kept the definition as minimal as possible. For details on what else you can do, you can refer to the documentation here.
To register the task, we use the
/api/metadata/taskdefs endpoint of the Conductor Server API.
We’ll use the Swagger interface to call the endpoint. Note that the endpoint expects a JSON array, so we wrap our definition in
After clicking Executewe can see the Task appearing in the Conductor UI:
We then create the Workflow in Conductor, using the
Again, I’ve kept the definition as minimal as possible. For details on the other properties you can use, you can refer to the documentation.
And check that the
SimpleSampleWorkflow shows up in the Conductor UI:
Now that we have our Workflow registered, we can run an instance of it using the
That’s great! The workflow is now running, waiting for a Worker to pick up the first Task, ie the
Next, we’ll implement the Worker process.
Using SuperSimpleConductor, creating a Worker is, well, super simple (see what I did there?).
First, we create a new background worker project.
dotnet new worker -f net5.0
Note that I am using .NET 5 here. SuperSimpleConductor supports .NET Standard 2.1 or higher, so whatever you pick here that satisfies that requirement is fine.
We won’t need the
Worker.cs file, so it’s safe to delete it.
We then add the SuperSimpleConductor packages from NuGet.
dotnet add package SuperSimpleConductor.ConductorClient
dotnet add package SuperSimpleConductor.ConductorWorker
And replace the code in
Program.cs class with the following:
Running the worker using
dotnet run gives us the following output:
Note that I changed the logging settings in
appSettings.Development.json to get better output (by default, it is set to only log
What we can see is the worker polling the Conductor queue and finding 2 tasks: the
SimpleSampleWorkerTask and the
_deciderQueue task. The latter is a system task, so the worker skips it.
It also notifies us that the
SimpleSampleWorkerTask is not registered with this worker. That’s because we haven’t done so yet.
So how do we do this?
As per the theme of this article, this is also super simple.
loads the specified assembly (in this case only the
conductor-worker assembly), and registers any class that derives from the
IWorkflowTask interface with the worker.
That means that all we need to do now is implement the
To implement the
SimpleSampleWorkerTaskcreate a new file
and paste in the following code:
Now, running the worker using
dotnet run shows us that the
SimpleSampleWorkerTask is picked up and executed:
And finally, the workflow in Conductor UI shows as
And that’s it! All that remains now is to replace the logging call in the
SimpleSampleWorkerTask class with an actual implementation.
And, of course, adding additional tasks to your workflow should be as simple as repeating the steps described above. Conductor can support very complex workflows using parallel tasks, decision trees, and all sorts of other goodness.
The only limit is your imagination.
Getting started with Netflix Conductor may seem daunting at first. But using the right libraries, getting started actually becomes super simple.
TestWorker class and all the above source code are available on GitHub as part of the SuperSimpleConductor project.