Getting Started with Netflix Conductor in .NET using SuperSimpleConductor | by Erik Burger | Mar, 2022

Netflix Conductor logo
Netflix Conductor logo from the Netflix Conductor GitHub repository.

Two of the key concepts of Netflix Conductor are Workers and Tasks. Workers are the microservices that contain the implementation of Tasks, which are executed by Conductor as part of a workflow. Tasks are the building blocks of workflows, ie the steps executed by the process that is represented by a workflow.

Libraries to implement Workers and Tasks are available for various programming languages, most notably Go, Java and Python. In this article, I will introduce a library for .NET called SuperSimpleConductor, which I developed and maintain.

In this article, I will use the SuperSimpleConductor library to show how easy it is to get started using Netflix Conductor using .NET.

Netflix Conductor is a workflow orchestration engine that was built by Netflix to “orchestrate microservices-based process flows” (source: Netflix Conductor documentation). The full feature list of Netflix Conductor is located here, but a few key features are:

  • Workflow and task definitions are implemented in JSON format. This allows for versioning and easy support for custom tools to create and manage workflows and tasks.
  • Tasks and Workers, ie, the building blocks that make up the workflows and the microservices that host them, respectively, are completely language agnostic, which allows the implementation to be done in the language best suited for the tasks.
  • The database architecture is pluggable, which means we can choose what database we want to use for Conductor. Conductor comes with numerous ready-made plugins, supporting (amongst others) Dynomite (also by Netflix), MySQL, and PostgreSQL.
  • Netflix Conductor comes with an (optional) UI, which allows insight and control over workflows, like (re)starting, pausing, stopping.
  • The entire project is open source and is being actively developed and supported.

In short, Netflix Conductor allows you to automate and manage processes that consist of several tasks (which might be processes in themselves), that may need to be executed in sequence, in a reliable, consistent, and scalable manner.

At my current project, we use Netflix Conductor and PostgreSQL to run scheduled ETL processes, create rapports, download and upload files, and for monitoring and alerting. We have also investigated using a conductor to automate the provisioning of user accounts and other infrastructure.

In order to code along with this article, you’ll need a running instance of Conductor. The easiest way to do this is to run Conductor locally using Docker. Using the article I wrote below, you should be up and running in a couple of minutes.

If you have followed the steps in the article, you should be able to access the Conductor UI at http://localhost:5000/, and the Conductor Server API at http://localhost:8080/swagger-ui/index.html .

The Conductor UI and Server API running side-by-side.

Now that we’ve got Conductor up and running, let’s get started using it!

Our first step is to register a Task with Conductor. Our task definition looks like this:

Here we’re creating the SimpleSampleWorkerTask Task.

I’ve kept the definition as minimal as possible. For details on what else you can do, you can refer to the documentation here.

To register the task, we use the /api/metadata/taskdefs endpoint of the Conductor Server API.

We’ll use the Swagger interface to call the endpoint. Note that the endpoint expects a JSON array, so we wrap our definition in [ ].

Registering a task using the Conductor Server API.

After clicking Executewe can see the Task appearing in the Conductor UI:

Task definition in Conductor UI.

We then create the Workflow in Conductor, using the api/metadata/workflow endpoint.

Again, I’ve kept the definition as minimal as possible. For details on the other properties you can use, you can refer to the documentation.

Registering a workflow using the Conductor Server API.

And check that the SimpleSampleWorkflow shows up in the Conductor UI:

Workflow definition in Conductor UI.

Now that we have our Workflow registered, we can run an instance of it using the api/workflow/{name} endpoint.

Workflow running in Conductor UI.

That’s great! The workflow is now running, waiting for a Worker to pick up the first Task, ie the SimpleSampleWorkerTask.

Next, we’ll implement the Worker process.

Using SuperSimpleConductor, creating a Worker is, well, super simple (see what I did there?).

First, we create a new background worker project.

mkdir conductor-worker
cd conductor-worker
dotnet new worker -f net5.0

Note that I am using .NET 5 here. SuperSimpleConductor supports .NET Standard 2.1 or higher, so whatever you pick here that satisfies that requirement is fine.

We won’t need the Worker.cs file, so it’s safe to delete it.

We then add the SuperSimpleConductor packages from NuGet.

dotnet add package SuperSimpleConductor.ConductorClient
dotnet add package SuperSimpleConductor.ConductorWorker

And replace the code in Program.cs class with the following:

Running the worker using dotnet run gives us the following output:

Running the worker.

Note that I changed the logging settings in appSettings.Development.json to get better output (by default, it is set to only log Information messages):

What we can see is the worker polling the Conductor queue and finding 2 tasks: the SimpleSampleWorkerTask and the _deciderQueue task. The latter is a system task, so the worker skips it.

It also notifies us that the SimpleSampleWorkerTask is not registered with this worker. That’s because we haven’t done so yet.

So how do we do this?

As per the theme of this article, this is also super simple.

In Program.csthe line

loads the specified assembly (in this case only the conductor-worker assembly), and registers any class that derives from the IWorkflowTask interface with the worker.

That means that all we need to do now is implement the SimpleSampleWorkerTask class.

To implement the SimpleSampleWorkerTaskcreate a new file

touch SimpleSampleWorkerTask.cs

and paste in the following code:

Now, running the worker using dotnet run shows us that the SimpleSampleWorkerTask is picked up and executed:

Running the worker with the task added.

And finally, the workflow in Conductor UI shows as Completed.

Workflow completed in Conductor UI.

And that’s it! All that remains now is to replace the logging call in the SimpleSampleWorkerTask class with an actual implementation.

And, of course, adding additional tasks to your workflow should be as simple as repeating the steps described above. Conductor can support very complex workflows using parallel tasks, decision trees, and all sorts of other goodness.

The only limit is your imagination.

Getting started with Netflix Conductor may seem daunting at first. But using the right libraries, getting started actually becomes super simple.

The TestWorker class and all the above source code are available on GitHub as part of the SuperSimpleConductor project.

Happy coding!

Leave a Comment