Orchestrating the Messy Stuff: Why We Use Kestra

When scheduled jobs become full workflows

A little while back I wrote about rpeat and why I think it's a better way to schedule jobs, and I really do mean it. I still have my own jobs running at home, and we have client projects here at Lander Analytics that rely on it too. When we have a job that needs to run on a schedule and give us easy access to the logs in the middle of the night, rpeat is the tool we reach for. It's a single binary, it's configured with plain readable JSON, and it runs shell, R, Python, Julia, or SQL without a rewrite, all with the kind of scheduled precision that would make even the most esteemed horologists jealous.

But not all of our work is discrete or cron-shaped. A fair amount of what we build for clients is less "run this script at 2am" and more "when a file lands, pull it apart, run three different transforms depending on what's inside, fan out over every account in it, and only then write the result somewhere." Once the work grows branches and loops and steps that each want their own environment, we reach for something built for that shape. For us, that something is Kestra.

What Kestra does that a job scheduler can't

Kestra is an open-source orchestration platform where you describe an entire workflow, a "flow", in declarative YAML. A flow has an id, a namespace, some inputs, and a list of tasks. Tasks hand data to each other by referencing outputs like {{ outputs.extract.uri }}, so the ordering is explicit rather than implied by whatever order things happen to fire in.

One of the most important parts of any codebase is transparency. It’s important to be able to look at the code and easily understand what’s going on. That’s where we find Kestra excels. Because a flow is defined declaratively, everything it does lives at the orchestration level, right there in the YAML for anyone to read. You could do a lot of what Kestra does with rpeat, but the logic would have to live inside the scripts. From rpeat's point of view, it ran a script and the script did something. When the pipeline takes a wrong turn, you’re usually back inside the scripts trying to reconstruct what happened and where.

Kestra flips that around. The shape of the pipeline is visible before you run it and legible after it breaks. When a run goes sideways, you look at the flow and see which step failed, instead of sprinkling print statements through a script and running it again. On top of that visibility you get the things you want the moment something runs unattended, retries, timeouts, and error handling that live in the flow itself rather than getting bolted on later.

The shapes a flow can take

The other thing I didn't appreciate until I'd built a few flows is how many shapes one can take. The simplest is the one everybody pictures, a straight line. Task A, then task B, then task C, each step waiting on the one before it. That sequential flow covers a surprising amount of real work, and if it were the whole story I'd probably still reach for rpeat.

But a flow doesn't have to be a straight line. It can branch, taking one path or another based on something it figured out a step earlier, so a file flagged as a full refresh goes down a different set of tasks than one that's just a daily delta. It can loop, running the same steps once for every item in a list. That's how a single ForEach handles a client file with two hundred accounts in it, instead of two hundred copy-pasted tasks. It can be event-triggered rather than scheduled, sitting idle until a file lands in S3 or a message hits a Kafka or Pub/Sub queue. That last one matters, because so much of our work runs when some other thing happened, not at a fixed time. And it can be composed from subflows, calling another flow the way a function calls a function, so our "clean and store one account's data" logic lives in its own flow that the bigger pipeline calls once per account.

What makes this useful, rather than just a list of features, is that none of those are separate products with separate mental models. They're all just flows. A single pipeline can be scheduled and loop over accounts and branch on each one and hand a chunk of the work to a subflow, all in the same YAML file. That combination is exactly the messy kind of work we build for clients at Lander Analytics.

Let the plugins handle the plumbing

A flow shape describes how the work is arranged, but every task still has to do the work. You might have a flow that pulls a file from S3, does some basic transformations, dumps it into Postgres, then posts to a Slack channel when it's done. This is where the plugins come in, and Kestra has enough of them to cover most of the plumbing we need. Most of the connecting-to-databases-and-clouds work you'd otherwise write by hand is already a task type you can drop into a flow.

That matters more than it sounds like it should. A big chunk of any real pipeline is the plumbing, the code that authenticates to a service, handles pagination, retries a flaky request, and parses whatever comes back. When that plumbing is a maintained plugin instead of a helper function buried in one of our scripts, we get to spend our time on the part that's actually specific to the client. A flow that reads from a database, runs a transform, and writes the result somewhere else is three plugins and a bit of glue, not three custom integrations we have to keep alive.

Running our own code is just another task type in the same list. Kestra has script tasks for R, Python, shell, and more, so the transform in the middle of that flow is an R or Python script that reads the previous task's output and hands its own output to the next one. The plugins handle the edges, our scripts handle the logic, and the flow wires them together.

Docker is the reason we can build these the way we do

One of the things that first drew us to Kestra is that you can choose from a variety of task runners from local on-host, to an off-host AWS batch job, a Kubernetes cluster, or, our favorite, a local Docker runner.

When you run a script natively in Kestra the environment isn't saved between runs. Every time the task fires, its dependencies have to be installed at runtime, from scratch. No one wants to sit around and wait for pip install or renv::restore() (even if you're installing binaries), and even those rely on the package server being available. Without providing a complete environment, you’re a sitting duck for a breaking change.

Using pre-built Docker containers allows you build an image once with all your dependencies baked in, and that environment persists across every run that uses it. Because Kestra saves the container images, you just need to make sure the server is available on the first run. You’ll be able to use that image forever. If you use images from a private registry, you can easily add your puller credentials and you’ll have everything available.

When we reach for Kestra

Not every scheduled job needs an orchestration platform. If a script just needs to run every night, write some logs and tell us when it fails, we still reach for rpeat. Kestra starts earning the extra machinery when the workflow branches, loops over hundreds of items, needs different environments or reuses the same logic across multiple pipelines. That's why we use it for the messy stuff: our R and Python code still handles the client-specific work, while Kestra keeps the order, environments, retries, inputs and outputs visible enough to understand what happens when something breaks.

Gus Lipkin

Data Scientist

Lander Analytics

---

Subscribe to our Substack and below to our monthly emails for practical AI strategies for your organization: what to build, what to avoid, and how to make systems reliable in the real world.

Work with us: If you want help identifying the right first workflow, building a permissioned knowledge base, or training your team to ship responsibly, reach out at info@landeranalytics.com.

About the author: Gus Lipkin is a Data Scientist at Lander Analytics, where he writes software for data science practitioners and consumers.