Putting my Ducks in a Row: Using Kestra and DuckDB's Quack Protocol Together
A practical look at combining Kestra orchestration with DuckDB's Quack protocol
I've written about two tools recently that I've been genuinely excited about. The first was Kestra, which I reach for when a job stops being a job and turns into a workflow with branching, looping, and subflows. The second was DuckDB's Quack protocol, which finally solved the "only one process can write to my DuckDB at a time" problem I'd been dancing around for years by turning one DuckDB instance into a little server that everyone else can connect to.
Writing about them separately was fun, but the whole reason I got interested in both was the picture of them working together. An orchestrator fanning work out across a bunch of parallel tasks, and a database that can actually take concurrent writes from all of them without me building a queue or partitioning files by hand. So I built a small demo to see if that picture held up, and it did, but not before teaching me a few things about where the seams are. The full code lives in ketra-quack-dev on GitHub; this post is a walk through of the architecture and the decisions I made along the way.
Finding a concurrency problem
The demo fetches live currency exchange rates and writes them into a DuckDB table, one row per run. It uses the Frankfurter API, which is a free, no-key exchange rate service, and it converts USD into five currencies: EUR, JPY, GBP, CNY, and CHF. The rates table is exactly what you'd expect, a timestamp column plus one column per currency.
While that data may be interesting for some people, I find the work itself the fun part. Rather than fetch all five rates in one script, I split it into a parent flow (fetch_exchange_rates) that fans out to five parallel child flows (convert), one per currency. Each child fetches its own rate and writes its own column. It's a deliberately small problem blown up into a concurrent one, because concurrency is the whole point of using quack here. If I only ever wrote one row from one process, I wouldn't need any of this.
The pieces and how they're wired
There are three long-running services in the stack, all joined on a named Docker network called quack-network:
A DuckDB Quack server, which is a persistent exchange.duckdb database with quack_serve() running against it. Inside the container, quack binds to port 9494, and Caddy reverse-proxies external port 8080 to it. (More on why Caddy is there in a moment.)
Kestra, the orchestrator, running the two flows.
PostgreSQL, which is just Kestra's own metadata backend and not part of the interesting path.
As much as Docker networking frustrates me, unfortunately it matters here. Kestra's Docker task runner spins up a fresh container for each quack-facing task, and it attaches each of those containers to quack-network by name so they can reach the duckdb service directly at duckdb:8080. Without that, the tasks and the database would be shouting into different rooms.
Early on I had it matching only http://localhost:8080 as a site block, which worked fine when I tested from my laptop and then broke the moment a container hit it using the duckdb service name instead, since the Host header no longer matched. The symptom was a cryptic Serialization Error: not enough data in buffer to fulfill read request, which sure doesn't say "your reverse proxy dropped the request." Switching Caddy to match :8080 on any host fixed it. It's the kind of bug where the error message doesn’t mean anything until you figure out the problem (like object of type 'closure' is not subsettable in R).
The Kestra flows
I started off with the parent flow fetch_exchange_rates which sets the time for the task so each returning row has the same time and therefore the same primary key. I also could’ve used an input for that if I wanted this to run for any arbitrary time with a default of now(). Then I’m pre-inserting the new row (more on why in a moment), before fanning out using a ForEach task to run my convert Subflow with no concurrency limits. The parent flow also has an errors section that lets me clean up any rows that were created then had something go wrong before the flow finished.
Why the parent pre-creates the row
My first instinct was to let each of the five children insert-or-upsert its own value, something like INSERT ... ON CONFLICT (ts) DO UPDATE, and let the database merge the five writes into one row. It's clean, it's stateless, and it does not work reliably.
DuckDB uses optimistic concurrency control, which means concurrent transactions that touch the same row conflict by design. With five children all racing to create the very first row for a brand-new timestamp, two of them can both look, both see "no row here yet," and both try the initial insert. One wins and the other gets a duplicate-key error, because ON CONFLICT resolves a conflict against a row that already exists, not against another insert happening at the same instant. It doesn't protect you from that particular race.
The fix is almost boring once you see it. The parent creates the row once, synchronously, before it fans out, a single plain INSERT with the timestamp, usd = 1.0, and every currency column left NULL. By the time the five children start, the row already exists, so each child only ever updates one column of an existing row. There's no insert race left to lose that would happen even if we used an UPSERT. If any child fails, the parent's error handler deletes that timestamp's row so a half-written run never sticks around.
I'll flag this because it bit me and it'll bite you: the right answer here is not to slap a retry on the child. The duplicate-key error looks transient, but it's telling you something real about your write pattern. Serializing that first insert is the actual fix.
Why ATTACH for the insert but quack_query() for the updates
The second decision came from quack itself, and it's the one I'd most want to save my future self from rediscovering. Quack gives you two ways to talk to a remote database, and the difference between them is really about where the query runs.
When you ATTACH 'quack:duckdb:8080', the remote database shows up as a catalog you query as if it were local. Your local DuckDB does the planning and coordination, treating the remote as a combination catalog and storage backend, pushing down filters where it can but otherwise running the show from the client side. quack_query() is the other model entirely: you hand it a SQL string, and per the quack docs that string "executes remotely and the server streams the result back." The whole statement runs on the server, against the server's own tables.
The two methods for connecting are why writing behaves differently. A plain INSERT through ATTACH works fine, because inserting into a remote table is something the attach layer supports cleanly. But an UPDATE through ATTACH fails with Binder Error: Can only update base table, because from the local planner's point of view the attached table isn't a local base table it's allowed to update in place, and an upsert-style INSERT ... ON CONFLICT fails with Not implemented Error: GetStorageInfo not implemented yet. These aren't bugs so much as the current edges of a beta protocol's remote catalog.
So the demo splits the difference along that line. The parent's plain INSERT goes through ATTACH, which is exactly the case ATTACH is good at. The children's UPDATEs go through quack_query(), which sidesteps the whole problem by shipping the update statement to the server and letting the server run it as an ordinary local UPDATE against its own base table. Once I understood that ATTACH plans locally while quack_query() executes remotely, the split stopped feeling like a workaround and started feeling like using each tool for what it's actually for.
Two smaller choices worth mentioning
A couple of the other decisions are less about limitations and more about keeping things simple.
The first is that each child does its HTTP fetch inside DuckDB itself, in the same invocation that writes the result. DuckDB can read straight from a URL, so the child points read_json_auto() at the Frankfurter endpoint, pulls the rate into a session variable, and then writes it, all in one duckdb call. I didn't add a separate Kestra task (or a separate shell curl step) to go fetch the number and pass it down the line. The database is perfectly capable of making the request on its own, so letting it do the fetching keeps each child down to a single task instead of a fetch-then-write handoff. It also keeps the quack_query() statement short and legible, since the alternative, embedding the whole HTTP fetch inside the SQL string I send over quack, means writing SQL that contains SQL, and the nested-quote escaping gets ugly fast.
The second is that every task that talks to quack runs the official duckdb/duckdb Docker image directly rather than Kestra's plugin-jdbc-duckdb. I wanted to use the JDBC plugin, I really did, since it would have been the tidy Kestra-native choice. But the JDBC driver couldn't reliably prepare statements against the quack wire protocol, and after enough failed attempts I stopped fighting it and just called the real duckdb CLI. (The image has no shell, so the tasks invoke /duckdb -c directly, which is a small quirk but an easy one.)
Wrapping up
None of this is production-grade, and it isn't trying to be. Quack is still beta, and I'd expect some of these sharp edges, especially the ATTACH update limitation, to soften as it moves toward a stable release. But the core service I wanted to see absolutely works: Kestra fans five parallel tasks out at a real DuckDB server, all five write concurrently, and they land in one clean row. No partitioned files, no queue, no lock dance.
If you want to actually run it, tear it apart, or point out something I did the hard way, the whole codebase is in kestra-quack-dev on GitHub, README and troubleshooting notes included. And if you haven't met either tool yet, the Kestra and Quack posts are the gentler introductions I'd start with before coming back here.
Gus Lipkin
Data Scientist
Lander Analytics
Subscribe to our Substack and below to our monthly emails for practical AI strategies for your organization: what to build, what to avoid, and how to make systems reliable in the real world.
Work with us: If you want help identifying the right first workflow, building a permissioned knowledge base, or training your team to ship responsibly, reach out at info@landeranalytics.com.
About the author: Gus Lipkin is a Data Scientist at Lander Analytics, where he writes software for data science practitioners and consumers.
