Bart Zwemmer

Today’s post is a short one. I’ve been working on a project that required supporting multiple data backends. To simplify querying and testing across systems, I created a uniform dataset available in several formats. To that effect I have created an open-source Docker Compose setup with S3 storage, and DuckDB, Oracle and PostgreSQL databases. Each of them contains the same Simpsons dataset from Kaggle. Dataset This dataset contains the characters, locations, episode details, and script lines for approximately 600 Simpsons…

Working on different projects in the same time, I aim to keep my development environment minimal, predictable, and free of hidden global dependencies. This post explains why that philosophy led me to replace pre-commit with prek — and why a single, fast, dependency-free binary fits better than a Python-based tool. Intro For about a year now, I’ve been using uv for Python project management. One of its most underrated benefits is that I no longer need a global Python installation.…

The world of Data Engineering – and LinkedIn as a consequence – is full with the latest and greatest tools for big, bigger, biggest data. But while everybody is chasing the latest functions of Databricks, Snowflake and DuckDB, there is a plethora of companies that are still running good old SQL databases in production. Often times, this is a proper choice and it works for many. As a consequence, data engineering is in practise often done in SQL databases like…

This is a demo of serving spatial data as vector tiles directly from DuckDB using the spatial extension and showing it on a map in a Flask app.

Snippets and tips to create a PySpark UDF

Learn how to run a throw-away pod in your Kubernetes cluster for easy debugging.