postgresql

Today’s post is a short one. I’ve been working on a project that required supporting multiple data backends. To simplify querying and testing across systems, I created a uniform dataset available in several formats. To that effect I have created an open-source Docker Compose setup with S3 storage, and DuckDB, Oracle and PostgreSQL databases. Each of them contains the same Simpsons dataset from Kaggle. Dataset This dataset contains the characters, locations, episode details, and script lines for approximately 600 Simpsons…

The world of Data Engineering – and LinkedIn as a consequence – is full with the latest and greatest tools for big, bigger, biggest data. But while everybody is chasing the latest functions of Databricks, Snowflake and DuckDB, there is a plethora of companies that are still running good old SQL databases in production. Often times, this is a proper choice and it works for many. As a consequence, data engineering is in practise often done in SQL databases like…