I love the ergonomics of dbt for building reproducible data pipelines with DuckDB -- particularly the ability to define output tables as individual `SELECT` queries and have them refer to each other -- but have always found the configuration and boilerplate excessive for the projects I work on. With the recent Rust rewrite of dbt Core, now seemed like a good time to consider the bits I actually need/use.
Recently, I realised that DuckDB itself can parse SQL queries into an AST representation using the `json_serialize_sql` function. duckdb.mk (~250 lines `include`d in your Makefile) takes advantage of this to generate Make rules for your models automatically, and works with both DuckDB 1.4.1+ and 2.0 preview.
Write a neat directory of SQL statements querying local and remote data, get a neat directory of Parquet files (plus a Mermaid diagram of the DAG) back. When you rebuild a table, duckdb.mk keeps track of dependencies (including remote resources queried over HTTP) and only rebuilds those that need it.
This system is a distillation of various ad hoc solutions cooked up while working on data journalism and research projects over the past 5-10 years, most recently doing Observable-based research notebooks with Data Desk: https://research.datadesk.eco/ It's very much a starting point for further development, and contributions in line with the general minimalist philosophy are very welcome!
Check out the `example/` directory for a simple test project that attributes satellite-observed methane plumes to oil and gas infrastructure using public data.
I love the ergonomics of dbt for building reproducible data pipelines with DuckDB -- particularly the ability to define output tables as individual `SELECT` queries and have them refer to each other -- but have always found the configuration and boilerplate excessive for the projects I work on. With the recent Rust rewrite of dbt Core, now seemed like a good time to consider the bits I actually need/use.
Recently, I realised that DuckDB itself can parse SQL queries into an AST representation using the `json_serialize_sql` function. duckdb.mk (~250 lines `include`d in your Makefile) takes advantage of this to generate Make rules for your models automatically, and works with both DuckDB 1.4.1+ and 2.0 preview.
Write a neat directory of SQL statements querying local and remote data, get a neat directory of Parquet files (plus a Mermaid diagram of the DAG) back. When you rebuild a table, duckdb.mk keeps track of dependencies (including remote resources queried over HTTP) and only rebuilds those that need it.
This system is a distillation of various ad hoc solutions cooked up while working on data journalism and research projects over the past 5-10 years, most recently doing Observable-based research notebooks with Data Desk: https://research.datadesk.eco/ It's very much a starting point for further development, and contributions in line with the general minimalist philosophy are very welcome!
Check out the `example/` directory for a simple test project that attributes satellite-observed methane plumes to oil and gas infrastructure using public data.