DuckDB: In-Process SQL OLAP Database for Files and Data Lakes

My notes

Summary

DuckDB positions itself as an in-process, columnar SQL database (“OLAP”) that queries Parquet, JSON, S3, and other data lake sources directly, without a separate server. It now also ships optional client-server support (“quack”), and the whole project (core plus extensions plus the DuckLake format) is MIT licensed.

Key Insight

• In-process, not client-server by default - DuckDB runs embedded inside the calling process (like SQLite), so there is no daemon to manage for local analytics work; a client-server mode is now available as an add-on for shared or remote use cases.
• Columnar engine built for analytics, not transactions - optimized for scanning and aggregating large columns rather than row-by-row writes, and it spills to disk so workloads bigger than available RAM still complete instead of running out of memory.
• Queries files and cloud storage without an import step - can run SQL directly against Parquet, JSON, and S3-hosted data lakes, skipping the usual “load into a database first” workflow.
• Extension-driven architecture - many core features are themselves implemented as extensions, which is why the feature set (S3, JSON, spatial, and more) can grow without bloating the base install.
• Fully permissive licensing - DuckDB core, its extensions, and the DuckLake format are all MIT-licensed, which matters for anyone evaluating it for commercial or embedded redistribution.