DuckLake vs Apache Iceberg on AWS (2026): SQL Catalog or S3 Tables

Choose DuckLake only when every reader and writer is DuckDB and the catalog runs on Aurora PostgreSQL or RDS for PostgreSQL, with Parquet on S3.

Teams that already operate an Iceberg lake should start from the S3 Tables reference architecture. That post covers compaction pricing. This one is the format choice.

DuckLake spec 1 shipped on 13 April 2026, with the reference ducklake extension in DuckDB 1.5.2. The specification is frozen with backward compatibility. The DuckLake release calendar still lists spec 1.1 as tentative for fall 2026. This guide is written against spec 1. DuckDB 1.5.0 and 1.5.1 speak spec 0.4. Do not attach a spec 1 catalog with those builds.

On AWS, keep Apache Iceberg on Amazon S3 Tables when Athena, Glue, Redshift, or Spark must read the table.

The trade-off is planning latency versus engine coverage. In a MotherDuck architecture walkthrough of the DuckDB design, each sequential object-storage GET is about 100 ms. Iceberg reads a metadata file, then a manifest list, then a manifest, so planning sits at about half a second before the query touches Parquet. DuckLake puts that metadata in the SQL catalog, so planning is one query. FactualMinds has not measured that half-second. It is DuckDB’s stated figure.

Artifacts: decision worksheet, attach SQL.

Published figure — MotherDuck’s DuckLake architecture deep dive, restating the DuckDB design: about 100 ms per object-storage GET, and about 0.5 s of sequential metadata reads before an Iceberg query starts. Not a FactualMinds benchmark. Not a client result.

What DuckLake is

DuckLake is an open lakehouse format from DuckDB Labs, MIT licensed. Data files are Parquet on object storage. All metadata lives in a SQL database that supports ACID transactions and primary keys. The v1.0 specification defines that catalog as 28 tables, including snapshots, file paths, column statistics, and the rows staged by data inlining.

The ducklake extension is the reference implementation. Official catalog databases for that extension are PostgreSQL, SQLite, and DuckDB. A single DuckDB file is enough on a laptop. The DuckLake FAQ names Amazon Aurora as a production catalog host, with the Parquet prefix on S3.

There is no separate catalog HTTP API. A commit is a SQL transaction: file path, stats, snapshot id, and the change log land together. Cross-table changes share that transaction. The May 2025 manifesto states the design target as a PostgreSQL catalog at hundreds of terabytes and thousands of compute nodes. That sentence is DuckDB’s claim. This post does not treat it as a measured limit.

What Apache Iceberg is

Apache Iceberg is the open table format AWS analytics engines already speak. Data is Parquet (or another Iceberg data format) on S3. Table state is a tree of files: a metadata JSON file, a manifest list, and manifest files that name the data files in a snapshot. A catalog — Glue Data Catalog or S3 Tables — stores the pointer to the current metadata file so readers agree on the latest snapshot.

That split is why Iceberg works across engines. Athena, Redshift Spectrum, Glue Spark, EMR, and DuckDB’s iceberg extension can all plan a query once they can read the catalog pointer and the metadata files. It is also why planning touches object storage several times before a single data file is opened.

On AWS, the maintained path is S3 Tables plus Glue 5.1. S3 Tables reduced compaction costs by up to 90% effective 1 July 2025, with a default 512 MB target file (AWS What’s New). Glue 5.1 ships Spark 3.5.6 and Iceberg 1.10+ (Glue 5.1 GA, November 2025). Scan-cost tactics for those tables are in the Athena Iceberg cost guide.

Why use DuckLake

Use it when the pain is the metadata path and the small-write path, and DuckDB is the only engine.

• One round trip to plan. The catalog database returns the file list. You skip the metadata-file → manifest-list → manifest chain. Who this is for: interactive DuckDB queries where half a second of planning dominates the scan.
• Small changes stay out of S3 until you flush them. v1.0 turns inlining on by default at 10 rows. An insert, update, and delete under that threshold write no Parquet file until CHECKPOINT. Who this is for: streams of row-sized changes that would otherwise create a tiny file per commit.
• Several tables commit together. Schema changes and data changes are rows in the same database transaction. Who this is for: a load that must not publish table A without table B.
• No new catalog service. If Aurora or RDS PostgreSQL is already operated, DuckLake adds tables to a database you know how to back up. Who this is for: a platform team that will not run a second catalog product.

Opinionated take: take DuckLake for a DuckDB-only lake with a Postgres catalog you already run. Do not take it to escape Glue while Athena is still a consumer. The half-second planning claim is real as a DuckDB design argument. Losing Athena is the price.

Why use Iceberg on AWS

Use it when more than one engine must read the same table, or when you want AWS to run maintenance.

• Athena, Redshift, Glue, and EMR read it. A table QuickSight hits through Athena is an Iceberg table. DuckLake is invisible to those services.
• S3 Tables runs compaction, snapshot expiry, and unreferenced-file removal. The July 2025 price cut is the reason to turn maintenance on before staffing a weekend compaction job. The worksheet in the modern data lake post is the cost model for that choice.
• Lake Formation is the grant plane. Column and table grants, cross-account shares, and Glue as the system of record do not apply to a DuckLake catalog in Aurora.
• The metadata survives without your OLTP database. If Glue and S3 are up, Athena can still plan. DuckLake cannot plan when Postgres is down.

When to use which

Match the first row that fits. The same rules are in the worksheet.

A batch-append DuckDB lake can live in either format. If an AWS analytics engine is on the roadmap for the next two quarters, start on Iceberg. Migrating the catalog later is a project. Picking Iceberg first is a catalog pointer.

How this works on AWS

Two stacks. They share S3. They do not share a catalog.

flowchart LR
  duckdb[DuckDB_1_5_2]
  aurora[Aurora_PostgreSQL_catalog]
  s3[S3_Parquet]
  athena[Athena_Glue_Redshift]
  s3tables[S3_Tables_Iceberg]
  glue[Glue_Data_Catalog]
  duckdb --> aurora
  duckdb --> s3
  athena --> s3tables
  athena --> glue

DuckLake path

1. Catalog. Aurora PostgreSQL or RDS for PostgreSQL, Multi-AZ if more than one writer depends on it. The extension creates the 28 catalog tables. Network path: the DuckDB host must reach port 5432. A laptop on a bastion or SSM port forward is enough for a prototype. Shared writers belong on EC2 or ECS in the same VPC, with the Aurora security group limited to that fleet.
2. Storage. An S3 prefix, for example s3://bucket/ducklake/. Files are immutable. The extension does not rewrite a key in place. A gateway endpoint keeps that traffic off the NAT gateway.
3. Compute. Any host running DuckDB 1.5.2 or a later 1.5.x build with the ducklake, postgres, and httpfs extensions. Many DuckDB processes can attach the same Postgres catalog. That is the multi-writer model. A single-file DuckDB catalog is not.
4. Auth. Postgres roles protect the catalog. An IAM principal protects the prefix. CREATE SECRET (TYPE s3, PROVIDER credential_chain) uses the instance role, environment, or the local AWS profile. DuckLake does not issue Lake Formation credentials.
5. What AWS will not query. Athena, Glue Data Catalog, Redshift Spectrum, EMR Spark, and Lake Formation do not read DuckLake. Community clients exist — Apache DataFusion, a MotherDuck Spark client, and Trino ports named in the v1.0 announcement. This guide does not treat those as AWS-native readers.

MotherDuck’s hosted catalog is a different product. It is not this architecture.

Iceberg path

Writers are Glue 5.1 jobs or Athena MERGE. Storage is an S3 table bucket with managed maintenance. The catalog pointer lives in S3 Tables and, when you want one directory for the account, in the Glue Data Catalog via SageMaker Lakehouse. Readers are Athena, Redshift, and QuickSight. Wiring, compaction settings, and the cost worksheet are in the S3 Tables reference architecture. Do not rebuild that lake here.

DuckDB can read those Iceberg tables. Current DuckDB docs still mark S3 Tables support as experimental.

Attach DuckLake to Aurora and S3

DuckDB 1.5.2 or later, ducklake extension 1.0, spec 1.0. Replace the host and the bucket. Do not commit a password. The same script is attach-ducklake-s3.sql.

INSTALL ducklake;
INSTALL postgres;
INSTALL httpfs;
LOAD ducklake;
LOAD postgres;
LOAD httpfs;

CREATE SECRET (
  TYPE s3,
  PROVIDER credential_chain,
  REGION 'us-east-1'
);

ATTACH 'ducklake:postgres:dbname=ducklake_catalog host=catalog.cluster-example.us-east-1.rds.amazonaws.com'
  AS lake (DATA_PATH 's3://example-bucket/ducklake/');

USE lake;

CREATE TABLE IF NOT EXISTS events (id BIGINT, status VARCHAR, ts TIMESTAMP);
INSERT INTO events VALUES (1, 'en route', TIMESTAMP '2026-04-13 00:00:00');

The connection string matches the DuckDB DuckLake usage docs: ducklake:postgres: plus a libpq string, and DATA_PATH set to the S3 prefix. One inserted row is under the 10-row inlining default, so ducklake_list_files('lake', 'events') returns no Parquet file until CHECKPOINT.

Read Iceberg on S3 Tables from DuckDB

DuckDB current docs, iceberg extension, S3 Tables support marked experimental. This attaches a table bucket. It does not create the lake. Production writes still belong in Glue or Athena, as in the reference architecture post.

INSTALL aws;
INSTALL httpfs;
INSTALL iceberg;

CREATE SECRET (
  TYPE s3,
  PROVIDER credential_chain
);

ATTACH 'arn:aws:s3tables:us-east-1:111122223333:bucket/example'
  AS ice (TYPE iceberg, ENDPOINT_TYPE s3_tables);

SHOW ALL TABLES;

ENDPOINT_TYPE s3_tables sets SigV4 and the regional S3 Tables REST endpoint. It cannot be combined with a hand-written ENDPOINT. Account 111122223333 in the ARN is the AWS documentation sample account, not a FactualMinds account.

What broke

What broke — Documented in the DuckDB DuckLake usage page, not observed in a FactualMinds engagement. ATTACH … (DATA_PATH 'other/', OVERRIDE_DATA_PATH true) does not update the path stored in the catalog. It hides files under the original DATA_PATH for that connection. Queries return no rows for data that is still on S3. Detection: ducklake_list_files names keys you cannot read, and a second session without the override still sees them. Recovery: detach, attach without OVERRIDE_DATA_PATH, or point DATA_PATH at the prefix recorded in the catalog.

A second failure is a design consequence, not an incident report. If Aurora is unavailable, every DuckDB session stops planning, including readers. The Parquet objects are still in the bucket. Iceberg on S3 Tables does not have that dependency: Athena plans from Glue and the metadata files. Run the DuckLake catalog Multi-AZ, snapshot it, and alarm on Postgres availability the same way you alarm on a warehouse. A single-AZ dev instance is fine for the prototype in the SQL file. It is the wrong catalog once a second writer exists.

What to Do This Week

1. List every engine that must read the table this quarter. Include QuickSight. If Athena, Redshift, Glue Spark, or EMR appears, stay on S3 Tables and stop.
2. If the list is DuckDB only, open the worksheet and fill your_row. Match it to a recommendation row. Do not average the rows.
3. If the match is DuckLake, run attach-ducklake-s3.sql against an empty prefix and a dev Aurora database. Run SELECT version(); first and stop if it is older than 1.5.2. Insert one row, call ducklake_list_files, then CHECKPOINT, and confirm a Parquet object appears under the prefix.

Reproduce this — Download decision-worksheet.csv. Fill the your_row engines, writer count, catalog, and write shape. Read the recommendation on the first matching rule row. The SQL file is the attach step when that recommendation is ducklake-on-aurora-and-s3 or ducklake-local-file-catalog.

What This Post Doesn’t Cover

• No FactualMinds benchmark of planning time, inlining, or Aurora catalog load. The 100 ms and 0.5 s figures are DuckDB’s, via the MotherDuck walkthrough.
• MotherDuck’s hosted DuckLake, where they run the catalog and the storage. The AWS path in this post is your Aurora database and your S3 prefix.
• A claim that registering existing Parquet, or DuckLake’s Iceberg-compatible deletion vectors, makes the table readable by Athena. Deletion vectors in v1.0 are experimental in DuckDB’s own announcement. Athena still needs an Iceberg catalog.
• DuckLake 1.1, still unshipped as of 22 September 2026. The v1.0 post schedules variant inlining for non-DuckDB catalogs, and multi-deletion-vector Puffin files, for 1.1. Roles, git-style branching, and incremental materialized views are on the v2.0 list, not 1.1.
• DataFusion, Spark, and Trino clients. Named in the v1.0 announcement. Not tested for this post. Not a substitute for Athena.

If You Only Do One Thing

Write down the engines. If any AWS analytics engine is on the list, the format is Iceberg on S3 Tables. DuckLake is the lake you run when that list is DuckDB and the catalog is Postgres you already operate.