Apache Iceberg vs Delta Lake: A Practical Guide for Beginners

If you have been working in the data engineering space over the last couple of years, you have probably heard the terms Iceberg and Delta Lake thrown around a lot. They often get mentioned in the same sentence as “lakehouse architecture” or “modern data stack”, and for someone new to the space, it can be confusing. Are they databases? Are they file formats like Parquet? Do they compete with Snowflake?

In this article, let us cut through the noise and take a practical look at Apache Iceberg and Delta Lake. We will cover what these table formats actually are, walk through the key differences, compare them side by side, and talk about when you would pick one over the other in real projects.

What Are Table Formats, Really?

Before comparing Iceberg and Delta Lake, we need to get clear on what a table format is. This was something I struggled with when I first came across the term.

When you store data in a data lake (say on S3 or GCS), you typically write Parquet files into folders. Over time you end up with a bunch of Parquet files scattered across directories. The problem is that a directory full of Parquet files is not really a “table”. There is no schema enforcement, no way to do safe updates or deletes, no concept of ACID transactions, and query engines have to scan through all those files to figure out what is in them.

A table format sits on top of your Parquet (or ORC, or Avro) files and provides a proper “table” abstraction. Think of it as the metadata layer that tells query engines what files exist, what schema they have, which files belong to which partition, and what has changed over time. Both Iceberg and Delta Lake do this — they just approach it differently.

The Short History

Delta Lake was created by Databricks and open-sourced in 2019. It started as an improvement on top of Spark’s Parquet-based tables and was deeply integrated into the Databricks ecosystem. Over time it has been opened up to work with other engines like Trino, Flink, and Rust-based tools.

Apache Iceberg was started at Netflix around the same time and donated to the Apache Foundation. Netflix had serious scale problems with Hive tables — partition listings were taking forever, and there was no snapshot isolation. Iceberg was built to solve those problems at Netflix-level scale from day one. It was designed to be engine-agnostic rather than tied to Spark.

How They Work Under the Hood

Let us look at the architectural differences. Both use a directory of metadata files that track your data files, but the approach differs.

Delta Lake uses a transaction log (_delta_log) stored as JSON files alongside your data. Every change to the table (insert, update, delete, schema change) gets written as a new JSON entry in the log. These log entries track which Parquet files are part of the current version of the table. When you read the table, the engine replays the log to figure out the current state. Periodically, Delta runs checkpoint files (Parquet snapshots of the log) so that you do not have to replay thousands of JSON entries every time.

my_table/
├── _delta_log/
│   ├── 000000.json
│   ├── 000001.json
│   ├── 000000.checkpoint.parquet
│   └── ...
├── part-00000.parquet
├── part-00001.parquet
└── ...

Iceberg takes a different approach. Instead of a sequential transaction log, Iceberg maintains a tree of metadata: a metadata.json file points to manifest lists, which point to manifest files, which track the actual data files. Each commit creates a new metadata.json that points at the updated tree. This design means Iceberg can resolve the state of a table in constant time — it does not need to replay history.

my_table/
├── metadata/
│   ├── v1.metadata.json
│   ├── v2.metadata.json
│   └── snap-*.avro
├── data/
│   ├── part-00000.parquet
│   └── ...
└── ...

Here is a quick comparison:

Feature	Delta Lake	Apache Iceberg
Metadata model	Sequential JSON transaction log + checkpoints	Tree-based: metadata → manifest list → manifest → data files
Partition evolution	Supported (since Delta 2.0)	Supported (built-in from the start)
Hidden partitioning	No (partitions are explicit directory paths)	Yes (partitions tracked in metadata, not directories)
Time travel	Query by version or timestamp	Query by snapshot ID or timestamp
Compaction	OPTIMIZE (bin-packing of small files)	rewrite_data_files and rewrite_manifests
Schema evolution	Add, rename, reorder, drop columns (some with caveats)	Add, rename, reorder, drop columns + type promotion
Engine support	Strongest with Spark/Databricks, growing elsewhere	Engine-agnostic by design (Spark, Trino, Flink, Hive, Presto, Snowflake, BigQuery)

A Quick Hands-On: Creating a Table with Each

Let us walk through creating a simple table with both formats so you get a feel for the difference.

Delta Lake (using PySpark)

from pyspark.sql import SparkSession

spark = SparkSession.builder \
    .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension") \
    .config("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog") \
    .getOrCreate()

# Write some data as a Delta table
df = spark.range(0, 1000).withColumnRenamed("id", "user_id")
df.write.format("delta").mode("overwrite").save("/tmp/delta_users")

# Read it back
spark.read.format("delta").load("/tmp/delta_users").show(5)

# Time travel — go back to the first version
spark.read.format("delta").option("versionAsOf", 0).load("/tmp/delta_users").show(5)

Iceberg (using PySpark)

spark = SparkSession.builder \
    .config("spark.sql.extensions", "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions") \
    .config("spark.sql.catalog.local", "org.apache.iceberg.spark.SparkCatalog") \
    .config("spark.sql.catalog.local.type", "hadoop") \
    .config("spark.sql.catalog.local.warehouse", "/tmp/iceberg_warehouse") \
    .getOrCreate()

# Create an Iceberg table using SQL
spark.sql("CREATE TABLE local.db.users (user_id BIGINT) USING iceberg")

# Insert data
df = spark.range(0, 1000).withColumnRenamed("id", "user_id")
df.writeTo("local.db.users").append()

# Time travel by snapshot ID
spark.sql("SELECT * FROM local.db.users VERSION AS OF 1234567890123456789").show(5)

The setup is slightly different but the developer experience is similar. The real differences show up when you need to do more advanced things.

Where Each Format Actually Shines

Go with Delta Lake if:

• Your stack is mostly Databricks and Apache Spark. The integration is first-class and you get features like Delta Live Tables, Unity Catalog, and OPTIMIZE out of the box.
• You want a single vendor to support most of your needs. Databricks has made Delta its default format and the tooling around it is mature.
• You need features like change data feed (CDF), column mapping for renaming, and generated columns. These are well supported.

Go with Iceberg if:

• You use multiple query engines. Iceberg works with Spark, Trino, Flink, Hive, Presto, Snowflake, BigQuery, Athena, Dremio, and more — often without needing custom connectors.
• You are not locked into a single compute platform. If your team uses Databricks for some workloads and Snowflake for others, Iceberg lets both engines read and write the same tables.
• Hidden partitioning matters to you. With Iceberg you do not need to include partition columns in your WHERE clauses explicitly, and you can change partitioning without rewriting data.

Things to Watch Out For

Neither format is perfect. Here are some practical gotchas I have seen:

Delta Lake limitations:

• The transaction log can grow large if you do frequent small writes. You need to run OPTIMIZE and checkpointing regularly.
• While Delta works with engines beyond Spark now, the support is not as seamless as Iceberg yet. Trino’s Delta connector, for example, has known gaps.
• If you are on a platform that does not support Delta natively (like Athena until recently), you might have trouble just reading the table.

Iceberg limitations:

• The catalog story can be confusing. Iceberg needs a catalog (Hive metastore, JDBC, REST, Glue, or a few others) and picking the right one depends on your environment. Getting the catalog setup wrong has tripped up many first-time users.
• Compaction is more manual. Delta’s OPTIMIZE command is straightforward; Iceberg’s rewrite_data_files and rewrite_manifests require more understanding of what you are doing.
• Some cloud warehouses claim Iceberg support but it is read-only or limited. Always check what “Iceberg support” actually means for your platform.

What Changes in Production

If you are just trying these out on your laptop with local files, both formats work fine. In production a few things change:

1. Catalog setup matters. For Iceberg in production on AWS you would probably use the Glue catalog or a REST catalog. For Delta, you would likely be on Databricks with Unity Catalog or use the Delta Lake Spark catalog managed by your orchestration layer.

2. You need a compaction strategy. Small files kill performance in both formats. Schedule regular compaction jobs (daily or after heavy ingestion windows) or configure your writers to avoid creating too many small files in the first place.

3. Locking and concurrency. If multiple writers are appending to the same table concurrently, you need to think about conflict resolution. Delta uses optimistic concurrency and can handle concurrent appends well. Iceberg supports similar patterns but the exact behavior depends on your catalog.

4. Retention for time travel. Both formats keep old snapshots. Set a retention policy (e.g., keep 7 days of history) and run a cleanup job, otherwise your metadata storage costs will creep up over time.

So Which One Should You Learn First?

If you are learning this for the first time and want to build something today:

• Start with Delta Lake if you are already comfortable with Spark and want the smoothest getting-started experience. The documentation is good, the Databricks community edition is free, and the _delta_log is easy to inspect manually to understand what is going on.

• Start with Iceberg if you are thinking about multi-engine architectures or want to understand the format that is becoming the industry standard for interoperability. Snowflake, BigQuery, and Athena all speak Iceberg now, and that trend is not slowing down.

Both are solid choices and learning one makes the other much easier to pick up. The concepts — ACID on data lakes, snapshot isolation, time travel, schema evolution — are shared between them. The differences are mostly in the implementation details and the ecosystem around each format.

At the end of the day, the “right” choice depends on where your data lives and what engines you need to read it. Do not overthink it — pick the one that fits your stack and start building.