Apache Iceberg vs Delta Lake: A Practical Guide for Beginners

In this article, let us compare Apache Iceberg and Delta Lake from a beginner’s point of view. Both formats help us build reliable tables on object storage, but choosing one only from a feature list can be confusing. We will look at what files they create, how to use them with Spark SQL, and what I would check before selecting one for a real project.

If you are currently writing Parquet files directly to S3, GCS, or ADLS, either format can make updates, schema changes, and concurrent writes much safer. The important point is that Iceberg and Delta Lake are table formats. They are not replacements for Spark, Databricks, Trino, or your cloud storage. They add metadata and transaction rules around files stored in that storage.

Why plain Parquet is not enough

Let us say we have a job that writes daily order data as Parquet. Reading all the files is straightforward, but a few practical problems appear as the pipeline grows:

1. How do we update an order without rewriting the complete dataset?
2. What happens if a job fails after writing only half of its output?
3. Can one job read the table while another job changes it?
4. How do we find the correct files without listing thousands of storage paths?
5. Can we safely rename or add columns?

A table format maintains metadata that answers these questions. It tracks which data files belong to a table and commits changes atomically. Readers see either the old version or the new version, rather than a partially written table.

Quick comparison

Area	Apache Iceberg	Delta Lake
Metadata	Metadata JSON, manifest lists, and manifest files	Transaction log containing JSON commits and checkpoints
Common fit	Open, multi-engine lakehouse environments	Databricks and Spark-focused environments
Partitioning	Hidden partitioning and partition evolution	Generated columns or liquid clustering, depending on platform
Engine support	Strong support across Spark, Flink, Trino, Athena, and others	Excellent in Spark and Databricks, with support in other engines improving
Time travel	Snapshot ID or timestamp	Table version or timestamp
Open source	Apache project	Linux Foundation project with an open protocol

This table is only a starting point. Engine and catalog support usually matter more than one isolated feature. A format can support an operation, but the engine version used by your team may not support it in the same way.

Creating an Iceberg table with Spark

For a simple local Spark example, we can configure a Hadoop catalog and use a local warehouse directory. In a cloud project, this would normally be a REST, Hive, Glue, or managed catalog, with the warehouse on object storage.

spark-sql \
  --packages org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.7.1 \
  --conf spark.sql.catalog.demo=org.apache.iceberg.spark.SparkCatalog \
  --conf spark.sql.catalog.demo.type=hadoop \
  --conf spark.sql.catalog.demo.warehouse=/tmp/iceberg-warehouse

We can then create and update a table using Spark SQL:

CREATE TABLE demo.sales.orders (
  order_id BIGINT,
  customer_id BIGINT,
  order_ts TIMESTAMP,
  amount DECIMAL(12, 2)
)
USING iceberg
PARTITIONED BY (days(order_ts));

INSERT INTO demo.sales.orders VALUES
  (1001, 45, TIMESTAMP '2026-10-12 10:30:00', 125.50),
  (1002, 46, TIMESTAMP '2026-10-12 11:00:00', 89.00);

UPDATE demo.sales.orders
SET amount = 130.00
WHERE order_id = 1001;

One useful Iceberg feature here is hidden partitioning. The application filters on order_ts; it does not need to know that the table is physically partitioned by day. We can later change the partition strategy without changing every query to reference a partition column.

Creating the same table with Delta Lake

For Delta Lake, a local Spark session can be started with the Delta package and extensions:

spark-sql \
  --packages io.delta:delta-spark_2.12:3.2.1 \
  --conf spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension \
  --conf spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog

The SQL is quite similar:

CREATE TABLE orders_delta (
  order_id BIGINT,
  customer_id BIGINT,
  order_ts TIMESTAMP,
  order_date DATE,
  amount DECIMAL(12, 2)
)
USING delta
PARTITIONED BY (order_date);

For this basic example, we need an order_date column if we want to partition by date. In Databricks, I would also evaluate liquid clustering instead of creating fixed partitions, especially when access patterns might change.

Delta Lake’s MERGE syntax is commonly used for incremental loads:

MERGE INTO orders_delta AS target
USING order_updates AS source
ON target.order_id = source.order_id
WHEN MATCHED THEN UPDATE SET *
WHEN NOT MATCHED THEN INSERT *;

Iceberg supports MERGE INTO with Spark as well. From the pipeline developer’s view, the code can therefore look nearly identical. The larger differences become visible in metadata, catalogs, maintenance, and support across query engines.

Time travel and recovery

Both formats retain table history. This is useful when a bad pipeline overwrites values or when we need to compare today’s result with yesterday’s result.

With Iceberg, we can inspect snapshots and query one of them:

SELECT * FROM demo.sales.orders.snapshots;

SELECT * FROM demo.sales.orders
VERSION AS OF 6384980821778528934;

With Delta Lake, we can inspect history and query a table version:

DESCRIBE HISTORY orders_delta;

SELECT * FROM orders_delta VERSION AS OF 3;

Time travel is not a backup by itself. Old data files are eventually removed by maintenance commands, and deleting the storage account or bucket still deletes the table. For production, backups and storage retention policies need to be designed separately.

Maintenance is part of the design

Table formats do not remove all operational work. Frequent streaming or micro-batch jobs can create many small files. Updates and deletes can also leave old files that are no longer referenced by the current table version.

For Iceberg, typical maintenance includes rewriting small data files, expiring snapshots, and removing orphan files. For Delta Lake, it normally includes compaction using OPTIMIZE where available and deleting unreferenced files using VACUUM.

Be careful with retention settings. Running an aggressive cleanup while a long query or delayed stream still uses an older snapshot can make that workload fail. In a simple demo, cleaning files after a few hours may look fine. In production, I would first measure the longest-running query, job retry window, and recovery requirement, then set retention with enough margin.

We also need to monitor metadata. Iceberg may accumulate snapshots and manifest files. Delta can accumulate transaction log entries before checkpoints are written. These are manageable, but they should be included in regular platform maintenance rather than handled only when queries become slow.

Which one should we choose?

I would start with the platforms that must read and write the table. If the project is mainly on Databricks and the team uses Databricks features for ingestion, optimization, governance, and sharing, Delta Lake is usually the easier choice. The integration is already there and the operational experience is straightforward.

If the table must be accessed equally from Spark, Flink, Trino, Athena, and possibly other engines, Iceberg is often a natural fit. Its catalog integrations and hidden partitioning are useful for an open data platform where no single processing engine owns the table.

Cloud services can also make the decision for us. Before choosing, I would create a small table in the exact services and versions planned for production. Test INSERT, MERGE, schema evolution, time travel, concurrent jobs, and cleanup. Documentation sometimes describes the newest format capability while a managed service supports an older specification or only a subset of write operations.

Another thing to check is migration. Converting a large existing Parquet table is not only a syntax change. We need to validate file discovery, partition values, permissions, catalog registration, and rollback. I prefer testing one non-critical table end to end before migrating the rest.

Conclusion

Iceberg and Delta Lake solve many of the same problems, and both are a large improvement over managing Parquet folders manually. Delta Lake fits naturally in a Databricks or Spark-centred stack, while Iceberg is attractive when several engines need equal access through an open catalog. For a real project, let the engines, catalog, and operational support decide the format, then prove the important write and recovery paths with a small table before committing to it.