Delta Lake Basics for Data Engineers: A Practical Guide

In this article, let us look at Delta Lake from the ground up — what it actually is, how it works under the hood, and how to start using it in your Spark jobs. If you have been building data pipelines on top of plain Parquet or CSV files and have run into problems like corrupt data from failed jobs, no way to update a single row without rewriting the whole partition, or downstream queries reading half-written data, Delta Lake is designed to fix exactly these things.

I am going to assume you know Spark basics but have not used Delta Lake yet, or you have used it through Databricks without really understanding what is happening under the hood. We will set up a local Delta table, walk through the key features, and look at a few things that can bite you when you take it to production.

What is Delta Lake?

At the storage level, Delta Lake is two things: Parquet files and a transaction log. The data itself is still stored as Parquet — the same columnar format you have been using for years. What Delta adds is a _delta_log directory alongside your Parquet files that tracks every write operation as a numbered commit.

Each commit in the log is a JSON file (or a checkpoint Parquet file when the log gets long) that records which data files were added and which were removed. When Spark reads a Delta table, it reads the transaction log first to figure out which Parquet files make up the current version of the table, then reads only those files. This is how it provides ACID transactions on top of object storage like S3 or ADLS — something that plain Parquet cannot do.

Here is what the directory structure looks like after a few writes:

/mnt/datalake/sales/
  _delta_log/
    00000000000000000000.json
    00000000000000000001.json
    00000000000000000002.json
  part-00000-abc.parquet
  part-00000-def.parquet
  part-00001-ghi.parquet

Because the data format is still Parquet, you are not locked into a particular engine. You can read Delta tables from Spark, Trino, Daft, DuckDB, and others. The format is open-source and governed under the Linux Foundation.

Why Not Just Use Parquet Directly?

If you have worked with plain Parquet on a data lake for a while, you know the pain points. Here is a quick comparison of what you get by switching to Delta:

Feature	Plain Parquet	Delta Lake
Atomic writes	No — partial files visible	Yes — commits are all-or-nothing
Upserts (UPDATE/DELETE)	Rewrite entire partition	Row-level via transaction log
Schema enforcement	Manual checks needed	Automatic on write
Schema evolution	Manual handling	Controlled via mergeSchema option
Time travel	No	Query any previous version
Compaction	Manual — you manage small files	OPTIMIZE command built in
Concurrent writes	Risk of corruption	Optimistic concurrency control

I am not saying Parquet is bad. It is still one of the best formats for analytical workloads. But Parquet alone was not designed to solve transactional problems. Delta layers those transactional guarantees on top of Parquet without changing the underlying data format.

Setting Up Delta Lake with Spark

You do not need Databricks to use Delta Lake. You can run it with open-source Spark by adding the Delta Lake dependency. Here is how to set it up in a Spark session:

from pyspark.sql import SparkSession

spark = SparkSession.builder \
    .appName("delta-lake-basics") \
    .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension") \
    .config("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog") \
    .getOrCreate()

The two configs are important. The first one registers Delta SQL extensions so you can use SQL syntax like CREATE TABLE ... USING DELTA. The second one tells Spark to use the Delta catalog, which is what lets you run Delta-specific operations like OPTIMIZE, VACUUM, and DESCRIBE HISTORY from SQL.

If you are using a Spark cluster managed by a platform like Databricks or EMR, the Delta libraries are usually pre-installed and you only need the config.

Creating Your First Delta Table

Let us create a small table to work with. We will simulate a simple orders dataset:

from pyspark.sql.types import StructType, StructField, StringType, DateType, DecimalType
from datetime import date

schema = StructType([
    StructField("order_id", StringType(), False),
    StructField("customer_id", StringType(), False),
    StructField("order_date", DateType(), False),
    StructField("amount", DecimalType(10, 2), False),
    StructField("status", StringType(), False)
])

data = [
    ("ORD001", "CUST01", date(2026, 1, 1), 150.00, "completed"),
    ("ORD002", "CUST02", date(2026, 1, 2), 200.00, "pending"),
    ("ORD003", "CUST01", date(2026, 1, 3), 75.50, "completed"),
]

df = spark.createDataFrame(data, schema)

df.write.format("delta") \
    .mode("overwrite") \
    .save("/tmp/delta/orders")

This creates a Delta table at /tmp/delta/orders. You can query it like any Spark table:

orders_df = spark.read.format("delta").load("/tmp/delta/orders")
orders_df.show()

If you want to register it in the Hive metastore so you and your team can query it with SQL, use saveAsTable instead of save:

df.write.format("delta") \
    .mode("overwrite") \
    .saveAsTable("sales.orders")

Now SELECT * FROM sales.orders works from any Spark session that shares the same metastore.

Schema Enforcement: Catching Bad Data Early

One of the features I have found most useful in practice is schema enforcement. When you write to a Delta table, it checks that the columns in your DataFrame match the table schema. If there is a mismatch — a column is missing, has the wrong data type, or has nulls in a non-nullable column — the write fails with a clear error message.

# This will fail — extra column not in the schema
bad_df = df.withColumn("extra_column", lit("oops"))
bad_df.write.format("delta") \
    .mode("append") \
    .save("/tmp/delta/orders")
# AnalysisException: A schema mismatch detected when writing to the Delta table

With plain Parquet, Spark would happily write this DataFrame and you would only discover the schema drift when a downstream query started failing with a confusing error. Schema enforcement catches these issues at write time, when you can still do something about them.

Schema Evolution: When You Actually Need to Add Columns

Schema enforcement blocks accidental schema changes. But sometimes you genuinely need to add a new column. Delta handles this through schema evolution:

df_with_discount = df.withColumn("discount", lit(0.0))
df_with_discount.write.format("delta") \
    .mode("append") \
    .option("mergeSchema", "true") \
    .save("/tmp/delta/orders")

The mergeSchema option tells Delta to add any new columns from the incoming DataFrame to the table schema. Existing rows get NULL for the new column, which is usually what you want. You can also set this globally with spark.databricks.delta.schema.autoMerge.enabled set to true, but I prefer doing it explicitly per write — accidental schema changes are common enough that I want the safety net on by default.

The Transaction Log: How Delta Keeps Everything Consistent

If there is one concept worth understanding properly, it is the transaction log. Every write to a Delta table creates a new JSON file in the _delta_log directory. Let us look at what is inside one:

{"commitInfo":{"timestamp":1705104000000,"operation":"WRITE","operationParameters":{"mode":"Append"}}}
{"add":{"path":"part-00000-abc.parquet","size":1234,"partitionValues":{},"dataChange":true}}

Each commit file contains actions — actions that add files, actions that remove files, and metadata changes. An UPDATE in Delta is really an atomic combination of: remove the old Parquet file, add a new Parquet file with the updated rows. An INSERT just adds new files. A DELETE just removes files.

Because the log is ordered and each commit references the previous state, reading the log from 00000.json up to the latest commit gives you a complete picture of which files make up the current table. This is called “replaying the log.”

When the log gets long (typically after 10 commits), Delta writes a checkpoint — a Parquet file that contains the merged state of all prior commits. This way Spark does not have to replay hundreds of JSON files every time it reads the table.

You can inspect the history of any Delta table from Spark:

from delta.tables import DeltaTable

delta_table = DeltaTable.forPath(spark, "/tmp/delta/orders")
delta_table.history().select("version", "timestamp", "operation", "operationParameters").show(truncate=False)

Or using SQL:

DESCRIBE HISTORY delta.`/tmp/delta/orders`;

Upserts with MERGE

One of the most common patterns in ETL pipelines is upserts — you have new data arriving and some of it might update existing rows. Delta supports this through the MERGE operation:

from delta.tables import DeltaTable

delta_table = DeltaTable.forPath(spark, "/tmp/delta/orders")

updates_df = spark.createDataFrame([
    ("ORD001", "CUST01", date(2026, 1, 1), 175.00, "completed"),  # updated amount
    ("ORD004", "CUST03", date(2026, 1, 5), 90.00, "pending"),    # new order
], schema)

delta_table.alias("target") \
    .merge(
        updates_df.alias("source"),
        "target.order_id = source.order_id"
    ) \
    .whenMatchedUpdate(set={"amount": "source.amount", "status": "source.status"}) \
    .whenNotMatchedInsert(values={
        "order_id": "source.order_id",
        "customer_id": "source.customer_id",
        "order_date": "source.order_date",
        "amount": "source.amount",
        "status": "source.status"
    }) \
    .execute()

This updates ORD001 in place and inserts ORD004. With plain Parquet, the easiest way to do this would be to rewrite the entire partition — which is expensive when your partitions are large. The MERGE operation in Delta only rewrites the files that contain matching rows, which is much more efficient.

A few things to be careful about with MERGE:

• If the source DataFrame has duplicate keys, the MERGE will fail. Delta enforces that the merge condition produces at most one match per target row. You need to deduplicate your source data before merging.
• The matched clause runs once per matching row. If you have multiple source rows that match the same target row, Delta throws an error rather than silently picking one.
• MERGE is not a replacement for SQL-style multi-table joins. It is designed for the specific pattern of “apply changes from one dataset to another based on a key.”

Maintaining Delta Tables: OPTIMIZE and VACUUM

As you keep writing to a Delta table — especially with lots of small inserts — you end up with many small Parquet files. This is the classic small-file problem, and it hurts read performance because Spark has to open and read metadata for every file.

Delta provides the OPTIMIZE command to compact small files into larger ones:

OPTIMIZE delta.`/tmp/delta/orders`;

This reads all the small files and rewrites them into fewer, larger files (default target is 1 GB per file). It does not change the data — it is purely a storage-level compaction. You can run OPTIMIZE on a schedule, or after a batch of writes that you know produced many small files.

VACUUM is the cleanup command. When you run UPDATE or DELETE, Delta does not physically delete the old Parquet files immediately. It only marks them as removed in the transaction log. The old files stay on disk because they might still be needed for time travel queries. VACUUM deletes files that are no longer referenced by any version of the table, but it keeps files needed for the retention period (default 7 days):

VACUUM delta.`/tmp/delta/orders` RETAIN 168 HOURS;

Be careful with VACUUM. Once you vacuum old files, you cannot time travel to versions older than the retention period. If you set the retention too low, you might break the ability to roll back a bad write.

Practical Things That Can Catch You

Here are a few things I have run into when using Delta Lake in real projects:

1. Concurrent writes on the same table. Delta uses optimistic concurrency control. If two jobs try to write to the same table at the same time, one will succeed and the other will get a ConcurrentAppendException. You need to handle retries in your job, or design your pipeline so that only one writer writes to a table at a time. In practice, I have found that structuring your pipeline as a single-writer-per-table avoids most of these issues.

2. Small file accumulation from streaming writes. If you use Delta with Spark Structured Streaming and set the trigger interval too low, you get a commit with a tiny file on every micro-batch. Running OPTIMIZE periodically handles this, but a better approach is to tune the trigger so each micro-batch processes enough data to produce reasonably sized files.

3. Schema evolution with nested structs. Adding a field to a nested struct works fine with mergeSchema, but changing the type of an existing nested field does not. Delta does not allow type changes on existing columns — you need to rewrite the table with the new schema.

4. Large transaction logs. If your table has thousands of commits without checkpoints, reading the log becomes slow. Delta handles checkpointing automatically, but if you are writing from a custom engine or an older Spark version, the checkpoint interval might need tuning.

5. Data skipping works best with sorted data. Delta collects column-level statistics (min/max per file) and uses them to skip files during reads. But this only helps if your data is naturally sorted on the filter column. If you query by customer_id but the data is written in random order, every file has a wide range of customer IDs and file skipping does not help much. Using ZORDER on commonly filtered columns can improve this significantly.

What Changes in Production

When you are doing a quick proof of concept, writing to a local path with mode("overwrite") on every run works fine. In production, here is what I would do differently:

• Use a managed metastore table instead of a path-based table. This gives you a single source of truth for table schemas and makes it easier for your team to discover tables.
• Set up a proper partitioning scheme. Choose partition columns based on your query patterns, not just the date. Over-partitioning creates as many problems as under-partitioning.
• Use GENERATED columns for partitioning if your partition column is derived from another column (like order_month from order_date). This avoids the common mistake of forgetting to populate the partition column in your ETL code.
• Schedule OPTIMIZE and VACUUM as separate maintenance jobs. Do not run them inside your ETL pipeline — you do not want a VACUUM failure to block your data pipeline.
• Set an appropriate log retention and file retention. The defaults (30 days for logs, 7 days for files) are reasonable for most use cases, but adjust them based on your compliance requirements and how far back you need to time travel.
• Monitor the size of your transaction log. If it grows too large, reads slow down. Most platforms expose metrics for this.

Wrapping Up

Delta Lake gives you database-like features on top of your data lake without locking you into a proprietary format. The two things that matter most day-to-day are schema enforcement (catching bad data at write time) and the transaction log (giving you ACID guarantees, upserts, and time travel).

If you are already using Spark and storing data in Parquet, moving to Delta is mostly a matter of changing format("parquet") to format("delta") and being thoughtful about your write patterns. The storage format stays the same — you are really just adding the transaction log layer on top.

I have found that the biggest benefit is not any individual feature, but that you stop having to work around the limitations of immutable files. Being able to update a single row or delete bad data without rewriting entire partitions changes how you design pipelines, and for the better.