Delta Lake Time Travel: A Practical Beginner Guide

In this article, let us look at Delta Lake time travel, how to query an older version of a table, and how it can help when a pipeline accidentally updates or deletes data. We will create a small Delta table, make a few changes, inspect its history, and read the table as it existed before those changes. This is useful for debugging data issues, comparing pipeline outputs, and recovering from simple mistakes without immediately reaching for a backup.

Time travel is one of the features that makes a Delta table different from a normal folder of Parquet files. Every write is recorded in the Delta transaction log. As long as the required log entries and data files are still available, Delta can reconstruct an earlier state of the table.

What does Delta Lake time travel store?

A Delta table contains data files and a _delta_log directory. The transaction log records operations such as appends, updates, deletes, and schema changes. It does not create a complete copy of the table for every operation. Instead, each table version points to the files that make up that version.

For our example, the table might go through the following versions:

Version	Operation	Table state
0	Initial write	Three customer rows
1	Update	One customer status changed
2	Delete	One customer removed

We can query by version number, which is normally the clearest option, or by timestamp when we know approximately when a change happened.

1. Create a sample Delta table

Let us start with a small table in PySpark. This example works in a Databricks notebook. It will also work in another Spark environment when Delta Lake is configured.

from pyspark.sql import Row

customers = [
    Row(customer_id=101, name="Anita", status="active"),
    Row(customer_id=102, name="Ben", status="active"),
    Row(customer_id=103, name="Chen", status="inactive")
]

df = spark.createDataFrame(customers)

(df.write
   .format("delta")
   .mode("overwrite")
   .saveAsTable("demo.customer_status"))

The first successful write creates version 0. We can confirm the current data using SQL.

SELECT *
FROM demo.customer_status
ORDER BY customer_id;

For a quick demo, a managed table is easier because we do not need to manage the storage path. In a production use case, the catalog, schema, location, access permissions, and table ownership should be decided before creating the table.

2. Make a few changes

Let us say customer 102 has closed the account. We update the status first.

UPDATE demo.customer_status
SET status = 'inactive'
WHERE customer_id = 102;

This creates another table version. Now let us simulate an incorrect delete made by a pipeline.

DELETE FROM demo.customer_status
WHERE customer_id = 101;

If we query the current table, customer 101 is missing. In a normal Parquet table, finding the previous state would require our own backup or snapshot process. With Delta Lake, we can inspect the transaction history first.

3. Inspect the table history

Run DESCRIBE HISTORY to see the operations recorded for the table.

DESCRIBE HISTORY demo.customer_status;

The output includes the version, timestamp, operation, user, notebook or job details, and operation metrics. The exact columns depend on the platform and operation. For debugging, I normally start with version, timestamp, operation, and operationMetrics.

We should see an initial write, an update, and a delete. If the delete is version 2, then version 1 is the table state immediately before it. Checking history is better than guessing a version because production tables can receive writes from multiple jobs.

4. Query an older version

We can query version 1 directly using SQL.

SELECT *
FROM demo.customer_status VERSION AS OF 1
ORDER BY customer_id;

Customer 101 should be present in this result. We can also query using a timestamp.

SELECT *
FROM demo.customer_status
TIMESTAMP AS OF '2026-09-08 10:15:00';

The timestamp must point to a time when that table state was available. Time zones can make timestamp queries confusing, especially when job logs, Spark sessions, and the person investigating the issue use different zones. When possible, I prefer to get the exact version from table history and use that.

PySpark supports the same approach through reader options.

old_df = (spark.read
    .format("delta")
    .option("versionAsOf", 1)
    .table("demo.customer_status"))

old_df.show()

For a path-based table, replace .table(...) with .load("/path/to/table"). A timestamp can be supplied using the timestampAsOf option.

5. Compare the current and previous data

Time travel is not only for recovery. It is quite useful when validating a pipeline change. We can compare the current version with an older version to find inserted, updated, or deleted rows.

WITH previous_data AS (
  SELECT * FROM demo.customer_status VERSION AS OF 1
),
current_data AS (
  SELECT * FROM demo.customer_status
)
SELECT
  COALESCE(p.customer_id, c.customer_id) AS customer_id,
  p.status AS previous_status,
  c.status AS current_status
FROM previous_data p
FULL OUTER JOIN current_data c
  ON p.customer_id = c.customer_id
WHERE p.status IS DISTINCT FROM c.status
   OR p.customer_id IS NULL
   OR c.customer_id IS NULL;

For a large table, do not compare every column and every row without thinking about cost. Filter to the affected partition or business date and compare using the table’s business key. Operation metrics from the history can also tell us whether a write changed far more rows than expected.

6. Restore the table after a mistake

Reading an older version does not change the current table. If we have confirmed that version 1 is correct, Delta supports restoring the table.

RESTORE TABLE demo.customer_status TO VERSION AS OF 1;

A restore is itself a new transaction. It does not remove versions 2 or 1 from history. For example, restoring version 1 after version 2 can create version 3 with the same logical data as version 1. This makes the recovery operation auditable.

I would not run RESTORE immediately on an important production table. First query the old version, validate row counts and key metrics, and check whether another job has written valid data after the bad operation. Restoring the complete table could remove those later valid changes. Sometimes the safer option is to read the missing rows from the old version and merge only those rows into the current table.

MERGE INTO demo.customer_status AS target
USING (
  SELECT *
  FROM demo.customer_status VERSION AS OF 1
  WHERE customer_id = 101
) AS source
ON target.customer_id = source.customer_id
WHEN NOT MATCHED THEN INSERT *;

Retention and VACUUM limitations

Time travel is not an unlimited backup. Delta needs the older data files to reconstruct an old version. The VACUUM command removes unreferenced files after the configured retention period. Once the files required by an older version are deleted, that version may remain visible in history but cannot necessarily be queried.

Be careful about reducing the vacuum retention period. A very short period can break long-running readers and remove the files needed for recovery. The default safety checks exist for a reason. In production, retention should match the recovery requirement, storage budget, write frequency, and any regulatory rules.

There are a few more points to keep in mind:

1. Time travel protects table versions, but it does not replace cross-region backups or disaster recovery.
2. Access to historical versions can expose old sensitive values after they were removed from the current table. Permissions and retention policies still matter.
3. Frequent updates and deletes can leave many old files until vacuum runs, increasing storage usage.
4. A timestamp query depends on table commit timestamps and should not be treated as a business-effective-date query.
5. Cloning, streaming, and concurrent writes need additional care before restoring a busy table.

Conclusion

Delta Lake time travel gives us a practical way to inspect and query earlier table states using a version or timestamp. The normal workflow is to check DESCRIBE HISTORY, query the suspected good version, compare it with the current data, and only then decide between a full restore and a targeted repair. It is very useful for day-to-day pipeline debugging, but retention and backup still need to be designed separately for a production platform.