A Beginner's Guide to Delta Lake Time Travel in Apache Spark

In this article, let us look at Delta Lake time travel — what it actually is, the problems it solves, and how to use it in your Spark jobs. If you have ever run a bad UPDATE or DELETE on a table and wished you could undo it, time travel is the feature you reach for. Even if you have not broken anything yet, it is one of those features that you will be glad exists when the day comes.

We will go through querying older versions of a table, restoring a table to a previous state, and a few practical things I have noticed while using this in projects.

What is Delta Lake Time Travel?

Every time you write to a Delta table — whether it is an INSERT, UPDATE, DELETE, or MERGE — Delta Lake creates a new version of the table. It does this by writing new Parquet files and recording the changes in a transaction log (the _delta_log directory you see under your table path). Each version is a snapshot of what the table looked like at that point in time.

Time travel simply means you can query any of those older snapshots. You can do this by specifying a version number or a timestamp, and Spark will read the table as it existed at that moment.

This is not a separate backup system. It works off the same transaction log that Delta already maintains for ACID guarantees. No extra infrastructure needed.

When Would You Use It?

Here are a few real scenarios where time travel has saved me:

• Bad MERGE or UPDATE: You ran a transformation that corrupted a column or updated the wrong rows. Instead of restoring from a backup, you just roll the table back one version.
• Debugging data issues: A downstream report is showing numbers that do not make sense. You can query the table as it looked yesterday and compare it with today to find what changed.
• Audit and reproducibility: Someone asks what the data looked like before the monthly cleanup job ran. You can point to an exact version and reproduce the state.
• Experimentation: You want to try a transformation on production data without committing it. You can query an older version, test your logic on it, and only write back when you are confident.

It is not a replacement for proper backups or snapshots in every case, but for many operational questions it is much faster than going to your backup tool.

How Time Travel Works Under the Hood

When you create a Delta table, a _delta_log folder appears alongside your data files. Inside it, you will see JSON files named like 00000000000000000000.json, 00000000000000000001.json, and so on. Each one represents a commit — a version of the table.

These JSON files record which Parquet files were added and which were removed in that commit. When you query version 5, Spark reads the log from version 0 to version 5 and reconstructs the set of Parquet files that made up the table at version 5. There is no data duplication — the same Parquet file can belong to multiple versions if it was not touched.

This is why you can time travel to any version without storing full copies of the data at each version. You only pay for the storage of the Parquet files that differ between versions.

Setting Up a Delta Table

Let us create a small Delta table to work with. If you already use Delta, this will look familiar.

from pyspark.sql import SparkSession
from pyspark.sql.types import StructType, StructField, StringType, IntegerType, DoubleType

spark = SparkSession.builder \\\n    .appName("delta-time-travel") \\\n    .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension") \\\n    .config("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog") \\\n    .getOrCreate()

schema = StructType([
    StructField("id", IntegerType()),
    StructField("product", StringType()),
    StructField("category", StringType()),
    StructField("price", DoubleType())
])

data = [
    (1, "Widget A", "Electronics", 29.99),
    (2, "Widget B", "Home", 14.50),
    (3, "Widget C", "Electronics", 45.00),
]

df = spark.createDataFrame(data, schema)
df.write.format("delta").mode("overwrite").save("/tmp/delta/sales")

This creates version 0 of our table with three rows.

Querying Older Versions

Now let us make a few changes so we have multiple versions to travel through.

# Version 1: INSERT a new row
from pyspark.sql import Row
new_row = spark.createDataFrame([Row(4, "Widget D", "Garden", 22.00)])
new_row.write.format("delta").mode("append").save("/tmp/delta/sales")

# Version 2: UPDATE prices
from delta.tables import DeltaTable

delta_table = DeltaTable.forPath(spark, "/tmp/delta/sales")
delta_table.update(
    condition = "category = 'Electronics'",
    set = {"price": "price * 1.10"}
)

# Version 3: DELETE a row
delta_table.delete("id = 2")

We now have four versions. Let us query version 0 — the original data before any changes:

# Time travel by version number
df_v0 = spark.read.format("delta") \\\n    .option("versionAsOf", 0) \\\n    .load("/tmp/delta/sales")
df_v0.show()

Output:

+---+--------+-----------+-----+
| id| product|   category|price|
+---+--------+-----------+-----+
|  1|Widget A|Electronics|29.99|
|  2|Widget B|      Home| 14.5|
|  3|Widget C|Electronics| 45.0|
+---+--------+-----------+-----+

You can also travel by timestamp:

# Time travel by timestamp
df_ts = spark.read.format("delta") \\\n    .option("timestampAsOf", "2025-12-15 10:00:00") \\\n    .load("/tmp/delta/sales")

Both options work the same way. Use version numbers when you know exactly which commit you want to inspect. Use timestamps when you want to see what the table looked like before a known incident time.

Restoring a Table to a Previous Version

Querying an old version is read-only. If you want to actually roll back the table — make the old version the current version — use RESTORE:

RESTORE TABLE delta.`/tmp/delta/sales` TO VERSION AS OF 1

Or in Python:

from delta.tables import DeltaTable

delta_table = DeltaTable.forPath(spark, "/tmp/delta/sales")
delta_table.restoreToVersion(1)

After this runs, version 4 becomes the restored version. The old versions 2 and 3 are not deleted — you can still time travel to them. Delta Lake never rewrites history. RESTORE creates a new version whose contents match the version you restored to. This means you can even undo a restore if you need to.

Let me say that again because it is important: RESTORE creates a new version. It does not delete anything. This is good for auditability but it also means your version count keeps growing and the bad data might still be in your storage if you do not clean it up.

Comparison: Time Travel vs Alternatives

Approach	What It Does	When To Use
Time Travel (versionAsOf)	Reads an old snapshot without changing the table	Debugging, comparisons, ad-hoc inspection
RESTORE	Creates a new version that matches an old one	Rolling back a bad deployment or bad data write
VACUUM	Physically deletes old Parquet files no longer needed	Storage cleanup, but removes ability to time travel beyond retention
External backups	A separate copy of the data at a point in time	Disaster recovery, regulatory archiving
Clone (shallow)	Creates a new table pointing to the same Parquet files	Testing transformations on a snapshot without affecting production

None of these replace the others. In practice you will use a mix. Time travel for quick operational fixes, VACUUM for storage management, and external backups for disaster recovery.

Practical Things to Be Careful About

Retention Period

By default, Delta Lake keeps the transaction log for 30 days. You can query any version within that window. If you try to time travel to a version older than the retention period, you will get an error.

-- Check current setting
DESCRIBE DETAIL delta.`/tmp/delta/sales`

You can configure this with delta.logRetentionDuration and delta.deletedFileRetentionDuration. The former controls how long the transaction log entries are kept. The latter controls how long the physical Parquet files are kept after they are marked for deletion.

If you need to keep history for longer — say for compliance reasons — increase the log retention. But remember that this means more storage and slower metadata operations over time, since Spark has to parse a longer log.

VACUUM Deletes Old Files

VACUUM removes Parquet files that are no longer referenced by any version within the retention window. Once you run VACUUM, you cannot time travel to versions that depended on those files.

-- Retain files needed for the last 168 hours (7 days)
VACUUM delta.`/tmp/delta/sales` RETAIN 168 HOURS

A common mistake: running VACUUM with the default retention of 7 days, then later discovering you need to time travel to a version from two weeks ago. The data is gone. Set your retention based on what your team actually needs, not the default.

Time Travel with Streaming

If you have a streaming job reading from a Delta table, time travel queries do not interfere with it. But if you RESTORE a table, the streaming job will see the restored version as a new set of changes. Depending on your streaming logic, this could cause duplicates or unexpected behaviour.

For production pipelines, coordinate restores with your downstream consumers. A quick message in the team channel saying “rolling back table X to version Y” goes a long way.

Not All Operations Are Atomic From the User’s Perspective

RESTORE is atomic at the Delta level — it creates a single new version. But if your downstream jobs read the table between the bad write and the restore, they already consumed bad data. Time travel lets you fix the table, but it does not undo the downstream impact. You still need to handle that separately.

Column Mapping and Schema Changes

If you have column mapping mode enabled in your Delta table, time travel across schema changes can get tricky. A column that was renamed between version 0 and version 5 will appear with different names depending on which version you query. Be aware of this if your table schema evolves.

What Changes in a Production Setup

The examples above write to a local path. In production, your Delta table will likely live in S3 or GCS, and you will use a metastore (like the Hive metastore or Unity Catalog in Databricks) to register it as a named table.

# Production pattern: use a catalog table, not a path
df.write.format("delta").mode("overwrite") \\\n    .saveAsTable("sales_db.sales")

# Time travel works the same way
spark.sql("SELECT * FROM sales_db.sales VERSION AS OF 3")

You will also want to think about:

• Access control: Who can run RESTORE? You probably do not want everyone with write access to roll back a table. In Databricks, you can restrict RESTORE to specific users or groups using table ACLs.
• Monitoring: Log every RESTORE operation. It is a significant event and your team should know when it happens.
• Retention configuration: Tune delta.logRetentionDuration and the VACUUM schedule based on your recovery SLAs. If your team expects to be able to roll back up to 90 days, you need to retain at least that long.
• Testing restores: Practice restoring a non-critical table. Know how long it takes for your table size. A 10 GB table restores quickly. A 10 TB table might take minutes. You do not want to learn this during an incident.

Wrapping Up

Delta Lake time travel is one of those features that seems simple on the surface — just query an old version, right? — but it has real depth when you start using it in day-to-day work. It saves you from bad writes, helps you debug data issues, and gives you an audit trail that is always on with no extra cost.

The key things to remember: time travel reads older versions without changing the table, RESTORE creates a new version that looks like an older one, and VACUUM permanently removes old data so be careful with your retention settings.

If you are already using Delta Lake, time travel is already available. You do not need to enable anything. Just start querying older versions and it will work.