Databricks Notebooks vs Jobs for Production Work

In this article, let us look at the difference between Databricks notebooks and Databricks Jobs, and where each one fits when we build a data pipeline. Notebooks are very convenient when we explore data or develop Spark code. The same convenience becomes a problem if we treat a manually run notebook as a production pipeline. We will take a small example, turn it into a scheduled Job, and discuss what I would change for a real production use case.

A point that confused me initially is that notebooks and Jobs are not exactly competing features. A notebook contains the code, while a Job (shown under Workflows in the Databricks UI) controls how and when that code runs. A Job can run a notebook, Python wheel, SQL task, JAR, or another Job. The practical decision is often not “notebook or Job”, but whether a notebook should run interactively or as a managed task.

A quick comparison

Area	Interactive notebook	Databricks Job
Main use	Exploration and development	Scheduled production execution
Trigger	A person clicks Run	Schedule, API, event, or another task
Compute	Usually an all-purpose cluster	Usually job compute or serverless
Retries	Manual	Configurable per task
Parameters	Widgets entered by a user	Job and task parameters
Monitoring	Notebook output	Run history, alerts, logs, task status
Multi-step flow	%run or manual order	Task dependencies in a DAG

For exploration, I would start with a notebook. It is easier to inspect a DataFrame, display a few records, and rerun one cell. Once the logic is repeatable and has a clear input and output, I would create a Job around it.

A sample notebook

For our use case, assume JSON files land in a volume and we need to write a daily sales summary to a Delta table. We can start with widgets so that the same notebook works for different dates and environments.

from pyspark.sql import functions as F

dbutils.widgets.text("run_date", "")
dbutils.widgets.dropdown("environment", "dev", ["dev", "prod"])

run_date = dbutils.widgets.get("run_date")
environment = dbutils.widgets.get("environment")

if not run_date:
    raise ValueError("run_date must be supplied as YYYY-MM-DD")

source_path = f"/Volumes/{environment}/landing/sales/{run_date}/*.json"
target_table = f"{environment}.reporting.daily_sales"

sales = spark.read.json(source_path)
summary = (
    sales.groupBy("store_id")
         .sum("amount")
         .withColumnRenamed("sum(amount)", "sales_amount")
         .withColumn("sales_date", F.lit(run_date).cast("date"))
)

This is enough for testing, but there are already things to fix. We should validate the input schema and make the write safe when the same date is rerun. An append creates duplicate rows after a retry. A simple approach is to register summary as a temporary view and use MERGE with sales_date and store_id as the key.

MERGE INTO prod.reporting.daily_sales AS target
USING daily_sales_update AS source
ON target.sales_date = source.sales_date
AND target.store_id = source.store_id
WHEN MATCHED THEN UPDATE SET *
WHEN NOT MATCHED THEN INSERT *

The retry behaviour matters because Jobs can retry a failed task automatically. If the code is not idempotent, a retry can make the data worse even though the Job eventually shows as successful.

Create the Job

From the Databricks workspace, open Workflows, choose Jobs, and click Create job. Add a task with these values:

1. Choose Notebook as the task type and select the notebook path.
2. Use job compute or serverless compute instead of a developer’s all-purpose cluster.
3. Add run_date and environment as task parameters.
4. Configure a timeout and retry count. For a daily batch, two retries with a few minutes between them is a reasonable start.
5. Add a schedule and select the correct timezone.
6. Configure a failure notification for the team rather than one person.

For the date parameter, we can use a dynamic value supplied by the Job, or let an upstream system pass it through the Jobs API. I prefer an explicit date instead of calling current_date() inside the notebook. It makes backfills and investigation easier. If yesterday’s pipeline must be rerun, we can submit yesterday’s date without changing code.

A basic job definition managed through a Databricks Asset Bundle can look like this:

resources:
  jobs:
    daily_sales:
      name: daily-sales-${bundle.target}
      schedule:
        quartz_cron_expression: "0 0 2 * * ?"
        timezone_id: "Australia/Melbourne"
        pause_status: UNPAUSED
      tasks:
        - task_key: aggregate_sales
          notebook_task:
            notebook_path: ../src/daily_sales.py
            base_parameters:
              environment: ${bundle.target}
              run_date: ""
          job_cluster_key: small_job_cluster
          max_retries: 2
          min_retry_interval_millis: 300000
      parameters:
        - name: run_date
          default: ""
      job_clusters:
        - job_cluster_key: small_job_cluster
          new_cluster:
            spark_version: 15.4.x-scala2.12
            node_type_id: Standard_D4ds_v5
            num_workers: 2

The exact runtime and node type depend on the cloud and workspace. I would not copy those values without checking what is supported. Keeping the Job definition in source control is the important part. It gives us a reviewable history and avoids manually recreating settings in every workspace.

Split a larger pipeline into tasks

It is common to see one large notebook with ingestion, transformation, quality checks, and publishing sections. That works for a demo, but failures become difficult to restart. In a Job, I would normally split it into tasks:

ingest_sales -> validate_sales -> aggregate_sales -> publish_sales

Each task should have a clear output and be safe to rerun. Databricks lets us define task dependencies, so aggregate_sales runs only after validation succeeds. Independent tasks can run in parallel. Use task values only for small metadata such as a row count or output path; do not try to pass a DataFrame between tasks. Persist actual data in Delta tables or files.

I also avoid using %run as the main orchestration mechanism. It is useful for small shared helpers, but a chain of %run calls hides dependencies inside notebook code. Job tasks make the order and failure point visible in the Workflows screen. Reusable Python code is better moved into modules or a wheel and tested separately.

What changes for production

For a simple demo, a notebook and one Job task are fine. In production, I would add the following:

• Deploy notebooks and Job configuration using Asset Bundles, Terraform, or the Jobs API.
• Use a service principal for the Job owner and run identity instead of a developer account.
• Store secrets in a secret scope or use the cloud identity. Never put credentials in notebook cells.
• Send alerts to a monitored location with the Job name, task name, run URL, and run date.
• Add data quality checks before publishing. A successful Spark command does not always mean the data is correct.
• Control the Databricks Runtime version and test upgrades before changing production.
• Set cluster policies, timeouts, and sensible scaling limits so one bad input cannot create a large bill.

There are limitations to be careful about. A notebook can contain hidden state if cells are run out of order during development. Restart the Python process and use Run all before calling it ready. Job clusters take time to start, although serverless compute can reduce this delay. Very small and frequent tasks may need a different compute choice.

Jobs provide orchestration inside Databricks, but they do not automatically replace every enterprise scheduler. If a pipeline coordinates many systems outside Databricks, an external orchestrator may still own the overall flow and call the Databricks Job. Also review permissions separately: users who can edit a notebook should not automatically be able to change or run a production Job.

Conclusion

Notebooks are a good place to explore data and develop the first version of Spark logic. For repeatable production execution, put that logic behind a Databricks Job with parameters, retries, monitoring, and dedicated compute. Start with one notebook task if the pipeline is small, then split it into clear tasks as dependencies and recovery requirements grow. The goal is not to remove notebooks, but to stop depending on a person to run them correctly.