Databricks Workflows for Scheduled Jobs: A Practical Guide

If you have worked with Databricks for a while, you have probably run notebooks manually or used an external scheduler like Apache Airflow to trigger them. But Databricks has its own orchestration layer called Workflows, and it handles a lot more than just running notebooks on a schedule.

In this article, let us walk through what Databricks Workflows can do, how to set up a scheduled job, handle task dependencies, pass parameters between tasks, and configure retries and alerts. We will also look at a few things that tripped me up the first time I used it in production.

What Are Databricks Workflows?

Databricks Workflows is a managed orchestration service inside the Databricks platform. You can use it to run notebooks, Python scripts, JAR jobs, dbt tasks, or Delta Live Tables pipelines. It supports scheduling (cron-based), event-driven triggers, and manual runs.

If you are comparing it with something like Airflow or AWS Step Functions, the biggest difference is that Workflows lives inside Databricks and talks directly to your clusters and notebooks — there is no external connection to manage, no DAG files to sync, and no separate infrastructure to maintain.

Here is a quick comparison to give you a sense of where it fits:

Feature	Databricks Workflows	Apache Airflow	AWS Step Functions
Managed infrastructure	Yes (no servers to run)	No (self-hosted or MWAA)	Yes (serverless)
Native Databricks integration	Excellent	Good (via operators)	Limited
DAG / task dependency support	Yes (multi-task jobs)	Yes (core feature)	Yes (state machines)
Event-driven triggers	Yes (file arrival, etc.)	Yes (via sensors)	Yes (native)
External system orchestration	Limited	Excellent (huge connector ecosystem)	Good (AWS services)
Learning curve for Databricks users	Low (GUI + JSON)	Medium-High	Medium

If all your workloads live inside Databricks, Workflows is often the simpler choice. If you need to orchestrate across systems outside Databricks, you might still need Airflow or a similar tool.

Creating Your First Scheduled Workflow

Let us create a job with a single notebook task and run it on a schedule. You can do this from the Databricks UI or using the CLI. I will show both.

From the UI

1. In your Databricks workspace, go to Workflows in the sidebar and click Create Job.
2. Give your job a name and add a task. Choose Notebook as the task type.
3. Select the notebook path, source (workspace or Git), and the cluster — you can use an existing all-purpose cluster or, better yet, create a job cluster that spins up and tears down automatically.
4. Under Schedule, add a cron expression. For example, 0 6 * * * runs it at 6 AM UTC every day.
5. Click Create and the job is live.

Using the Databricks CLI

If you prefer to manage things in code, here is how the same job looks as a JSON definition:

{
  "name": "daily-customer-ingestion",
  "schedule": {
    "quartz_cron_expression": "0 0 6 * * ?",
    "timezone_id": "Australia/Melbourne",
    "pause_status": "UNPAUSED"
  },
  "tasks": [
    {
      "task_key": "ingest_customers",
      "notebook_task": {
        "notebook_path": "/Shared/ingestion/customer_ingest",
        "source": "WORKSPACE"
      },
      "existing_cluster_id": "1234-567890-abcdef12",
      "max_retries": 2,
      "timeout_seconds": 3600
    }
  ]
}

You can deploy this using databricks jobs create --json-file job-spec.json.

A quick note: I prefer using job clusters over existing all-purpose clusters for scheduled jobs. They cost the same per DBU but shut down when the job finishes, so you are not paying for idle time. The cold start is usually 2-5 minutes, which is fine for a daily pipeline.

Multi-Task Jobs and Dependencies

Most real-world pipelines are not a single notebook. You might have an ingestion step, a transformation step, and a validation step running in sequence — or in parallel.

Databricks Workflows supports multi-task jobs where you can define dependencies using depends_on:

{
  "tasks": [
    {
      "task_key": "ingest_orders",
      "notebook_task": { "notebook_path": "/Shared/pipeline/ingest" },
      "job_cluster_key": "main-cluster"
    },
    {
      "task_key": "transform_orders",
      "notebook_task": { "notebook_path": "/Shared/pipeline/transform" },
      "depends_on": [ { "task_key": "ingest_orders" } ],
      "job_cluster_key": "main-cluster"
    },
    {
      "task_key": "validate_orders",
      "notebook_task": { "notebook_path": "/Shared/pipeline/validate" },
      "depends_on": [ { "task_key": "transform_orders" } ],
      "job_cluster_key": "main-cluster"
    }
  ],
  "job_clusters": [
    {
      "job_cluster_key": "main-cluster",
      "new_cluster": {
        "spark_version": "14.3.x-scala2.12",
        "node_type_id": "i3.xlarge",
        "num_workers": 2
      }
    }
  ]
}

Notice that I used a shared job cluster (job_cluster_key) across all three tasks. This means the same cluster stays alive across the entire job run and only shuts down when the last task completes. It saves both time and money compared to spinning up a new cluster per task.

Something I learned the hard way: if a task fails and you have retries configured, the cluster stays up during retries. But if you set max_retries and all retries fail, the downstream tasks do not run. The job marks them as “skipped”.

Passing Parameters Between Tasks

One of the more useful features is passing outputs from one task to the next. You can use task values for this.

In your notebook, you set a value like this:

dbutils.jobs.taskValues.set(key="load_date", value="2026-03-10")

And in a downstream notebook, you reference it using the task key:

load_date = dbutils.jobs.taskValues.get(
    taskKey="ingest_orders",
    key="load_date"
)

This is cleaner than writing files to DBFS or using global temp views just to pass a string between notebooks. You can also use job parameters — these are key-value pairs you define when creating the job and reference with ,, or custom parameters like `` in notebooks or SQL widgets.

Notifications and Alerting

By default, a failed job just sits there silently until you check the UI. That is not great for production.

You can configure email notifications per job:

"email_notifications": {
  "on_failure": ["alerts@yourcompany.com"],
  "on_success": [],
  "no_alert_for_skipped_runs": true
}

I usually skip notifications on success for jobs that run reliably. If they start failing, that is when I want to know. For critical pipelines, you can also add a webhook notification destination — this lets you post to Slack, Teams, or any HTTP endpoint when a run fails.

To set up a webhook destination, go to Admin Settings → Notification Destinations → Add Destination, point it at your Slack webhook URL, and reference the destination ID in the job config.

Practical Things to Watch Out For

Here is a list of things I ran into or had to fix later:

1. Job cluster cold starts are real. If you have a job that needs to run exactly at 9:00 AM and your SLA is tight, factor in 3-5 minutes for cluster provisioning. Either schedule it 5 minutes earlier or use an always-on (job) compute pool if your organisation supports it.

2. Concurrent runs conflict silently. By default, if a scheduled run overlaps with a previous run that is still in progress, the new run is skipped with no error. You can change this with max_concurrent_runs, but think about whether your pipeline can safely run concurrently before upping it.

3. Notebook parameters from widgets can be confusing. If you are using dbutils.widgets.get() in your notebook and expecting a job parameter to fill it, the widget name must match the parameter name exactly. A typo means the notebook runs with a null or default value, and there is no warning.

4. Git-based notebooks need the repo to be synced. If your notebook source is a Git folder rather than the workspace, make sure your repo is configured to pull the right branch. A job referencing a notebook from a stale branch is a fun way to debug at 3 AM.

5. Costs add up for frequent small jobs. Each job run spins up a cluster, and if the cluster runs for 5 minutes but your notebook only uses 1 minute of compute, you still pay for the full 5-minute DBU minimum. If you have 100 such jobs running every hour, the minimum billing adds up. Batch them into fewer, larger jobs if it makes sense.

Production Considerations

In a demo, you create a job in the UI, run it once, and call it done. In production, here is what I would add:

• Infrastructure as code. Use the Databricks CLI or Terraform provider (databricks_job resource) to manage job definitions. This gives you version control, code review, and the ability to roll back.
• Environment separation. Use different Databricks workspaces for dev, staging, and production, or at least use different job resource prefixes and isolated catalogs.
• Monitoring dashboard. Set up a Databricks SQL dashboard that queries the system table system.workflow.job_runs so you can see run history, durations, and failure rates over time.
• Run repair workflows. If a job fails mid-pipeline (say task 3 of 5), you can repair and re-run from the failed task without re-executing tasks 1 and 2. This is available from the UI under the failed run.

Wrapping Up

Databricks Workflows is a solid choice for orchestrating jobs that live entirely within Databricks. It is simpler to set up than external schedulers, and the multi-task dependency support covers most pipeline patterns. Combined with job clusters, task values, and webhook alerts, you get a respectable production setup without needing a separate orchestration platform.

It is not a replacement for Airflow if you are managing pipelines across a dozen different services. But if your world is Databricks notebooks, Delta tables, and Unity Catalog, Workflows is probably already good enough — and you do not need to manage a single extra server.