Getting Started with Apache Spark on Databricks: A Practical Guide

In this article let us walk through getting started with Apache Spark on Databricks. If you have been building data pipelines using traditional ETL tools or scripting languages like Python and are now looking to move to distributed processing, Databricks gives you a managed Spark environment that takes away a lot of the infrastructure headache.

We will spin up a Databricks workspace, create a cluster, run some PySpark code in a notebook, and then talk about what changes when you move this to a production job. Along the way I will point out the things I wish someone had told me when I first started with Spark on Databricks.

Why Databricks for Apache Spark?

You can run Apache Spark in a few different ways — on-premises clusters, EMR on AWS, Dataproc on GCP, or using a managed service like Databricks. Each has its place, but here is a quick comparison based on what I have seen in real projects:

Setup	Best For	What You Manage
Self-managed (on-prem / EC2)	Full control, fixed workloads	Nodes, networking, Spark config, tuning
EMR / Dataproc	Cloud-native, transient clusters	Cluster config, bootstrap scripts, logging
Databricks	Teams that want to skip infra, focus on code	Almost nothing — cluster config only
AWS Glue	Simple serverless ETL	Job config, IAM, and patience with cold starts

Databricks wins when you want to get to writing Spark code without spending a day configuring worker nodes. The trade-off is cost — Databricks is not cheap, especially if you leave clusters running. But for teams where engineering time is the bottleneck, it usually pays off.

Step 1: Setting Up a Databricks Workspace

If you do not already have a Databricks account, you can sign up for a trial on their website. If your organisation already uses AWS, Azure, or GCP, you will likely spin up Databricks through the respective marketplace so it deploys into your cloud account.

Once your workspace is ready, the first thing you see is the Databricks UI with the sidebar. The key sections you will use most often:

• Workspace — where your notebooks live, organised into folders
• Compute — cluster management (called “Compute” in newer UI, used to be “Clusters”)
• Jobs & Workflows — for scheduling production jobs
• Catalog — Unity Catalog for managing tables, schemas, and data access

If your workspace has Unity Catalog enabled (most new ones do), you get a proper three-level namespace — catalog.schema.table — instead of the old database.table model. This matters later when you start managing access at scale.

Step 2: Creating Your First Cluster

Go to Compute and click “Create Compute.” You will see a few cluster types:

• All-Purpose Clusters — for interactive development, notebooks, ad-hoc work
• Job Clusters — for scheduled jobs, cheaper because they terminate after the job finishes

For learning, pick an all-purpose cluster. Keep it small:

Cluster mode: Single Node
Databricks Runtime: 14.3 LTS (or latest LTS)
Node type: m5d.xlarge (or equivalent, ~4 vCPU / 16 GB)
Terminate after: 30 minutes of inactivity

A single-node cluster is more than enough for learning Spark. Spark still runs in local mode and you can do all the DataFrame operations you would on a multi-node cluster — just without the distributed execution. Once your code works, you can scale it up later.

A quick note on Databricks Runtime: the LTS (Long Term Support) versions are what you want for anything that is not a throwaway experiment. They are more stable and get security patches longer.

Step 3: Running Your First PySpark Notebook

Create a new notebook in your Workspace. Set the default language to Python and attach it to the cluster you just created.

Databricks notebooks come with a Spark session pre-configured as the spark variable. You do not need to create a SparkSession yourself — it is already there. This trips people up when they copy-paste code from a local PySpark setup.

Let us start with something simple. We will create a DataFrame from a small CSV stored as a string, do a basic aggregation, and write the output as a Delta table:

from pyspark.sql.functions import col, sum as spark_sum

# Sample sales data
csv_data = """date,product,category,amount
2026-01-01,widget_a,electronics,150
2026-01-01,widget_b,furniture,320
2026-01-02,widget_a,electronics,200
2026-01-02,widget_c,furniture,180
2026-01-03,widget_a,electronics,175
2026-01-03,widget_b,furniture,410
"""

# Read CSV into a DataFrame
df = (spark.read
  .option("header", True)
  .option("inferSchema", True)
  .csv(spark.createDataFrame([(csv_data,)], ["csv_string"])
       .selectExpr("explode(split(csv_string, '\\n')) as csv_string")
       .where("csv_string != ''")
       .selectExpr("split(csv_string, ',') as cols")
       .selectExpr(
           "cols[0] as date",
           "cols[1] as product",
           "cols[2] as category",
           "cast(cols[3] as double) as amount"
       ))

df.show()

Actually, that is overly complicated for a demo. In practice you would read from cloud storage. Here is what that looks like with a file in DBFS or cloud storage:

# Read a CSV from cloud storage (S3 / ADLS / GCS)
df = spark.read \
    .option("header", True) \
    .option("inferSchema", True) \
    .csv("dbfs:/mnt/data/sales_2026.csv")

# Or if you are reading from a mounted S3 bucket
# df = spark.read.csv("s3://my-bucket/sales/sales_2026.csv", header=True, inferSchema=True)

df.printSchema()
df.show(5, truncate=False)

Now let us do a simple aggregation:

# Total sales by category
category_totals = df.groupBy("category") \
    .agg(spark_sum("amount").alias("total_sales")) \
    .orderBy(col("total_sales").desc())

category_totals.show()

# Write the result as a Delta table
category_totals.write \
    .mode("overwrite") \
    .format("delta") \
    .saveAsTable("default.sales_by_category")

Notice we used .format("delta") instead of parquet or csv. Databricks defaults to Delta Lake format, and you should too. Delta gives you ACID transactions, time travel, schema enforcement, and the ability to upsert with MERGE. Once you get used to Delta, going back to plain Parquet feels like stepping back in time.

Step 4: Querying with Spark SQL

Databricks notebooks let you mix Python and SQL in the same notebook using magic commands. This is one of the nicest features for data exploration:

%sql
SELECT 
    category,
    COUNT(DISTINCT product) as product_count,
    ROUND(SUM(amount), 2) as total_sales
FROM default.sales_by_category
GROUP BY category
ORDER BY total_sales DESC

The %sql magic switches the cell to SQL mode and runs it against the Spark session. The result shows up as a table in the notebook output, and you can click to turn it into a quick chart directly in the UI.

If you want to use the SQL result back in Python:

sql_result = spark.sql("""
    SELECT category, SUM(amount) as total
    FROM default.sales_by_category
    GROUP BY category
""")
sql_result.show()

The spark.sql() method returns a DataFrame, so you can chain it into further transformations or write it out.

Step 5: Notebooks vs Jobs — What Changes in Production

Running code in a notebook is great for development. But when you are ready to run this on a schedule, you need to move it to a job. Here is what I have learned the hard way:

Notebooks are for exploration, not for production. Notebooks have hidden state — variables linger between cells, you might have run cells out of order, and someone else opening the notebook can break things by running half of it. A production job should be reproducible every time from scratch.

The better approach is to write your Spark logic as Python modules or .py files and use spark-submit through a Databricks job. If you must use a notebook as a job, at least make it idempotent — clear your output tables or use overwrite mode, and do not rely on variables from earlier cells that might not have run.

A production pipeline on Databricks typically looks like this:

# main.py — production Spark job
from pyspark.sql import SparkSession
from pyspark.sql.functions import col, current_timestamp

def transform_sales(input_path: str, output_table: str):
    spark = SparkSession.builder.getOrCreate()
    
    df = spark.read \
        .option("header", True) \
        .option("inferSchema", True) \
        .csv(input_path)
    
    result = df.groupBy("category") \
        .agg({"amount": "sum"}) \
        .withColumn("processed_at", current_timestamp())
    
    result.write \
        .mode("overwrite") \
        .format("delta") \
        .saveAsTable(output_table)

if __name__ == "__main__":
    transform_sales("s3://my-bucket/input/", "default.sales_summary")

You deploy this as a Python script in a Databricks job, configured with a job cluster that spins up when the job starts and terminates when it finishes. Job clusters are cheaper because you only pay for the time they run.

Practical Caveats and Tips

Here are a few things that caught me off guard when I started with Spark on Databricks:

Small files are the enemy. If you are writing Parquet or Delta from a job that processes small batches, you end up with hundreds of tiny files that kill read performance. Use .coalesce(1) sparingly (it kills parallelism), and instead enable optimizeWrite in your Delta table settings or run OPTIMIZE after large writes.

Auto-scaling is not magic. Databricks clusters can auto-scale, but Spark does not redistribute existing data to new executors. If your cluster scales up mid-job, the new executors only handle new tasks. For batch jobs, a fixed-size cluster tuned to your data size usually performs better and costs less.

Watch your shuffle partitions. The default spark.sql.shuffle.partitions is 200. If you are processing 10 GB of data across 4 executors, 200 partitions are fine. If you are processing 10 MB, you are wasting time on scheduling overhead. Set it to roughly 2-3x the number of executor cores for best performance.

Photon is worth trying. If your Databricks plan includes Photon (the vectorised query engine), enable it on your SQL warehouses and job clusters. For DataFrame-heavy workloads it can cut runtime in half with no code changes. It does not help much with UDFs or RDD-based code, though.

Use dbutils for utility operations. Databricks provides the dbutils module for things like listing files, mounting storage, and passing parameters into notebooks. For example, dbutils.fs.ls("dbfs:/mnt/data/") shows you files in a mount point without needing to use Spark.

This article covered the basics of getting Spark running on Databricks — creating a workspace, spinning up a cluster, writing your first PySpark code, and the key differences between development and production. The real learning happens when you start building actual pipelines, dealing with skewed data, and tuning jobs that cost real money every time they run. Start small, use Delta from day one, and move your logic into Python scripts sooner than you think you need to.