Batch vs Streaming: A Practical Guide for Beginner Data Engineers

In this article, let us look at what batch and streaming processing actually mean for a data engineer, and more importantly, when you should reach for one over the other. If you are just getting started with building data pipelines, the distinction can feel academic until you pick the wrong one and end up with a pipeline that either misses data or costs twice as much as it should.

We will go through practical examples using tools you are likely to encounter — Spark for batch and a simple streaming setup with Kafka — and talk through the decisions that actually matter when you are building something for the first time.

What is Batch Processing?

Batch processing is exactly what it sounds like. You collect data over a period of time, then process it all at once. Think of it as doing your laundry on Sunday rather than washing one shirt every time you wear it.

In a typical batch pipeline, you might have a CSV file landing in a GCS bucket every hour, and a Spark job that picks it up, transforms it, and writes the results to BigQuery. The key thing is that the job runs on a schedule — every hour, every day, whatever makes sense — and processes whatever data has accumulated since the last run.

Here is a simple PySpark batch job that reads customer orders from GCS and aggregates them:

from pyspark.sql import SparkSession
from pyspark.sql.functions import sum, col

spark = SparkSession.builder \
    .appName("daily-order-aggregation") \
    .getOrCreate()

# Read all order files from the past day
orders_df = spark.read \
    .option("header", "true") \
    .csv("gs://my-bucket/orders/2026-03-31/*.csv")

# Aggregate total spend per customer
daily_summary = orders_df \
    .groupBy("customer_id") \
    .agg(sum("order_amount").alias("total_spend"))

# Write results to BigQuery
daily_summary.write \
    .format("bigquery") \
    .option("table", "sales.daily_customer_summary") \
    .mode("append") \
    .save()

This job runs once a day, takes a couple of minutes, and costs very little. If it fails, you just re-run it against the same input files and you get the same result. That idempotency is one of the biggest practical advantages of batch — you can always hit retry and fix things.

What is Streaming Processing?

Streaming means you process each record (or small groups of records) as they arrive, rather than waiting for a batch window to close. If batch is laundry day, streaming is a conveyor belt that washes each shirt the moment it comes off your back.

Here is a similar aggregation, but this time using Spark Structured Streaming to read from Kafka:

from pyspark.sql import SparkSession
from pyspark.sql.functions import from_json, col, window, sum
from pyspark.sql.types import StructType, StructField, StringType, DoubleType

spark = SparkSession.builder \
    .appName("realtime-order-aggregation") \
    .getOrCreate()

schema = StructType([
    StructField("customer_id", StringType()),
    StructField("order_amount", DoubleType()),
    StructField("timestamp", StringType())
])

# Read from Kafka continuously
orders_stream = spark \
    .readStream \
    .format("kafka") \
    .option("kafka.bootstrap.servers", "localhost:9092") \
    .option("subscribe", "orders") \
    .load() \
    .select(from_json(col("value").cast("string"), schema).alias("data")) \
    .select("data.*")

# 5-minute tumbling window aggregation
realtime_summary = orders_stream \
    .withWatermark("timestamp", "2 minutes") \
    .groupBy(
        window(col("timestamp"), "5 minutes"),
        col("customer_id")
    ) \
    .agg(sum("order_amount").alias("total_spend"))

realtime_summary.writeStream \
    .outputMode("append") \
    .format("bigquery") \
    .option("table", "sales.realtime_customer_summary") \
    .option("checkpointLocation", "gs://my-bucket/checkpoints/") \
    .start() \
    .awaitTermination()

Notice a few extra things in the streaming version. We have watermarks to handle late-arriving data, checkpoint locations so the job can recover if it crashes, and the query never stops — it just keeps running. This is the first thing beginners often miss: a streaming job is not a script you run and forget. It is a service.

Batch vs Streaming: A Practical Comparison

Aspect	Batch	Streaming
Latency	Minutes to hours	Seconds to sub-second
Cost	Lower — pay only when running	Higher — always-on infrastructure
Complexity	Simpler — retry on failure	More complex — handle out-of-order data, backpressure, state recovery
Throughput	Very high for large volumes	Lower per-record but continuous
Use case examples	Daily reports, ML training data, data warehousing	Fraud detection, live dashboards, alerting
State management	Stateless by nature	Stateful — needs checkpointing
Error handling	Re-run the whole batch	Per-record dead-letter queues, partial recovery
Tools	Spark, Dataflow batch, dbt, Airflow	Kafka, Flink, Spark Structured Streaming, Dataflow streaming

When Should You Use Which?

Here is the rule of thumb I use when someone asks me this at work:

Start with batch unless you have a concrete reason not to.

Batch is easier to build, test, and debug. You can look at the input files, run the job, check the output, and fix what is wrong. With streaming, debugging often means trawling through logs and trying to reproduce the exact sequence of events that caused a problem.

That said, here are the cases where streaming is genuinely the right call:

1. The business needs results in seconds, not hours. If you are building a fraud detection system, you cannot wait for a daily batch. The transaction needs to be flagged before it completes.

2. The data volume is so high that batch windows become impractical. If you are processing clickstream data from millions of users, hourly batches might take longer than an hour to run — you are falling behind before you even start.

3. The downstream system expects a continuous feed. If your ML model needs real-time features, or your dashboard needs live metrics, batch will not cut it.

But here is the thing many tutorials do not tell you: most “real-time” pipelines in production are actually micro-batch pipelines with a 30-second or 1-minute trigger. True event-by-event streaming is rare and overkill for most use cases. Even Spark Structured Streaming, despite the name, processes data in micro-batches under the hood.

Common Gotchas Beginners Hit

Thinking streaming is just “batch but faster.” It is not. Streaming requires you to think about out-of-order events, late data, exactly-once semantics, and what happens when your job restarts. These are not small details — they are the majority of the work.

Underestimating the operational burden. A batch job that runs for 10 minutes a day and then stops is easy to manage. A streaming job that runs 24/7 means you now have an always-on service — alerts, monitoring, restarts, scaling. Your team owns it like any other production service.

Ignoring the cost difference. Streaming infrastructure runs continuously. If you use a managed service like Dataflow, a streaming job that does very little work still costs you the baseline worker cost every hour. For a batch job, you only pay while it is running. I have seen teams burn through budgets because they used streaming for something that could have been a 15-minute daily batch.

Not having a backfill strategy. With batch, you can always go back and reprocess historical data. With streaming, you typically process data and move on. If your streaming logic has a bug, you need a separate batch pipeline to fix historical data. Plan for this from day one.

What Changes in Production

For a production batch pipeline, the main things you add are:

• Idempotency: The job should produce the same result if run twice with the same input.
• Retry logic: Use something like Airflow or Cloud Composer to orchestrate retries and dependencies.
• Partition handling: Make sure late-arriving files get picked up by designing your input paths carefully.

For a production streaming pipeline:

• Dead-letter queues: Messages that fail to process should not block everything. Route them to a DLQ topic.
• Alerting on lag: Set up monitoring for consumer lag. If your streaming job falls behind, you need to know before the data becomes useless.
• State store backups: Your checkpoint data is the state of your pipeline. If you lose it, you lose where you were in the stream.
• Autoscaling: Streaming workloads are rarely flat. Plan for peak hours being different from the middle of the night.

Wrapping Up

If you are building your first data pipeline, do yourself a favour and start with batch. Get comfortable with the data, the transformations, the destination schema — all the things that have nothing to do with streaming complexity. Once the batch version is running smoothly and you find yourself needing lower latency, then ask whether streaming is worth the operational overhead.

The gap between batch and streaming has narrowed a lot. Tools like Delta Live Tables and Spark Structured Streaming make the code look almost identical. But the operational differences are still very real, and that is where you will spend most of your time — not writing the pipeline, but keeping it running.