Orchestrating AWS Glue Jobs with Step Functions: A Practical Guide

In this article, let us look at how to use AWS Step Functions to orchestrate Glue jobs. If you have been building data pipelines on AWS, you probably started with a single Glue job — pull some data, transform it, write it back to S3. That works fine until you need to chain multiple jobs together. Maybe job B should only run after job A finishes, and job C needs to run if either A or B fails. This is where Step Functions come in.

I have seen teams reach for Airflow or a custom scheduler for this kind of thing, but if you are already on AWS and have simple orchestration needs, Step Functions can do the job with way less overhead. No servers to manage, no DAG files to maintain, just a state machine defined in JSON (or, better yet, deployed with Terraform or CDK).

What We Will Build

We will build a simple ETL pipeline with three Glue jobs:

1. Raw to Staging — read raw CSV files from S3, do basic type casting, write to Parquet in a staging bucket
2. Staging to Curated — join staging data with lookup tables, apply business logic, write to a curated bucket
3. Data Quality Check — run row counts and null checks, publish results to CloudWatch

The Step Functions state machine will run these in sequence, and if the quality check fails, it will send an SNS notification instead of proceeding silently.

Here is a rough picture of what the state machine looks like:

Start → Raw-to-Staging → Staging-to-Curated → Quality-Check
                                                    |
                                           ┌────────┴────────┐
                                           ↓                  ↓
                                      Pass (Success)     SNS Notify (Fail)

Why Step Functions Over Other Options

Before we jump into the code, let us talk about when Step Functions make sense versus other tools.

Approach	Good For	Not Great For
Step Functions + Glue	Simple linear or branching pipelines, small number of jobs	Complex dynamic DAGs, heavy fan-out patterns
Airflow / MWAA	Large DAGs with complex dependencies, rich scheduling	Small teams who don’t want to manage infra
EventBridge + Lambda	Event-driven triggers, lightweight transforms	Long-running jobs (Lambda has a 15-minute limit)
Custom Scheduler (ECS/EKS)	Full control, any runtime	You now own a scheduler — maintenance burden

For our use case — three Glue jobs in sequence with basic error handling — Step Functions is a good fit. No need to spin up an Airflow cluster.

Step 1: Create the Glue Jobs

Let us assume we already have our three Glue jobs created. For this article we will keep them simple. Each job is a PySpark script that reads from one S3 location and writes to another.

Here is a stripped-down version of what the raw-to-staging job might look like:

import sys
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from awsglue.context import GlueContext

args = getResolvedOptions(sys.argv, ['JOB_NAME', 'source_path', 'dest_path'])

sc = SparkContext()
glueContext = GlueContext(sc)
spark = glueContext.spark_session

df = spark.read.option("header", "true").csv(args['source_path'])
df = df.withColumn("amount", df["amount"].cast("double"))
df = df.withColumn("event_date", df["event_date"].cast("date"))
df.write.mode("overwrite").parquet(args['dest_path'])

Notice we are passing source_path and dest_path as job arguments. This is important — Step Functions will pass these dynamically so the same job can be reused across different runs without hardcoding paths.

Step 2: Define the State Machine

Now the interesting part. We define our state machine using Amazon States Language (ASL). Here is what the definition looks like:

{
  "Comment": "Simple ETL pipeline with Glue jobs",
  "StartAt": "RawToStaging",
  "States": {
    "RawToStaging": {
      "Type": "Task",
      "Resource": "arn:aws:states:::glue:startJobRun.sync",
      "Parameters": {
        "JobName": "raw-to-staging",
        "Arguments": {
          "--source_path": "s3://my-bucket/raw/2026/06/23/",
          "--dest_path": "s3://my-bucket/staging/2026/06/23/"
        }
      },
      "Retry": [
        {
          "ErrorEquals": ["Glue.AWSGlueException"],
          "IntervalSeconds": 60,
          "MaxAttempts": 2,
          "BackoffRate": 2
        }
      ],
      "Next": "StagingToCurated"
    },
    "StagingToCurated": {
      "Type": "Task",
      "Resource": "arn:aws:states:::glue:startJobRun.sync",
      "Parameters": {
        "JobName": "staging-to-curated",
        "Arguments": {
          "--source_path": "s3://my-bucket/staging/2026/06/23/",
          "--dest_path": "s3://my-bucket/curated/2026/06/23/"
        }
      },
      "Retry": [
        {
          "ErrorEquals": ["States.ALL"],
          "IntervalSeconds": 30,
          "MaxAttempts": 3,
          "BackoffRate": 2
        }
      ],
      "Next": "QualityCheck"
    },
    "QualityCheck": {
      "Type": "Task",
      "Resource": "arn:aws:states:::glue:startJobRun.sync",
      "Parameters": {
        "JobName": "data-quality-check",
        "Arguments": {
          "--source_path": "s3://my-bucket/curated/2026/06/23/",
          "--threshold": "0.95"
        }
      },
      "Next": "CheckQualityResult"
    },
    "CheckQualityResult": {
      "Type": "Choice",
      "Choices": [
        {
          "Variable": "$.JobRunState",
          "StringEquals": "SUCCEEDED",
          "Next": "PipelineSuccess"
        }
      ],
      "Default": "NotifyFailure"
    },
    "NotifyFailure": {
      "Type": "Task",
      "Resource": "arn:aws:states:::sns:publish",
      "Parameters": {
        "TopicArn": "arn:aws:sns:us-east-1:123456789012:etl-alerts",
        "Message": "Data quality check failed for pipeline run. Check CloudWatch logs."
      },
      "End": true
    },
    "PipelineSuccess": {
      "Type": "Succeed"
    }
  }
}

A few things worth pointing out here:

Using .sync is important. The glue:startJobRun.sync resource waits for the Glue job to complete before moving to the next state. Without .sync, Step Functions would fire the job and immediately move on, which is not what we want for a sequential pipeline. But note — this means your state machine execution will run for as long as your Glue jobs run, and you get billed for that duration.

Retry logic is built in. Each state has a Retry block. The first job only retries on Glue.AWSGlueException, which covers things like concurrent run limits being hit. The second job retries on States.ALL because we want to catch anything that goes wrong at that stage. The backoff rate of 2 means the wait time doubles each retry — 30 seconds, then 60, then 120.

Choice state for branching. The CheckQualityResult state looks at the output of the quality check job. Step Functions stores each state’s output in $, so we can access $.JobRunState to decide what to do next.

Step 3: Passing Parameters Dynamically

Hardcoding S3 paths in the state machine is not great. In practice, you would pass the date or partition as input when starting the execution:

{
  "run_date": "2026-06-23",
  "source_bucket": "my-bucket"
}

And then reference it in your state machine using States.Format or the $ path syntax:

"--source_path": "States.Format('s3://{}/{}/raw/', $.source_bucket, $.run_date)"

This is one of those things that looks clean in the docs but can get tricky when you have deeply nested paths. I have found it easier to use a small Lambda function as the first step in the state machine to build all the paths and pass them forward, rather than doing complex string formatting inside ASL.

Things to Watch Out For

Concurrency limits on Glue. By default, AWS limits how many concurrent Glue job runs you can have per account. If your pipeline triggers often, you might hit this limit. The retry logic we added helps, but in production you should request a limit increase or add a queue-like pattern.

Cost of waiting. Step Functions charges per state transition and per duration of execution. If your Glue job takes 20 minutes, your state machine is RUNNING for 20 minutes and you pay for that time. For long-running jobs, the cost adds up. Compare with something like EventBridge Scheduler that just fires and forgets.

Error messages can be vague. When a Glue job fails inside Step Functions, the error you get is sometimes just “Glue.AWSGlueException” with not much else. You will need to go into the Glue console or CloudWatch to figure out what actually happened. I recommend logging the Step Functions execution ARN inside your Glue job so you can trace back from either side.

State machine size limit. ASL has a 1 MB size limit for the state machine definition. For a few jobs this is fine, but if you have twenty or thirty states with inline code, you hit it fast. At that point, either split into multiple state machines or move the logic into Lambda functions.

Passing large payloads between states. Step Functions has a 256 KB limit on the data passed between states. If your Glue job outputs a huge result object, trim it before passing it along. Most of the time you only need a status flag and maybe a row count.

What Would Change in Production

For a real production pipeline, here is what I would add on top of what we built:

1. Parameter Store or Secrets Manager for sensitive config instead of hardcoding SNS topic ARNs.
2. A dead-letter queue pattern. Instead of just sending an SNS alert on failure, push the failed execution details to an SQS queue that a support engineer (or another automation) can pick up.
3. Step Functions execution history logging to S3. By default, execution history is kept for 90 days. For audit purposes, you probably want longer retention.
4. CloudWatch alarms on ExecutionsFailed metric. Don’t rely on someone seeing the SNS email. Wire up an alarm that pages the on-call person.
5. Tests for the state machine itself. Use the Step Functions Local Docker image to test your ASL without deploying to AWS every time.

Wrapping Up

Step Functions plus Glue is a solid combination for simple ETL orchestration on AWS. You get retry logic, branching, and error handling out of the box without managing any infrastructure. It is not a replacement for Airflow if you have a hundred interdependent jobs, but for the use case we walked through — three jobs in sequence with quality checks — it works well and keeps things simple.

We could extend this further by adding parallel branches, map states for processing multiple partitions at once, or integrating with EventBridge for scheduled triggers. But the foundation we built here covers a lot of ground already.

If you are already on AWS and your orchestration needs are modest, give Step Functions a try before reaching for a heavier workflow tool. You might be surprised how far it gets you.