AWS Step Functions for Data Pipeline Orchestration: A Practical Guide

In this article, let us build a small data pipeline using AWS Step Functions and understand where it fits compared with running the same workflow in Airflow. We will use Step Functions to validate an input file, run an AWS Glue job, check the result, and send a notification if something fails. This approach is useful when most of the pipeline already runs on AWS managed services and we do not want to maintain a separate orchestration server.

The example is simple, but it includes the parts that usually matter in a real pipeline: passing parameters, waiting for a long-running job, retries, timeouts, and a separate failure path.

What we are building

Assume a CSV file lands in an S3 bucket. An EventBridge rule starts the state machine and passes the bucket and object key. The workflow performs these steps:

1. Invoke a Lambda function to validate the file name and size.
2. Start an AWS Glue job and wait for it to complete.
3. Invoke another Lambda function to record success.
4. Publish a message to an SNS topic if any step fails.

The state machine does not process the data itself. It only coordinates services and maintains the workflow state. The Glue job still contains our Spark transformation code.

Step Functions compared with Airflow

Area	AWS Step Functions	Apache Airflow
Hosting	Fully managed AWS service	Managed service or infrastructure to maintain
Workflow definition	Amazon States Language JSON or YAML	Python DAG
Best fit	AWS service orchestration and event-driven flows	Complex data workflows across many systems
Pricing	Per state transition or duration	Environment and worker cost
Local development	More limited	Easier to run and test locally

For our use case, Step Functions is a good fit because Lambda, Glue, S3, EventBridge, and SNS have native integrations. If the workflow had hundreds of SQL tasks across different clouds, Airflow might be easier to manage.

1. Create the validation Lambda

Let us first create a small Python Lambda function. It checks that the input is a CSV file and is not empty.

import boto3

s3 = boto3.client("s3")

def lambda_handler(event, context):
    bucket = event["bucket"]
    key = event["key"]

    if not key.lower().endswith(".csv"):
        raise ValueError(f"Unsupported file type: {key}")

    response = s3.head_object(Bucket=bucket, Key=key)
    if response["ContentLength"] == 0:
        raise ValueError("Input file is empty")

    return {
        "bucket": bucket,
        "key": key,
        "size": response["ContentLength"]
    }

The Lambda execution role needs s3:GetObject permission for the input prefix. In production, I would restrict this to a particular bucket and prefix instead of using a wildcard resource.

2. Pass arguments to the Glue job

Our Glue job can read the bucket and key as command-line arguments. A shortened PySpark example looks like this:

import sys
from awsglue.utils import getResolvedOptions
from awsglue.context import GlueContext
from pyspark.context import SparkContext

args = getResolvedOptions(
    sys.argv,
    ["JOB_NAME", "SOURCE_BUCKET", "SOURCE_KEY"]
)
glue_context = GlueContext(SparkContext.getOrCreate())
spark = glue_context.spark_session

source = f"s3://{args['SOURCE_BUCKET']}/{args['SOURCE_KEY']}"
df = spark.read.option("header", True).csv(source)
clean_df = df.dropDuplicates()

clean_df.write.mode("overwrite").parquet(
    "s3://demo-curated-bucket/orders/"
)

For a demo, overwrite keeps the example easy to understand. For a production pipeline, overwriting one fixed path could remove earlier data. I would normally partition by ingestion date or business date and make the write idempotent so rerunning the same file does not create duplicates.

3. Define the state machine

The below Amazon States Language definition connects the main services. Replace the function ARN, Glue job name, and SNS ARN with values from your account.

{
  "StartAt": "ValidateInput",
  "States": {
    "ValidateInput": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": {
        "FunctionName": "arn:aws:lambda:ap-southeast-2:123456789012:function:validate-file",
        "Payload.$": "$"
      },
      "OutputPath": "$.Payload",
      "Retry": [{
        "ErrorEquals": [
          "Lambda.ServiceException",
          "Lambda.TooManyRequestsException"
        ],
        "IntervalSeconds": 2,
        "MaxAttempts": 3,
        "BackoffRate": 2
      }],
      "Catch": [{
        "ErrorEquals": ["States.ALL"],
        "ResultPath": "$.error",
        "Next": "NotifyFailure"
      }],
      "Next": "RunGlueJob"
    },
    "RunGlueJob": {
      "Type": "Task",
      "Resource": "arn:aws:states:::glue:startJobRun.sync",
      "Parameters": {
        "JobName": "orders-csv-to-parquet",
        "Arguments": {
          "--SOURCE_BUCKET.$": "$.bucket",
          "--SOURCE_KEY.$": "$.key"
        }
      },
      "TimeoutSeconds": 3600,
      "Retry": [{
        "ErrorEquals": ["Glue.ConcurrentRunsExceededException"],
        "IntervalSeconds": 30,
        "MaxAttempts": 2,
        "BackoffRate": 2
      }],
      "Catch": [{
        "ErrorEquals": ["States.ALL"],
        "ResultPath": "$.error",
        "Next": "NotifyFailure"
      }],
      "Next": "RecordSuccess"
    },
    "RecordSuccess": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": {
        "FunctionName": "arn:aws:lambda:ap-southeast-2:123456789012:function:record-success",
        "Payload.$": "$"
      },
      "End": true
    },
    "NotifyFailure": {
      "Type": "Task",
      "Resource": "arn:aws:states:::sns:publish",
      "Parameters": {
        "TopicArn": "arn:aws:sns:ap-southeast-2:123456789012:data-pipeline-alerts",
        "Subject": "Data pipeline failed",
        "Message.$": "States.JsonToString($)"
      },
      "Next": "PipelineFailed"
    },
    "PipelineFailed": {
      "Type": "Fail",
      "Error": "PipelineFailed"
    }
  }
}

The important part for Glue is the .sync suffix. Without it, Step Functions waits only for the StartJobRun API call, not for the Glue job itself. With glue:startJobRun.sync, the next state runs after Glue reports a successful completion.

The Catch blocks preserve the error under error and move to the notification state. Notice that validation errors are not retried. Retrying a bad file will not make it valid. Only transient Lambda throttling and Glue concurrency errors have retries.

4. Give the state machine access

The Step Functions execution role needs permission to invoke the two Lambda functions, start and inspect the Glue job, and publish to the SNS topic. The actions are similar to these:

{
  "Effect": "Allow",
  "Action": [
    "lambda:InvokeFunction",
    "glue:StartJobRun",
    "glue:GetJobRun",
    "glue:GetJobRuns",
    "glue:BatchStopJobRun",
    "sns:Publish"
  ],
  "Resource": "*"
}

This policy is shortened for readability. For an actual project, split permissions by service and specify the exact function, job, and topic ARNs. The Glue job role separately needs access to the source and destination S3 paths.

5. Start and inspect an execution

We can test the state machine from the AWS CLI:

aws stepfunctions start-execution \
  --state-machine-arn arn:aws:states:ap-southeast-2:123456789012:stateMachine:orders-pipeline \
  --input '{"bucket":"demo-landing-bucket","key":"orders/2026-11-10/orders.csv"}'

Open the execution in the Step Functions console to see which state is running and the input and output at each step. I find this visual execution history useful when a parameter disappears between tasks. Enable CloudWatch Logs for the state machine, while the detailed Spark logs remain under the Glue job run.

Things to be careful about

Standard workflows can run for up to one year and provide exactly-once workflow execution semantics. This suits long-running Glue jobs. Express workflows are intended for high-volume, short-duration workloads and have different execution and logging behaviour. I would use a Standard workflow for this pipeline.

Step Functions state input and output has a 256 KiB limit. Pass S3 locations between states, not file contents or large query results. For many files, a Distributed Map state can help, but limit concurrency so that it does not overwhelm Glue job quotas or a destination database.

Finally, make every task idempotent. Step Functions can retry a task after a timeout even if the earlier request reached the service. Use the source file key or an execution ID as a processing key. In a production use case, I would record completed inputs in DynamoDB or a control table, configure alarms for failed executions, encrypt logs, and deploy the state machine and IAM roles through Terraform or CloudFormation.

Conclusion

We created a practical Step Functions workflow that validates an S3 file, waits for a Glue transformation, records success, and sends failures to SNS. Step Functions removes most orchestration infrastructure when a pipeline is built around AWS services. The main work is still in safe retries, restricted IAM roles, idempotent tasks, and useful logs. For a small AWS-native pipeline, I would start with this approach and move to a broader orchestrator only when the workflow needs it.