Cost Optimization Basics for AWS Data Pipelines: A Practical Guide

In this article let us look at something that sneaks up on every data team: AWS costs. Not the big, obvious ones like provisioned Redshift clusters or Kinesis shards. The quiet ones. The Glue job that runs twice as long as it needs to. The S3 bucket with five years of intermediate build artifacts nobody remembers. The NAT gateway quietly charging you for cross-AZ traffic you did not even know was happening.

I have worked on pipelines where the infra cost was nearly double what it should have been, and the fixes took maybe a couple of afternoons. No architecture rewrites. Just looking at what was actually running, what was actually being stored, and trimming what did not need to be there.

This article covers the places I now check first when reviewing AWS pipeline costs. It is not a definitive list, but it is the checklist that has caught the most waste in my experience.

Where Pipeline Costs Actually Come From

If you break down a typical batch pipeline on AWS, the bill comes from roughly four buckets:

Cost Area	What Drives It	How It Shows Up
Compute	Glue DPU-hours, EMR instance-hours, Lambda GB-seconds	The biggest line item in most pipelines
Storage	S3 GB-months, EBS volumes, snapshots	Grows quietly month over month
Data Transfer	Cross-AZ, cross-region, NAT gateway, S3 egress	Hardest to spot because it is scattered across services
Idle/Orphaned	Unattached EBS volumes, old snapshots, unused endpoints, development resources left running	Pure waste — no value at all

Each of these behaves differently. Compute you can tune. Storage you can lifecycle. Data transfer you can re-architect slightly. Orphaned resources you just delete.

Let us walk through each one with concrete examples.

1. Glue Jobs: The DPU Trap

AWS Glue charges by DPU-hour (Data Processing Unit). The default allocation for a Glue job is 10 DPUs, and if your job is not CPU-bound, you are burning money for nothing.

Here is what I do. First, check the job metrics in the Glue console or via CloudWatch. Look at the glue.driver.ExecutorAllocated metric versus the actual CPU and memory utilisation. If the executors are sitting at 30% CPU and your job takes 20 minutes to finish at 10 DPUs, try it at 6 DPUs and check if the runtime actually changes much.

A quick way to experiment without touching production: duplicate the job, set --worker-type to G.1X and --number-of-workers to something lower, run it against the same input, and compare:

# In your Glue job script, you can log resource usage
import psutil
import os

logger.info(f"CPU count: {psutil.cpu_count()}")
logger.info(f"Memory: {psutil.virtual_memory().total / (1024**3):.1f} GB")

Run it once, check the logs, and you will know if the job actually needs all those DPUs.

Also — and this one is annoyingly common — turn off job bookmarks if you do not use them. Bookmarks cause Glue to scan state files in S3 before every run, adding small but real cost and latency. If your pipeline already handles incremental logic elsewhere, the bookmark is just dead weight.

# CloudFormation / Terraform: set job bookmark to "disable"
# In console: Job details → Advanced properties → Job bookmark → Disable

2. S3 Storage That Never Goes Away

S3 is cheap, which is exactly why people forget about it. A few terabytes of old data at Standard tier is not noticeable month one. By month twelve, you have paid for it twelve times.

The single highest-ROI change I have made on multiple projects: lifecycle policies on intermediate buckets. These are the buckets that hold things like raw JSON extracts before transformation, Parquet build artifacts, temporary join results. You do not need them after the pipeline completes.

{
  "Rules": [
    {
      "Id": "ExpireIntermediateAfter3Days",
      "Status": "Enabled",
      "Filter": { "Prefix": "intermediate/" },
      "Expiration": { "Days": 3 }
    },
    {
      "Id": "TransitionLogsToIA",
      "Status": "Enabled",
      "Filter": { "Prefix": "logs/" },
      "Transitions": [
        { "Days": 30, "StorageClass": "STANDARD_IA" }
      ]
    }
  ]
}

For the main data lake, tiering is the play. Move anything older than 90 days to Intelligent-Tiering or straight to Infrequent Access if you are confident the access pattern is low. The savings are significant — IA is roughly half the cost of Standard for storage, though you trade it for a retrieval charge. So do not tier data that your daily pipelines actively read.

Also, versioning. If you have versioning enabled on a bucket where objects get overwritten frequently (like a staging area), every old version stays and you keep paying for it. Add a noncurrent version expiration rule:

{
  "NoncurrentVersionExpiration": { "NoncurrentDays": 7 }
}

3. Data Transfer Costs Nobody Talks About

This one is tricky because the AWS bill does not give you a single clean line for it. Data transfer costs are embedded in per-service charges.

The biggest offenders I have run into:

• Cross-AZ traffic between Glue and an RDS instance in a different AZ. If your Glue job runs in us-east-1a and pulls from an RDS writer in us-east-1b, every byte transferred costs $0.01/GB in each direction. It adds up fast on large datasets. Keep resources in the same AZ when possible, or use VPC endpoints to avoid NAT gateway charges for S3/DynamoDB access from private subnets.

• NAT gateway for S3 access from private subnets. Glue jobs in a private subnet that reach S3 through a NAT gateway pay data processing charges on the NAT. Instead, create an S3 VPC endpoint — it is free, and traffic routes directly to S3 without touching the NAT.

• CloudWatch logs with aggressive retention. Glue and Lambda are verbose loggers by default. A Glue job writing 50 MB of logs per run, running hourly, generates ~36 GB of logs a month. At $0.50/GB ingested, that is $18/month per job. Set log retention explicitly:

# Terraform example for a CloudWatch log group
resource "aws_cloudwatch_log_group" "glue_job_logs" {
  name              = "/aws-glue/jobs/my-job"
  retention_in_days = 7
}

4. Orphaned Resources: The Silent Leak

Every environment that has existed for more than six months has these. The dev Glue job that was replaced by a new version but never deleted. The EBS volume left behind after an EC2 instance was terminated. The S3 bucket from a PoC that everyone forgot about.

I keep a simple script that I run once a month to surface things that look abandoned:

#!/bin/bash
# Find S3 buckets with no recent activity (approximation via CloudTrail or trusted advisor)
# List unattached EBS volumes
aws ec2 describe-volumes \
  --filters Name=status,Values=available \
  --query 'Volumes[*].[VolumeId,Size,CreateTime]' \
  --output table

# List Glue jobs and their last run time
aws glue get-jobs --query 'Jobs[*].[Name,LastModifiedOn]' --output table

You do not need anything fancy. Even a manual scan once a quarter catches enough to pay for the time spent.

Things to Be Careful About

Cost optimization has a couple of traps worth flagging:

Do not optimise too early. If your pipeline is still changing shape — schemas evolving, volume unpredictable — keep things simple. The cost of an over-tuned setup that breaks on edge cases is higher than a few months of slightly inflated AWS bills.

IA and Glacier have retrieval costs. Moving data to cheaper tiers saves on storage but charges you when you read it back. If you have ad-hoc analysts running Athena queries against historical data, moving it to Glacier Deep Archive will surprise someone with a bill. Understand your access patterns before tiering.

Reserved capacity commits you to spend. Glue and Redshift offer reserved capacity at a discount, but only if you are certain the workload stays. For pipelines still growing, start with on-demand, track utilisation for a quarter, then commit.

Spot instances are not a silver bullet. Spot can cut EMR costs by 60-70%, but if your SLA does not tolerate retries, the unpredictability is not worth the savings. Use spot for fault-tolerant workloads (batch ETL, model training checkpoints) and keep on-demand for time-sensitive production jobs.

What Changes in Production

In a demo or a side project, you can hand-tune a few jobs and call it done. In a production environment with dozens of pipelines, you need a slightly more systematic approach:

• Budget alerts first. Set a monthly budget in AWS Budgets with an 80% threshold alert. It is the cheapest insurance against a runaway job or a misconfigured loop.
• Tag everything. Cost allocation tags (like Team, Pipeline, Environment) let you break down the bill by pipeline or team. Without tags, you are staring at an aggregated bill and guessing.
• Automate the cleanup. Use AWS Config rules or a simple Lambda cron to flag unattached resources. Even an alert is better than nothing.
• Review quarterly. Costs drift. The S3 bucket that made sense at 100 GB might be wasteful at 10 TB. The Glue job that was fine at 10 DPUs might now process 5x the data and actually need tuning up rather than down.

Wrapping Up

AWS data pipeline costs do not usually explode overnight. They drift. A few extra DPUs here, a forgotten bucket there, some cross-AZ traffic that grew with the data volume. None of it is catastrophic on its own, but together it adds up.

The good news is that the fixes are usually small. A lifecycle policy. An S3 endpoint. A lower DPU count tested against real data. You do not need to redesign your architecture to cut 20-30% off the bill, and the time invested pays back within the same month.

Start with the orphaned resources — they are pure waste and take minutes to clean. Then the lifecycle policies. Then tune compute. By the time you have done those three, you have probably caught 80% of what is leaking.