A Practical Guide to Data Platform Architecture on AWS for Small Teams

When you are a small team—maybe two or three engineers—building a data platform on AWS, the usual enterprise architectures with five layers of tooling do not fit. You cannot afford to run Airflow, dbt, Spark on EMR, Kafka, and a dozen other services with just a couple of people on rotation. The architecture that works for a 50-person data org will drown a small team in operational overhead.

This article walks through the AWS services and patterns I have found practical for a small team data platform. The goal is not a perfect architecture on a whiteboard. It is something you can build, maintain, and extend with limited people while still being production-grade.

The Core Stack

For a small team, every piece of the platform needs to earn its place. Here is what I have landed on after a few iterations:

Service	Role	Why This, Not Something Else
S3	Data lake storage	Obvious choice. Cheap, durable, no servers to manage.
Glue Crawler + Catalog	Schema discovery and metastore	More hands-off than running Hive Metastore yourself.
Athena	Ad-hoc querying	No cluster to manage. Pay per query, scales to zero.
Glue ETL (Spark) or Lambda	Data transformation	Glue for heavier transforms, Lambda for light ones.
Step Functions	Orchestration	Serverless, decent UI, easy to wire up in Terraform.
Terraform	Infrastructure as code	CDK is fine too. Pick one and stick with it.

The theme is serverless wherever possible. A small team has no business managing Spark clusters, Airflow instances, or Kafka brokers if they can avoid it.

Setting Up the Foundation: S3 and IAM

Before anything else, get your S3 bucket structure and permissions right. I use a simple layout:

s3://mycompany-data-lake/
  raw/
    <source-system>/
      <yyyy>/<mm>/<dd>/
  curated/
    <dataset>/
  analytics/
    <dataset>/

The raw layer holds data as it arrives—CSV, JSON, Parquet, whatever the source gives you. The curated layer has cleaned, deduplicated data in Parquet. The analytics layer holds business-level aggregations, usually as Parquet tables that feed dashboards.

For permissions, do not use a single bucket policy that lets everything talk to everything. This is where small teams often cut corners, and it bites later. Create separate IAM roles for ingestion, transformation, and querying. Use least privilege from day one. It adds maybe 30 minutes of Terraform work upfront and saves days of headaches later.

# IAM role for Glue jobs
resource "aws_iam_role" "glue_job_role" {
  name = "data-platform-glue-job-role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect = "Allow"
      Principal = { Service = "glue.amazonaws.com" }
      Action = "sts:AssumeRole"
    }]
  })
}

resource "aws_iam_role_policy" "glue_job_datalake_access" {
  name = "glue-job-datalake-access"
  role = aws_iam_role.glue_job_role.id

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect   = "Allow"
        Action   = ["s3:GetObject", "s3:PutObject", "s3:DeleteObject"]
        Resource = "${aws_s3_bucket.datalake.arn}/raw/*"
      },
      {
        Effect   = "Allow"
        Action   = ["s3:GetObject", "s3:PutObject"]
        Resource = "${aws_s3_bucket.datalake.arn}/curated/*"
      }
    ]
  })
}

A small note on IAM: when you are the only person writing policies, you will be tempted to use s3:* on "Resource": "*" to unblock yourself. I have done this. It works. Then six months later you wonder why a Lambda function somehow has write access to your production billing bucket. Be stricter than you think you need to be.

Ingestion: Keep It Simple

For most small teams, data comes from a few sources—a transactional database, an API, maybe some third-party SaaS tools, and internal CSV exports.

Database ingestion is the easiest to over-engineer. If you have a PostgreSQL or MySQL database, AWS DMS (Database Migration Service) works reasonably well for full-load and ongoing CDC. Set it up to write Parquet files to your S3 raw bucket. It is not perfect—DMS has quirks with certain data types and the error logging is sometimes unhelpful—but for the price of zero servers to manage, it is hard to beat.

API-based sources work well with Lambda and EventBridge. Write a small Lambda that calls the API, transforms the response into JSON or Parquet, and writes it to S3. Schedule it with EventBridge rules. The code for these is usually 50 to 100 lines of Python. Do not overthink it.

import boto3
import requests
import json
from datetime import datetime, timezone

def handler(event, context):
    resp = requests.get("https://api.example.com/v1/orders",
                         headers={"Authorization": f"Bearer {API_KEY}"})
    resp.raise_for_status()
    data = resp.json()

    now = datetime.now(timezone.utc)
    key = f"raw/orders-api/{now.year}/{now.month:02d}/{now.day:02d}/orders_{now.isoformat()}.json"

    s3 = boto3.client("s3")
    s3.put_object(
        Bucket="mycompany-data-lake",
        Key=key,
        Body=json.dumps(data),
        ContentType="application/json"
    )

File-based sources like CSV dumps or spreadsheet exports are the messiest in practice. Someone always emails a CSV with the column names changed from last week. I have found it best to set up a dedicated S3 prefix like raw/manual-uploads/ and use a Glue Crawler to infer the schema. If the schema drifts, the crawler picks it up, and Athena queries catch the mismatch. It is not elegant, but it keeps the data available without building a full validation pipeline for ad-hoc files.

Transformation: Glue When You Need It, Views When You Do Not

The temptation with a data platform is to transform everything into perfect star schemas on day one. Resist this. For a small team, a lot of analytical questions can be answered by Athena views over the curated layer. It is faster to iterate and costs nothing to maintain.

I use Glue ETL jobs only when:

• The transform is too heavy for a SQL view (large joins, complex window functions).
• The output needs to be materialized for performance reasons—dashboards cannot wait 30 seconds per query.
• The data needs deduplication or complex business logic that SQL handles poorly.

For everything else, a view in Athena gets the job done. You can always materialize it later if the query becomes too slow.

-- Athena view for daily order metrics
CREATE OR REPLACE VIEW curated.daily_order_metrics AS
SELECT
    date(order_timestamp) AS order_date,
    customer_region,
    COUNT(*) AS order_count,
    SUM(order_total) AS revenue,
    COUNT(DISTINCT customer_id) AS unique_customers
FROM curated.orders
WHERE order_status != 'CANCELLED'
GROUP BY 1, 2;

When you do need a Glue job, write it as a simple script, not an over-abstracted framework. A single Python file that reads from S3, does the transform, and writes the output back is easier to debug than a ten-file project with custom decorators and shared utility libraries that three people have to understand.

Orchestration: Step Functions Over Airflow

This is the hill I will die on for small teams: use Step Functions, not Airflow. Running an Airflow instance means managing a scheduler, a web server, worker nodes, a metadata database, and dealing with upgrades. It is a part-time job by itself. For a team of two or three, that part-time job becomes someone’s full-time headache.

Step Functions are serverless. You define a state machine as JSON (or better, in Terraform), and AWS runs it. The trade-off is that Step Functions have less flexibility than Airflow’s DAGs—the fan-out patterns are more limited and the native retry options are simpler. But for a pipeline that does ingestion, triggers a Glue job, waits for it, then sends a Slack notification, Step Functions is more than enough.

{
  "Comment": "Daily ETL pipeline",
  "StartAt": "IngestData",
  "States": {
    "IngestData": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:us-east-1:123456789:function:ingest-data",
      "Next": "TransformData"
    },
    "TransformData": {
      "Type": "Task",
      "Resource": "arn:aws:states:::glue:startJobRun.sync",
      "Parameters": { "JobName": "curated-transform-job" },
      "Next": "NotifySlack"
    },
    "NotifySlack": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:us-east-1:123456789:function:notify-slack",
      "End": true
    }
  }
}

The glue:startJobRun.sync integration is particularly useful—it waits for the Glue job to finish before moving on, so you do not need to poll for completion yourself.

One thing to plan for: Step Functions execution history is limited to 90 days. If you need long-term audit trails of pipeline runs, log the execution details to CloudWatch or S3 yourself from within the state machine.

Monitoring and Alerting

For a small team, the most important metric is: did the pipeline run, and did it produce reasonable output?

Set up CloudWatch alarms on Step Function execution failures. The alarm can trigger an SNS topic that sends an email or a Slack message. Do not over-instrument at first—a single alarm that fires when a pipeline fails is worth more than ten dashboards nobody looks at.

For data quality, I use a simple approach: after each transformation job, run a validation query that checks row counts and a few business invariants. If the checks fail, fail the Step Function state. This is not as sophisticated as Great Expectations or dbt tests, but it catches 80% of issues with 10% of the effort.

# Simple row count check in a Lambda
import boto3
import time

def handler(event, context):
    athena = boto3.client("athena")
    response = athena.start_query_execution(
        QueryString="SELECT COUNT(*) FROM curated.daily_orders WHERE order_date = CURRENT_DATE",
        QueryExecutionContext={"Database": "curated"},
        ResultConfiguration={
            "OutputLocation": "s3://mycompany-data-lake/athena-results/"
        }
    )
    query_id = response["QueryExecutionId"]

    # Poll until complete
    while True:
        status = athena.get_query_execution(QueryExecutionId=query_id)
        state = status["QueryExecution"]["Status"]["State"]
        if state in ("SUCCEEDED", "FAILED", "CANCELLED"):
            break
        time.sleep(2)

    if state != "SUCCEEDED":
        raise Exception(f"Validation query failed: {state}")

What Changes in Production vs. a Demo

If you follow this guide as-is, you will have a working data platform. But there are a few things I would add before calling it truly production-ready:

1. Cross-account access: Keep your data lake in a separate AWS account from your application infrastructure. Use S3 bucket policies and IAM roles for cross-account access. This limits blast radius if someone accidentally opens a bucket to the public.

2. KMS encryption: Enable server-side encryption with KMS on your S3 buckets. It is a checkbox in Terraform and adds real protection against data leaks.

3. Lake Formation: If you need fine-grained access control—different analysts need different table-level permissions—use Lake Formation instead of raw IAM policies on S3. It adds complexity, but it is better than maintaining hundreds of IAM policies by hand.

4. GitHub Actions for CI/CD: Every Lambda, Glue script, and Terraform change goes through pull requests and automated deployments. With a small team, you can get away with manual deploys for a while, but setting up CI/CD early pays back quickly. Once your pipeline is processing real business data, breaking it with a bad deploy is a bad look.

Limitations and Things to Watch Out For

This approach is not right for every team. Here is where it falls short:

• High-volume streaming: If you need sub-second latency on streaming data, you need Kinesis or Kafka. S3 plus Lambda plus Athena is fundamentally a batch-oriented architecture.

• Large Spark workloads: Glue ETL is convenient but the cold-start times are noticeable—two to five minutes sometimes. If you have jobs that need to run every few minutes, Glue is the wrong tool. Consider EMR Serverless or running Spark on ECS.

• Complex DAGs: Step Functions are great for linear pipelines. If you have pipelines with dynamic branching, conditional logic, or hundreds of tasks, Step Functions become hard to read and debug. At that scale, consider Airflow or MWAA (the managed version).

• Cost visibility: Serverless billing—pay-per-query in Athena, per-job in Glue—is convenient but can lead to surprise bills if queries or jobs are not optimized. Set up AWS Budgets alerts early and keep an eye on the Athena query history for expensive full scans.

• Glue Crawler quirks: The crawler sometimes infers wrong data types, especially with CSV files that have inconsistent formatting. When it guesses a numeric column as a string because one row has a stray character, you will spend a morning debugging why your join produces no results. Validating the inferred schema against a known schema is worth the extra step.

Wrapping Up

Building a data platform with a small team is about making peace with trade-offs. You will not have the fanciest architecture, and that is fine. What matters is that the data is available, reliable, and the team is not burning out maintaining infrastructure.

Start with S3, Athena, and a few Glue jobs wired together with Step Functions. Add complexity only when the current setup actually hurts—not because a blog post or conference talk says you should. The best data platform for a small team is the one you can explain to a new hire in a day and debug at 2 AM without wanting to quit.