Getting Started with AWS Glue Crawlers: A Practical Guide

If you have worked with data lakes on AWS, you have probably come across AWS Glue crawlers. In simple terms, a crawler scans data stored in S3, figures out the schema, and populates the Glue Data Catalog with table definitions. Once the table is in the catalog, you can query the data using Athena, Redshift Spectrum, or even Glue ETL jobs.

This article walks through what Glue crawlers do, how to set one up, what configuration options actually matter, and a few things I wish I knew before running them in production.

What Problem Do Crawlers Solve?

Imagine you have a bucket full of Parquet files coming in from some upstream pipeline. The files have columns and nested structures. You could manually write CREATE TABLE DDL in Athena every time the schema changes — but nobody wants to do that.

A Glue crawler automates that. It reads a sample of the data, infers columns and types, and writes the metadata to the Data Catalog. It also handles partitioning: if your S3 path looks like s3://bucket/logs/year=2026/month=06/day=15/, the crawler picks up those partition keys automatically and adds them to the table definition.

Setting Up Your First Crawler

Let us go through the steps to create a crawler that reads CSV files from S3 and creates a table in the Data Catalog.

1. Have Your Data Ready in S3

Create a bucket and upload a few sample files. For this walkthrough, let us assume the path is:

s3://my-company-logs/website-traffic/year=2026/month=06/day=15/

Inside that prefix, drop a couple of CSV files. Make sure the first row has headers — the crawler will use them as column names.

2. Create a Database in the Glue Data Catalog

Before running the crawler, you need a database for it to write to. In the Glue console, go to “Databases” and click “Add database.” Give it a name like website_logs_db. That is all you need for now.

3. Set Up an IAM Role for the Crawler

The crawler needs permission to read from S3 and write to the Data Catalog. Create an IAM role with a policy like this:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::my-company-logs",
        "arn:aws:s3:::my-company-logs/*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "glue:CreateTable",
        "glue:UpdateTable",
        "glue:GetTable",
        "glue:GetDatabase",
        "glue:CreateDatabase"
      ],
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents"
      ],
      "Resource": "arn:aws:logs:*:*:*"
    }
  ]
}

Also attach the AWS managed policy AWSGlueServiceRole to the role — it covers the basics like writing logs and accessing the catalog.

4. Create and Run the Crawler

Now head to the Glue console, go to “Crawlers,” and click “Add crawler.”

Step 1 — Crawler source type: Choose “Data stores” if your data is in S3 (or JDBC for databases). For this example, pick S3.

Step 2 — Data store: Under “Include path,” enter s3://my-company-logs/website-traffic/. If you have data across multiple buckets or prefixes, you can add more paths.

Step 3 — IAM role: Pick the role you created earlier.

Step 4 — Schedule: You can set it to run on a schedule (hourly, daily, etc.) using a cron expression, or leave it as “Run on demand” and trigger it manually. For a first run, choose on-demand.

Step 5 — Output: Select the database website_logs_db. You can also add a table prefix if you want, like traffic_ — the crawler will name the table traffic_website_traffic instead of just website_traffic.

Click through and run the crawler. It usually takes 1-2 minutes depending on how much data you have.

5. Verify the Table in Athena

Once the crawler finishes, hop over to the Athena console. Run:

SHOW TABLES IN website_logs_db;

You should see your new table. Then try a quick select:

SELECT * FROM "website_logs_db"."website_traffic" LIMIT 10;

If the table has partitions, Athena will show them as columns you can filter on:

SELECT * FROM "website_logs_db"."website_traffic"
WHERE year = '2026' AND month = '06'
LIMIT 10;

Configuration Options That Actually Matter

There are a few settings in the crawler configuration that are worth paying attention to.

Option	What it does	Recommendation
Schema updates	Whether the crawler updates the table when the schema changes	Set to “Update the table definition in the Data Catalog” for most use cases
Object deletion	What happens when the crawler finds objects deleted from S3	Set to “Mark the table as deprecated” or “Delete tables and partitions” depending on your retention policy
Partition indexing / Configuration options	Crawler can create partition indexes for faster Athena queries	Enable if you have more than a few hundred partitions
Classifiers	Custom logic for parsing file formats the crawler does not recognize well	Only needed for edge cases like oddly-formatted CSVs or custom delimiters
Crawler lineage	Tracks data lineage when moving catalog tables	Enable in production to understand where tables came from

A common mistake is leaving schema updates as “Ignore” and wondering why new columns do not show up. If your schema evolves (and it will), set the crawler to update the table definition.

Crawler Classifiers: When You Need Them

Built-in classifiers handle JSON, CSV, Parquet, Avro, ORC, and XML reasonably well. But when your files do not follow the standard patterns, you need to write a custom classifier.

For example, take a CSV file that uses a pipe delimiter instead of a comma. The built-in CSV classifier will get confused. You can create a custom classifier with a GROK pattern or specify the delimiter explicitly:

{
  "Classification": "csv",
  "CsvHeader": ["PRESENT"],
  "Delimiter": "|",
  "QuoteSymbol": "\""
}

Assign this classifier to your crawler in the “Custom classifiers” step and it will apply the right parsing rules.

Things I Learned the Hard Way

1. Crawlers sample data, they do not scan everything. By default, the crawler reads only a sample of files to infer the schema. If your data has inconsistencies — say most files have a column as int but a few have it as string — the crawler might pick the wrong type based on whichever files it samples. You can increase the sample size, but it adds runtime.

2. Hive-style partitions are your friend. If your S3 path does not use key=value format, the crawler will not pick up partitions automatically. s3://bucket/data/2026/06/15/ will not create partitions unless you specifically tell the crawler to treat subdirectories as partition keys. Stick to year=2026/month=06/day=15/.

3. Crawlers overwrite manual changes carefully. If you manually add a column to the table DDL in the Data Catalog and then re-run the crawler, it may drop that column if the source data does not have it. You can control this with the “Schema updates” setting, but it is better to keep your source-of-truth clear: either the crawler owns the schema or you do, not both.

4. Costs can sneak up on you. Glue crawlers are billed per minute (in DPU hours). Running them every 15 minutes on a data lake with millions of objects will add up. Use S3 event notifications to trigger crawlers only when new data arrives, rather than running on a fixed schedule.

5. Nested schemas in JSON can get unwieldy. The crawler flattens nested JSON into struct columns, which is fine, but deeply nested data can result in very wide tables that are painful to query. Consider transforming nested data before landing it in the data lake if you plan to query it frequently.

Production Considerations

In a production setup, you would not be clicking through the console. You would define the crawler in Terraform or CloudFormation:

resource "aws_glue_crawler" "website_traffic" {
  database_name = aws_glue_catalog_database.website_logs.name
  name          = "website-traffic-crawler"
  role          = aws_iam_role.glue_crawler.arn

  s3_target {
    path = "s3://my-company-logs/website-traffic/"
  }

  schema_change_policy {
    update_behavior = "UPDATE_IN_DATABASE"
    delete_behavior = "DEPRECATE_IN_DATABASE"
  }

  configuration = jsonencode({
    Version = 1.0
    CrawlerOutput = {
      Partitions = { AddOrUpdateBehavior = "InheritFromTable" }
    }
  })
}

You would also add a schedule or, better, trigger the crawler from a Lambda that listens to S3 events. This way the catalog stays up to date without burning DPU hours on empty crawls.

Another production practice is to separate raw and curated zones in your data lake. The crawler on the raw zone picks up whatever schema the source system produces. Downstream ETL (Glue jobs or something like dbt) transforms the data and writes to curated, where a separate crawler maintains a cleaner, well-documented schema.

Wrapping Up

AWS Glue crawlers are one of those services that sound simple but have enough knobs that getting them wrong can cause headaches. The core loop is straightforward — point it at data, get a table — but paying attention to schema evolution, partition detection, and the right schedule makes the difference between a setup that runs quietly in the background and one that breaks every time a new data format shows up.

Once you have the crawler populating the catalog, you are in a good spot to start querying with Athena, running Glue ETL jobs, or exposing the data through QuickSight. It is the first building block, and it pays to get it right.