S3 Data Lake Folder Design — A Practical Guide to Getting It Right

If you have worked on building a data lake on S3, you know the question comes up pretty fast — how should I organise the folders? It sounds trivial. You create a few prefixes, drop files in, and move on. But after six months of pipelines writing data every hour, you end up with a directory layout nobody understands, partition schemes that make querying slow, and a security model held together by hope.

In this article, let us walk through how to design S3 folder structures that actually hold up over time. I will cover zone-based layouts, partition naming conventions, what changes when you use tools like Athena or Glue, and a few things I have seen go wrong in production.

Why Folder Design Matters More Than You Think

S3 is not a file system. There are no real directories. But the prefix structure directly affects three things: query performance, access control, and the mental model of anyone who needs to find data.

When you run a query in Athena, it uses the prefix to prune which files to scan. If your partition scheme is s3://my-lake/sales/, Athena scans everything under sales/. If you partition by date as s3://my-lake/sales/year=2026/month=04/day=07/, Athena only touches the partitions you filter on. That can be the difference between a query that finishes in five seconds and one that scans a terabyte.

Access control works the same way. IAM policies can grant or deny access at the prefix level. If your raw zone and curated zone share a common root and someone fat-fingers a policy, you might accidentally let analysts read raw PII data. A well-designed folder layout makes it easy to write tight, auditable permissions.

The Zone-Based Layout

Most data lakes I have worked on follow some version of the medallion architecture — raw, cleaned, curated — but the folder layout needs to be explicit. Here is a pattern I have found to work well:

s3://my-data-lake/
  raw/
    system-a/
      entity-name/
        ingest_date=2026-04-07/
          file_001.parquet
  cleaned/
    system-a/
      entity-name/
        year=2026/month=04/day=07/
          file_001.parquet
  curated/
    business-domain/
      aggregate-name/
        snapshot_date=2026-04-07/
          file_001.parquet

Let me explain each layer.

Raw is the landing zone. Files arrive exactly as the source system produced them. Do not rename, compress, or convert files here — treat it as immutable. The partition key at this layer is usually ingest_date (the date the file was received) rather than the business date, because you want to be able to trace back when something landed. I have also seen teams add an _incoming/ or _staging/ subfolder within raw where files sit until a validation job picks them up.

Cleaned is where your first transformation happens. Column names get standardised, data types are enforced, nulls are handled. The partition key here should be a meaningful business or event date — year, month, day — so that downstream consumers can filter efficiently.

Curated is the consumption layer. This is where you build aggregation tables, join across domains, and expose data products to analysts. Domain-driven subfolders (sales/, inventory/, customer/) work better than system-driven ones here, because the consumer of a curated table cares about what the data means, not which operational system it came from.

Partition Naming: Hive-Style vs Custom

If you use Athena, Glue, or EMR with Hive metastore, stick to Hive-style partition naming: key=value. Athena will automatically detect partitions named this way when you run MSCK REPAIR TABLE or use partition projection.

year=2026/month=04/day=07/

If you use a custom format like 2026/04/07/, Glue can still crawl it, but you need to configure the crawler to recognise the pattern, and you lose the self-describing nature of Hive-style paths. I have seen both work. Hive-style is easier to debug because the partition values stare you in the face when browsing the console.

One trap to avoid: inconsistent depth. If your raw layer uses ingest_date=YYYY-MM-DD but your cleaned layer uses year=/month=/day=/, consuming tools need to handle both. Pick one convention per layer and enforce it. Your future self — and your on-call rotation — will thank you.

Partition Granularity: Day, Hour, or Something Else?

This depends on query patterns. If your analysts query data by date range and the daily volume is under a few hundred megabytes, daily partitions are fine. If you ingest streaming data and users frequently query a specific hour, go hourly.

Here is a rough guide:

Granularity	Daily Volume	Query Pattern	Overhead
Yearly	< 100 MB	Annual reports	Very low
Monthly	100 MB – 1 GB	Monthly rollups	Low
Daily	1 GB – 50 GB	Daily dashboards	Moderate
Hourly	> 50 GB or streaming	Real-time or near-real-time	High

Too many small partitions creates a long list for metadata tools to enumerate. I have seen pipelines that create per-minute partitions — Athena took over 30 seconds just to list the partition directories before it could start scanning data. There is a trade-off.

Folder Naming Conventions

Be boring with your naming conventions. It pays off.

Source system names: use short, consistent codes — erp, crm, webhooks — rather than full vendor names that might change (salesforce-prod-na, salesforce-prod-eu).

Table and entity names: use snake_case or kebab-case and stick to it across every layer. Nothing worse than user_events in raw and userEvents in curated with no mapping between them.

No special characters: spaces, slashes, or unicode characters in prefix names will eventually break something. S3 allows almost any character in a key, but that does not mean EMR, Glue, or your ETL tool handles them gracefully.

Date formats: YYYY-MM-DD and YYYY/MM/DD are the two most common. ISO 8601 sorts correctly as a string, so use it.

Access Control and Folder Design

IAM policies use the s3:prefix condition to limit access. If your layout is:

s3://my-lake/raw/erp/employees/
s3://my-lake/cleaned/erp/employees/
s3://my-lake/curated/hr/employee_summary/

You can give the HR analytics team read access to curated/hr/* and no access to raw/ or cleaned/. If the raw employee data contains PII, that separation matters.

A pattern I like:

• Data engineers get write access to raw/ and cleaned/ plus read on curated/
• Analysts get read-only on curated/ only
• A single automation role does the raw → cleaned → curated writes

Lake Formation can add column-level and row-level controls on top of this, but the prefix-level IAM design is your first line of defence. Get that right before layering on more tools.

Things I Have Seen Go Wrong

Flat raw zone with no subfolders. All source systems dumping into raw/ with no system-level prefix. After three source systems you cannot tell who produced what. Retrofitting is painful.

Date-only partitions in a cleaned layer with late-arriving data. If your pipeline processes a file three days late and writes it under the ingest date, but the query filters on the event date, the data is effectively invisible. Decide whether to use ingest date or event date — and if events can arrive late, partition by event date and handle updates with upserts or snapshots.

Mixing file formats in the same prefix. Some files as JSON, some as Parquet, in the same directory. Athena or Glue will choke unless you configure things carefully. Standardise on a single format per table — Parquet with Snappy compression is a safe default.

Schemas drifting across partitions. A new field added in February but not backfilled means partitions from January have fewer columns. If you use schema merging, be explicit about it. If you do not, newer partitions may fail to read with old schemas.

What Changes in Production

In development, you might have a single S3 bucket and a handful of prefixes. In production, you need to consider:

• Separate buckets for raw vs. curated — not strictly necessary, but makes costing and access boundary simpler.
• Object lifecycle policies — transition raw data to S3-IA after 30 days, archive to Glacier after 90 days. Design prefixes so these policies can target the right data without sweeping up curated tables.
• Cross-account access — if another AWS account (or a partner) needs to read curated data, their access pattern should be on a dedicated bucket or prefix with a well-defined resource policy.
• Data catalog registration — every new table or partition should be registered in Glue Catalog automatically as part of the pipeline, not as an afterthought. A table that exists in S3 but not in Glue is invisible to Athena and Redshift Spectrum.

Wrapping Up

Folder design in S3 is one of those things that feels unimportant until it is not. Get the zone layout right early, pick partition conventions and stick to them, and think about who needs access to what before you start writing data. The goal is that someone joining the team six months later can browse the bucket, understand the structure without a wiki page, and write a correct IAM policy on their first attempt.

We covered the basic zone layout, partition naming, access control, and a few production considerations. The patterns here are simple, but the consistency is what makes them work. Once you have this foundation, tools like Glue, Athena, and Lake Formation slot in much more cleanly.