In this article let us look at something that sneaks up on every data team: AWS costs. Not the big, obvious ones like provisioned Redshift clusters or Kinesis shards. The quiet ones. The Glue job t...
When I joined my first data engineering team, we had pipelines running every day and nobody really checked if the data was right. We found out something was broken only when a stakeholder pinged us...
Schema changes in a data pipeline are like changing the wheels on a moving car. You cannot just stop everything, and yet one wrong move can break reports, dashboards, and ML models that depend on t...
In this article, let us look at building reliable backfills in data pipelines — what backfills actually are, the patterns that work in practice, the things that go wrong, and how to make them borin...
In this article let us look at dbt (data build tool) — what it actually is, why analytics engineering teams reach for it, and how to get a basic project running. I will walk through setting up mode...
If you have data sitting in S3 and someone asks you a question that needs SQL to answer, you basically have two paths: spin up something that loads the data somewhere first, or query it where it li...
You have probably heard the phrase “S3 is schema on read” a hundred times. What people say less often is that your folder structure becomes your schema — whether you planned it that way or not. In...
When I started building data pipelines, I kept hearing “batch” and “streaming” thrown around like they were completely different worlds. It took me a while to realise they are not as far apart as t...
In this article, let us look at what ETL and ELT actually mean in practice — not the textbook definitions you have already read a hundred times, but what happens when you sit down to build a pipeli...
In this article, let us look at the Medallion Architecture — a data design pattern that organises your data lake into three layers: Bronze, Silver, and Gold. If you have heard terms like “bronze ta...