In this article, let us look at Delta Lake time travel — what it actually is, the problems it solves, and how to use it in your Spark jobs. If you have ever run a bad UPDATE or DELETE on a table an...
In this article, let us understand what AWS Glue job bookmarks are, how they work, when to use them, and most importantly, when not to. If you have ever run a Glue job that processed the same data ...
Notebooks are where most people start with Databricks, and for good reason. They are interactive, visual, and great for figuring things out. But as soon as you move from exploration to something th...
When you are a team of two or three engineers and someone says “we need a data platform”, the first thing that comes to mind is not Kubernetes clusters and multi-region failover. It is usually “how...
When you are a small team—maybe two or three engineers—building a data platform on AWS, the usual enterprise architectures with five layers of tooling do not fit. You cannot afford to run Airflow, ...
In this article, let us walk through building a simple Change Data Capture (CDC) ingestion pipeline. If you have been only running full-load or batch ETL jobs so far, CDC gives you a way to capture...
In this article let us look at two things that separate a production ETL job from one that just happens to work on a Tuesday: retries and idempotency. Most of us have written pipelines that run fin...
When you start using Terraform, you usually run it from your laptop. That works fine when you are the only person touching the infrastructure. But as soon as a second person joins, or you need to d...
In this article, let us look at how to manage Terraform state when you are working in a small team. If you have used Terraform alone on your local machine, you might have just ignored the terraform...
In this article, let us look at how to think about partitioning when you are building data lake tables. Partitioning is one of those things that seems straightforward when you read the docs, but wh...