In this article, let us look at Delta Lake from the ground up — what it actually is, how it works under the hood, and how to start using it in your Spark jobs. If you have been building data pipeli...
In this article let us walk through getting started with Apache Spark on Databricks. If you have been building data pipelines using traditional ETL tools or scripting languages like Python and are ...
In this article, let us look at AWS Glue and Athena side by side — not the textbook definitions, but how they actually behave when you are building an analytics pipeline. If you have stared at the ...
In this article, I want to walk through the Apache Spark transformations that I reach for every day in real pipelines. There are dozens of transformations in the Spark API, but in practice, a handf...
In this article, let us look at Delta Lake time travel — what it actually is, the problems it solves, and how to use it in your Spark jobs. If you have ever run a bad UPDATE or DELETE on a table an...
In this article, let us understand what AWS Glue job bookmarks are, how they work, when to use them, and most importantly, when not to. If you have ever run a Glue job that processed the same data ...
Notebooks are where most people start with Databricks, and for good reason. They are interactive, visual, and great for figuring things out. But as soon as you move from exploration to something th...
When you are a team of two or three engineers and someone says “we need a data platform”, the first thing that comes to mind is not Kubernetes clusters and multi-region failover. It is usually “how...
When you are a small team—maybe two or three engineers—building a data platform on AWS, the usual enterprise architectures with five layers of tooling do not fit. You cannot afford to run Airflow, ...
In this article, let us walk through building a simple Change Data Capture (CDC) ingestion pipeline. If you have been only running full-load or batch ETL jobs so far, CDC gives you a way to capture...