Why GitHub Actions for Data Pipelines? Most data engineers I know started out running pipelines manually. You write a Spark job, test it on your local, copy the script to a Cloud Storage bucket, t...
If you are getting started with Airflow and want to build your first DAG to move some data around, this article walks you through exactly that. We will build a simple ETL pipeline that extracts dat...
If you have been working in the data engineering space over the last couple of years, you have probably heard the terms Iceberg and Delta Lake thrown around a lot. They often get mentioned in the s...
In this article, let us look at Delta Lake from the ground up — what it actually is, how it works under the hood, and how to start using it in your Spark jobs. If you have been building data pipeli...
In this article let us walk through getting started with Apache Spark on Databricks. If you have been building data pipelines using traditional ETL tools or scripting languages like Python and are ...
In this article, let us look at AWS Glue and Athena side by side — not the textbook definitions, but how they actually behave when you are building an analytics pipeline. If you have stared at the ...
In this article, I want to walk through the Apache Spark transformations that I reach for every day in real pipelines. There are dozens of transformations in the Spark API, but in practice, a handf...
In this article, let us look at Delta Lake time travel — what it actually is, the problems it solves, and how to use it in your Spark jobs. If you have ever run a bad UPDATE or DELETE on a table an...
In this article, let us understand what AWS Glue job bookmarks are, how they work, when to use them, and most importantly, when not to. If you have ever run a Glue job that processed the same data ...
Notebooks are where most people start with Databricks, and for good reason. They are interactive, visual, and great for figuring things out. But as soon as you move from exploration to something th...