In this article, let us go through the Apache Spark transformations that I use most often while building data pipelines. Knowing the syntax is useful, but it is equally important to know what Spark...
In this article, let us look at Delta Lake time travel, how to query an older version of a table, and how it can help when a pipeline accidentally updates or deletes data. We will create a small De...
In this article, let us understand AWS Glue job bookmarks and build a small example using files in S3. Job bookmarks are useful when a Glue job runs on a schedule and should process only new data i...
In this article, let us look at the difference between Databricks notebooks and Databricks Jobs, and where each one fits when we build a data pipeline. Notebooks are very convenient when we explore...
In this article, let us look at a practical data platform architecture on GCP for a small team. The aim is not to create a platform with every possible feature. We want something that can ingest fi...
Most data pipelines I have worked on start with a simple batch pull — run a query, dump the results, load them somewhere. But at some point, when a table grows to a few hundred million rows scrapin...
In this article, let us look at how to handle retries and idempotency in ETL jobs. If you have built pipelines that process data at any reasonable scale, you have probably seen jobs fail for reason...
In this article, let us go through setting up a CI/CD pipeline for Terraform using GitHub Actions. If you have been running Terraform from your local machine so far, this will help you move to a wo...
Terraform state is one of those things that works fine until it doesn’t. When you are the only person running terraform apply, keeping a local terraform.tfstate file is not a big deal. But once a s...
Partitioning is one of those things that sounds simple when you read the docs but gets tricky fast when you actually need to make it work on real data. In this article let us walk through how to ch...