In this article, let us understand AWS Glue job bookmarks and build a small example using files in S3. Job bookmarks are useful when a Glue job runs on a schedule and should process only new data i...
In this article, let us look at the difference between Databricks notebooks and Databricks Jobs, and where each one fits when we build a data pipeline. Notebooks are very convenient when we explore...
In this article, let us look at a practical data platform architecture on GCP for a small team. The aim is not to create a platform with every possible feature. We want something that can ingest fi...
Most data pipelines I have worked on start with a simple batch pull — run a query, dump the results, load them somewhere. But at some point, when a table grows to a few hundred million rows scrapin...
In this article, let us look at how to handle retries and idempotency in ETL jobs. If you have built pipelines that process data at any reasonable scale, you have probably seen jobs fail for reason...
In this article, let us go through setting up a CI/CD pipeline for Terraform using GitHub Actions. If you have been running Terraform from your local machine so far, this will help you move to a wo...
Terraform state is one of those things that works fine until it doesn’t. When you are the only person running terraform apply, keeping a local terraform.tfstate file is not a big deal. But once a s...
Partitioning is one of those things that sounds simple when you read the docs but gets tricky fast when you actually need to make it work on real data. In this article let us walk through how to ch...
In this article let us demystify the medallion architecture — what people mean when they talk about landing, bronze, silver, and gold layers, how to actually build them, and where things get messy ...
In this article, let us look at BigQuery external tables — what they are, when they make sense, and how to actually set one up over data sitting in Google Cloud Storage. If you have CSV or Parquet ...