Most data pipelines I have worked on start with a simple batch pull — run a query, dump the results, load them somewhere. But at some point, when a table grows to a few hundred million rows scrapin...
In this article, let us look at how to handle retries and idempotency in ETL jobs. If you have built pipelines that process data at any reasonable scale, you have probably seen jobs fail for reason...
In this article, let us go through setting up a CI/CD pipeline for Terraform using GitHub Actions. If you have been running Terraform from your local machine so far, this will help you move to a wo...
Terraform state is one of those things that works fine until it doesn’t. When you are the only person running terraform apply, keeping a local terraform.tfstate file is not a big deal. But once a s...
Partitioning is one of those things that sounds simple when you read the docs but gets tricky fast when you actually need to make it work on real data. In this article let us walk through how to ch...
In this article let us demystify the medallion architecture — what people mean when they talk about landing, bronze, silver, and gold layers, how to actually build them, and where things get messy ...
In this article, let us look at BigQuery external tables — what they are, when they make sense, and how to actually set one up over data sitting in Google Cloud Storage. If you have CSV or Parquet ...
In this article, let us look at how to use AWS Step Functions to orchestrate Glue jobs. If you have been building data pipelines on AWS, you probably started with a single Glue job — pull some data...
If you have worked with data lakes on AWS, you have probably come across AWS Glue crawlers. In simple terms, a crawler scans data stored in S3, figures out the schema, and populates the Glue Data C...
When our team first migrated from the default Hive metastore in Databricks to Unity Catalog, I’ll be honest — it wasn’t something any of us were excited about. Governance tools rarely are. But a fe...