In this article let us see how to handle retries and idempotency in ETL jobs, and why this matters when a pipeline fails halfway and we need to run it again without creating bad data. Most teams st...

In this article, let us look at how to set up a proper CI/CD pipeline for Terraform using GitHub Actions. If you have been running Terraform from your local machine, you might have noticed it works...

In this article, I want to walk through how we approach partitioning for data lake tables. I have seen this done wrong enough times that I think it is worth writing down what actually works in prac...

In this article let us walk through the medallion architecture pattern — landing, bronze, silver, and gold layers — and why teams use this approach when building data lakehouses. If you are coming ...

In this article let us see how to create external tables in BigQuery on top of files stored in GCS, why you might choose this approach, and what limitations you should keep in mind before using it ...

In this article, let us see how to use AWS Step Functions together with AWS Glue for simple orchestration, and why this is often a good choice when you do not want to build a full scheduler or a he...

In this article, let us see how to get started with AWS Glue crawlers, what problem they solve, and why you might want to use them in a simple data lake setup. If you are keeping files in S3 and wa...

In this article, let us understand the basics of Unity Catalog in Databricks, why someone would use it, and how to think about it when you are just getting started. If you have been working with Da...

In this article let us go through how to use Databricks Delta tables in analytics pipelines and why this approach is useful when you want something more reliable than plain parquet files. If you ar...

In this article let us go through some practical basics for reducing BigQuery cost before it turns into a painful surprise in your monthly bill. If you are using BigQuery for analytics, reporting, ...