If you have worked with BigQuery for any reasonable amount of time, you have probably stared at a query that scanned a few hundred gigabytes when it really only needed a few megabytes. That is usua...
In a previous article we looked at Terraform for managing data infrastructure on GCP. Terraform is great, but if your team works mostly in AWS and writes Python or TypeScript all day, you might fin...
In this article, let us look at AWS Step Functions and how you can use it to orchestrate your data pipelines. If you have been building data pipelines for a while, you have probably used tools like...
Infrastructure as code has moved from being a niche platform-engineering skill to something every data engineer bumps into. A few years ago, my job stopped at making the pipeline run — someone else...
In this article let us look at how you can use GitHub Actions to set up a simple CI/CD pipeline for your data workflows. If you are building dbt models, Airflow DAGs, or Python-based ETL scripts an...
In this article, we will build a simple but real ETL pipeline using Apache Airflow. If you have been writing cron jobs or one-off scripts to move data around and are looking for something more mana...
If you have spent any time in the data engineering space over the last couple of years, you have probably heard about table formats — Delta Lake, Apache Iceberg, Apache Hudi. They all sound like th...
In this article, let us go through Delta Lake and understand what it actually does for a data engineer. If you have been working with plain Parquet files on a data lake and running into issues like...
In this article let us walk through what it actually looks like to run Apache Spark on Databricks when you are coming from a self-managed Spark background. If you have been running Spark on EMR, Cl...
In this article, let us go through the Apache Spark transformations that come up again and again in real pipelines. I am not going to list every function in the API. Instead, I will focus on the on...