When I started building data pipelines on Databricks, the orchestration piece was always the awkward part. You would write your notebooks, get the transformations working, and then you had to figur...
When I first started working with GCP, I kept hearing about Dataflow but wasn’t quite sure where it fit. We had Cloud Functions for light event-driven work, Composer (managed Airflow) for orchestra...
If you have worked with BigQuery for any reasonable amount of time, you have probably stared at a query that scanned a few hundred gigabytes when it really only needed a few megabytes. That is usua...
In a previous article we looked at Terraform for managing data infrastructure on GCP. Terraform is great, but if your team works mostly in AWS and writes Python or TypeScript all day, you might fin...
In this article, let us look at AWS Step Functions and how you can use it to orchestrate your data pipelines. If you have been building data pipelines for a while, you have probably used tools like...
Infrastructure as code has moved from being a niche platform-engineering skill to something every data engineer bumps into. A few years ago, my job stopped at making the pipeline run — someone else...
In this article let us look at how you can use GitHub Actions to set up a simple CI/CD pipeline for your data workflows. If you are building dbt models, Airflow DAGs, or Python-based ETL scripts an...
In this article, we will build a simple but real ETL pipeline using Apache Airflow. If you have been writing cron jobs or one-off scripts to move data around and are looking for something more mana...
If you have spent any time in the data engineering space over the last couple of years, you have probably heard about table formats — Delta Lake, Apache Iceberg, Apache Hudi. They all sound like th...
In this article, let us go through Delta Lake and understand what it actually does for a data engineer. If you have been working with plain Parquet files on a data lake and running into issues like...