In this article, let us look at what batch and streaming processing actually mean for a data engineer, and more importantly, when you should reach for one over the other. If you are just getting st...
If you have spent any time in data engineering, you have probably heard the ETL vs ELT debate. It gets brought up in every data platform discussion, and honestly, most of what you read about it is ...
If you have spent any time reading about data lakehouses or Databricks, you have probably heard the term “medallion architecture” thrown around. It sounds fancy, but at its heart the idea is simple...
If you have worked with Databricks for a while, you have probably run notebooks manually or used an external scheduler like Apache Airflow to trigger them. But Databricks has its own orchestration ...
When you are building data pipelines on GCP, you will eventually hit a point where Cloud Functions or simple scheduled scripts don’t cut it anymore. Maybe your CSV files grew from a few MBs to a fe...
In this article let us look at AWS CDK — the Cloud Development Kit — and how data platform teams can use it to manage their infrastructure. If you have been using Terraform or CloudFormation to spi...
In this article, let us look at AWS Step Functions and how we can use it to orchestrate data pipelines. If you have been chaining Lambda functions together or relying on cron jobs to kick off seque...
Infrastructure as code (IaC) has become so common now that it is no longer just a skill for platform engineering or DevOps teams. If you are building data pipelines today, especially on cloud platf...
Why GitHub Actions for Data Pipelines? Most data engineers I know started out running pipelines manually. You write a Spark job, test it on your local, copy the script to a Cloud Storage bucket, t...
If you are getting started with Airflow and want to build your first DAG to move some data around, this article walks you through exactly that. We will build a simple ETL pipeline that extracts dat...