AWS CDK basics for data platform teams
In this article, let us see how to get started with AWS CDK for data platform work and why you might want to use it instead of creating resources manually from the AWS Console. If your team builds ...
Data Dev Hub
Using AWS Step Functions to Orchestrate a Data Pipeline
In this article let us see how to use AWS Step Functions for orchestrating a simple data pipeline and why this approach is useful when you want something more structured than a set of Lambda trigge...
Terraform basics for data engineering infrastructure
Infrastructure as code is no longer something only a platform or DevOps team needs to care about. In many data engineering projects, the same person who builds a pipeline also needs to provision th...
Using GitHub Actions for simple data pipeline CI/CD
In this article, let us see how to use GitHub Actions for a simple data pipeline CI/CD setup and why this approach is useful for a small team. If you have a lightweight ETL job, a dbt project, or a...
Building your first Airflow DAG for ETL
In this article, let us build our first Apache Airflow DAG for a simple ETL pipeline. If you are starting with data engineering, you will quickly notice that writing the transformation code is only...
Iceberg vs Delta Lake for beginners
In this article let us understand Apache Iceberg and Delta Lake from a beginner point of view, why these table formats became popular, and when you might choose one over the other. If you are build...
Getting Started with Delta Lake: A Practical Guide for Data Engineers
Getting Started with Delta Lake: A Practical Guide for Data Engineers If you’re in the data engineering space, you’ve probably heard the buzz around Delta Lake. It’s been gaining traction as a go-...
BigQuery Data Transfer Service: What, why and how?
In this article let us uncover the BigQuery Data Transfer service, what is it, when to use it, what its limitations are and a sample terraform code to deploy one onto GCP. What is BigQuery Data Tr...
Productionizing dbt as a Cloud Run Job: Infrastructure Management with Terraform and CI/CD with GitHub Actions - Part 3
This would be the last part of our three-part series to run a dbt job in GCP. In part 1 we went through the setup needed to run a dbt job on our local system. Next, we saw how to create the infrast...
Productionizing dbt as a Cloud Run Job: Infrastructure Management with Terraform and CI/CD with GitHub Actions - Part 2
This is the second part of the three-part series to get a dbt job into production. Previously we saw how to get the dbt job running locally. Our end goal would be to deploy the dbt job as a Cloud R...