Iceberg vs Delta Lake for beginners
In this article let us understand Apache Iceberg and Delta Lake from a beginner point of view, why these table formats became popular, and when you might choose one over the other. If you are build...
Data Dev Hub
Getting Started with Delta Lake: A Practical Guide for Data Engineers
Getting Started with Delta Lake: A Practical Guide for Data Engineers If you’re in the data engineering space, you’ve probably heard the buzz around Delta Lake. It’s been gaining traction as a go-...
BigQuery Data Transfer Service: What, why and how?
In this article let us uncover the BigQuery Data Transfer service, what is it, when to use it, what its limitations are and a sample terraform code to deploy one onto GCP. What is BigQuery Data Tr...
Productionizing dbt as a Cloud Run Job: Infrastructure Management with Terraform and CI/CD with GitHub Actions - Part 3
This would be the last part of our three-part series to run a dbt job in GCP. In part 1 we went through the setup needed to run a dbt job on our local system. Next, we saw how to create the infrast...
Productionizing dbt as a Cloud Run Job: Infrastructure Management with Terraform and CI/CD with GitHub Actions - Part 2
This is the second part of the three-part series to get a dbt job into production. Previously we saw how to get the dbt job running locally. Our end goal would be to deploy the dbt job as a Cloud R...
Productionizing dbt as a Cloud Run Job: Infrastructure Management with Terraform and CI/CD with GitHub Actions - Part 1
In this multi-part series let us try to get a dbt job running as a Cloud Run job with the infrastructure managed by Terraform and CI/CD with GitHub Actions Part 1: Setup to run the dbt job locally...
Automating Cloud Functions with GCS: A Terraform and GitHub Actions Guide
In a previous article, we saw how to set up Terraform on our local system and manage the resources in GCP. In this article, let us take it one step further and add GitHub Action to the mix. Let us...
Getting started with Terraform with GCP
Infrastructure as code (IaC) has become so popular these days that it is no longer a skill of a particular team like platform engineering or DevOps. When I used to build Apache Spark pipelines usi...
Manage Multiple gcloud accounts in CLI
Manage Multiple gcloud accounts If your organisation has multiple gcloud accounts or you want to manage both your company and your personal gcloud account using the CLI you could follow the below s...
Containerise your Python Flask using Docker and deploy it onto Heroku
Containerise your Python Flask using Docker and deploy it onto Heroku Python Flask application can be directly deployed onto Heroku. But sometimes you would want to have more control on your deplo...