Most data incidents in batch systems are not caused by a lack of compute. They are caused by weak orchestration design. I have seen pipelines where jobs run successfully, yet data still fails bus...
Many pipeline incidents are not infrastructure failures. They are expectation failures. Example: producer changes a field type downstream SQL silently breaks dashboards and features becom...
If you’re building a new data platform on AWS, medallion architecture (bronze/silver/gold) is one of the cleanest patterns to start with. But many teams either: under-design it (everything in ...
In this article let us uncover the BigQuery Data Transfer service, what is it, when to use it, what its limitations are and a sample terraform code to deploy one onto GCP. What is BigQuery Data Tr...
When teams move to AWS data platforms, one of the first architecture debates is: Should we do everything in Glue? Can Athena replace most transforms? Where does dbt actually fit? The wron...
This would be the last part of our three-part series to run a dbt job in GCP. In part 1 we went through the setup needed to run a dbt job on our local system. Next, we saw how to create the infrast...
This is the second part of the three-part series to get a dbt job into production. Previously we saw how to get the dbt job running locally. Our end goal would be to deploy the dbt job as a Cloud R...
In this multi-part series let us try to get a dbt job running as a Cloud Run job with the infrastructure managed by Terraform and CI/CD with GitHub Actions Part 1: Setup to run the dbt job locally...
In a previous article, we saw how to set up Terraform on our local system and manage the resources in GCP. In this article, let us take it one step further and add GitHub Action to the mix. Let us...
Infrastructure as code (IaC) has become so popular these days that it is no longer a skill of a particular team like platform engineering or DevOps. When I used to build Apache Spark pipelines usi...