If you work with data pipelines long enough, you will eventually hit a point where your Parquet files stop being enough. You need upserts, you need to roll back a bad run, you need to know who chan...
If you have worked with BigQuery for any reasonable amount of time, you have probably hit that moment where you look at the billing dashboard and think — wait, that cannot be right. BigQuery makes ...
If you have been running data pipelines on AWS for a while, you have probably looked at the monthly bill and wondered where half of it came from. It is easy to spin up an EMR cluster, throw data in...
In this article, let us go through the data quality checks that every beginner data team should add to their pipelines. These are not fancy checks that need a dedicated data observability platform....
In this article, let us look at one of those problems that nobody talks about until it hits them in production: schema evolution. You have a pipeline that has been running fine for months. Then som...
In this article let us look at one of the most common yet rarely discussed parts of a data engineer’s job — backfills. If you have worked on data pipelines for any length of time, you have probably...
In this article let us look at dbt (data build tool) and how analytics engineering teams can use it to bring software engineering practices to their data transformations. If your team writes a lot ...
In this article, let us look at how to use AWS Athena to query files sitting in your S3 data lake. If you have been following the data lake series, we already looked at how to design your S3 folder...
If you have worked on building a data lake on S3, you know the question comes up pretty fast — how should I organise the folders? It sounds trivial. You create a few prefixes, drop files in, and mo...
In this article, let us look at what batch and streaming processing actually mean for a data engineer, and more importantly, when you should reach for one over the other. If you are just getting st...