In this article, let us see how to handle schema evolution in a data pipeline without breaking all the jobs that depend on it. This becomes important when your source system adds a column, renames ...
In this article let us see how to build reliable backfills in data pipelines, why we need them, and what things usually break when we run them in a hurry. Backfills sound simple at first. We missed...
In this article, let us see the basics of dbt and why analytics engineering teams use it so often. If your team is writing SQL transformations directly in the warehouse, copying queries between not...
In this article, let us see how to use Amazon Athena to query files from a data lake without building a separate database server or Spark job. This approach is useful when we already have data land...
In this article, let us see how to design folder structure in an S3-based data lake, why it matters early, and what kind of layout usually works well for a simple project without making production ...
In this article, let us understand what batch processing and streaming mean in data engineering, when to use each one, and why both still matter in real projects. If you are just getting started, t...
In this article let us understand ETL and ELT with practical examples, why teams choose one over the other, and what changes when we move from a small demo pipeline to a production data platform. I...
In this article let us see medallion architecture explained in a simple way, why teams use it, and how you can implement it without turning a small pipeline into a complicated platform project. If ...
In this article let us see how to use Databricks Workflows for scheduled jobs, why you might choose it over bringing in a separate orchestrator, and what it looks like in a simple real project. If ...
In this article let us see how to get started with GCP Dataflow for a simple batch pipeline, and why you might choose this approach when you want Google to manage most of the heavy lifting for runn...