In this article, let us understand the basics of Delta Lake and build a small pipeline using Apache Spark. A normal data lake is easy to start with because we can write Parquet files directly to ob...
In this article, let us get started with Apache Spark on Databricks by building a small data pipeline in a notebook. We will create compute, load data, clean and aggregate it, and finally write the...
In this article, let us compare AWS Glue and Amazon Athena and see where each service fits in an analytics pipeline. These services are often mentioned together because both work well with data in ...
In this article, let us go through the Apache Spark transformations that I use most often while building data pipelines. Knowing the syntax is useful, but it is equally important to know what Spark...
In this article, let us look at Delta Lake time travel, how to query an older version of a table, and how it can help when a pipeline accidentally updates or deletes data. We will create a small De...
In this article, let us understand AWS Glue job bookmarks and build a small example using files in S3. Job bookmarks are useful when a Glue job runs on a schedule and should process only new data i...
In this article, let us look at the difference between Databricks notebooks and Databricks Jobs, and where each one fits when we build a data pipeline. Notebooks are very convenient when we explore...
In this article, let us look at a practical data platform architecture on GCP for a small team. The aim is not to create a platform with every possible feature. We want something that can ingest fi...
Most data pipelines I have worked on start with a simple batch pull — run a query, dump the results, load them somewhere. But at some point, when a table grows to a few hundred million rows scrapin...
In this article, let us look at how to handle retries and idempotency in ETL jobs. If you have built pipelines that process data at any reasonable scale, you have probably seen jobs fail for reason...