In this article, let us build a small data pipeline using AWS Step Functions and understand where it fits compared with running the same workflow in Airflow. We will use Step Functions to validate ...
In this article, let us understand the Terraform basics that are useful for a data engineer and build a small piece of infrastructure in GCP. We will create a GCS bucket and a BigQuery dataset, con...
In this article, let us set up a simple CI/CD process for a Python data pipeline using GitHub Actions. This is useful when the pipeline code is already in GitHub and we want every change tested bef...
In this article, let us build our first Airflow DAG for an ETL use case using the TaskFlow API. We will read a daily CSV file, validate and transform the records, and load them into Postgres. This ...
In this article, let us compare Apache Iceberg and Delta Lake from a beginner’s point of view. Both formats help us build reliable tables on object storage, but choosing one only from a feature lis...
In this article, let us understand the basics of Delta Lake and build a small pipeline using Apache Spark. A normal data lake is easy to start with because we can write Parquet files directly to ob...
In this article, let us get started with Apache Spark on Databricks by building a small data pipeline in a notebook. We will create compute, load data, clean and aggregate it, and finally write the...
In this article, let us compare AWS Glue and Amazon Athena and see where each service fits in an analytics pipeline. These services are often mentioned together because both work well with data in ...
In this article, let us go through the Apache Spark transformations that I use most often while building data pipelines. Knowing the syntax is useful, but it is equally important to know what Spark...
In this article, let us look at Delta Lake time travel, how to query an older version of a table, and how it can help when a pipeline accidentally updates or deletes data. We will create a small De...