In this article, let us look at how to manage Terraform state when you are working in a small team. If you have used Terraform alone on your local machine, you might have just ignored the terraform...
In this article, let us look at how to think about partitioning when you are building data lake tables. Partitioning is one of those things that seems straightforward when you read the docs, but wh...
If you have spent any time around Databricks or lakehouse architecture, you have probably heard the terms landing, bronze, silver, and gold thrown around. They sound like something out of a fantasy...
In this article let us look at BigQuery external tables — what they are, when they make sense, and how to set one up over data sitting in Google Cloud Storage (GCS). If you have CSV, JSON, Parquet,...
In this article let us look at how to use AWS Step Functions to orchestrate Glue jobs. If you have been running Glue jobs individually or glueing them together with Lambda functions and CloudWatch ...
In this article, let us look at AWS Glue crawlers — what they do, how to set one up, and where they fit in a data pipeline. If you have been dumping files into S3 and then wondered how to make them...
If your team has been using Databricks for a while, you probably started with multiple workspaces — one for dev, one for prod, maybe one per squad. Each workspace has its own Hive Metastore, its ow...
In this article, let us look at Databricks Delta tables — what they are, why you would use them in an analytics pipeline, and how to set them up from the ground up. If you have worked with data lak...
BigQuery is one of those services where the pricing model can surprise you if you are not paying attention. A query that runs in 3 seconds today might scan 2 TB of data and cost you a few dollars t...
In this article let us look at something that sneaks up on every data team: AWS costs. Not the big, obvious ones like provisioned Redshift clusters or Kinesis shards. The quiet ones. The Glue job t...