Terraform Basics for Data Engineering Infrastructure
In this article, let us understand the Terraform basics that are useful for a data engineer and build a small piece of infrastructure in GCP. We will create a GCS bucket and a BigQuery dataset, connect the resources using Terraform, and see the commands used to review and apply the changes. This approach is useful when a pipeline needs repeatable infrastructure instead of a list of manual console steps.
A data pipeline normally depends on more than Python or SQL code. It might need buckets, datasets, service accounts, PubSub topics, or scheduled jobs. Creating these manually is fine for a quick experiment, but it becomes difficult to reproduce the same setup in development, test, and production. Terraform lets us keep that infrastructure as code and review its changes in Git.
How Terraform fits into a data project
Terraform reads configuration files, compares the configuration with the infrastructure it already manages, and produces a plan. Once we approve the plan, Terraform calls the cloud provider APIs to make the changes.
| Terraform item | What it means in our project |
|---|---|
| Provider | The cloud or service Terraform connects to |
| Resource | Something Terraform creates, such as a bucket |
| Variable | An input that can change between environments |
| Output | A useful value shown after deployment |
| State | Terraform’s record of the resources it manages |
For our use case, we will use the Google provider. The same Terraform concepts apply to AWS and Azure, though the resource names and authentication methods are different.
Prerequisites
We need Terraform, the Google Cloud CLI, and a GCP project where we are allowed to create storage buckets and BigQuery datasets. Check the installations with:
1
2
terraform version
gcloud version
For this local example, we can authenticate using Application Default Credentials:
1
gcloud auth application-default login
This opens a browser and stores local credentials for the Google provider to use. We should not commit credential files to Git. In a production CI/CD workflow, I would use Workload Identity Federation instead of downloading a long-lived service account key.
Create the Terraform files
Let us create a new directory and add three files:
1
2
3
4
terraform-data-platform/
├── main.tf
├── variables.tf
└── outputs.tf
In main.tf, configure Terraform and the Google provider first:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
terraform {
required_version = ">= 1.6.0"
required_providers {
google = {
source = "hashicorp/google"
version = "~> 6.0"
}
}
}
provider "google" {
project = var.project_id
region = var.region
}
Pinning a provider version avoids an unexpected major upgrade when another engineer runs the code later. The ~> 6.0 constraint allows compatible updates within version 6.
Now add a bucket for incoming files and a BigQuery dataset for processed data:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
resource "google_storage_bucket" "landing" {
name = "${var.project_id}-data-landing"
location = var.region
uniform_bucket_level_access = true
force_destroy = false
lifecycle_rule {
condition {
age = 30
}
action {
type = "SetStorageClass"
storage_class = "NEARLINE"
}
}
}
resource "google_bigquery_dataset" "analytics" {
dataset_id = "analytics"
location = var.bigquery_location
labels = {
managed_by = "terraform"
environment = var.environment
}
}
I have kept force_destroy as false. If files exist in the bucket, Terraform will refuse to delete it. This can feel inconvenient during a demo, but it is a useful guardrail because destroying a bucket should not silently remove pipeline data.
Add the inputs in variables.tf:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
variable "project_id" {
type = string
description = "GCP project ID"
}
variable "region" {
type = string
default = "australia-southeast1"
}
variable "bigquery_location" {
type = string
default = "australia-southeast1"
}
variable "environment" {
type = string
default = "dev"
}
Finally, add values that may be needed by pipeline code in outputs.tf:
1
2
3
4
5
6
7
output "landing_bucket_name" {
value = google_storage_bucket.landing.name
}
output "analytics_dataset_id" {
value = google_bigquery_dataset.analytics.dataset_id
}
Initialize and review the plan
Run Terraform initialization from the directory containing the files:
1
2
3
terraform init
terraform fmt -check
terraform validate
terraform init downloads the provider and creates a dependency lock file. We should commit .terraform.lock.hcl because it records the provider version selected for the project. We should not commit the .terraform directory or local state files.
Create a plan by passing the project ID:
1
2
3
terraform plan \
-var="project_id=my-gcp-project" \
-out=tfplan
Read the plan before applying it. Terraform will show two resources to add for a new project. If it also shows an unexpected delete or replacement, stop and understand why. A plan is much more useful as a review step than treating terraform apply as just another command.
Apply the saved plan:
1
terraform apply tfplan
After the deployment, terraform output displays the bucket and dataset names. We can also verify them using the cloud CLI:
1
2
gcloud storage buckets describe gs://my-gcp-project-data-landing
bq show my-gcp-project:analytics
Understanding Terraform state
Terraform writes a terraform.tfstate file locally by default. This file maps the resource blocks in our code to actual cloud resources. It may also contain sensitive values, so it should not be committed to Git.
Local state is acceptable while learning, but it is not suitable when a team or CI/CD workflow runs Terraform. In production I would configure a remote GCS backend, enable bucket versioning, restrict access to the deployment identity, and separate state by environment. A backend configuration could look like this:
1
2
3
4
5
6
terraform {
backend "gcs" {
bucket = "company-terraform-state"
prefix = "data-platform/dev"
}
}
The state bucket must exist before Terraform can use it. It is common to create this bootstrap resource separately rather than managing it from the same state stored inside the bucket.
Making the example production ready
This example creates resources, but a real data platform needs a few more decisions. Use separate projects or at least separate state for dev and production. Keep IAM permissions small instead of assigning broad editor roles. Add labels for ownership and cost reporting. Enable logging, retention, and encryption settings based on the data classification.
We should also run terraform fmt, terraform validate, and terraform plan in pull requests. The apply step should run only from a protected branch and use short-lived cloud authentication. For shared modules, pin versions so that production does not change just because a module’s main branch changed.
One thing to be careful about is changing arguments that force resource replacement. Renaming a bucket, changing some locations, or changing a resource address can result in Terraform planning to destroy and recreate it. For data infrastructure, that replacement may mean downtime or data loss. Always inspect the plan and use migration commands such as terraform state mv or moved blocks when refactoring resource names.
Clean up the demo
When the resources are no longer needed, review a destroy plan first:
1
2
terraform plan -destroy -var="project_id=my-gcp-project"
terraform destroy -var="project_id=my-gcp-project"
The bucket must be empty because we set force_destroy to false. This is intentional. Remove only the test files that are safe to delete, then run the destroy command again.
Terraform gives data engineers a repeatable way to create the cloud resources around a pipeline. Starting with a bucket and a dataset makes the provider, variables, plan, apply, outputs, and state easier to understand. Once these basics are working, we can use the same structure for service accounts, PubSub topics, orchestration services, and the rest of the data platform.