Terraform for Data Engineers: A Practical Guide to Managing Infrastructure as Code

Infrastructure as code (IaC) has become so common now that it is no longer just a skill for platform engineering or DevOps teams. If you are building data pipelines today, especially on cloud platforms like GCP, AWS, or Azure, you will eventually need to manage your own infrastructure. The days where a separate team would spin up clusters for you are fading.

In this article, we will walk through the basics of Terraform from a data engineer’s perspective. We will cover core concepts, set up Terraform with GCP, provision a real data stack (BigQuery dataset, GCS bucket, and a Pub/Sub topic), and talk about what changes when you move from a local demo to a production pipeline.

Why Data Engineers Should Care About Terraform

When I worked on Cloudera-based Spark pipelines a few years ago, my job stopped at making the pipeline run. Someone else managed the Hadoop cluster, the YARN queues, the HDFS directories. All of that changed when we moved to GCP. Suddenly, every data engineer had to own their infrastructure — storage buckets, BigQuery datasets, Pub/Sub topics, Cloud Functions. If you wanted a new dataset, you wrote the Terraform for it.

There are a few reasons this shift happened:

1. Speed. If you have to raise a ticket and wait for another team to create a bucket, your development cycle slows down. With IaC, you open a PR, it gets reviewed, and the resource is created automatically.
2. Reproducibility. You can spin up an identical dev environment in a separate project without clicking through the GCP console for an hour.
3. Auditability. Every resource change is tracked in git. You know who changed what and why.

Terraform is not the only IaC tool out there, but it is the most widely adopted. It is cloud-agnostic, has a massive provider ecosystem, and uses a declarative language (HCL) that is easier to read than YAML-based alternatives once you get used to it.

Core Concepts (Without the Jargon Overload)

Before we write any code, let us cover the four things you need to understand.

Providers

A provider is the plugin that lets Terraform talk to a specific platform — GCP, AWS, Azure, Datadog, PagerDuty, whatever. You declare which providers you need, and Terraform downloads them when you run terraform init.

terraform {
  required_providers {
    google = {
      source  = "hashicorp/google"
      version = "~> 5.0"
    }
  }
}

Resources

Resources are the actual things you want to create. A GCS bucket, a BigQuery dataset, a Pub/Sub topic — each is a resource block.

resource "google_storage_bucket" "raw_data" {
  name     = "my-project-raw-data"
  location = "australia-southeast1"
}

State

Terraform keeps a state file that maps your configuration to the real resources it created. Without state, Terraform would not know what it already built. By default, this is a local file called terraform.tfstate, but for anything collaborative, you should store it remotely (more on this later).

Modules

Modules are reusable groups of resources. If you find yourself creating the same set of resources for every data pipeline — say, a GCS bucket, a BigQuery dataset, and a service account — you can wrap them into a module and call it with different parameters.

Comparison: Terraform vs Manual Console vs Other IaC Tools

Approach	Good For	Not Good For
GCP Console (ClickOps)	Quick experiments, one-off resources	Anything that needs to be reproduced, reviewed, or audited
gcloud CLI scripts	Simple automation, learning	Gets messy with many resources, no state tracking
Terraform	Multi-resource stacks, teams, production	Very simple use cases where a script is enough
Pulumi	Teams that prefer general-purpose languages (Python, TypeScript)	Smaller community, fewer examples online
Deployment Manager (GCP native)	GCP-only environments	Lock-in, less community support compared to Terraform

For most data engineering teams, Terraform hits the sweet spot. It is mature, well-documented, and there are examples for almost every GCP service you would use.

Hands-On: Provisioning a Data Engineering Stack on GCP

Let us build something real. We will provision three resources that almost every data pipeline touches: a GCS bucket for raw file storage, a BigQuery dataset for analytics, and a Pub/Sub topic for event-driven triggers.

Prerequisites

You need the following before we start:

• Terraform installed (brew install terraform on macOS)
• A GCP project with billing enabled
• A service account with the right permissions (Storage Admin, BigQuery Data Editor, Pub/Sub Editor)
• The service account key JSON file downloaded to your local system

Create that service account from IAM & Admin → Service Accounts in the GCP console. Give it the roles I mentioned above, create a key in JSON format, and save it somewhere safe — not inside your repo.

Step 1: Project Structure

Create a directory for your Terraform code. A simple structure looks like this:

terraform/
├── main.tf          # provider and resource definitions
├── variables.tf     # input variables
├── terraform.tfvars # variable values (gitignored if sensitive)
└── outputs.tf       # output values

Step 2: Define the Provider

In main.tf, configure the Google provider:

terraform {
  required_providers {
    google = {
      source  = "hashicorp/google"
      version = "~> 5.0"
    }
  }
}

provider "google" {
  credentials = file(var.gcp_sa_key_path)
  project     = var.project_id
  region      = var.region
}

Step 3: Variables

In variables.tf:

variable "project_id" {
  description = "GCP Project ID"
  type        = string
}

variable "region" {
  description = "GCP region"
  type        = string
  default     = "australia-southeast1"
}

variable "gcp_sa_key_path" {
  description = "Path to the service account key file"
  type        = string
}

In terraform.tfvars (add this to .gitignore):

project_id      = "my-data-project"
region          = "australia-southeast1"
gcp_sa_key_path = "/secure/path/keys.json"

Step 4: Define the Resources

Now the interesting part. Add these to main.tf:

# GCS bucket for raw data landing
resource "google_storage_bucket" "raw_data" {
  name          = "${var.project_id}-raw-data"
  location      = var.region
  force_destroy = false

  versioning {
    enabled = true
  }

  lifecycle_rule {
    condition {
      age = 90
    }
    action {
      type = "Delete"
    }
  }
}

# BigQuery dataset
resource "google_bigquery_dataset" "analytics" {
  dataset_id = "analytics_data"
  location   = var.region

  labels = {
    environment = "dev"
    managed_by  = "terraform"
  }
}

# Pub/Sub topic for pipeline events
resource "google_pubsub_topic" "pipeline_events" {
  name = "pipeline-events"

  message_retention_duration = "86400s" # 24 hours
}

A few things to call out here. The GCS bucket has versioning enabled and a lifecycle rule that deletes objects older than 90 days — this is something you would want in a real data lake to manage costs. The BigQuery dataset has labels, which become really useful when you have dozens of datasets across multiple environments and need to track what is what.

Step 5: Apply

Run the following:

terraform init      # downloads the Google provider
terraform plan      # shows what will be created
terraform apply     # creates the resources

After terraform apply, Terraform will show you what it created and write it to the state file. You can now see your bucket, dataset, and topic in the GCP console — all created in under a minute from a few lines of HCL.

State Management: The Thing Nobody Talks About Enough

When you run Terraform from your laptop, the state file sits on your local filesystem. That works for learning, but it breaks down fast in a team. If two people run terraform apply from different machines with different state files, you will end up with conflicting or orphaned resources.

The fix is remote state. For GCP, the simplest approach is storing state in a GCS bucket:

terraform {
  backend "gcs" {
    bucket = "my-project-terraform-state"
    prefix = "data-engineering/state"
  }
}

You create this bucket once manually (or with a separate bootstrap Terraform config), and from then on, all team members share the same state. Terraform also supports state locking through the GCS backend, so two people cannot run apply at the same time and corrupt the state.

Limitations and Things to Watch Out For

1. State drift. Terraform only knows what is in its state file. If someone manually changes a resource in the console, Terraform will not know until the next plan. In production, lock down console access and enforce that all changes go through Terraform.

2. Secrets in state. Even if you use variables for sensitive values, the state file can contain them in plain text (especially for resources that store secrets, like database passwords). Treat your state file like a secret — store it in a private bucket with restricted access.

3. Provider lag. GCP releases new features faster than the Terraform provider updates. You might find that a newer BigQuery feature or a specific Pub/Sub setting is not yet available in the provider. Check the provider changelog before assuming something is supported.

4. Import pain. If you already have resources created manually, you need to terraform import them one by one to bring them under management. There is no “import everything” button. For large existing environments, this can take a while.

5. Destroy is scary. terraform destroy tears down everything. In a production project, I recommend setting prevent_destroy = true on critical resources like production datasets or buckets with customer data.

What Changes in Production

The example above works for a personal project or a dev environment. Here is what you would add before using it in production:

• Remote state with a GCS backend. Non-negotiable for any shared environment.
• CI/CD pipeline (GitHub Actions, Cloud Build). You do not run terraform apply from your laptop in production. A PR merge triggers a plan, and an approval step triggers the apply.
• Separate state per environment. A dev, staging, and prod folder or workspace, each with its own state file and its own GCP project.
• IAM bindings as code. Instead of manually assigning roles, define them in Terraform so access control is versioned and reviewable.
• Terragrunt for DRY configs. If you have many environments with similar setups, Terragrunt helps reduce duplication. Plain Terraform works fine for simpler setups though.
• Service account key rotation. Long-lived keys are a security risk. Use Workload Identity Federation when running from CI/CD so you do not need keys at all.

Wrapping Up

Terraform is one of those tools that feels like overhead when you are just trying to get a pipeline working. But once you have used it on a real project, going back to clicking through the console or writing ad-hoc gcloud scripts feels messy and error-prone.

Start small. Write Terraform for your next new bucket or dataset. Store the state remotely. Add it to CI/CD. Over time, you will build up a library of modules that makes spinning up data infrastructure faster and more reliable than the old way ever was.

We covered the basics here. In a future article, we will look at how to tie Terraform into a GitHub Actions pipeline so that every infrastructure change goes through the same review and deploy process as your application code.