Getting Started with GitHub Actions for Data Pipeline CI/CD — A Practical Guide

Why GitHub Actions for Data Pipelines?

Most data engineers I know started out running pipelines manually. You write a Spark job, test it on your local, copy the script to a Cloud Storage bucket, then go to the console and trigger the job. Maybe you send the output to a colleague on Slack. That works for a while — until you have twelve pipelines, three environments, and someone accidentally deploys a broken transformation to production on a Friday evening.

CI/CD for data pipelines isn’t talked about as much as CI/CD for backend services, but the problems are the same. You want to know that your code works before it reaches production. You want changes to be traceable. You want rollbacks to be straightforward. You don’t want to remember which GCS path has the correct version of your Glue script.

GitHub Actions gives you a way to do all of this without managing a separate CI/CD server. Since your code already lives in GitHub, the workflow definitions live right next to it. In this article, we’ll walk through setting up GitHub Actions for a typical data engineering setup — Spark jobs, dbt models, and infrastructure managed through Terraform. We’ll keep it practical and cover what actually matters day to day.

Setup Overview

Here is what we’ll build:

• A lint and test workflow that runs on every pull request — checks Python/SQL formatting, runs unit tests on transformation logic
• A build and deploy workflow that packages a Spark job, uploads it to GCS, and updates a Cloud Composer DAG
• A Terraform plan/apply workflow for infrastructure changes
• A dbt CI workflow that runs dbt test and dbt build against a slimmed-down CI schema

All of these will run directly from GitHub Actions. No Jenkins, no Cloud Build (though you could swap that in later if needed).

Prerequisites

Before we start writing workflows, make sure you have:

1. A GitHub repository with your pipeline code
2. A GCP project (I’m using GCP for examples, but the concepts apply to AWS/Azure too)
3. A service account with appropriate permissions — Storage Admin for GCS uploads, Composer Admin if you’re updating DAGs
4. The service account key stored as a GitHub secret (we’ll call it GCP_SA_KEY)

A note on service account keys: long-lived keys are not ideal for production. If your organization supports Workload Identity Federation, use that instead. We’ll go with keys here because they’re simpler to set up and understand first.

Step 1: Lint and Test on Pull Requests

This is the workflow you’ll run the most, so let’s start here. Create .github/workflows/pr-checks.yml:

name: PR Checks

on:
  pull_request:
    branches: [main]

jobs:
  lint-and-test:
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.11'

      - name: Install dependencies
        run: |
          pip install -r requirements.txt
          pip install flake8 black pytest

      - name: Lint with flake8
        run: flake8 src/ --max-line-length=100

      - name: Check formatting with black
        run: black --check src/

      - name: Run unit tests
        run: pytest tests/ -v

This is straightforward — it checks out the code, installs dependencies, lints, checks formatting, and runs tests. The key thing is that this runs on every pull request targeting main. Nobody merges without these passing.

For a Spark pipeline, your unit tests don’t need a real Spark cluster. Use pyspark with local mode in your test fixtures:

# tests/conftest.py
import pytest
from pyspark.sql import SparkSession

@pytest.fixture(scope="session")
def spark():
    return SparkSession.builder \
        .master("local[2]") \
        .appName("unit-tests") \
        .config("spark.sql.shuffle.partitions", "2") \
        .getOrCreate()

Then your tests can use the fixture to create DataFrames and test transformations without needing a cluster.

Step 2: Build and Deploy a Spark Job

Once a PR is merged to main, you want the pipeline to deploy automatically. Here’s a workflow that packages a PySpark job and deploys it:

name: Deploy Spark Job

on:
  push:
    branches: [main]
    paths:
      - 'spark-jobs/**'

jobs:
  build-and-deploy:
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v4

      - name: Authenticate to GCP
        uses: google-github-actions/auth@v2
        with:
          credentials_json: $

      - name: Package Spark job
        run: |
          cd spark-jobs/customer_etl
          zip -r ../../customer_etl.zip .

      - name: Upload to GCS
        run: |
          gsutil cp customer_etl.zip gs://my-pipeline-artifacts/spark-jobs/customer_etl-$.zip
          gsutil cp customer_etl.zip gs://my-pipeline-artifacts/spark-jobs/customer_etl-latest.zip

      - name: Update Cloud Composer DAG
        run: |
          gcloud composer environments storage dags import \
            --environment my-composer-env \
            --location us-central1 \
            --source dags/customer_etl_dag.py \
            --destination /home/airflow/gcs/dags/

A few things worth pointing out here:

• The paths filter means this only runs when something in spark-jobs/ changes. No point redeploying if someone only updated a README.
• We upload two copies — one tagged with the commit SHA and one as -latest. The SHA-tagged version lets you roll back to an exact point in time. The -latest version is what your DAG references if you don’t want to update DAG code on every deploy.
• We’re using google-github-actions/auth@v2 instead of manually copying the key. This is the recommended way.

Step 3: Terraform Plan and Apply

If your infrastructure is managed with Terraform (and it should be), you can use GitHub Actions to plan on PRs and apply on merge:

name: Terraform

on:
  pull_request:
    paths:
      - 'terraform/**'
  push:
    branches: [main]
    paths:
      - 'terraform/**'

jobs:
  terraform:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      pull-requests: write

    steps:
      - uses: actions/checkout@v4

      - name: Setup Terraform
        uses: hashicorp/setup-terraform@v3
        with:
          terraform_version: "1.7"

      - name: Terraform Format
        run: terraform fmt -check -recursive terraform/

      - name: Terraform Init
        run: |
          cd terraform
          terraform init

      - name: Terraform Plan
        if: github.event_name == 'pull_request'
        run: |
          cd terraform
          terraform plan -no-color -out=tfplan
        
      - name: Terraform Apply
        if: github.event_name == 'push' && github.ref == 'refs/heads/main'
        run: |
          cd terraform
          terraform apply -auto-approve tfplan

For Terraform state, use a GCS backend — never commit state files to git. Your backend.tf would look like:

terraform {
  backend "gcs" {
    bucket = "my-terraform-state-bucket"
    prefix = "data-pipelines"
  }
}

Step 4: dbt CI with Slim CI

If you use dbt, GitHub Actions can run tests against a dedicated CI schema so you catch model issues before they hit production:

name: dbt CI

on:
  pull_request:
    paths:
      - 'dbt/**'

jobs:
  dbt-test:
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v4

      - name: Setup Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.11'

      - name: Install dbt
        run: pip install dbt-bigquery

      - name: Run dbt tests
        env:
          DBT_PROFILES_DIR: ./dbt
        run: |
          cd dbt
          dbt deps
          dbt build --select state:modified+ --defer --state ./prod-artifacts \
            --target ci

The state:modified+ selector only builds models that changed and their downstream dependents, which keeps CI fast even with hundreds of models. You’ll need a separate step to pull the production manifest artifacts for the --state flag to work — storing those as build artifacts or in GCS works well.

Comparison: CI/CD Approaches for Data Pipelines

Here’s how GitHub Actions stacks up against other options I’ve seen teams use:

Approach	Setup Effort	Cost	Flexibility	Best For
GitHub Actions	Low to medium	Free tier covers most use cases	High — write any workflow	Teams already on GitHub, small to medium pipelines
Cloud Build / CodeBuild	Medium	Pay per build minute	Medium — good GCP/AWS integration	Heavily cloud-native setups
Jenkins (self-hosted)	High	Server cost only	Very high	Large orgs with complex, multi-repo pipelines
Manual deploys	None	Free	N/A	Tiny teams, low stakes — but you’ll regret it eventually

GitHub Actions hits a sweet spot for most data teams. You’re not maintaining a server, the YAML syntax is readable, and the marketplace has actions for most things you’d need.

Things to Watch Out For

Runner resource limits. The free runners have 7GB of RAM and 14GB of disk. If your Spark unit tests load large datasets, they might fail. Use sampled or synthetic test data instead.

Secret management. GitHub Secrets are encrypted and reasonably secure, but they’re still static values. Rotate service account keys regularly, and move to Workload Identity Federation when you can. The setup is more involved but eliminates long-lived credentials.

Workflow sprawl. It’s easy to end up with twenty workflow files that all do slightly different things. Use reusable workflows (.github/workflows/_shared.yml) once you find yourself copying YAML. For example, a shared workflow for GCP auth + Python setup saves you 15 lines per workflow.

Cold starts. Serverless runners experience occasional cold start delays — usually 10-30 seconds. Not a dealbreaker for CI/CD, but noticeable if you’re iterating quickly.

GitHub Actions is not a scheduler. Don’t try to replace Airflow with GitHub Actions. You can use scheduled workflows (cron triggers) for lightweight periodic tasks, but it’s not designed for orchestrating multi-step pipeline DAGs with retries, backfills, and SLAs.

What Changes for a Production Setup

For a simple demo, the workflows above are enough to get going. Here’s what I’d add for a real production environment:

1. Environment approvals. Use GitHub Environments to require manual approval before deploying to production. You don’t want a merged PR to automatically touch your prod Composer environment at 2 AM.

2. Matrix builds. If your pipeline needs to run on Python 3.10, 3.11, and 3.12, use a matrix strategy instead of hardcoding one version.

3. Integration tests against a staging environment. Unit tests cover logic, but they don’t catch GCP API changes or IAM misconfigurations. Have a workflow that deploys to a staging project and runs an end-to-end test.

4. Notifications. Add a Slack or email notification step on failure. Nobody checks the Actions tab unless you tell them to.

5. Build caching. Cache your pip packages with actions/cache to speed up runs. A typical data pipeline venv can take 2-3 minutes to install from scratch.

Here’s a quick cache setup you can drop into any workflow:

- name: Cache pip packages
  uses: actions/cache@v4
  with:
    path: ~/.cache/pip
    key: $-pip-$
    restore-keys: |
      $-pip-

Wrapping Up

GitHub Actions isn’t the fanciest CI/CD tool out there, but it does the job for data pipelines without adding complexity you don’t need. Start with the PR checks workflow — that alone catches most issues before they reach production. Add the deploy workflow once you’re confident the tests are solid. Bring in Terraform and dbt workflows as your project grows.

The best CI/CD setup is the one your team actually uses. GitHub Actions has a low enough barrier that people won’t skip it, and that’s honestly most of the battle.