Using GitHub Actions for Simple Data Pipeline CI/CD

In this article, let us set up a simple CI/CD process for a Python data pipeline using GitHub Actions. This is useful when the pipeline code is already in GitHub and we want every change tested before merging, then deployed in a repeatable way after it reaches the main branch. We will keep the example small, but the same structure can be expanded for Airflow DAGs, AWS Glue jobs, Databricks notebooks, or container-based pipelines.

For our use case, the pipeline reads a CSV file, removes rows without a customer ID, calculates the total amount, and writes a Parquet file. We will use pytest for validation and copy the packaged code to an S3 bucket during deployment.

What CI and CD mean for a data pipeline

A data pipeline deployment can mean uploading a Python file to S3, publishing a container image, or synchronising DAG files to an Airflow environment. For this example, we will split the process into two workflows.

Workflow	Trigger	What it does
CI	Pull request	Installs dependencies, checks formatting, and runs tests
CD	Push to main	Repeats tests, packages the pipeline, and uploads it to S3

I prefer keeping CI and CD separate because a pull request should never deploy anything. It also makes permissions easier to understand since only the deployment workflow needs cloud access.

1. Create the sample pipeline

Let us use the below project structure.

data-pipeline/
├── .github/workflows/
├── pipeline/
│   ├── __init__.py
│   └── transform.py
├── tests/
│   └── test_transform.py
└── requirements.txt

Our transformation is intentionally simple.

# pipeline/transform.py
import pandas as pd

def transform_orders(input_path: str, output_path: str) -> None:
    orders = pd.read_csv(input_path)
    orders = orders.dropna(subset=["customer_id"]).copy()
    orders["total_amount"] = orders["quantity"] * orders["unit_price"]
    orders.to_parquet(output_path, index=False)

The requirements.txt file contains both runtime and test packages for this demo.

pandas==2.2.3
pyarrow==17.0.0
pytest==8.3.3
ruff==0.7.1

In a bigger project, I would keep runtime and development dependencies in separate files or use a tool such as Poetry. The important part is to pin versions so a package update does not suddenly break the workflow.

2. Add a useful pipeline test

A workflow that only checks whether Python imports correctly does not give us much confidence. Let us test the actual data behaviour.

# tests/test_transform.py
import pandas as pd
from pipeline.transform import transform_orders

def test_transform_orders(tmp_path):
    input_file = tmp_path / "orders.csv"
    output_file = tmp_path / "orders.parquet"

    pd.DataFrame([
        {"customer_id": "C001", "quantity": 2, "unit_price": 12.5},
        {"customer_id": None, "quantity": 1, "unit_price": 8.0},
    ]).to_csv(input_file, index=False)

    transform_orders(str(input_file), str(output_file))
    result = pd.read_parquet(output_file)

    assert len(result) == 1
    assert result.iloc[0]["customer_id"] == "C001"
    assert result.iloc[0]["total_amount"] == 25.0

This checks the two rules that matter: invalid customer rows are removed and the amount is calculated correctly. For a real pipeline, I would also test empty files, unexpected column types, duplicate records, and late-arriving data.

3. Create the pull request workflow

Create .github/workflows/ci.yml with the following configuration.

name: Pipeline CI

on:
  pull_request:
    branches: [main]

permissions:
  contents: read

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - name: Check out repository
        uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.12"
          cache: pip

      - name: Install dependencies
        run: pip install -r requirements.txt

      - name: Check code
        run: ruff check pipeline tests

      - name: Run tests
        run: pytest -q

When a pull request targets main, GitHub creates a clean runner, installs Python, and runs the checks. The pip cache reduces downloads but does not cache the complete installed environment.

After committing this file, open a pull request with a deliberately failing assertion once. We could see that the merge check fails and the output is available in the Actions tab. Once branch protection requires Pipeline CI, the change cannot be merged until the job passes.

4. Authenticate to AWS without access keys

The deployment workflow needs access to S3. We could store an AWS access key in GitHub Secrets, but a long-lived key is not a good production setup. GitHub Actions supports OpenID Connect, which allows AWS to issue short-lived credentials for each workflow run.

Create an IAM OIDC provider for token.actions.githubusercontent.com, then create a role whose trust policy limits access to the repository and main branch. A simplified condition looks like this.

{
  "Condition": {
    "StringEquals": {
      "token.actions.githubusercontent.com:aud": "sts.amazonaws.com",
      "token.actions.githubusercontent.com:sub":
        "repo:my-org/data-pipeline:ref:refs/heads/main"
    }
  }
}

The role only needs permission to write to the required S3 prefix. Avoid giving the workflow access to every bucket. Add the role ARN as a repository variable named AWS_DEPLOY_ROLE_ARN.

5. Create the deployment workflow

Now create .github/workflows/deploy.yml.

name: Pipeline CD

on:
  push:
    branches: [main]

permissions:
  contents: read
  id-token: write

concurrency:
  group: pipeline-production
  cancel-in-progress: false

jobs:
  deploy:
    runs-on: ubuntu-latest
    environment: production
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
          cache: pip

      - name: Install dependencies and test
        run: |
          pip install -r requirements.txt
          ruff check pipeline tests
          pytest -q

      - name: Create deployment package
        run: |
          mkdir build
          cp -R pipeline build/
          tar -czf orders-pipeline-$.tar.gz -C build .

      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: $
          aws-region: ap-southeast-2

      - name: Upload pipeline artifact
        run: |
          aws s3 cp orders-pipeline-$.tar.gz \
            s3://company-data-artifacts/orders-pipeline/

The commit SHA gives us an immutable artifact. If a deployment causes a problem, we know exactly which version was uploaded and can point the job back to an earlier artifact. The concurrency setting also prevents two production deployments from running at the same time.

GitHub environments can add a manual approval before the job starts. Whether to enable this depends on the team, but I normally use it for production and allow lower environments to deploy automatically.

Things to be careful about

Do not put database passwords, cloud keys, or connection strings directly in workflow YAML. Use OIDC where supported and GitHub Secrets for values that still need storage. A secret can still be exposed by a careless shell command, so avoid printing environment variables in debug steps.

Pinning actions to tags such as @v4 is convenient, but a stricter organisation might pin each action to a full commit SHA. Dependabot can then raise pull requests when a newer action version is available.

This demo packages only our source code. A production pipeline may need a Docker image, database migration, infrastructure deployment, data quality checks, and a rollback step. I would also add separate development and production AWS roles, restrict deployments using GitHub environments, and send deployment results to the team’s monitoring system.

One more limitation is that successful unit tests do not prove the pipeline can access its real data sources. Integration tests can run against a small test bucket or temporary database, but keep them separate from fast pull request tests if they take several minutes or cost money.

Conclusion

We created a small CI workflow that validates every pull request and a CD workflow that deploys only from main. Even for a simple data pipeline, this removes manual copying and gives us a record of what was tested and deployed. Once this basic flow works, it is easier to add environment approvals, integration tests, container builds, or infrastructure deployment based on what the project actually needs.