Small Team Data Platform Architecture on GCP — A Practical Guide

When you are a team of two or three engineers and someone says “we need a data platform”, the first thing that comes to mind is not Kubernetes clusters and multi-region failover. It is usually “how do we get something working this week that will not fall apart next month.”

In this article, I will walk through a data platform architecture on GCP that works for small teams. This is not a reference architecture from a cloud vendor whitepaper — it is the kind of setup I have seen work in practice when there are more pipelines to build than people to build them.

If you have already read the AWS version of this article, the principles are similar — keep things simple, pick managed services, and avoid building anything you do not have to. The GCP flavour leans heavily on BigQuery, Cloud Storage, and Cloud Run, which are the three services I end up reaching for in almost every project.

What a Small Team Actually Needs

Before talking about services, let us be clear about what we are trying to do. A small team data platform usually needs to:

• Ingest data from a few sources (databases, APIs, third-party SaaS tools)
• Store raw data somewhere cheap and durable
• Transform it into something queryable
• Serve it to dashboards, reports, or operational systems
• Do all of this without someone babysitting it every day

That is it. You do not need a real-time streaming engine, a feature store, or a data mesh. You need something that runs reliably and does not take half your week to maintain.

The Core Services

Here is the stack I would start with. Every service here is fully managed, which is the whole point — small teams do not have time to patch servers.

Service	Role	Why This One
Cloud Storage	Data lake / landing zone	Cheap, scales infinitely, no operations
BigQuery	Warehouse and query engine	No clusters to manage, pay per query
Cloud Run	Execution environment for pipelines	Serverless containers, scales to zero
Pub/Sub	Message bus between components	Fully managed, no broker to run
Cloud Scheduler	Cron for triggering pipelines	Built-in, no infrastructure
Dataform	Transformations (dbt alternative)	Native to BigQuery, free tier available

If your sources are mostly databases, you might also add Dataflow or the BigQuery Data Transfer Service, but I would try to avoid Dataflow for a small team unless you really need streaming. Dataflow is powerful but it adds complexity you might not need on day one.

Laying It Out: The Architecture

Here is a typical flow:

1. Cloud Scheduler triggers a Cloud Run job on a schedule.
2. The Cloud Run service pulls data from a source (say, a Postgres database or a REST API), does some light validation, and writes the raw data as JSON or Parquet files into a Cloud Storage bucket. This is your landing zone.
3. Another Cloud Run service (or the same one, if it is simple enough) picks up new files and loads them into BigQuery as external tables or by running a LOAD job.
4. Dataform runs on a schedule to transform the raw data into cleaned, modelled tables inside BigQuery.
5. Your BI tool (Looker Studio, Metabase, whatever) points at the Dataform output datasets.

That is the whole thing. Five services, three of which you barely need to configure.

Let us go through each piece with some actual code and configuration so this is not just a diagram in your head.

Step 1: Cloud Storage as the Landing Zone

Create a GCS bucket. That is step one. I usually create two — one for raw ingestion and one for Dataform artefacts, but you can start with one.

gsutil mb -l australia-southeast1 gs://my-data-platform-raw

Use a consistent folder structure inside the bucket. Something like:

gs://my-data-platform-raw/
  postgres/
    public.users/
      ingest_date=2025-11-25/
        data_001.parquet
  stripe/
    charges/
      ingest_date=2025-11-25/
        data_001.json

The ingest_date partition helps when you want to query the raw files directly with BigQuery external tables later. It also makes it easy to clean up old data without scripting anything complicated.

Step 2: Cloud Run for Ingestion

Cloud Run is great for small team pipelines because you write a simple script, package it in a Docker container, and it just runs. No cluster, no node pools, no Kubernetes manifests.

Here is a minimal ingestion script in Python that pulls from a Postgres database and writes to GCS:

import os
import psycopg2
import pyarrow as pa
import pyarrow.parquet as pq
from google.cloud import storage
from datetime import datetime

def ingest_table(table_name):
    conn = psycopg2.connect(os.environ["DATABASE_URL"])
    cursor = conn.cursor()
    cursor.execute(
        f"""SELECT * FROM {table_name}
           WHERE updated_at >= CURRENT_DATE - INTERVAL '1 day'"""
    )

    rows = cursor.fetchall()
    columns = [desc[0] for desc in cursor.description]
    table = pa.Table.from_pydict(
        {col: [row[i] for row in rows] for i, col in enumerate(columns)}
    )

    date_str = datetime.utcnow().strftime("%Y-%m-%d")
    filename = f"/tmp/{table_name}_{date_str}.parquet"
    pq.write_table(table, filename)

    client = storage.Client()
    bucket = client.bucket(os.environ["GCS_BUCKET"])
    blob = bucket.blob(
        f"postgres/{table_name}/ingest_date={date_str}/data_000.parquet"
    )
    blob.upload_from_filename(filename)

    cursor.close()
    conn.close()
    return len(rows)

if __name__ == "__main__":
    rows = ingest_table("public.users")
    print(f"Ingested {rows} rows")

You deploy this to Cloud Run with a Dockerfile. Set --max-instances=1 because you do not need concurrency for a scheduled job. Start with 512 MiB memory and increase if you hit OOM errors.

gcloud run deploy ingest-users \
  --source . \
  --region australia-southeast1 \
  --memory 512Mi \
  --max-instances 1 \
  --set-env-vars DATABASE_URL=...,GCS_BUCKET=my-data-platform-raw

Then create a Cloud Scheduler job to hit the Cloud Run endpoint on a schedule:

gcloud scheduler jobs create http ingest-users-daily \
  --schedule "0 3 * * *" \
  --uri "https://ingest-users-xxx.a.run.app" \
  --http-method POST \
  --oidc-service-account-email "sa-pipelines@project.iam.gserviceaccount.com"

The --oidc-service-account-email flag is important — it handles authentication to Cloud Run without you having to manage API keys or tokens manually.

Step 3: Loading into BigQuery

Once the files land in GCS, you need them in BigQuery. There are a few ways to do this:

• BigQuery external tables: Query the Parquet/JSON files directly from GCS. Good for ad-hoc exploration, bad for performance on larger datasets.
• LOAD DATA or bq load: Load the files into native BigQuery tables. This is what you want for your bronze layer.
• BigQuery Data Transfer Service: Scheduled, managed loads from GCS. Works if you do not need to transform the data on the way in.

For a small team, I usually go with a simple bq load command wrapped in a Cloud Run job or even a Cloud Function. Here is what the load SQL looks like:

LOAD DATA INTO raw.users
FROM FILES (
  format = 'PARQUET',
  uris = ['gs://my-data-platform-raw/postgres/public.users/ingest_date=2025-11-25/*.parquet']
);

Run this after the ingestion step completes. You can chain them — have the ingestion Cloud Run service call the load job via the BigQuery API, or run a separate scheduled job that loads everything new in one go.

One thing I learned the hard way: if you use WRITE_APPEND, make sure your ingestion job is idempotent. There is nothing worse than waking up to duplicate rows because the scheduler fired twice. Add a dedup step in your Dataform transforms, or use a merge pattern if your source has a reliable primary key.

Step 4: Transforming with Dataform

This is where the raw data becomes useful. Dataform is basically dbt for BigQuery — you write SQLX files, define your models, and Dataform compiles them into BigQuery SQL and runs them in the right dependency order.

A simple Dataform model looks like this:

-- definitions/users_clean.sqlx
config {
  type: "table",
  schema: "silver",
  tags: ["daily"]
}

SELECT
  id AS user_id,
  LOWER(email) AS email,
  COALESCE(status, 'unknown') AS status,
  TIMESTAMP(created_at) AS created_at,
  TIMESTAMP(updated_at) AS updated_at
FROM ${ref("raw", "users")}
WHERE email IS NOT NULL
QUALIFY ROW_NUMBER() OVER (
  PARTITION BY id ORDER BY updated_at DESC
) = 1

The QUALIFY clause at the end is deduplication — it keeps only the latest version of each row by id. This is one of those things that you do not think about until you have duplicates in production and suddenly a dashboard shows double the actual revenue.

Set up a release configuration in Dataform that runs on a schedule and you have a transformation layer you did not have to build from scratch. It handles incremental tables, assertions, and documentation out of the box.

One caveat with Dataform: the free tier limits you to 500 compilation units per month across your entire Google Cloud organisation, not per project. If you have multiple Dataform repositories, keep an eye on this. The paid tier removes the limit but costs per compilation.

Step 5: Orchestration — Keep It Stupid Simple

The orchestration question is where small teams often over-engineer. You do not need Airflow for five pipelines. Here is my rough decision guide:

Number of pipelines	What to use
1–3	Cloud Scheduler + Cloud Run
3–8	Cloud Scheduler + Cloud Run, with Pub/Sub for dependencies
8–20	Cloud Workflows or a lightweight managed orchestrator
20+	Consider Cloud Composer (managed Airflow), but know what you are signing up for

Cloud Workflows is underrated for this kind of setup. It connects Cloud Run, BigQuery, and Pub/Sub with a YAML definition, and you do not have to manage a single server. The syntax is a bit verbose but it works:

- runIngest:
    call: http.post
    args:
      url: https://ingest-users-xxx.a.run.app
      auth:
        type: OIDC
- loadBigQuery:
    call: googleapis.bigquery.v2.jobs.insert
    args:
      projectId: my-project
      body:
        configuration:
          query:
            query: >
              LOAD DATA INTO raw.users
              FROM FILES (
                format='PARQUET',
                uris=['gs://my-data-platform-raw/postgres/public.users/*']
              )
- runDataform:
    call: http.post
    args:
      url: https://dataform.googleapis.com/v1beta1/projects/.../workflowInvocations
      auth:
        type: OIDC

Things That Will Bite You

BigQuery costs at small scale are negligible, but at medium scale they catch people off guard. If you are running SELECT * queries for dashboards that refresh every five minutes, you are going to pay for it. Use partitioned and clustered tables, and set up custom quotas in BigQuery so nobody accidentally runs a 10 TB scan. It takes five minutes to configure and might save you a difficult conversation with your finance team.

Cloud Run has a 60-minute timeout (or 3600 seconds for Cloud Run jobs). If your ingestion takes longer than that, Cloud Run is the wrong tool. Split the work into smaller chunks or use a Compute Engine VM with a startup script and shut it down when done.

IAM will be the thing that slows you down. GCP IAM is less confusing than AWS IAM, but you will still spend time getting the service account permissions right. Start with the minimum set of roles and add as you go. For a pipeline service account, you typically need roles/storage.objectUser, roles/bigquery.dataEditor, and roles/run.invoker on the specific resources, not the whole project.

Monitoring is easy to skip and hard to retrofit. At a minimum, set up Cloud Monitoring alerts for Cloud Run job failures and BigQuery query errors. Cloud Logging is decent for debugging but you want the alert before you need the log.

What Changes in Production

Everything above works for a production setup, but here is what I would tighten before calling it done:

• Terraform everything. Do not click around in the console. All the GCS buckets, Cloud Run services, Scheduler jobs, and IAM bindings should be in Terraform. It is extra work up front but it means you can recreate the whole platform if something goes wrong, and your teammate can understand what exists without asking you.
• Use a separate project for production. GCP projects are free and they give you isolation. Keep your dev pipelines in one project and production in another. If someone messes up a dev deployment, production keeps running.
• Set up a CI/CD pipeline. GitHub Actions with Terraform works well. Run terraform plan on PRs and terraform apply on merge to main. Store the Terraform state in a GCS bucket with versioning turned on so you can roll back if needed.
• Add data quality checks. Dataform has built-in assertions. Use them. A pipeline that runs successfully but loads zero rows is a silent failure that will sit there until someone notices a dashboard is wrong.

Wrapping Up

Building a data platform for a small team on GCP is more about discipline than technology. Pick a handful of managed services, write your infrastructure as code, and resist the urge to add complexity you do not need yet. Cloud Storage, BigQuery, Cloud Run, Pub/Sub, and Dataform cover a huge amount of ground before you need anything more sophisticated.

The hard part is not the architecture. It is keeping the architecture simple when everyone around you is talking about data meshes and streaming pipelines. Ignore them until you have a problem that actually needs those things.