Scoring 25 Million Voter Records: An ML Pipeline Walkthrough

Key numbers

• 25.3M voters scored for turnout and persuasion
• 250.9M vote-history rows processed
• $2.365B in federal contributions matched back to individuals
• 3 states running from one codebase, each gated by an automated end-to-end verifier
• PostgreSQL database at 33GB+

The problem with a statewide voter file

Statewide voter files are large, inconsistent, and useless on their own. A raw file gives a campaign a name, an address, a party registration, and a vote-history string that looks like a sequence of yes/no flags across a dozen past elections. It does not tell anyone who is worth a door knock, who is worth a mail piece, or who already gave to a federal candidate last cycle under a slightly different spelling of their name.

That gap, between "we have the data" and "we can act on the data," was the entire problem. The firm needed three things layered on top of the raw files: a score for how likely each voter is to turn out, a score for how persuadable they are on a given issue set, and a link between the voter file and federal contribution records so campaign staff could see money and turnout in the same view. None of that existed. It had to be built as a pipeline, not a one-time analysis, because every new state means a new file with its own quirks.

The architecture: graph, scores, matches, maps

The system has four layers that run in sequence for every state: ingest into a unified voter-and-donor graph, score each voter for turnout and persuasion, match federal contribution records back to individuals in that graph, and serve the result through choropleth district maps and dashboards built in Next.js.

Raw statewide voter file + FEC data

Python ETL profile, clean, load

Unified voter-and-donor graph (Postgres, 33GB+)

ML scoring pipeline turnout + persuasion

FEC contribution match $2.365B linked to voters

Next.js dashboard choropleth district maps

Playwright verifier gates every page, every state

verifier watches the whole pipeline, not just the UI

The voter-and-donor graph is the core data model. Every other layer reads from it or writes back into it. That decision, treating the graph as the source of truth rather than a byproduct of the dashboard, is what let three states run on one codebase instead of three forked projects.

Scoring: turnout and persuasion per voter

The scoring pipeline runs in Python against the Postgres graph. For each voter it reads the features the graph already holds and produces two scores: a turnout probability and a persuasion score. The feature set and the model choices are the client's, so what follows is a simplified illustration of the pipeline shape rather than the code or the schema that shipped:

# Simplified illustration of a scoring-pipeline stage. Generic shape only:
# the real feature set, queries, and models are the client's.
import pandas as pd
from sklearn.pipeline import Pipeline

def extract_features(conn, state: str) -> pd.DataFrame:
 """Read the per-record features the graph already holds for one state."""
 return pd.read_sql(FEATURE_QUERY, conn, params={"state": state})

def score_batch(features: pd.DataFrame, turnout_model: Pipeline,
 persuasion_model: Pipeline) -> pd.DataFrame:
 features["turnout_score"] = turnout_model.predict_proba(features)[:, 1]
 features["persuasion_score"] = persuasion_model.predict_proba(features)[:, 1]
 return features[["voter_id", "turnout_score", "persuasion_score"]]

def write_scores(conn, scores: pd.DataFrame) -> None:
 """Write scores back onto the same records they were derived from."""
 scores.to_sql("voter_scores", conn, if_exists="append",
 index=False, method="multi", chunksize=5000)

Feature extraction, model score, write-back. Nothing exotic. What matters is not which model wins a bake-off but that every voter in every state goes through the same three steps, on the same schema, so a new state does not require new code, only a new run of the pipeline against a newly loaded file.

Matching federal contributions back to individuals

Federal contribution records do not arrive pre-linked to a voter file. Names are misspelled, addresses shift between an employer's mailing address and a home address, and the same person can appear under three name variants across cycles. So the pipeline needs an entity-resolution step: a stage that decides, record by record, when a contribution and a voter describe the same individual, and that runs before the dashboard ever renders a dollar figure next to a name.

How that step is tuned is the client's intellectual property, and it is the part of a build like this that is genuinely hard to get right, because both a false match and a missed match are expensive in different directions. What we can say is the outcome: $2.365 billion in contributions came through that step linked to individual voter records in the graph, visible next to each voter's turnout and persuasion scores.

Verifying a data-heavy dashboard with Playwright

A dashboard sitting on top of 25.3 million scored voters and 250.9 million vote-history rows fails in ways a small app does not. A choropleth map can render with the wrong color scale for one state and no one notices until a campaign staffer flags a district that looks flipped. The Playwright gate exists to catch exactly that, on every page, before anything ships. A simplified illustration of the shape of one of those checks:

import { test, expect } from '@playwright/test';

const STATES = ['state-a', 'state-b', 'state-c'];

for (const state of STATES) {
 test(`district map renders and totals reconcile for ${state}`, async ({ page }) => {
 await page.goto(`/dashboard/${state}/districts`);

 await expect(page.getByTestId('choropleth-map')).toBeVisible();

 const voterTotal = await page.getByTestId('total-voters-scored').innerText();
 const dbTotal = await fetchExpectedVoterCount(state);
 expect(parseInt(voterTotal.replace(/,/g, ''))).toBe(dbTotal);

 const contributionTotal = await page.getByTestId('matched-contributions').innerText();
 expect(contributionTotal).toMatch(/^\$[\d,]+$/);

 await page.getByTestId('district-select').selectOption({ index: 0 });
 await expect(page.getByTestId('turnout-score-panel')).toBeVisible();
 await expect(page.getByTestId('persuasion-score-panel')).toBeVisible();
 });
}

This is not a smoke test run once at launch. It runs against every state, on every deploy, and it checks numbers against the database, not just that a component rendered. If a state's totals drift from what the graph actually holds, the pipeline fails before the change reaches the dashboard.

Onboarding a new state without re-engineering

The claim "one command onboards a new state" is not a script someone runs by hand and hopes for the best. It is a pipeline stage sequence: profile the file, score it, verify every page, go live. Each stage is a discrete, repeatable job with its own inputs and outputs, and each one has to pass before the next one starts.

1. Profile the file. New state file lands, ETL inspects its schema, flags anomalies (missing fields, unexpected encodings, date format drift), and normalizes it into the same shape every other state uses.
2. Score it. The ML scoring pipeline runs turnout and persuasion models against the newly loaded voters, writing scores back into the graph.
3. Verify every page. The Playwright gate runs against the new state's dashboard routes, checking that totals reconcile and every panel renders before anyone on the campaign side sees it.
4. Go live. The state flips from staging to production behind the dashboard, joining the others on the same codebase.

The reason this matters is that campaign timelines do not wait for a re-architecture. A firm bringing on a fourth or fifth state needs that state live in days, not another development cycle, and the only way to guarantee that is to have already paid the engineering cost of making the pipeline state-agnostic the first time.

What ships to the firm, not to us

The Postgres graph, the ETL jobs, and the Next.js dashboard all ship into the firm's own repository and cloud account, the same way every build we run does. The firm owns the system outright, and nothing in it depends on a service only we control. That ownership model, and the security posture behind it (BAAs signed on request, a SOC 2 Type II report available under NDA), are described in full on our security page. For political and campaign data specifically, the sensitivity is different from healthcare PHI, but the discipline is the same: audit what moves, verify before it ships, and never make the client dependent on infrastructure they cannot see.

Why a pod, not a single hire, for this kind of build

A build like this needs ETL engineering, ML scoring, database work at scale, and frontend and test engineering, running in parallel, not sequentially. That is close to what our pods are built for: a pod lead plus a bench of engineers shipping weekly, sized to the build rather than to a single job description. A firm that tried to hire this out one role at a time (a data engineer, then an ML engineer, then a frontend engineer) would spend the better part of a hiring cycle before the first state ever scored a single voter.

The pattern here, unified data model first, scoring pipeline second, verification gate before anything ships, is not specific to political data. The same shape shows up whenever a client has large, messy, high-stakes data and needs a system that produces a score or a decision a human can act on, which is most of what we cover in our guide to custom AI development and in the deeper walkthrough of what LLM development services actually involve when the workload includes model calls, not just classical ML scoring.

The short version

A statewide voter file and a stack of federal contribution records are not, by themselves, anything a campaign can act on. We built a unified voter-and-donor graph, an ML scoring pipeline for turnout and persuasion, a contribution-matching step, and a choropleth dashboard on top, all gated by an automated Playwright verifier before anything ships. The result: 25.3 million voters scored, 250.9 million vote-history rows processed, $2.365 billion in matched federal contributions live behind the dashboard, and three states running from a single codebase, each one onboarded through the same repeatable profile-score-verify-go-live sequence rather than a fresh build.