Skip to main content

Geocode a large address list (there is no batch endpoint, and that is fine)

There is no /batch endpoint here. Every geocode is one POST /api/v1/geocode with one address. That sounds like a problem for a 400,000-row HMDA file until you look at what a batch endpoint actually buys you, which is mostly a way to wait longer for the same work.

What you want instead is three things: stop geocoding the same address twice, keep a bounded number of requests in flight, and be able to stop and restart without losing the work. This post builds that in about 60 lines.

What you'll need

  • An API key (X-API-Key: gck_...). New accounts get 5,000 geocodes a month free.
  • Python 3.9+ with requests.
  • A CSV with an address column.

Step 1: deduplicate first

This is the single largest saving and almost everyone skips it. Address lists built from transaction records repeat heavily — the same branch, the same servicer address, the same handful of corporate mailing addresses, thousands of times.

import csv
from collections import OrderedDict

def norm(address):
    """The cache key. Must be used everywhere, or variants miss the cache."""
    return " ".join(address.split()).upper()

def unique_addresses(path):
    """{normalised key: one original spelling to send to the API}."""
    seen = OrderedDict()
    with open(path, newline="", encoding="utf-8") as fh:
        for row in csv.DictReader(fh):
            seen.setdefault(norm(row["address"]), row["address"])
    return seen

Normalising whitespace and case before the comparison is what makes this work: 123 Main St and 123 MAIN ST become one lookup, not two.

Keep that norm() in one place and use it for both the dedupe and the cache lookup. Deduplicating on the normalised form but caching under the raw string is an easy mistake, and it quietly reintroduces every duplicate you just removed.

Step 2: cache to disk, so a crash is not a restart

Write every result as you get it. SQLite is enough, and it makes the job resumable — a rerun skips everything already stored.

import json
import sqlite3

def open_cache(path="geocodes.db"):
    db = sqlite3.connect(path)
    db.execute("""
        CREATE TABLE IF NOT EXISTS geocode (
            address TEXT PRIMARY KEY,
            payload TEXT NOT NULL
        )
    """)
    db.commit()
    return db

def cached(db, key):
    row = db.execute(
        "SELECT payload FROM geocode WHERE address = ?", (key,)
    ).fetchone()
    return json.loads(row[0]) if row else None

def store(db, key, payload):
    db.execute(
        "INSERT OR REPLACE INTO geocode (address, payload) VALUES (?, ?)",
        (key, json.dumps(payload)),
    )
    db.commit()

Storing results is worth saying out loud: you are allowed to. There is no caching clause here and no expiry on what you geocoded. That is not true of every provider — see how this compares if you are migrating from one that restricts it.

Step 3: a bounded worker pool

The per-key burst ceiling is 300 requests per minute by default — and that is a rate, not a concurrency limit. Eight workers against an API that answers in 200ms is about 40 requests a second, which is eight times over the line. Pool size does not bound your rate; you have to pace the requests yourself.

At 300/minute a 100,000-address job takes about five and a half hours. Dedupe first (step 1) and that is usually a much smaller number — but plan for the job to run for a while, which is exactly why step 2 made it resumable.

import threading
import time
from concurrent.futures import ThreadPoolExecutor, as_completed

import requests

API = "https://weganar.com/api/v1/geocode"
WORKERS = 8
PER_MINUTE = 300

class Pacer:
    """Hold the whole pool to a global request rate.

    The workers exist to hide latency, not to raise throughput -- the ceiling is
    requests per minute across the key, however many threads ask for them.
    """

    def __init__(self, per_minute):
        self.interval = 60.0 / per_minute
        self.lock = threading.Lock()
        self.next_at = 0.0

    def wait(self):
        with self.lock:
            now = time.monotonic()
            sleep_for = max(0.0, self.next_at - now)
            self.next_at = max(now, self.next_at) + self.interval
        if sleep_for:
            time.sleep(sleep_for)

def geocode_one(session, api_key, address, pacer, attempts=4):
    """One address, with bounded backoff on 429. Bounded matters: an unbounded
    retry turns a sustained rate limit into a hang you cannot interrupt."""
    for attempt in range(attempts):
        pacer.wait()
        r = session.post(
            API,
            headers={"X-API-Key": api_key, "Content-Type": "application/json"},
            json={"address": address},
            timeout=30,
        )
        if r.status_code != 429:
            r.raise_for_status()
            return r.json()
        # Backed off rather than dropped: the address is still worth a result.
        time.sleep(2 ** attempt)
    raise RuntimeError(f"rate limited after {attempts} attempts: {address}")

def run(addresses, api_key, db):
    """`addresses` is the {key: original} mapping from unique_addresses()."""
    todo = {k: v for k, v in addresses.items() if cached(db, k) is None}
    print(f"{len(addresses) - len(todo)} already cached, {len(todo)} to fetch")
    pacer = Pacer(PER_MINUTE)

    with requests.Session() as session, ThreadPoolExecutor(WORKERS) as pool:
        futures = {
            pool.submit(geocode_one, session, api_key, original, pacer): key
            for key, original in todo.items()
        }
        for i, future in enumerate(as_completed(futures), 1):
            key = futures[future]
            try:
                store(db, key, future.result())
            except Exception as exc:
                print(f"failed: {key}: {exc}")
            if i % 500 == 0:
                print(f"  {i}/{len(todo)}")

The pacer is what keeps you under the limit; the 429 branch is the safety net for when your estimate of the API's latency is wrong. That branch matters more than the concurrency number. Rate limits are a signal to slow down, not an error to log and move past — dropping the address there is how you end up with a 3% hole in your output that nobody notices until someone asks why a county is under-represented.

Step 4: write the enriched file

def enrich(in_path, out_path, db):
    with open(in_path, newline="", encoding="utf-8") as src, \
         open(out_path, "w", newline="", encoding="utf-8") as dst:
        reader = csv.DictReader(src)
        writer = csv.DictWriter(
            dst,
            fieldnames=reader.fieldnames + [
                "lat", "lon", "census_block", "tract", "fips",
                "location_code", "match_code",
            ],
        )
        writer.writeheader()
        for row in reader:
            g = cached(db, norm(row["address"])) or {}
            block = g.get("censusBlock") or ""
            row.update({
                "lat": g.get("lat", ""),
                "lon": g.get("lon", ""),
                "census_block": block,
                # State(2) + county(3) + tract(6) = the 11-digit tract GEOID.
                "tract": block[:11],
                "fips": g.get("fips", ""),
                "location_code": g.get("locationCode", ""),
                "match_code": g.get("matchCode", ""),
            })
            writer.writerow(row)

Note the dedupe key is rebuilt here from the original row, so the output has one line per input row even though the API saw each distinct address once.

Checking the result before you trust it

locationCode tells you what the coordinate actually is. For anything regulatory, count them before you file:

from collections import Counter

counts = Counter(
    (cached(db, key) or {}).get("locationCode", "NO_MATCH") for key in addresses
)
print(counts.most_common())

ROOFTOP and ADDRESS_POINT are structure-level. PARCEL is the parcel centroid. SEGMENT is interpolated along a street and is the one to look at twice — it is a position on a block face, not a building. If your SEGMENT share is above a percent or two, check whether those addresses are rural or simply malformed.

Why not just use the Census Geocoder?

For a one-off file under 10,000 rows, do. It is free and authoritative, and we say so on the comparison page. The 10,000-record cap per batch file is what pushes people to something else once a job stops being a one-off.

Where to go next

Run this on your own addresses

Start free with 5,000 geocodes a month, Census block included.

Get your API key — 150 free credits

More guides

Enrich a CSV of addresses with Census demographics (Python) Cache-first geocoding, and when to re-geocode Add rooftop geocoding to a Django app