Skip to content

Configuration reference

Field-by-field schema for every YAML configuration file in the repo, plus the environment variables the tools honor. Guides in the Guides section explain the why of each field; this page lists the what, the type, and the default.

Normalize mapping YAML

File path: normalize/mappings/<type>.yaml where <type> is one of yellow, green, fhv, fhvhv.

Emitted by normalize <type> on bootstrap (first run against a type with no mapping) and amended in place on subsequent runs with unresolved items. Hand-edited by the user between runs to accept or reject SUGGESTED entries and to fill in ack_date: on TODO blocks. Consumed by every subsequent normalize <type> run.

Top-level fields

Field Type Required Description
target string yes Filename (basename) of the target parquet in raw/<type>/**/. Pins the schema that every historical file will be conformed to. Bootstrap picks the newest file at generation time; humans can bump it manually when a new "latest" appears.
renames dict[str, str] no Map of <old_name>: <new_name>. Both must be strings. Applies when the historical column name is <old_name> and the target has <new_name>.
lossy_casts dict[str, LossyCastEntry] no Explicit acknowledgments for type casts that could lose data (range or precision). Keyed by the target column name.
acknowledged_data_loss dict[str, DataLossEntry] no Explicit acknowledgments for historical columns that have data but no place to land in the target schema (and no rename mapping). Keyed by the historical column name.
value_maps dict[str, dict] no Column value remapping, keyed by column name. Each entry is either a bare {source_value: target_value} map, or {map: {...}, on_unmapped: "error"\|"null"} (default policy "error") to control behavior for values not present in the map.

Unknown top-level keys are rejected by the loader.

LossyCastEntry fields

Field Type Required Description
from string yes Historical column's DuckDB type (DOUBLE, INTEGER, VARCHAR, etc.).
to string yes Target column's DuckDB type. Cast target used at runtime.
ack_date string (ISO-8601 date) yes Anything truthy counts as acknowledgment; convention is YYYY-MM-DD. Its presence is what the tool enforces.
ack_by string no Recommended for audit trails. Any identifier (GitHub handle, email, name).
reason string no Recommended. One-sentence justification. Survives amend cycles; free-form YAML body comments do not.

DataLossEntry fields

Field Type Required Description
ack_date string (ISO-8601 date) yes Same convention as LossyCastEntry.ack_date.
ack_by string no Recommended.
reason string no Recommended.

Automatic behavior (no mapping entry needed)

  • All-null column drop. Historical column always-null across every file AND missing from target → dropped. No data lost.
  • Missing-in-historical null-fill. Column present in target but missing from a given historical file → filled with NULL.
  • Safe widening cast. Type change where the target can represent every possible source value (INT → BIGINT, FLOAT → DOUBLE, VARCHAR(10) → VARCHAR(50)) → auto-cast.

What triggers an error

  • Historical column has non-null data, is missing from target, and has no entry in renames: or acknowledged_data_loss:.
  • Type cast between historical and target could lose data (range or precision) and has no entry in lossy_casts:.
  • lossy_casts: or acknowledged_data_loss: entry missing ack_date:.

On error, the tool amends the mapping file with new SUGGESTED/TODO entries for every unresolved item, prints the consolidated report, and exits 1. Nothing is written to raw-normalized/.

Emitted YAML shape

On bootstrap the tool writes a timeline header (one comment line per detected schema-drift transition) followed by target:, renames:, lossy_casts:, and acknowledged_data_loss:. Every SUGGESTED rename appears as a commented # SUGGESTED (confidence X%, data-verified) — uncomment to accept: line above a commented # <old>: <new> line. Every TODO ack appears as a commented block ending in # ack_date: (empty — you fill it in).

On amend, existing semantic entries survive, new SUGGESTED/TODO items are appended below them, and the timeline header is regenerated. Free-form body comments in the YAML do not survive — PyYAML strips them on parse. Put reasoning in the reason: field of an ack entry, or in the git commit message.

Fully-worked example

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
# Generated by `normalize yellow` on 2026-07-22.
# Detected drift transitions:
#   2013-01 -> 2015-01: renamed pu_datetime->tpep_pickup_datetime; added PULocationID
#   2016-06 -> 2016-07: dropped pickup_latitude, pickup_longitude
#   2020-01 -> 2020-02: added congestion_surcharge
#   2023-01 -> 2023-02: added airport_fee

target: yellow_tripdata_2024-01.parquet

renames:
  pu_datetime: tpep_pickup_datetime
  do_datetime: tpep_dropoff_datetime

lossy_casts:
  passenger_count:
    from: DOUBLE
    to: BIGINT
    ack_date: 2026-07-21
    ack_by: andrekamman
    reason: "Values are logically integer; the 2011-2014 DOUBLE was a data-entry choice."

acknowledged_data_loss:
  pickup_latitude:
    ack_date: 2026-07-21
    ack_by: andrekamman
    reason: "Coarse-grained location data superseded by PULocationID zones."
  pickup_longitude:
    ack_date: 2026-07-21
    ack_by: andrekamman
    reason: "Same rationale as pickup_latitude."

Loader-rejected shapes

The loader is strict — anything outside the listed structure raises MappingError:

  • Missing file → Mapping file not found: {path}.
  • Empty file → Empty mapping file: {path}.
  • YAML that parses to something other than a dict → Mapping file must be a YAML mapping.
  • Unknown top-level key → Unknown top-level key: {key} (only target, renames, lossy_casts, acknowledged_data_loss, value_maps are allowed).
  • Missing target: → Missing required field: target.
  • renames: where a key or value is not a string → Rename entry must be string->string, got {…}: {…}.
  • lossy_casts.<col> that is not a dict → lossy_casts.{col} must be a dict.
  • lossy_casts.<col> missing from:, to:, or ack_date: → lossy_casts.{col}: missing required field {field}.

The strictness is deliberate: silent tolerance of a mistyped field name would defeat the whole "explicit acknowledgment" model.

Loader and orchestrator configuration

There is no YAML config file for the loader or the orchestrator — taxi-load and taxi-run are configured entirely through CLI flags (--host, --port, --database, --schema, --user, --data-dir/--input-dir, --flush-rows, --full-refresh, --dry-run, and, for taxi-run, --recent, --skip-download, --download-only, --load, --sample) plus one required environment variable, MSSQL_PASSWORD, whenever a load is involved. See Loader and Orchestrator for the full flag reference.

Environment variables

Variable Used by Description
MSSQL_PASSWORD taxi-load / taxi-run SQL Server password. Required whenever a load happens: always for taxi-load, and for taxi-run only when --load is passed. There is no --password flag — this is deliberate so the password never appears in ps output or shell history via an argument.
HTTPS_PROXY / HTTP_PROXY httpx (downloader), uv, gh Standard HTTP proxy env vars. The downloader honors them implicitly via httpx.
NO_PROXY httpx (downloader), uv, gh Standard bypass list. Include localhost if you run SQL Server locally and need it to bypass a corporate proxy.

--data-dir is the shared "base directory" flag across the downloader, normalize, loader, and orchestrator — each tool resolves its own subdirectory beneath it (<data-dir>/raw, <data-dir>/raw-normalized, etc.). The loader also accepts --input-dir to point directly at a normalized-parquet directory, overriding the --data-dir-derived path.