Skip to content

Configuration reference

Field-by-field schema for every YAML configuration file in the repo, plus the environment variables the tools honor. Guides in the Guides section explain the why of each field; this page lists the what, the type, and the default.

Normalize mapping YAML

File path: normalize/mappings/<type>.yaml where <type> is one of yellow, green, fhv, fhvhv.

Emitted by normalize <type> on bootstrap (first run against a type with no mapping) and amended in place on subsequent runs with unresolved items. Hand-edited by the user between runs to accept or reject SUGGESTED entries and to fill in ack_date: on TODO blocks. Consumed by every subsequent normalize <type> run.

Top-level fields

Field Type Required Description
target string yes Filename (basename) of the target parquet in raw/<type>/**/. Pins the schema that every historical file will be conformed to. Bootstrap picks the newest file at generation time; humans can bump it manually when a new "latest" appears.
renames dict[str, str] no Map of <old_name>: <new_name>. Both must be strings. Applies when the historical column name is <old_name> and the target has <new_name>.
lossy_casts dict[str, LossyCastEntry] no Explicit acknowledgments for type casts that could lose data (range or precision). Keyed by the target column name.
acknowledged_data_loss dict[str, DataLossEntry] no Explicit acknowledgments for historical columns that have data but no place to land in the target schema (and no rename mapping). Keyed by the historical column name.

Unknown top-level keys are rejected by the loader.

LossyCastEntry fields

Field Type Required Description
from string yes Historical column's DuckDB type (DOUBLE, INTEGER, VARCHAR, etc.).
to string yes Target column's DuckDB type. Cast target used at runtime.
ack_date string (ISO-8601 date) yes Anything truthy counts as acknowledgment; convention is YYYY-MM-DD. Its presence is what the tool enforces.
ack_by string no Recommended for audit trails. Any identifier (GitHub handle, email, name).
reason string no Recommended. One-sentence justification. Survives amend cycles; free-form YAML body comments do not.

DataLossEntry fields

Field Type Required Description
ack_date string (ISO-8601 date) yes Same convention as LossyCastEntry.ack_date.
ack_by string no Recommended.
reason string no Recommended.

Automatic behavior (no mapping entry needed)

  • All-null column drop. Historical column always-null across every file AND missing from target → dropped. No data lost.
  • Missing-in-historical null-fill. Column present in target but missing from a given historical file → filled with NULL.
  • Safe widening cast. Type change where the target can represent every possible source value (INT → BIGINT, FLOAT → DOUBLE, VARCHAR(10) → VARCHAR(50)) → auto-cast.

What triggers an error

  • Historical column has non-null data, is missing from target, and has no entry in renames: or acknowledged_data_loss:.
  • Type cast between historical and target could lose data (range or precision) and has no entry in lossy_casts:.
  • lossy_casts: or acknowledged_data_loss: entry missing ack_date:.

On error, the tool amends the mapping file with new SUGGESTED/TODO entries for every unresolved item, prints the consolidated report, and exits 1. Nothing is written to raw-normalized/.

Emitted YAML shape

On bootstrap the tool writes a timeline header (one comment line per detected schema-drift transition) followed by target:, renames:, lossy_casts:, and acknowledged_data_loss:. Every SUGGESTED rename appears as a commented # SUGGESTED (confidence X%, data-verified) — uncomment to accept: line above a commented # <old>: <new> line. Every TODO ack appears as a commented block ending in # ack_date: (empty — you fill it in).

On amend, existing semantic entries survive, new SUGGESTED/TODO items are appended below them, and the timeline header is regenerated. Free-form body comments in the YAML do not survive — PyYAML strips them on parse. Put reasoning in the reason: field of an ack entry, or in the git commit message.

Fully-worked example

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
# Generated by `normalize yellow` on 2026-07-22.
# Detected drift transitions:
#   2013-01 -> 2015-01: renamed pu_datetime->tpep_pickup_datetime; added PULocationID
#   2016-06 -> 2016-07: dropped pickup_latitude, pickup_longitude
#   2020-01 -> 2020-02: added congestion_surcharge
#   2023-01 -> 2023-02: added airport_fee

target: yellow_tripdata_2024-01.parquet

renames:
  pu_datetime: tpep_pickup_datetime
  do_datetime: tpep_dropoff_datetime

lossy_casts:
  passenger_count:
    from: DOUBLE
    to: BIGINT
    ack_date: 2026-07-21
    ack_by: andrekamman
    reason: "Values are logically integer; the 2011-2014 DOUBLE was a data-entry choice."

acknowledged_data_loss:
  pickup_latitude:
    ack_date: 2026-07-21
    ack_by: andrekamman
    reason: "Coarse-grained location data superseded by PULocationID zones."
  pickup_longitude:
    ack_date: 2026-07-21
    ack_by: andrekamman
    reason: "Same rationale as pickup_latitude."

Loader-rejected shapes

The loader is strict — anything outside the listed structure raises MappingError:

  • Missing file → Mapping file not found: {path}.
  • Empty file → Empty mapping file: {path}.
  • YAML that parses to something other than a dict → Mapping file must be a YAML mapping.
  • Unknown top-level key → Unknown top-level key: {key} (only target, renames, lossy_casts, acknowledged_data_loss are allowed).
  • Missing target: → Missing required field: target.
  • renames: where a key or value is not a string → Rename entry must be string->string, got {…}: {…}.
  • lossy_casts.<col> that is not a dict → lossy_casts.{col} must be a dict.
  • lossy_casts.<col> missing from:, to:, or ack_date: → lossy_casts.{col}: missing required field {field}.

The strictness is deliberate: silent tolerance of a mistyped field name would defeat the whole "explicit acknowledgment" model.

K6 load-test config YAML

File path: k6-loadtest/config.yaml. Typically copied from k6-loadtest/config.sample.yaml and edited.

Consumed by uv run k6-preprocess, which validates the file, exports data, and writes a K6 test bundle to k6-loadtest/output/. Validation failures raise ConfigError and the tool exits 1.

Top-level structure

Three required top-level keys:

  • data_sources: — dict of <name>: <SourceConfig>. One entry per logical dataset (e.g. yellow_trips, green_trips).
  • targets: — dict of <name>: <TargetConfig>. One entry per SQL Server connection.
  • scenarios: — dict of <name>: <ScenarioConfig>. Each scenario ties one data source to one target and defines the K6 workload.

Missing any of the three raises Missing required section: <name>.

Data source config (per source)

Fields common to both modes:

Field Type Required Description
mode string no parquet (default) or synthetic. Anything else raises a ConfigError.
key_columns list[str] yes Columns used as primary key for UPDATE and DELETE statements. Every name listed here must exist as a key in columns:.
columns dict yes Column definitions. Shape differs per mode — see below.

Parquet-mode-only fields:

Field Type Required Description
path string (glob) yes Glob pattern like raw/yellow/**/*.parquet. Validated at preprocess time; if the glob matches zero files, preprocess raises path {…} matches no files.
chunk_size int no Rows per output JSON chunk. Larger chunks mean fewer bulk-insert round-trips at runtime but more RAM held by each VU's SharedArray slice.
max_rows int no Optional cap on rows loaded from parquet. Useful for reducing preprocess time on very large datasets.

Synthetic-mode-only fields:

Field Type Required Description
columns dict[str, ColumnSpec] yes Every column entry must be a dict with at least a type: key; missing type raises synthetic column {…} must have a 'type' field.

ColumnSpec fields (synthetic mode):

Field Type Required Description
type string yes One of int, bigint, float, datetime, date, string, bool. Maps to SQL Server types: INT, BIGINT, FLOAT, DATETIME2, DATE, NVARCHAR(MAX), BIT.
min number or ISO-8601 string no Lower bound for random generation. For datetime/date, use quoted ISO-8601 strings.
max number or ISO-8601 string no Upper bound. Same shape as min.

In parquet mode, columns is a dict of <canonical_name>: <parquet_column_name> — a rename map from the source parquet's column name to the name used in the generated SQL.

Synthetic-mode example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
yellow_trips:
  mode: synthetic
  key_columns: [pickup_time, dropoff_time]
  columns:
    pickup_time:
      type: datetime
      min: "2026-01-01"
      max: "2026-03-01"
    passenger_count:
      type: int
      min: 1
      max: 6

Parquet-mode example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
green_trips:
  mode: parquet
  path: raw/green/2026/*.parquet
  chunk_size: 5000
  max_rows: 1000000
  key_columns: [pickup_time, dropoff_time]
  columns:
    pickup_time: lpep_pickup_datetime
    dropoff_time: lpep_dropoff_datetime
    passenger_count: passenger_count

Target config

Field Type Required Description
host string yes Hostname or IP address of the SQL Server.
port int yes Typically 1433.
database string yes Target database name. Must already exist.
username string yes SQL Server login.
password string yes Password value or a ${VAR} placeholder. The preprocessor preserves ${VAR} verbatim in the generated manifest so K6 does the substitution at runtime — passwords never enter test.js or the generated manifest.
table string yes Target table name for INSERT/UPDATE/DELETE statements.

Missing any of these fields raises Target {…}: missing required field {…}.

Scenario config

Field Type Required Description
target string yes Name of an entry in the top-level targets: dict. Reference is validated.
data_source string yes Name of an entry in the top-level data_sources: dict. Reference is validated.
ordering string no parallel (default) or sequential. Controls whether insert/update/delete phases run interleaved or in order.
workload dict[str, int] yes Percentages for insert, update, delete. Must sum to exactly 100 or preprocess raises workload percentages must sum to 100, got {N}.
think_time dict no Bounds for random per-iteration sleep. Keys: min:, max:. Values are duration strings like 200ms or 1.5s — the parser rejects anything else.
k6 dict yes K6 executor configuration passed through verbatim to the generated test.js. Fields depend on executor; commonly executor: (ramping-vus, constant-vus, shared-iterations, etc.), startVUs:, stages: (list of {duration, target}), or duration:.

Scenario example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
cdc_stress_test:
  target: local_server
  data_source: yellow_trips
  ordering: parallel
  workload:
    insert: 80
    update: 15
    delete: 5
  think_time:
    min: 200ms
    max: 1s
  k6:
    executor: ramping-vus
    startVUs: 2
    stages:
      - duration: 1m
        target: 20
      - duration: 5m
        target: 20
      - duration: 1m
        target: 0

Environment variables

Variable Used by Description
OUTPUT_DIR downloader Overrides the default raw/ output directory. Absolute or relative to CWD. Common WSL usage: OUTPUT_DIR=/mnt/c/Users/$USER/taxi-data to write to the Windows filesystem instead of the WSL2 VHDX.
MSSQL_PASSWORD K6 test.js (via ${MSSQL_PASSWORD} in the sample password: value) SQL Server password. The variable name is a convention — any ${VAR} placeholder in the target's password: field is preserved through preprocess and expanded by K6 at runtime, so you can name it anything.
HTTPS_PROXY / HTTP_PROXY curl (downloader), uv, gh Standard HTTP proxy env vars. The downloader honors them implicitly via curl.
NO_PROXY curl (downloader), uv, gh Standard bypass list. Include localhost if you run SQL Server locally and need it to bypass a corporate proxy.