Ingest documentation

From any API to tables you can query

Ingest authors the connector, runs the pipeline and lands typed, deduplicated tables in a destination you own. These pages cover what it connects, what it writes, and how to set each destination up.

Start here

Data sources
Every source: the connectors certified against a real API today, and the vendors Ingest builds on request.
The tables Ingest creates
What lands in your warehouse (data tables, child tables, quarantine, the three _ingest_ control tables) and the semantics of each.
Machine-readable index
The same contents as plain text, for AI crawlers and agents.

Connectors

abilene_311
rest · public_data · beta
albanyny_311
rest · public_data · beta
annarbor_311
rest · public_data · beta
annearundelcounty_311
rest · public_data · beta
asana
rest · productivity · beta
austintexas_311
rest · public_data · beta
baltimore_311
rest · public_data · beta
batonrouge_311
rest · public_data · beta
beaumont_311
rest · public_data · beta
bellevue_311
rest · public_data · beta
bellingham_311
rest · public_data · beta
berkeley_311
rest · public_data · beta
bethlehempa_311
rest · public_data · beta
bloomingtonin_311
rest · public_data · beta
bocaraton_311
rest · public_data · beta
boston_311
rest · civic · beta
bridgeport_311
rest · public_data · beta
brockton_311
rest · public_data · beta
buffalony_311
rest · public_data · beta
cambridge_311
rest · public_data · beta
charlestoncountysc_311
rest · public_data · beta
charlottenc_311
rest · public_data · beta
chulavista_311
rest · public_data · beta
cincinnati_311
rest · public_data · beta
cityofchicago_311
rest · public_data · stable
cityofnewyork_311
rest · public_data · stable
claytoncountyga_311
rest · public_data · beta
cleveland_311
rest · public_data · beta
cobbcountyga_311
rest · public_data · beta
columbus_311
rest · public_data · beta
dallas_311
rest · public_data · beta
davenport_311
rest · public_data · beta
denver_311
rest · public_data · beta
detroit_311
rest · public_data · beta
elgin_311
rest · public_data · beta
encinitas_311
rest · public_data · beta
everettwa_311
rest · public_data · beta
federalway_311
rest · public_data · beta
fortlauderdale_311
rest · public_data · beta
gilbert_311
rest · public_data · beta
honolulu_311
rest · public_data · beta
howardcountymd_311
rest · public_data · beta
huntingtonwv_311
rest · public_data · beta
huntsville_311
rest · public_data · beta
indy_311
rest · public_data · beta
jerseycity_311
rest · public_data · beta
kcmo_311
rest · public_data · beta
kentwa_311
rest · public_data · beta
knoxcountytn_311
rest · public_data · beta
lacity_311
rest · public_data · beta
lexingtonky_311
rest · public_data · beta
lincolnne_311
rest · public_data · beta
livonia_311
rest · public_data · beta
lubbock_311
rest · public_data · beta
maconbibb_311
rest · public_data · beta
manchesternh_311
rest · public_data · beta
mauicounty_311
rest · public_data · beta
medfordma_311
rest · public_data · beta
memphis_311
rest · public_data · beta
minneapolis_311
rest · public_data · beta
montgomerycountymd_311
rest · public_data · beta
multnomahcounty_311
rest · public_data · beta
naperville_311
rest · public_data · beta
nashua_311
rest · public_data · beta
nashville_311
rest · public_data · beta
newarknj_311
rest · public_data · beta
nola_311
rest · public_data · beta
norfolk_311
rest · public_data · beta
oakland_311
rest · public_data · beta
omaha_311
rest · public_data · beta
openbrewerydb
rest · food_and_drink · stable
paterson_311
rest · public_data · beta
peoria_311
rest · public_data · beta
pinellascounty_311
rest · public_data · beta
portlandme_311
rest · public_data · beta
provo_311
rest · public_data · beta
raleigh_311
rest · public_data · beta
renton_311
rest · public_data · beta
rickandmortyapi
rest · entertainment · stable
sacramento_311
rest · public_data · beta
sanfrancisco_311
rest · public_data · beta
sangabriel_311
rest · public_data · beta
sanmateocounty_311
rest · public_data · beta
scottsdale_311
rest · public_data · beta
seattle_311
rest · public_data · beta
somervillema_311
rest · public_data · stable
springfieldmo_311
rest · public_data · beta
stpetersburg_311
rest · public_data · beta
toledo_311
rest · public_data · beta
topeka_311
rest · public_data · beta
vallejo_311
rest · public_data · beta
washingtondc_311
rest · public_data · beta
waynecountymi_311
rest · public_data · beta
westdesmoines_311
rest · public_data · beta
wichita_311
rest · public_data · beta
youngstown_311
rest · public_data · beta

96 certified, and 909 more vendors Ingest builds on request. Browse them all.

API kinds

File
Buckets and local paths via fsspec: `csv`, `jsonl`, `json`, `parquet`, `xlsx`.
GraphQL
One document per resource, POSTed as `{query, operationName, variables}`.
gRPC
A channel per resource, TLS by default, with auth carried in metadata.
REST
Ingest's original engine. Only `{method,url,headers,params,data,json,timeout,allow_redirects,ssl,proxy}` are forwarded to the transport.
SOAP
Envelope templates rendered with XML-escaped parameters, parsed with `lxml`.
SQL
Databases read through DB-API or ADBC with chunked fetch.
WebSocket
A bounded batch window over a stream, not real-time delivery.

Destinations

Amazon AthenaAmazon Athena
Iceberg tables in S3 and the AWS Glue Data Catalog, ready to query with Athena. Supports append, replace, and merge.
BigQuery
Load jobs straight from the worker. Native `MERGE`. No bucket to provision.
ClickHouse
Native insert. Merge uses `ReplacingMergeTree` keyed by `dedup_sort`.
Databricks
Loads through a Unity Catalog volume into Delta tables. Native MERGE for upserts.
DuckDB
A local development destination. Not available on the hosted worker fleet.
ABAzure Blob Storage
Parquet files in your Azure container, authorised by a connection string.
Google Cloud Storage
Parquet files in your own GCS bucket. Ingest holds no credential: your runs authenticate with the AWS identity your workspace already has.
Apache Iceberg
Iceberg tables in your REST catalog. Catalog and storage authentication must be supported by the setup form.
Amazon S3
Parquet files in your S3 bucket, written under a role you own and can revoke.
Generic SQL
MySQL with batched inserts and transactional merge emulation. Review the current TLS setup limitation before connecting.
MotherDuckMotherDuck
DuckDB, hosted. Same adapter, same semantics.
Microsoft SQL ServerSQL Server
Not yet available as an Ingest destination.
Postgres
The conformance oracle. `ON CONFLICT` merge, transactional.
Amazon RedshiftRedshift
COPY from your S3 staging prefix into Redshift, with native MERGE for upserts.
Snowflake
Tables in your Snowflake database, authenticated with your Ingest workspace identity. No stored Snowflake key or password.