How does Ingest work?

You pick a connector, paste credentials, choose endpoints, and pick a destination. Ingest deploys a managed pipeline that fetches data on a schedule, handles auth and retries automatically, and lands each endpoint as an Iceberg table in Snowflake or your own S3 bucket.

The pieces

Connectors: one per source API. Each knows how to authenticate, paginate, handle rate limits, and parse the provider's response shape.
Endpoints: the individual resources inside a connector (e.g. asana.tasks, hubspot.contacts). Each endpoint becomes its own table.
Pipelines: a connector + the endpoints you enabled + a destination + a schedule.
Schedules: how often the pipeline runs. Cron-style.
Runs: a single execution of a pipeline. Records duration, row count, bytes, errors.
Iceberg tables: where data lands. Open format, queryable from Snowflake or any Iceberg-compatible engine.

Step by step

You configure a pipeline: pick the connector, paste credentials, tick endpoints, choose destination.
Ingest deploys it: credentials get stored in AWS Secrets Manager, the pipeline is provisioned.
Each run fetches data per endpoint, paginates the full result set, handles 429s with backoff, resolves the auth credential, and writes results to your Iceberg table.
Snowflake reads the Iceberg table directly via an external volume. Your dbt models and queries see fresh data after each run completes.
The dashboard shows row counts per endpoint and pipeline health.

What you don't handle

API authentication, rate-limit backoff, pagination, retries, response parsing.
Compute provisioning and scaling.
Storing the API credentials securely.

Frequently asked

Do I need to manage any infrastructure?

No. Pipelines run on Ingest-managed compute. The marketing site describes Ingest as fully managed and serverless.

What happens when an API returns an error?

The runtime retries transient failures (5xx, 429) with exponential backoff. Auth failures surface in the dashboard so you can update the credential.