How Ingest Works
A serverless, event-driven framework that moves data from any API into Snowflake Iceberg tables.
The Ingest Pipeline
REST API → queryable Snowflake table, in five stages.
Request generation
Declarative models define what API calls to make, endpoints, parameters, incremental date ranges. Scheduled on your cadence.
JSON request payloads
event-driven
Event routing
Requests trigger processing automatically. Ordered delivery with deduplication. No polling, no idle compute.
Ordered message queue
queued
API execution
Generic runtime dynamically assembles auth, pagination, rate limiting, and retry logic per connector. No code deploys per API.
Raw API responses
processed
Iceberg tables
Incremental merge into Snowflake Iceberg tables. Open format. Parquet on S3. Full DML, time travel, schema evolution.
Queryable tables, no vendor lock-in
secured
PII masking
Role-based column masking at the query layer. Full redaction, SHA-256 hashing, or IP anonymization. Clients never see raw PII.
Secure, compliant client access
Self-tuning
Continuously monitors memory, duration, and error rates, auto-adjusts batch sizes. No manual tuning.
| Signal | Response |
|---|---|
| Low memory usage | Larger batches, fewer invocations, lower cost |
| High memory usage | Smaller batches, prevent failures |
| Timeout or OOM | Halve batch size immediately |
| Stuck at minimum | Alert, requires investigation |
Security architecture
Built into the architecture, not bolted on.
Zero passwords: OIDC-based identity federation for all service-to-service auth. Short-lived tokens only.
Vault-managed credentials: API keys resolved at runtime from encrypted secret store. Never in code or config.
Tenant isolation: Separate storage, databases, and IAM policies per client. Row-level security on control plane.
Bring your own bucket: Data stays in your AWS account. Cross-account write via IAM role.
Client dashboard
Full visibility into your pipelines, no black box.
Row counts
Real-time per table
Pipeline health
Status per connector
Schedule control
Hourly, daily, or custom
Usage & billing
Per-GB transparency
Ready to get started?
Join the waitlist and see how Ingest can streamline your data pipelines and eliminate API debt.
Sign up