SQLflow should add support for ingesting arbitrary JSON over HTTP webhooks. This would allow SQLFlow to listen to webhooks from any webhook provider (such as stripe, slack, shopify, auth0, github, zapier, hubspot, etc).
The goal is to provide a reliable, low friction way to begin to archive and work with webhook data:
SQLFlow HTTP Webhook Source (this task)
-> durable (disk-backed) handler for buffering data (#34)
-> Sink (kafka, s3 parquet, RDMBS, etc)
This story should add:
- New HTTP Source (flask is used for the debug endpoint, but have also used FastAPI)
- Benchmarks to understand realistic performance of the HTTP endpoint to guide potential users
SQLflow should add support for ingesting arbitrary JSON over HTTP webhooks. This would allow SQLFlow to listen to webhooks from any webhook provider (such as stripe, slack, shopify, auth0, github, zapier, hubspot, etc).
The goal is to provide a reliable, low friction way to begin to archive and work with webhook data:
SQLFlow HTTP Webhook Source (this task)
-> durable (disk-backed) handler for buffering data (#34)
-> Sink (kafka, s3 parquet, RDMBS, etc)
This story should add: