Kafka event streaming lets your application pull events from Upvest in real time, instead of hosting an endpoint for Upvest to push to. Events are held in a durable, ordered queue that you read at your own pace, with full control over replay and retention.
Webhooks and Kafka event streaming deliver the same events and both guarantee delivery. Pick the one that fits your architecture. See the Webhooks vs Kafka event streaming section for more information.
Leveraging event streaming provides several benefits:
- Real time messaging: Receive events the moment they occur, with no polling.
- Replay events: Rewind and reprocess past events during incident recovery or a migration.
- Independent consumers: Several of your services can read the same events by default.
Apache Kafka holds events in a durable, append-only queue. Each event is added to the end of the queue and remains in place for the retention period. Your application tracks its position with an offset, so a consumer that restarts resumes exactly where it left off, with no skipped or repeated events.
| Concept | Meaning |
|---|---|
| Topic | A named queue for a category of events. Upvest provisions your topics. |
| Partition | A subdivision of a topic for parallel consumption. Each partition is independently ordered. |
| Producer | The system writing events. Upvest is the producer. |
| Consumer | Your application which reads the events from a topic. |
| Consumer group | A set of consumer instances that share the workload. Each event goes to one instance in the group. |
| Offset | The position of a message in a partition. |
| Broker | A Kafka server node. This is the server Upvest provides as your entry point. |
Both methods deliver the same events and both guarantee delivery. The difference is how you receive events.
The table below provides a high-level comparison of webhooks vs streaming.
| Functionality | Webhooks | Kafka event streaming |
|---|---|---|
| Delivery | Upvest pushes HTTPS requests to your endpoint | Your application pulls from a durable queue |
| Your infrastructure | A public HTTPS endpoint | An outbound Kafka client, no inbound endpoint |
| Missed events | Upvest retries on failure | You resume from your last offset |
| Replay | Not available | Rewind your offset to replay history |
| Setup | Lower initial effort | Higher initial effort, very resilient once running |
Kafka event streaming is the stronger choice when you need to replay historical events, when several services consume the same events independently, or when your application benefits from controlling its own read position.