Implementing Kafka event streaming

Upvest provisions access for you. Before you connect, make sure you have received the following from your Upvest contact:

• Bootstrap server. The hostname and port your Kafka client connects to.
• Username and password. Your credentials for the cluster.
• Simple Authentication and Security Layer (SASL) mechanism. The authentication mechanism to configure on your client.

Connections are authenticated over TLS using a publicly trusted certificate authority, so you do not need any custom certificate authority configuration. Use the SASL mechanism, username, and password exactly as provided by Upvest.

You can connect using any standard Kafka client to complete the connection.

Your topics are exclusive to your organisation. Upvest provisions two types of topics:

Upvest gives you the exact topic names to use. Use each name as provided, without adding a prefix or suffix.

Each topic has 6 partitions. Kafka only guarantees message order within a single partition — not across the whole topic.

Upvest assigns each event's partition key when it publishes to your topic. This is not configurable from your side; Upvest sets it based on your event volume and ordering needs. Depending on your topic, the key is one of:

Tenant keying does not spread events across partitions. Your topic already contains only your organisation's events, so every event carries the same key and lands on a single partition. Tenant keying gives you full ordering across your entire event stream, but none of the parallelism benefit of spreading load across multiple partitions. Choose user or entity keying if you want your events spread across partitions, at the cost of a narrower ordering guarantee.

Entity keying only orders events about the exact same resource. It does not order events across related resources. For example, an order and a payment linked to that order are two separate resources — their events are not guaranteed to arrive in relative order, even under entity keying. (Under the hood, "the same resource" means the event's payload carries the same object.id.)

Events with different keys can interleave on different partitions, and arrive in a different order than they occurred. If your consumer logic depends on order between two events, confirm both concern the same resource for your topic's key.

Contact your Upvest representative to confirm which key applies to your topic. Upvest will notify you in advance of any change to your topic's partition key, since it changes which events remain ordered relative to one another.

Each message value is a single event, encoded as a JSON UTF-8 string. The event has the same structure as the events delivered over webhooks, so you can reuse the same parsing logic.

Each event contains the following fields:

The event payload contains the exact same fields as the related webhook. The object you receive on a topic matches the object in the webhook for that event type.

The only difference is the envelope. A webhook request bundles several events under a top-level payload array; Kafka delivers one event per message, so there is no payload wrapper to unpack.

Each message also carries Kafka headers for routing without parsing the body:

Kafka preserves the order of messages within a partition only. See Partitioning for how Upvest assigns events to partitions on your topic, and design your consumer to handle events independently unless they share a partition key.

Example message value

A single USER.CREATED event, as delivered on a topic. For the exact fields of this event, see the user event reference:

{
    "id": "fbecea50-2f35-4969-96af-342271da9eca",
    "created_at": "2021-07-21T14:10:00.00Z",
    "type": "USER.CREATED",
    "object": {
        "id": "83d83ec2-d2ca-49ff-bbea-b92b5c3be202",
        "type": "USER"
    }
}

Parse the message value as JSON before you access any field, and handle unknown fields gracefully, because new fields can be added in non-breaking releases.

Use a consumer group ID that is unique to your application. A clear convention is to name it after the service or use case that consumes the events, for example notifications-consumer or reconciliation. Run multiple instances with the same group ID to share the partition workload automatically.

Because Kafka assigns each partition to exactly one consumer instance within a group at a time, running more instances than your topic has partitions (6) leaves the extras idle. Size your consumer count up to, but not beyond, the partition count.

Your topics are exclusive to your organisation. An event your consumer cannot process can block it from receiving later events on that partition. We cannot clear these events for you, but we recommend the following.

For an event that keeps failing, retry a limited number of times, then store it in your own DLQ with its partition and offset, commit the offset, and continue. Reprocess the stored events once the cause is fixed.

When you store an event, later events for the same object can overtake it. If order matters, we recommend also storing those later events until you reprocess the stored event. Use object.id, or for events without one, such as cash balances and positions, their identifying fields.

To replay, stop your consumers and reset your consumer group's offsets to an earlier offset or timestamp. This works within the topic's retention period, for example 14 days for a Sandbox topic.

Before you integrate, verify the connection with kcat, a command-line Kafka client. Install it on your platform, then run the commands below, filling in the SASL mechanism Upvest provided.

kcat -b <BOOTSTRAP_SERVER> \
     -X security.protocol=SASL_SSL \
     -X sasl.mechanism=<SASL_MECHANISM> \
     -X sasl.username=<USERNAME> \
     -X sasl.password=<PASSWORD> \
     -L

kcat -b <BOOTSTRAP_SERVER> \
     -X security.protocol=SASL_SSL \
     -X sasl.mechanism=<SASL_MECHANISM> \
     -X sasl.username=<USERNAME> \
     -X sasl.password=<PASSWORD> \
     -C -t <TOPIC_NAME> -o beginning

Credentials are issued separately per environment. Sandbox credentials do not work against production, and production credentials do not work against sandbox.