Home › Learn › Real-time analytics › Contracts and bindings

Real-time analytics · page 8 of 11

Contracts and bindings at every hop

The spec at every hop of a streaming pipeline: contract, promises and binding, with Avro, CloudEvents, ODCS and AsyncAPI, plus ordering, delivery, deletes, late events and replay.

↑ All of real-time analyticsReal time, explainedLambda vs KappaComponents, left to rightChange capture toolsWhich route?Eight designsDesigns on each cloudContracts and bindingsPush to the screenWhat it costsEstablished technologies

In plain words

One small pattern, repeated

Every hop has the same shape: a producer, a topic with a contract, and a consumer. A stream processor consumes one topic and produces another, so the pattern repeats along the line. Each topic gets the same spec, in three parts.

ProducerCDC or appTopic 1contract + bindingProcessorconsumes 1,produces 2Topic 2contract + bindingConsumerstore, pushcontract: schema, key,meaning, compatibilitybinding: Kafka topic,partitions, guaranteesthe same three-part spec repeats at every hop
Producer, topic, consumer, and again. The contract says what; the binding says how.

The spec

Three parts per topic

PartWhat it pins downStandard to use
Contract (what)Event schema, the key, what each field means, compatibility rules (usually backward compatible)Avro or Protobuf in a schema registry; CloudEvents for the envelope: id, source, type, time
PromisesOwner, freshness target, quality checks, how long events are keptODCS, the Open Data Contract Standard
Binding (how)The technology for this hop: Kafka topic name, partitions, consumer group, delivery guarantee; or the WebSocket or SSE details for the screenAsyncAPI, which has a bindings section for Kafka, WebSocket, MQTT, AMQP and more

AsyncAPI fits the idea of coding to an interface. The channel and message are the interface; the bindings block is the adapter. Moving from Kafka to Event Hubs changes the binding, not the contract.

Every hop

Five promises every hop must state

PromiseWhat to decide
OrderingOrder is only kept within one partition, so pick the key with care, for example the order id.
DeliveryAt least once is the default, so consumers must cope with duplicates: upsert by key, for example ClickHouse ReplacingMergeTree. Use exactly once (Kafka transactions, Flink checkpoints) only where it is truly needed.
DeletesChange capture sends tombstone events for deletes; the serving store must apply them.
Late eventsWatermarks and an allowed lateness in the stream processor say how long to wait.
ReplayHow to rebuild: from the log, or from the lakehouse copy, and how to make sure nothing is counted twice.

FAQ

Questions people ask

What is AsyncAPI?

An open standard for describing event-driven interfaces: channels (topics), messages and their schemas, and bindings for each protocol, such as Kafka or WebSocket. It is to events what OpenAPI is to REST.

What is the difference between a contract and a binding?

The contract says what an event is: schema, key, meaning. The binding says how it travels on one hop: which topic, how many partitions, which delivery guarantee. Changing technology changes the binding only.

Do I need exactly-once delivery?

Rarely. At-least-once with consumers that upsert by key gives the same result for most analytics. Use exactly-once for money movements and counters that cannot be rebuilt.

Next

Where to go next

Next page: Push to the screen: snapshot first, then small changes

↑ All of real-time analytics · Data platform architecture · Key concepts