A Million People want a Million Things and They Want Them Now: How to Meet Personalised User Demands in Realtime

As the amount of data in Fintech and cryptocurrency applications has increased, Kafka has become the norm for wrangling this surge at speed and scale. However, the need to send personalized data streams to users has become paramount. Your mobile day trader only wants what they can use, and neither you nor they want the data bill for everything they are not using. You also don’t want topics per user. This is where it started, but as we’ve encountered more Kafka users, the needs are increasingly similar - whether it's large-scale chat systems, betting applications, or social media. Data access patterns are not but all need to be handled fairly. In this session, I’ll walk you through designing a scaling consumer system to allow for customized user streams in high cardinality data topics, all while minimizing the data burden on user devices. I’ll be demonstrating a few use cases of the system and running through the various tradeoffs that can be made to both improve user latency and add high availability to the system. Follow along to learn about the key tradeoff of user experience vs cost, latency vs flexibility, and the increasing demands of user-facing systems.


Roman de Oliviera

Solutions Engineer @Ably

Roman is a Solutions Engineer at Ably where he helps customers realize the benefits of infrastructure that powers realtime digital experiences and creates solutions unique to the customers' challenges. Prior to Ably, Roman worked at MuleSoft as Lead Solution Engineer.

Read more


Tuesday Oct 25 / 10:35AM PDT ( 50 minutes )




Kafka Streaming Blockchain Database Development


From the same track

Session Reliability

How to Build Reliable Systems Under Unpredictable Conditions

Tuesday Oct 25 / 11:50AM PDT

Today's success of businesses depends on their systems' stability and reliability. Downtimes are painful: They cause you headaches and cost your business money. Providing reliable services to your customers is essential to success.

Speaker image - Benjamin Wilms
Benjamin Wilms

Co-founder & Product minded engineer @Steadybit

Session Architecture

Architecting Your Data Platform Across Clusters, Regions, and Clouds

Tuesday Oct 25 / 02:55PM PDT

Data platform teams are increasingly challenged with accessing multiple data stores that are separated from compute engines, such as Spark, Presto, TensorFlow or PyTorch.

Speaker image - Adit Madan
Adit Madan

Director of Product Management @Alluxio

Session Sustainability

Bringing Green, Sustainable Software Solutions Into the Enterprise

Tuesday Oct 25 / 05:25PM PDT


Speaker image - Adam  Jordan
Adam Jordan

Distinguished Software Engineer @Shell

Session Event-Driven Architecture

A Guide to the Event-Native World of Open Standards: AsyncAPI, CloudEvents, and Shared Signals & Events

Tuesday Oct 25 / 04:10PM PDT

More and more, Event-Driven Architecture (EDA) is being adopted by organizations.

Speaker image - Linus Hakansson
Linus Hakansson

Vice President of Product @ Gravitee