Introduction to Real-Time Training and Scoring in AI/ML

In the rapidly evolving landscape of AI/ML, the shift from batch to real-time data processing is significant. It impacts how quickly and dynamically we can learn from data, leading to more responsive AI applications. In this session we will explore the shift from batch analytics to real-time decision-making for AI/ML use cases.

We will start by fostering a broad understanding of how to train and score a real-time model on streaming data, facilitated by Kafka/Redpanda, in contrast with traditional batch-processing methods. We will discuss the important aspects of time series data and time-aware features in the context of real-time analytics. Additionally, we will cover how to merge multiple data streams for more complex feature creation and scoring.

This session will include a demonstration using flight and weather data. We'll apply real-time streams to anticipate future air traffic, illustrating a simple application of these concepts that attendees can extend to their own use cases. We will also explore the implications for MLOps in a streaming environment. We'll discuss the adjustments required for real-time data handling and strategies to address the issue of missing data in a real-time setup and how to make decisions when parts of the data streams fail.


Wes Wagner

Solutions Engineer @Redpanda

Wes Wagner is a Solutions Engineer at Redpanda Data, specializing in AWS, Azure, and Google Cloud platforms.

With over 20 years of experience in the IT industry, Wes has honed his skills in cloud services, data science, software engineering, and machine learning. Before joining Redpanda, Wes excelled as a Customer Facing Data Scientist at DataRobot, assisting customers to leverage machine learning and realize the value of augmented intelligence. Wes holds professional certifications in cloud and AI technology platforms, and also has an MBA from Portland State University and a Bachelor's in Computer Systems Analysis from Miami University.

Wes is excited to share his insights on building ML models from streaming data at QCon San Francisco '23.

Read more

Session Sponsored By

Redpanda is a Kafka®-compatible streaming data platform that is proven to be 10x faster and 6x lower total costs.


Wednesday Oct 4 / 01:35PM PDT ( 50 minutes )


Pacific LM


Video is not available


Slides are not available


From the same track


Can’t Apps and Databases All Just Get Along?

Wednesday Oct 4 / 11:45AM PDT

Availability is a tricky thing. In order for your tier 0 endpoints to be available all their hard dependencies have to be available. For Stytch that means basically network, compute, database, and messaging providers.

Speaker image - Joshua Hight
Joshua Hight

Software Engineer @Stytch


Simple Platform Engineering and Simply High Cardinality Observability

Wednesday Oct 4 / 02:45PM PDT

Platform engineering transforms repetitive Engineering requirements into standardised and dependable services, or "paved roads". These paved roads offer engineering teams a smooth way to develop and deploy business logic layers continuously.

Speaker image - Piyush  Verma
Piyush Verma

Co-Founder and CTO @Last9


LLMs + Knowledge Graphs = Better Together

Wednesday Oct 4 / 03:55PM PDT

LLMs are often like the know-it-all at a bar - they can quickly and confidently produce realistic sounding answers to just about any question - even if the answers are complete fabrications.

Speaker image - Mark Quinsland
Mark Quinsland

Senior Field Engineer @Neo4j


Designing AI Agents with System Thinking

Wednesday Oct 4 / 10:35AM PDT

AI agents are a perfect fit for serverless environments, as they can leverage the power of the cloud and the edge for fast and accurate decision making.  It is often the case that AI applications do not require the costly vector database and compute solutions which are often used today, to s

Speaker image - Logan Grasby
Logan Grasby