The Journey to a Million Ops / Sec / Node in Venice

Venice is an open-source derived data platform developed by LinkedIn. It is used mainly for ML feature storage, which requires the ability to refresh data at very high throughput, and to look it up with low latency.

The lion share of the project is in Java, although it also leverages RocksDB and ZSTD via JNI. On the Java side, every bit of performance which can be squeezed is fair game.

After briefly presenting Venice, this talk then deep dives into some of the tricks we have employed in our relentless pursuit to lower read latency and to reach 1M operations per second per node.


Speaker

Alex Dubrouski

Technical Lead of Server Performance Team @LinkedIn

Alex joined LinkedIn in 2020 and since then has been working on optimizing server-side performance primarily for Java based applications. Alex also contributes performance patches to OSS projects like OpenJDK, Log4J and Venice.

Before LinkedIn, Alex spent 4 years working on performance and infrastructure at Pandora Media.

Read more
Find Alex Dubrouski at:

Speaker

Gaojie Liu

Senior Staff Software Engineer @LinkedIn, Open Source Contributor @Venice, a Massive Scalable Derived Data Platform

Gaojie joined LinkedIn in 2016 and since then, he has been working on Venice, a massively scalable derived data platform. He has been working on various aspects of Venice, such as new feature development, performance tuning and architecture evolution.

Prior to LinkedIn, Gaojie had worked in Yahoo for about 5 years, where he was mainly developing Yahoo Search Gateway platform.

Read more
Find Gaojie Liu at:

Date

Tuesday Oct 3 / 11:45AM PDT ( 50 minutes )

Location

Pacific DEKJ

Topics

Distributed Systems Performance Horizontal Scalable Feature Store Java Data Platforms

Share

From the same track

Session Java

How Netflix Really Uses Java

Tuesday Oct 3 / 10:35AM PDT

Netflix is (in)famous for the way we develop our systems. Micro Services, RxJava, Hystrix and Spring Cloud Netflix are just a few things that developers often associate with Netflix.

Speaker image - Paul Bakker

Paul Bakker

Java Platform @Netflix, Java Champion, and Co-Author of "Java 9 Modularity"

Session jvm

Optimizing JVM for the Cloud: Strategies for Success

Tuesday Oct 3 / 03:55PM PDT

Embracing cloud-native is more than just a trend; it's a strategic move for modern businesses. As organizations pivot from monolithic structures to agile micro-services, the role of the JVM becomes increasingly significant.

Speaker image - Tobi Ajila

Tobi Ajila

Eclipse OpenJ9 JVM Engineer @IBM, Optimizing JVMs for Cloud Environments

Session Java

The Keys to Developer Productivity: Collaborate and Innovate

Tuesday Oct 3 / 05:05PM PDT

In the JVM track at QCon SF, the speakers talked about the role of collaboration and innovation and how this is impacting their business.

Speaker image - Heather VanCura

Heather VanCura

Vice President, Community Engagement @Oracle, Director & Chairperson at Java Community Process (JCP) Program, MySQL Community/DevRel, and Board Member

Session

Unconference: JVM Trends

Tuesday Oct 3 / 02:45PM PDT

What is an unconference? An unconference is a participant-driven meeting. Attendees come together, bringing their challenges and relying on the experience and know-how of their peers for solutions.

Session

Harnessing Exotic Hardware: Charting the Future of JVM Performance

Tuesday Oct 3 / 01:35PM PDT

Monica Beckwith offers a deep dive into the intricate world of the JVM and its evolving relationship with exotic hardware in her enlightening presentation at QConSF's JVM trends track.

Speaker image - Monica Beckwith

Monica Beckwith

Java Champion, Author of JVM Performance Engineering, JVM Performance Expert @Microsoft, First Lego League Coach, Advocate for STEAM + kids