Relational Data at the Edge

Data storage and access at the edge delivers massive performance gains by reducing location-sensitive latency. Cloudflare operates a distributed cross-region database architecture at the edge, serving over 50TB of online transactional data over millions of requests across our entire security and web performance product suite. Storing and managing relational data at the edge carries a unique set of challenges dictated by timeless CAP constraints and highly variable load conditions, requiring careful tradeoffs.

How can we minimize replication lag, choose replication strategies (asynchronous, streaming, logical, custom tooling) that balance consistency and performance, optimize request routing, and even speed up split brain cluster resynchronization? Achieving high availability storage performance at the edge demands innovative techniques involving custom performance optimizations across our anycast BGP network, cluster management layer, and down to the core Postgres recovery system.

So, you still want to build your database architecture at the edge?

You’ll learn about Cloudflare’s entire edge database architecture, unique challenges and practices for data replication, failover and recovery, and custom performance techniques. We'll also peek into the exciting future landscape of edge storage.

What's the focus of your work these days?

Building a managed Postgres environment at Cloudflare, working on building edge applications, and contributing to Postgres and related open-source projects.

What's the motivation for your talk at QCon San Francisco 2023?

To help everyone understand the challenges of data at the edge and what are the ways to solve them.

How would you describe your main persona and target audience for this session?

Someone who is a Systems Engineer, SRE or even a developer who is curious in understanding edge computing and how to enable it through Postgres.

Is there anything specific that you'd like people to walk away with after watching your session?

I would expect attendees will learn something about the limitations/challenges of data at the edge, and opportunities if done right.


Speaker

Justin Kwan

Software Engineer Intern - iCloud Edge @Apple, Previously @Cloudflare

Justin Kwan is a Software Engineer Intern at Apple and a CS major at the University of Waterloo. He is currently focused on building high-throughput DDoS mitigation that protects iCloud’s data plane. Formerly at Cloudflare, Justin focused on multitenant resource isolation, storage reliability, chaos experimentation across distributed database infrastructure, and patching the performance of large open-source systems such as Postgres and PgBouncer. His excitement lies in the realms of distributed networking and storage, systems software development, kernel-layer observability, and the intricacies of database query execution.

Read more
Find Justin Kwan at:

Speaker

Vignesh Ravichandran

Engineering Manager @Cloudflare, Contributor to Postgres, Previously at Ticketmaster

Engineering Manager of Databases at Cloudflare. Manages a team of engineers across the world who is responsible for ensuring the databases are available, reliable, and highly performant 24*7*365. Created spinup and other projects. He is a frequent author and speaker at various open-source conferences.

Read more
Find Vignesh Ravichandran at:

Date

Monday Oct 2 / 11:45AM PDT ( 50 minutes )

Location

Ballroom BC

Topics

Database Edge Computing Cloud Computing Postgres SQLite Distributed Systems Architecture High Availability OLTP

Share

From the same track

Session Serverless

AWS Lambda Under the Hood

Monday Oct 2 / 10:35AM PDT

AWS Lambda is a serverless compute service running at a massive scale! Supporting packages of up to 10GiB while allowing over 15K new containers per second (for a single customer) and serving millions of TPS across millions of unique workflows is a challenging problem.

Speaker image - Mike Danilov
Mike Danilov

Senior Principal Engineer @AWS Lambda

Session Distributed Systems

Managing 238M Memberships at Netflix

Monday Oct 2 / 02:45PM PDT

Have you ever wondered what goes on behind the scenes when you sit back, relax and watch Netflix? How does Netflix own and operate their system of record for all members making sure they continue to be in good standing and get the best experience possible?

Speaker image - Surabhi Diwan
Surabhi Diwan

Senior Software Engineer @Netflix

Session

Unconference: Architectures You've Always Wondered About

Monday Oct 2 / 03:55PM PDT

What is an unconference? An unconference is a participant-driven meeting. Attendees come together, bringing their challenges and relying on the experience and know-how of their peers for solutions.

Session K8s

NIST 800-207A: Implementing Zero Trust Architecture

Monday Oct 2 / 01:35PM PDT

Zero Trust is all about replacing implicit trust based on perimeter security and network access with explicit trust based on identity and runtime authorization.

Speaker image - Zack Butcher
Zack Butcher

Founding Engineer @Tetrateio & NIST co-author on security, prev core services @GoogleCloud

Session Recommender Systems

Unpacking how Ads Ranking Works @Pinterest

Monday Oct 2 / 05:05PM PDT

In this session, we delve into the dynamic world of social media advertising. Facebook, Snap, Pinterest, Twitter, and many others generate the majority of their revenue from targeted ads.

Speaker image - Aayush Mudgal
Aayush Mudgal

Senior Machine Learning Engineer @Pinterest, Focusing on Privacy Safe Recommender Systems, IIT Kanpur Alumnus