The Rise of the Serverless Data Architectures

QCon New York 2023

Session Serverless

The Rise of the Serverless Data Architectures

Tuesday Jun 13 / 01:40PM EDT, Salon D

Abstract

For a while, it looked like Serverless was just a convenient way to run stateless functions in the cloud. But in the last year we’ve seen the rapid rise in serverless data stores. This includes data stores that are designed for serverless workloads, ones that abstract the underlying infrastructure and scale based on usage.

In this talk we’ll explore the implications of serverless workloads on the design of data stores. Historically, data stores were designed for predictability - providing consistent performance for stable workloads over long-lived low-latency connections. Serverless workloads still require predictable performance, but workload variability is expected, connections are short lived and have wide geographical distribution. As a result, data architectures evolved toward more flexible scalability in order to be cost-efficient to operate with highly variable workloads.

As examples of the changes in workloads and architectures, I’ll share lessons learned while leading the cloud native Kafka team at Confluent, and while building the data architecture at Nile, a still stealthy startup. I’ll also share insights from recent research papers that I believe will shape the data architecture space in the coming years. Of course, Serverless data architecture may or may not be a good fit for your use-case. As we explore these architectures in depth, you’ll acquire the tools you need to decide when to use them.

Topics

Serverless Database Architecture Edge Distributed Systems
76% senior dev or higher
1:11 speaker ratio
60+ practitioners

QCon New York 2023 is a three day conference for senior software engineers, architects and team leads. An international program committee of working engineers selects every session. Patterns and practices, not products and pitches.

Share

From the same track

Tuesday 13 June

10:35 Salon E Session Streaming Laying the Foundations for a Kappa Architecture - The Yellow Brick Road Sherin Thomas Staff Software Engineer @Chime 11:50 Salon E Session Stream Processing Streaming from Apache Iceberg - Building Low-Latency and Cost-Effective Data Pipelines Steven Wu Software Engineer @Apple and Apache Iceberg PMC 13:40 Salon D Session Serverless The Rise of the Serverless Data Architectures Gwen Shapira Founder @Nile, PMC Member @Kafka 14:55 Salon E Session Data Architecture Building a Large Scale Real-Time Ad Events Processing System Chao Chu Software Engineer @DoorDash 16:10 Salon E Session Architecture Enabling Remote Query Execution Through DuckDB Extensions Stephanie Wang Founding Engineer @MotherDuck 17:25 Carroll Gardens Unconference Unconference: Modern Data Architecture & Engineering Ben Linders Independent Consultant in Agile, Lean, Quality and Continuous Improvement