Speaker
Abstract
In the ever changing landscape of big data, focus is slowly moving away from batch and towards realtime analytics. Data Science workflows are evolving to adapt to this changing landscape. Realtime analytics are limited only by the underlying architecture that enables low latency ingestion, processing and serving of high throughput data. Lambda architecture unified batch and realtime to provide low latency computations with eventual correctness. However the challenges of maintaining two different codebases(among other things) made operations hard.
The founder of Kafka talks about a streaming first kappa architecture in this 2014 article. A streaming first, single path solution that can handle realtime processing as well as reprocessing and backfills. Kappa architecture makes sense, however it is 2023 and it has a long way to go for full scale adoption. A paradigm shift is required in how we design data infrastructure. Now, instead of worrying about two codebases, we need to worry about bootstrapping from a stream, backfills from history, idempotent sinks to handle reprocessing etc.
In this presentation, I will talk about strategies to evolve your Data Infrastructure to enable Kappa architecture in your organization. An iterative roadmap to move away from Lambda, while ensuring minimum disruption to end-users. I will be using real world examples from tech companies as case studies. By the end of this presentation you will walk away with a concrete roadmap for designing a data platform built on Kappa architecture.
Interview
I work as a Staff Engineer for Chime, which is a leading fintech company. In this role I’m helping build out a robust data platform that can serve analytics, fraud and risk, machine learning, experimentation etc. My main focus has been to provide capabilities to our users to derive real time insights from our data. Apart from that I’ve been focussing a lot on data governance and discovery as well as the usability of our data platform. Basically, figuring out how we can provide value to our end-users so they can find and use data with speed and ease without worrying about the internals of it all.
This is probably a very wordy way to say my current role touches upon all aspects of data :)
On the side, I have also been playing the role of an advisor to nonprofits and helping them scale their data infrastructure and enable a platform for ML and AI exploration. Very recently I worked with NASA’s SpaceML program to productionize a self supervised learning system that ingested petabytes of earth’s imagery and allowed scientists to search for weather phenomena such as hurricanes et al from these images.
I’ve been in the data streaming world for a long time, have worked with Flink, Kafka, AWS Kinesis, Lambda architecture, Kappa architecture, all the fun stuff. One common problem I’ve seen in organizations is that they struggle to evolve their existing architecture to make use of the cutting edge streaming technologies like Flink, Kafka infinite retention, Beam model and all. A big reason for this is a mindset shift, not to mention operational challenges that come with big change.
With this talk I hope to give people a framework to think about the goals of their data platform and work backwards to slowly evolve their architecture and modernize it with minimum disruption to their current way of doing things.
My talk is targeted towards staff engineers, architects, product and organization leads. Speaking of persona - anybody who is working with real-time data, streaming, or online training would gain something from this talk. However, folks who work for traditional data orgs where 80 - 100% of use cases are in batch but are thinking about modernizing their architecture will find the most value out of it.
- Modern streaming technologies have evolved in leaps and bounds.
- Anyone can enable real-time application in their organization without the level of operational overhead involved in traditional system.
- Even though streaming and real time is the hot topic, it does not mean batch data is going away. A typical organization should be equipped to support any kind of application on the latency and consistency spectrum.
I learned a lot of things but my favorite talk was Laura Mcguire’s talk about Exploring Costs of Coordination During Outages. I never thought about incident response in that manner and it shed so much light on what we could be focussing on.
Overall I had a lot of great discussions about machine learning platforms, kubernetes and just generally it was nice to hear about different problems spaces.
Topics
QCon New York 2023 is a three day conference for senior software engineers, architects and team leads. An international program committee of working engineers selects every session. Patterns and practices, not products and pitches.
Part of the track
Modern Data Architecture & Engineering Hosted by Allen Wang Senior Staff Engineer @DoorDashFrom the same track
Tuesday 13 June
10:35 Salon E Session Streaming Laying the Foundations for a Kappa Architecture - The Yellow Brick Road Sherin Thomas Staff Software Engineer @Chime In the ever changing landscape of big data, focus is slowly moving away from batch and towards realtime analytics. Data Science workflows are evolving to adapt to this changing landscape. 11:50 Salon E Session Stream Processing Streaming from Apache Iceberg - Building Low-Latency and Cost-Effective Data Pipelines Steven Wu Software Engineer @Apple and Apache Iceberg PMC Apache Flink is a very popular stream processing engine featuring sophisticated state management, even-time semantics, exactly-once state consistency. For low latency processing, Flink jobs typically consume data from streaming sources like Apache Kafka. 13:40 Salon D Session Serverless The Rise of the Serverless Data Architectures Gwen Shapira Founder @Nile, PMC Member @Kafka For a while, it looked like Serverless was just a convenient way to run stateless functions in the cloud. But in the last year we’ve seen the rapid rise in serverless data stores. 14:55 Salon E Session Data Architecture Building a Large Scale Real-Time Ad Events Processing System Chao Chu Software Engineer @DoorDash Two years ago, we embarked on building DoorDash's ad platform from the ground up. Today, our platform handles over 2 trillion events every day and our advertising business has experienced significant growth in recent years, becoming a key area of focus for the company. 16:10 Salon E Session Architecture Enabling Remote Query Execution Through DuckDB Extensions Stephanie Wang Founding Engineer @MotherDuck DuckDB is a high-performance, embeddable analytical database system that has gained massive popularity in the last few years. 17:25 Carroll Gardens Unconference Unconference: Modern Data Architecture & Engineering Ben Linders Independent Consultant in Agile, Lean, Quality and Continuous Improvement What is an unconference? An unconference is a participant-driven meeting. Attendees come together, bringing their challenges and relying on the experience and know-how of their peers for solutions.