What’s easier than building a data pipeline nowadays? You add a few Apache Kafka clusters and a way to ingest data (probably over HTTP), design a way to route your data streams, add a few stream processors and consumers, integrate with a data warehouse... wait, this looks like a lot of things, doesn’t it? And you probably want to make it highly scalable and available too.
Join this session to learn best practices for building a data pipeline, drawn from my experience at Activision/Demonware. I'll share the lessons learned about scaling pipelines, not only in terms of volume, but also in terms of supporting more games and more use cases. You'll also hear about message schemas and envelopes, Apache Kafka organization, topics naming conventions, routing, reliable and scalable producers and the ingestion layer, as well as stream processing.
Apache Kafka is a distributed streaming platform. It is de facto a standard for building data pipelines and it solves a lot of different use-cases around data processing: it can be used as a message queue, distributed log, stream processor, etc.
We'll start with some theory behind data pipelines in general, then continue with fundamental concepts behind Kafka. We'll also discover important components like Kafka Streams and Kafka Connect.
Topics include
Prerequisites & Requirements
Basic Java 8 or Scala knowledge is preferable. Please install Docker and Docker Compose if you want to run examples locally.
