Yaroslav Tkachenko

Software Architect at Activision

Talk

Building Scalable and Extendable Data Pipeline for Call of Duty Games: Lessons Learned
YET TO BE SCHEDULED
Topics:
kafka
data pipelines
stream processing
Level:
Intermediate

Your rating:
0/5

What’s easier than building a data pipeline nowadays? You add a few Apache Kafka clusters and a way to ingest data (probably over HTTP), design a way to route your data streams, add a few stream processors and consumers, integrate with a data warehouse... wait, this looks like a lot of things, doesn’t it? And you probably want to make it highly scalable and available too.

Join this session to learn best practices for building a data pipeline, drawn from my experience at Activision/Demonware. I'll share the lessons learned about scaling pipelines, not only in terms of volume, but also in terms of supporting more games and more use cases. You'll also hear about message schemas and envelopes, Apache Kafka organization, topics naming conventions, routing, reliable and scalable producers and the ingestion layer, as well as stream processing.


Workshop

Building Data Pipelines with Apache Kafka
Wednesday 9:00 - 17:00
Topics:
kafka
data pipelines
stream processing
Level:
Intermediate
Your rating:
0/5

Apache Kafka is a distributed streaming platform. It is de facto a standard for building data pipelines and it solves a lot of different use-cases around data processing: it can be used as a message queue, distributed log, stream processor, etc.

We'll start with some theory behind data pipelines in general, then continue with fundamental concepts behind Kafka. We'll also discover important components like Kafka Streams and Kafka Connect.

Topics include

  • Data pipelines 101: ingestion, storage, processing
  • Kafka fundamentals: topics, partitions, brokers, replication, etc.
  • Producer and Consumer APIs
  • Kafka Streams as a processing layer
  • Kafka Connect for integrating with external systems
  • Kafka best practices and tuning

Prerequisites & Requirements

Basic Java 8 or Scala knowledge is preferable. Please install Docker and Docker Compose if you want to run examples locally.

About

Yaroslav Tkachenko is a software engineer interested in distributed systems, microservices, functional programming, modern cloud infrastructure and DevOps practices. Currently Yaroslav is a Software Architect at Activision Blizzard, working on a Big Data platform for Call of Duty games.

Prior to joining Activision Yaroslav held various leadership roles in multiple startups. He was responsible for designing, developing, delivering and maintaining platform services and cloud infrastructure for mission critical systems.

Create account

You need an account to save events or rate talks.

Already have an account? Sign in

Sign in

Sign in to save events or rate talks.