Data processing at Spotify using Scio

Two years ago, Spotify introduced Scio, an open-source Scala framework to develop data pipelines and deploy them on Google Dataflow. In this talk, we will discuss the evolution of Scio, and share the highlights of running Scio in production for two years. We will showcase several interesting data processing workflows ran at Spotify, what we learned from running them in production, and how we leveraged that knowledge to make Scio faster, and safer and easier to use.

Session length

40 minutes

Language of the presentation

English

Target audience

Intermediate: Requires a basic knowledge of the area

Who is your session intended to

People interested in big data processing using Scala

Speaker

Julien Tournay (Data engineer at Spotify)

• Scala world
• scala.io
• scale by the bay

Contributes

• Scio