Coding / Programming Videos

Post your favorite coding videos and share them with others!

Andrew Montalenti & Keith Bourgoin – Real-time streams and logs with Storm and Kafka

Download Video link >



Andrew Montalenti & Keith Bourgoin – Real-time streams and logs with Storm and Kafka

PyData SV 2014
Some of the biggest issues at the center of analyzing large amounts of data are query flexibility, latency, and fault tolerance. Modern technologies that build upon the success of “big data” platforms, such as Apache Hadoop, have made it possible to spread the load of data analysis to commodity machines, but these analyses can still take hours to run and do not respond well to rapidly-changing data sets.

A new generation of data processing platforms — which we call “stream architectures” — have converted data sources into streams of data that can be processed and analyzed in real-time. This has led to the development of various distributed real-time computation frameworks (e.g. Apache Storm) and multi-consumer data integration technologies (e.g. Apache Kafka). Together, they offer a way to do predictable computation on real-time data streams.

In this talk, we will give an overview of these technologies and how they fit into the Python ecosystem. This will include a discussion of current open source interoperability options with Python, and how to combine real-time computation with batch logic written for Hadoop. We will also discuss Kafka and Storm’s alternatives, current industry usage, and some real-world examples of how these technologies are being used in production by Parse.ly today.

source

Bookmark(0)
 

 

View Comments source >

Transcript view all >

00:07 so let me introduce Andrew mantel NT

00:12 who's the CTO and founder of parsley

00:15 which is startup doing working with

00:18 content publishers to provide them

00:20 analytics and more relevance to reach

Leave a Reply

Please Login to comment
  Subscribe  
Notify of
Translate »