Velocity 2015: Building Real-Time metrics pipeline by AOL
Velocity 2015, the international convention dedicated to web performance and operations, is being held this year in Amsterdam.
On location in Amsterdam the first day, we attended a very interesting presentation by AOL (now recentered on media). The topic was “Building real-time metrics pipelines”.
It was a presentation of their real-time data collection and processing platform.
It was exciting to discover that the objectives of this platform are to make information from a variety of sources (client devices, systems internal and external to AOL) available to diverse “consumers”, among which:
- real-time visualization tools,
- real-time analytics services,
- other services.
The designers of this platform set themselves ambitious goals because they plan to collect and analyze 1.3 billion events each day.
From a technical point of view, data are seen as a flow or pipeline which the parts that make up the system are connected to. To create this pipeline and enable various consumers to connect to it, AOL used RabbitMQ. Note that RabbitMQ is written in Erlang, a language that is widely used in the world of telecommunications for real-time purposes.
Fig1: data flowing through the pipeline and the consumers connected to it
Micro-services, developed on the basis of NodeJs, then make it possible to access data to serve the aims of the platform, among which aims are visualization and analytics.
Consolidation and aggregation of data are processed with ElasticSearch which provides fast querying, analysis, and display.
Future work that AOL announced involves exploiting these data, particularly for the purposes of real time predictive performance analysis as well as real-time social sentiment analysis, which currently is all the rage. Lastly, AOL announced that it was looking to migrate the platform to Amazon AWS. Here’s proof that this type of infrastructure requires constant development, which is hard to do with in-house infrastructure management.