Outcome of the PoQEMoN project (FUI 16) for metrology of mobile services and networks
The mission of the PoQEMoN* project is to define the reference systems in France and in Europe for measuring the perceptual quality of mobile networks and services. This project was supported by the Systematic competitiveness cluster, by the French government through its FUI interministerial fund (BPI France and the French ‘DGE’ directorate), as well as by the Ile-de-France region.
9 partners (Orange, Altran, Next Interactive Media, Genymobile, Data Publica, HEC, INRIA-Orpailleur and UPEC-Lissi) completed the project, which ended on 1 July 2016. The results of the project were presented to funding agencies as well as to a number of players in the target ecosystem.
ip-label, as project leader, ensured coordination of partners, oversaw the progress of work on the project, and handled relations with public authorities. Project management activities were formalized in 12 meetings of the steering committee.
Results of the project
Metrology and testing infrastructure
ip-label and Genymobile worked together to define the code libraries for developing the metrology tools. Genymobile contributed its expertise in uncovering low-level APIs on Android systems which make it possible to capture metrics at the level of the TCP stack. ip-label developed the metrology framework. ip-label also deployed a network of testing servers located in Europe, Asia, and North America with a view to delivering a measurement tool with worldwide capabilities.
This network was deployed in the Amazon cloud on servers with high network traffic capacities. The servers were qualified by ip-label and Genymobile using a battery of automated tests. A dynamic routing system was set up so that testing could be conducted on the target server closest to the web user.
Application of mobile video QoE measurement
The partners worked on an application dedicated to studying QoE measurement and the correlations between QoS and QoE. Mobile streaming video was the usage they chose. The partners relied on the metrology framework to define how the application would operate. Altran coordinated this SP to develop an application in two parts:
- a questionnaire to fill in by the user to determine his/her profile and collect the assessment of viewing quality
- a YouTube player/viewing tool for collecting information about the quality of the network at the level of the device.
How data would be structured was also defined to enable automation of the collection of results into a database for conducting later studies. The HEC took part in defining test campaigns. When the specifications had been drawn up, it conducted a few series of tests, and implemented task sharing with the UPEC: while the UPEC would concentrate on conducting tests intensively on the video measurement application, the HEC would focus its efforts on the crowdsourcing application (01Net SpeedTest).
QoE model and QoS/QoE correlation
This part of the project meant creating a QoE model on the basis of the results of preceding work. For this evaluation campaigns were set up by the UPEC and Orange. In all, 2100 evaluations were conducted by 218 users, a very significant database for this type of test. The UPEC proposed a formal model to structure the data that was collected. This database made it possible to refine the statistical analyses which deal with the interaction between the various factors that affect the user’s perception. In this way it was possible to ascertain discriminating characteristics and thus select the best factors that affect users with respect to their contexts. These analyses enabled the UPEC to propose a prediction model of the user’s actual experience based on a multi-modular, adaptive, self-regulating approach involving selected factors and using artificial intelligence tools. Lastly, the UPEC started an experimentation and validation campaign of the proposed module by integrating it into an MPEG-Dash video player.
Orange devoted special attention to the correlation between quality of service (QoS), which can be measured using automated tools, and the user’s quality of experience (QoE), which requires customer surveys that are impossible to implement on a grand scale. Orange worked from data that was propagated from the video demonstrator, which ran an application for mobiles to enable users to watch YouTube videos. After a video played, a questionnaire was automatically sent to the user to collect his/her qualitative experience. Orange then compared its vision (quality of service) with the quality perceived by the user (quality of experience) as well as with the data fed back by the device itself via VLC metrics. From this study a number of lessons were learned about the operation of video streams distributed by YouTube. Thus, thanks to the studies conducted within the PoQEMoN project, Orange was able to elicit metrics that were straightforward and therefore usable by operators to predict their users’ QoE on YouTube type content.
01Net SpeedTest crowdsourcing application
In this part of the project, the partners worked, on the basis of the results of preceding work, to develop and deploy a measurement application, 01Net SpeedTest, for consumer audiences. These developments were carried out based also on the results of the SP5 concerning the anonymization of data in conformity with current regulations. Next Interactive supplied a design of the application. An extensive series of tests led to the validation of the application in diverse and varied contexts, and also populated the measurement database which is visible to the user in the form of a map of the results. The results of the measurements were structured within a database which in turn allowed the different tools of SP5 to be set up. Data Publica also contributed to the project by using and testing the application, and then supplying feedback about its use.
HEC contributed a great deal to validating the prototype by conducting a first series of measurements centered on the Paris region. After this first phase, HEC conducted measurement campaigns whose aim was to demonstrate the utility of 01Net SpeedTest and the value of its measurements for communications network subscribers, telecom operators, local authorities, and more broadly, transport infrastructure operators and players in mobility. Test campaigns were carried out in Paris on two operators, and then a wider campaign looked at 99.92% of the infrastructures of the SNCF national railway network open to regular passenger traffic. More than 100,000 measurements thus were made were made throughout the duration of the project.
Data anonymization, analysis, mining and visualization
Data Publica took charge of coordinating this work with INRIA-Orpailleur, ip-label, and HEC. Data Publica’s specific contribution was to develop a data visualization to present the results of the project visually and comprehensibly. Development was carried out in two phases:
- Creation of a first POC on the scale of the Paris metro system. This first POC enabled testing of the first ideas for representing and selecting the analyses to highlight.
- Expansion to the scale of metropolitan France on SNCF railways. After integrating the lessons learned from the first POC, a new data visualization was developed to cover the whole of France and display the whole of results from the SNCF network by integrating data from the SNCF’s Open Data API.
In addition we created a tool which allows interactive input along the path of SNCF passenger railways. HEC used this tool to map quality on the routes of commercial rail lines.
The work conducted by the INRIA focused on developing anonymization algorithms and data visualization. Anonymization within the PoQEMoN project is essential at several levels. First of all, project data are collected participatively from users; this requires anonymization methods at the source. In addition, the consortium intends to publish and analyze some of the data to transform them into knowledge in this field. Such work requires data mining algorithms which inherently offer guarantees in terms of privacy.
To implement algorithms we first worked to elaborate a data analysis web demonstrator. This demonstrator enabled us in a first stage to visualize the data gathered by the project, and then to apply data mining algorithms to analyze and better understand them. The second part of our project focused on implementing data anonymization algorithms based on reputed scientific publications in the field. This part relied on association rule extraction methods and improving an anonymization procedure to apply it effectively in the context of the project.
Furthermore, the INRIA, with the project team Orpailleur, also played a part in outreach, communicating about the project, for example, at the INRIA-Industry convention in Bordeaux in 2015.
Dissemination and economic model
HEC conducted a study on the dissemination and valorization of results, especially on the 01Net SpeedTest crowdsourcing application, beyond the project’s lifetime. In close partnership with ip-label and Next Interactive, ways have been finalized to continue exploiting the data obtained by the 01Net SpeedTest app to make them available as “open data”, accessible to consumers at large. In partnership with the group La Poste, the three partners are responding to the call for expression of interest issued by the ARCEP (French national communications regulator) about crowdsourcing. Lastly, various players in mobility are being approached: transport operators, local authorities, consumer associations and others, to disseminate widely or in specific target niches the results gleaned from the crowdsourcing application.
The number of actions undertaken to disseminate the project rose at the same pace as the progress of project studies. HEC in particular presented the results at the Mobile World Congress in Barcelona in February 2016, at its own booth in addition to ip-label’s in the French Tech pavilion.
*PoQEMON : Platform for Quality Evaluation of Mobile Networks.