The quality of experience, QoE, is an important issue for network operators and more generally for service providers over Internet, so for Orange as it has both roles.
However, this concept is complex as it deals with different components which impact the customer’s satisfaction: richness and price of the offer, hot line responsiveness and accuracy, subjective quality of the provided service, etc. Thus, the quality of experience is a complex mix of many aspects of services over Internet. Among these, the quality of the information transfer through the network plays a central role. This aspect is generally, defined/ understood /measured by quality of service parameters describing the network performance (packets loss, delay, jitter, etc.) Beyond the quality of service, others factors are involved: generating packet flow by the service platforms servers (flat flow vs. flow in bursts packets), restitution protocols of information (throughput adjustment, delay before display, etc.) and quality of display on the user’s device (typically on player for video flow)
To investigate these complex links between network and service, the PoQeMoN project was initiated within System@tic cluster under the leadership of IP Label and others key actors, namely Orange, the ALTRAN group, the Data-Publica company, the INRIA Orpailleur laboratory, Genymobile, the Next Radio group, HEC and the Paris Est Créteil University.
The PoQeMoN project intends to answer to the following issues:
- To help operators to better understand the quality of service they really deliver to users by providing them reliable and economical tools that can be used for audit campaign (coverage, troubleshooting, benchmark) with reduced costs.
- To help companies to improve performance of access to the information systems in mobility context (choice of right operator, improvement of various point of infrastructure with operator agreement, …)
- To help users to know the real quality of service delivered by mobile operators, by giving some understandable quality indicators, with the objective to help them in choosing an operator not only on prices but on the required quality regarding theirs needs and usages too.
Even if every year ARCEP, the French regulation authority, studies are published to qualify the mobile networks quality in France with speed tests and confirm the excellence of Orange networks, the PoQeMoN project aims to better understand the relationship between quality of service and user experience.
The PoQeMoN project is focusing in mobile networks (3G and 4G) and gives a full end-to-end evaluation system of quality of service (QoS) of these networks, of the user experience of the service (QoE) to establish a link between these two aspects.
To this goal, the project developed four studies axes:
- To accurately measure end-to-end quality of mobile networks using an application embedded in the device,
- To model the user experience (QoE) for mobile applications, extending the model available for voice quality evaluation (ITU 862/MOS PESQ) to various applications, focusing on video on a first step,
- To build and to process relations between QoE and QoS in mobile networks, enabling operators to have a global vision of the delivered quality and the perceived one.
- To visualize in an efficient and simple way all the collected data (tables, maps, etc.)
As relation between quality of service (QoS), measurable by operator, and quality of experience (QoE) is not simple to highlight and strongly depends of the experienced service, two applications are developed in an Android environment (version 4.2.2 and higher):
- The first one is a speed test that allows testing the network quality of user’s operator by measurements made on download and upload throughput (by file download), the latency (by ping). This application is available here. https://play.google.com/store/apps/details?id=com.nextradiotv.speedtest01net&hl=fr
- The second one offers to visualize short Youtube video (60 s of duration) and systematically asks the user perception about the content seen. A first survey collects information about the user as this could impact its perception (like his familiarity or not with Youtube viewing). The second survey collects the user perception. This application is available here https://5g-token.col.ip-label.net/web-rtrnetztest/apk/StreamVLC.apk
The speed test application gives at the end of the test a summary of outcomes and more useful their interpretation thanks to a colored line that shows if measured KPI are in line with, for example, a video viewing. Furthermore, all kinds of collected data are represented on a map. It is a collaborative one as every user of this application has access to results of all users. Thus, an user is able to compare its operator with others ones in his points of interest (workplace, daily journey, home, …) This map allows the viewing of all tests with different filters: all operators or a specific one, all access mode (2G/3G/4G) or one in particular, on a customizable observation period, zoom on a city/ zone… The user has his test history too. An example of results at the end of a test and the available map in the application are shown in Figure 1.
Figure 1 : Screenshots of the speed test application
In Figure 2, the survey of the second application is shown.
Figure 2 : Screenshots of the video survey
Beyond these specific applications developed in the PoQeMon project and illustrated above, this project implements for the first time in France a collaborative approach in network metrology domain. Users run some tests and report their results (including subjective ones) on servers which are then used to qualify the quality of experience in French networks. This collaborative approach between users, network operators, service providers and test agencies is an axis to further explore to give back to all actors a shared vision of the quality. The end to end approach requires user collaboration but work with every content, every protocol, encrypted or not. This is important because today most content is encrypted and moreover new transports protocols like QUIC (introduced by Google) also forbid basic QoS measurement like packet loss. Thus the “crowd measurement” would restore this lost equilibrium.