Telchemy - VQmon/EP

Home

Products

VQmon Performance

VQmon Algorithm Description

VQmon was developed specifically for measuring the performance of IP-based multimedia communications, e.g., Voice and Video over IP, and traditional TDM/PCM systems.

IP impairments typically result from network congestion, which is usually caused by excessive data traffic and is hence strongly time varying. VQmon measures the distribution of lost and discarded packets using a multi-state Markov Model that is designed to detect two primary states within calls: the burst state, in which the rate of packet loss and discard is high enough to cause noticeable degradation in quality, and the gap state. Call quality is calculated separately for each state and combined using a perceptual model that represents how people react to time-varying call quality.

Telchemy has devoted over six years to studying the impact of IP impairments on VoIP and Video codecs. For VoIP, Telchemy has characterized all the various rates and variations of G.711, G.722, G.722.1, G.722.2, G.723.1, G.726, G.728, G.729, GSM HR, GSM FR, GSM EFR, AMR NB, AMR WB, SMV, EVRC, Global IP Solutions' ILBC, Audiocodes' NetCoder, Broadcom's BV16, Lucent's SX7300 and SX9600, and many other narrowband and wideband codecs. For each rate or variant of each codec, 560 tests were typically conducted using P.862, resulting in a database with approximately 60,000 test results. In addition, the results of 26 subjective tests have been collected, covering a wide range of codecs and conditions. This large database of test data has allowed Telchemy to develop accurate codec models, and hence accurate call quality estimates.

In addition to IP impairments, VQmon can incorporate signal level, noise level, echo return loss, and delay, allowing call quality metrics to reflect a wide range of conditions. This allows VQmon to be used in pure PCM/TDM applications as well as hybrid TDM/VoIP systems.

VQmon Correlation with Subjective Test Data

VQmon correlates very well with subjective test data, as shown in the two charts below. The first chart shows how the Listening Quality MOS score calculated by VQmon compares to the ACR MOS score for thirty different codecs obtained from a series of subjective tests under conditions of zero impairments. Each point in the scatter diagram shows the VQmon MOS score vs the ACR test MOS Score for one of the codecs. The trend line through the points in the scatter diagram closely follows the ideal 45 degree characteristic and the points can be seen to cluster closely around this line.

The second chart below compares VQmon's estimated MOS score with subjective test data from COMSAT labs, for the G.711 codec with packet loss concealment under a wide range of packet loss conditions. This codec is one of the most commonly used, and hence this is a key result. The chart shows that VQmon closely tracks the subjective test score.

These two charts clearly show that VQmon accurately predicts MOS score, under the test conditions used.

VQmon Correlation with P.862 Test Data

P.862 is widely used and accepted as an intrusive test method; i.e., one in which an audio file is input to a system and the output audio file compared to the input. This is a computationally intensive process, but it does produce quite stable perceptual quality measurements.

The first chart compares VQmon's estimated P.862 score to a measured P.862 value for the G.711 codec with packet loss concealment. It can be clearly seen that VQmon closely tracks the measured P.862 scores, obtaining a correlation factor of 0.993.

The second chart compares VQmon's estimated P.862 score to measured values for the 3G cellular AMR codec at 12.2 kbits/sec. This again shows a very high degree of correlation.

VQmon vs. the E Model

VQmon does use elements of the ITU G.107 E Model, in addition to the perceptual model defined in ETSI TS 101 329-5 Annex E, and incorporates a series of highly efficient integer approximations to the very complex floating point equations in G.107. In addition to the accurate MOS scores and R factors produced by VQmon, the algorithm also outputs a "G.107 R factor", which is calculated in accordance with G.107. This value does not correlate with subjective or objective test data as do VQmon's call quality metrics; however, it is a requirement in some countries.

The first chart below shows how accurately VQmon's highly efficient algorithms are able to approximate the E Model's floating point equations. The chart shows the difference between VQmon's MOS calculation and that obtained from the E Model under a very wide range of conditions and codec types. The error is between -0.02 and +0.06, which corresponds to an accuracy of -1.0/+1.5 percent.

Test conditions included SLR 0-18dB, TELR 15-65dB, noise -50/-65dBm0, delay 0-400mS, packet loss 0-20 percent and seven different types of codec.

The E Model does not consider the effects of time varying IP impairments, which can lead to significant loss of accuracy under conditions that can occur quite frequently. The chart below shows how VQmon and E Model compare under conditions of strongly varying packet loss. These results are based on data presented by France Telecom to the ITU, in which they generated audio files of 3 minutes duration containing strongly varying packet loss conditions and obtained subjective test results. VQmon and the E Model were both used to determine estimated MOS scores and these plotted against the MOS scores obtained by France Telecom.

Each point in the scatter diagram shows how for one test condition, the MOS score estimated by VQmon or the E Model compared with the subjective test scores obtained by France Telecom. It can be clearly seen that the E Model points are more widely dispersed and tend to estimate a MOS score that is approximately 0.5 too low, whereas VQmon's estimated MOS scores cluster more tightly with the ideal line and the trend line through the VQmon points intersects the ideal trend line at (2.7,2.7).

These charts clearly show that VQmon is able to very accurately approximate the much more computationally intensive E Model under steady-state conditions, as well as outperforming the E Model under conditions of time-varying packet loss/discard.

Conclusions

VQmon is a highly efficient and accurate call quality measurement algorithm. Telchemy has invested considerable time and resource into detailed analysis of codec performance, algorithm development and refinement, which has resulted in VQmon producing accurate and repeatable call quality metrics.