System for the Evaluation of Audio Quality

Audio quality can be measured either subjectively, with human subjects who compare audio stimuli in listening tests and generate opinion scores, or objectively, with computations performed on the audio stimuli being tested. In both cases, the goal is to obtain a number that reliably represents the perceived audio quality.

CRC-SEAQ (System for the Evaluation of Audio Quality) is a Windows 95/98/NT/2000 software package for the subjective and/or objective evaluation of the quality of audio signals, including speech. With its intuitive graphical user interfaces, CRC-SEAQ provides an environment and a comprehensive set of tools for users who wish to accurately measure audio quality while developing or evaluating audio technologies, products and equipment. The system was developed by researchers at the Communications Research Centre, in Ottawa, Canada who have been involved for many years in the development of subjective and objective methods for audio quality measurements.

CRC-SEAQ can be configured to include a Subjective Test Module and/or an Objective Test Module. CRC-TimeSync, a companion program for time synchronizing audio files, can also be included as a module in CRC-SEAQ or licensed separately.

Subjective Test Module

The Subjective Test Module is an audio playback system with a series of control buttons and features (triple-stimulus A-B-C or multi-stimulus presentation, switching, zooming, looping and scoring) provided in three easy-to-use graphical user interfaces (GUI). The triple-stimulus GUI is designed to conduct ITU-R Recommendation BS.1116 compliant subjective tests and is particularly well suited for detecting and rating signals with small audible impairments. The multi-stimulus GUIs on the other hand are ideal when signals (up to 12) with medium and large impairments need to be compared and rated reliably. The audio sequences to be compared are read from files and played via any audio adapter with Windows MME support. The Subjective test Module supports a variety of audio signals and file formats,sampling frequencies and resolutions.

The Subjective Test Module is available in a 2-channel version, for mono and stereo signals, or a multichannel version which supports up to 16 audio channels.

The Subjective Test Module can be used for:

• Formal and informal listening tests
• Evaluating different implementations of audio processing equipment such as audio and speech codecs, 3D audio systems, etc.
• Status or reality checks during audio system development
• Perception and psychoacoustics research
• Demonstrations involving the comparison of different audio stimuli

Key Features:

• Three intuitive graphical user interfaces (user selectable)
• Compliant with ITU-R Recommendations for testing small,medium and large audible impairments
• Seamless switching between signals during playback
• Versatile zooming on any portion of the audio sequences under test
• Automatic repeat (looping) of audio sequences
• Audio sequences easily grouped into listening sessions
• Built-in scoring window. Test scores readily imported into spreadsheets or other analysis software
• Standard and user-definable rating scales
• Compatible with virtually any audio card that provides Windows MME support
• Mono, 2-channel and multichannel audio sequences Supports 8, 16, 20, 24, and 32 bits/sample, and normalized floating point
• Supports sampling rates of 8, 11.025, 22.05, 32, 44.1,48, 88.2 and 96 kHz Supports Intel and Motorola byte ordering for audio samples
• Compatible with Microsoft WAV and headerless PCM files

Objective Test Module

The Objective Test Module is an "electronic ear" which performs an objective measurement of perceived audio quality using a computerized model of the human ear. This model, known as PEAQ, was developed jointly by experts from eight leading research laboratories in the world, including the CRC, and is an international standard (ITU-R Recommendation BS.1387).The model processes the two audio signals to be compared (namely the original reference signal and an impaired version to be evaluated) and calculates instantaneously a quality rating similar to the mean rating that would be obtained if a formal subjective test were conducted. The basic (FFT-based) version of the ITU-R PEAQ model is implemented in the Objective Test Module.

The measuring instrument was calibrated using data from eight subjective listening tests conducted according to ITU-R Recommendation BS.1116. Much of the audio data in those tests were obtained from digital audio codecs. As a result, the measurement method is considered most reliable for evaluating codec quality. Other types of distortions may require re-calibration of the instrument.

The Objective Test Module is useful for:

• Evaluating different implementations of audio processing equipment
• Testing equipment or circuits before they are put into service
• Automated monitoring of the quality of an audio transmission in service
• Identifying the type and implementation of a particular audio codec
• Characterizing the performance of a coding algorithm during development
• Optimizing the cost and performance of a transmission network under given constraints

The Objective Test Module has three modes of operation:

• On-line Mode: This mode is used for automated real-time audio quality monitoring. The software monitors an incoming audio signal in order to recognize and capture a pre-defined reference test signal. Once captured, the impaired test signal is automaticallysynchronized with the unimpaired version stored on hard disk, and an objective quality measure is done.
• Off-line Mode: The two signals to be compared (i.e. original reference signal and impaired version) are stored as files on the user's PC. Both files can be automatically synchronized before an objective quality measure is calculated.
• Play and Measure Mode: This mode simplifies the task of measuring the quality of external audio devices. A reference signal is looped through the device under test and recorded back onto a PC. Once the recording is stopped the software automaticallysynchronizes the impaired signal with the reference signal before performing an audio quality assessment.

Key features:

• Three modes of operation: On-line, Off-line, Play and Measure
• Audio files may be specified interactively or via a batch file
• Analysis window duration and overlap is user-selectable
• Up to 12 model output variables may be displayed
• Interactive zoom and measurement of any portion of the audio sequences under test
• Automatic temporal alignment of audio sequences under test
• Audio files playable via the system audio card
• Supports mono and 2-channel audio sequences
• Supports 16, 20, 24, and 32 bits/sample, and normalized floating point
• Supports Intel and Motorola byte ordering for audio samples
• Supports sampling rates of 44.1 and 48 KHz
• Compatible with Microsoft WAV and headerless PCM files

Hardware Requirements

Pentium Pro or Pentium II CPU 400MHz or better, Minimum 128MB RAM, 5200rpm or faster hard drive (IDE or SCSI), Microsoft compatible three-button mouse, Audioadapter with Windows MME support, Windows 95/98/NT4/2000/XP.

CRC-SEAQ

For More Information

Mr. Louis Thibault, Manager, Advanced Audio Systems
Communications Research Centre Canada
3701 Carling Ave., P.O. Box 11490, Station H
Ottawa, Ontario, Canada K2H 8S2
Tel: 1 613 990 4349
Fax: 1 613 993 9950
Email: louis.thibault@crc.gc.ca
www.crc.gc.ca/aas