Towards the Next Generation of Web-based Experiments: A Case Study Assessing Basic Audio Quality Following the ITU-R Recommendation BS.1534 (MUSHRA)

Michael Schoeffer, Fabian-Robert Stöter
Bernd Edler, Jürgen Herre

Towards the Next Generation of Web-based Experiments: A Case Study Assessing Basic Audio Quality Following the ITU-R Recommendation BS.1534 (MUSHRA)

Towards the Next Generation of Web-based Experiments: A Case Study Assessing Basic Audio Quality Following the ITU-R Recommendation BS.1534 (MUSHRA)

Towards the Next Generation of Web-based Experiments: A Case Study Assessing Basic Audio Quality Following the ITU-R Recommendation BS.1534 (MUSHRA)

Towards the Next Generation of Web-based Experiments: A Case Study Assessing Basic Audio Quality Following the ITU-R Recommendation BS.1534 (MUSHRA)

Towards the Next Generation of Web-based Experiments: A Case Study Assessing Basic Audio Quality Following the ITU-R Recommendation BS.1534 (MUSHRA)

Audio Quality

Evaluating Impairment

How do we measure audio quality?

Audio Assessment

  • Objective Methods
    • Simple physical measures (SNR...)
    • Perceptually motivated (PEAQ, POLQA)
  • Subjective Methods

Subjective Evaluation

  1. Conducting listening experiments
  2. Time-Consuming
  3. Limited reproducibility in case no formal evaluation method was used

Evaluating Audio Codecs

Evaluating Audio Codecs

ITU-R BS.1534

ITU Recommendation

  • ITU-R BS.1534-0 published 2001
  • 2nd revision just recently published
  • introducing changes in the test design as well as statistical analysis of the results

MUSHRA

MUltiple Stimuli

MUltiple Stimuli

Basic Audio Quality

“a single, global attribute to judge any and all detected differences between the reference and the object.”
 

MUltiple Stimuli

Hidden Reference

...and Anchor

Anchors?

  • Ensure the the full range of the grading scale is used
  • 3.5 kHz / 7.0 kHz low pass filtered signals
  • Need to meet specific filter design criteria

Implementation

Requirements

  • Good performance (avoiding dropped samples)

Good Performance

Requirements

  • Good performance (avoiding dropped samples)
  • Sample accurate switching
  • No crossfading!
  • Instead: Fade-out and Fade-in of 5ms length

Switching

Not with <embed> or <audio>!

Web Audio API

Software Design

  • HTML5, JavaScript and Web Audio API
  • All processing in ScriptProcessorNode
  • UI in jQuery and jQuery Mobile
  • Easy to extend, modular software design
  • Easy to use for experimenters

Config in YAML Pages


pages:
  - type: mushra
    id: Item 1
    name: Orchestra
    content: Try to concentrate on the violins!
    showWaveform: true
    reference: reference.wav
    createAnchor35: true
    createAnchor70: true
    stimuli:
      C1: codec1.wav
      C2: codec2.wav
      C3: codec3.wav

Demo

Outlook

Web Based Experiments

The future of auditory web experiments

  • enabling large scale experiments
  • using technologies like
    • WebSockets
    • WebRTC
  • Experiment server (including live statistic)
  • still room for improvements
  • e.g. dropped samples, latency, channel mapping

Conclusion

  • Web Audio API will bring new types of auditory experiments to the web
  • Case Study: webMUSHRA - a (hopefully) recommendation-compliant MUSHRA software
    • Source code release in preparation

References / Links

webMUSHRA
Logo