Department of Computational Perception
Department of
Computational Perception
 Johannes Kepler Universität Linz

Home  –  Mission  –  Teaching  –  People  –  Research  –  Media  –  Awards  –  Impressum

MUSIC UNDERSTANDING & TRACKING

<< back to overview


One of our central research fields in computational perception is computers that "understand" certain aspects of music (that is, can identify/recognise things like melody, harmony, tempo, rhythm, instrumentation, mood, style, ... in a music recording) and, based on that, can recognize, interpret, and categorize music in useful ways. Such abilities are needed for many novel applications such as New Music Interfaces and TV & Radio Analysis. In addition, we develop computers that can analyze and follow music in real time, while it is being played, and that ultimately will interact with human musicians in musically meaningful ways.

Feature Extraction, Structure Recognition, Music Understanding

In general, we are interested in any project that aims at developing methods for the automatic recognition of musical entities or structures (melody, rhythm, segmentation, ...) in music recordings, for the recognition and categorisation of music, new means of interaction with music, new ways of displaying and visualising musical information, controlling or modifying music in interesting ways, etc. The following list of example projects is only a suggestion; we are open to any creative ideas you may have. The little video to the right of the list visualizes certain sound/rhythm characteristics in different kinds of music (from work by a former Ph.D. student of ours).

Example projects:
Contact: Bernhard Niedermayer, Gerhard Widmer
Feature extraction video by former PhD student Elias Pampalk

Real-time Music Following and Interpretation

At our institute, we are developing very efficient and reliable algorithms for "listening to" and tracking music in real time, and for aligning music audio streams to the printed score. Based on that, there is a lot of exciting possibilities for real-time applications. The following list gives a few examples. (Again: if you have other ideas, please let us know!)
The little video on the right (and Prof. Widmer as amateur pianist) shows an application of our real-time music tracking algorithm: an automatic page turning device for musicians that listens and knows when it is time to turn the page.

Example projects:
  • Automatic (reactive and pro-active) real-time accompaniment of soloists
  • Real-time Music Visualisation
  • Real-time Tempo Estimation
  • Implementing a real-time score follower for the iPad
  • Real-time Detectors for
    • Music
    • Speech
    • Silence
    • Applause
    • ...
Contact: Andreas Arzt, Gerhard Widmer
Our automatic reactive page turner

Real-time Music Interaction and Control


Although computers offer virtually unlimited freedom in the control, synthesis, and processing of sound, this potential has hardly been unlocked to performing musicians, because of the lack of intuitive control interfaces. Devices such as the Wiimote or Kinect would enable sound control by natural gestures of the body, relieving performers from the necessity to use computer keyboard and display. Gesture-tracking could also open up new possibilities for non-musicians, for instance by inferring intended musical character automatically by capturing arm gestures to create new interfaces to music searching and browsing or by capturing characteristics of the user's movement, and transfer them to a playing piece, in such a way that the user has real-time control over (for example) tempo and loudness of the music. The video on the right shows Prof. Widmer contactlessly controlling an expressive piano performance on a Bösendorfer CEUS Grand Piano using a MIDI Theremin.

Example projects:
  • Directing music by arm gestures
  • Estimation of musical character from arm gestures
  • Gesture-based audio-processing/synthesis for artistic purposes
  • Virtual music instruments
  • Air Guitar Hero
Contact: Maarten Grachten, Gerhard Widmer
CEUS and the MIDI Theremin


last edited by pk on May 3, 2012