Department of Computational Perception
Department of
Computational Perception
Johannes Kepler Universitšt Linz

Home  –  Mission  –  Teaching  –  People  –  Research  –  Media  –  Awards  –  Impressum

Towards Expressivity-aware Computer Systems in Music

ERC Advanced Grant 2015, European Research Council (ERC)

Call identifier: ERC-2014-AdG

Project Number: 670035

Principal Investigator: Gerhard Widmer

Project Period: 2016 - 2020

Funding Amount: € 2,318,750.00

Copyright (2015): Austrian Science Fund (FWF) / Ethan Vincent

The Con Espressione Game ...

Do you have a minute to listen to some bits of music (and maybe contribute some empirical data to the project)? This is what the project is about:

>     The Con Espressione Game     <

... or, if you have very little time: here's a short version with only one piece:

>     The Very Short Con Espressione Game     <

Project Summary

What makes music so important, what can make a performance so special and stirring? It is the things the music expresses, the emotions it induces, the associations it evokes, the drama and characters it portrays. The sources of this expressivity are manifold: the music itself, its structure, orchestration, personal associations, social settings, but also -- and very importantly -- the act of performance, the interpretation and expressive intentions made explicit by the musicians through nuances in timing, dynamics etc.

Thanks to research in fields like Music Information Research (MIR), computers can do many useful things with music, from beat and rhythm detection to song identification and tracking. However, they are still far from grasping the essence of music: they cannot tell whether a performance expresses playfulness or ennui, solemnity or gaiety, determination or uncertainty; they cannot produce music with a desired expressive quality; they cannot interact with human musicians in a truly musical way, recognising and responding to the expressive intentions implied in their playing.

The project is about developing machines that are aware of certain dimensions of expressivity, specifically in the domain of (classical) music, where expressivity is both essential and -- at least as far as it relates to the act of performance -- can be traced back to well-defined and measurable parametric dimensions (such as timing, dynamics, articulation). We will develop systems that can recognise, characterise, search music by expressive aspects, generate, modify, and react to expressive qualities in music. To do so, we will (1) bring together the fields of AI, Machine Learning, Music Information Retrieval (MIR), and Music Performance Research; (2) integrate theories from Musicology to build more well-founded models of music understanding; (3) support model learning and validation with massive musical corpora of a size and quality unprecedented in computational music research.

In terms of computational methodologies,*) we will rely on, and improve, methods from Artificial Intelligence - particularly: probabilistic models (for information fusion, tracking, reasoning and prediction); machine learning - particularly: deep learning techniques (for learning musical features, abstractions, and representations from musical corpora, and for inducing mappings for expression recognition and prediction); audio signal processing and pattern recognition (for extracting musical parameters and patterns relevant to expressivity); and information theory (for modelling musical expectation, surprise, uncertainty, etc.). This will be combined with high-level concepts and models of structure perception from fields like systematic and cognitive musicology, in order to create systems that have a somewhat deeper 'understanding' of music, musical structure, music performance, and musical listening, and the interplay of these factors in making music the expressive and rewarding art that it is. (A more detailed discussion of how we believe all these things relate to each other can be found in the "Con Espressione Manifesto").

With this research, we hope to contribute to a new generation of MIR systems that can support musical services and interactions at a new level of quality, and to inspire expressivity-centered research in other domains of the arts and human-computer interaction (HCI).

*) If you are very good in at least two of these domains and if you have a strong personal interest in (classical) music, consider applying for a PhD or post-doctoral position!)


Strategic Document: The "Con Espressione Manifesto"

Want to know about our view on the current research landscape, and what research needs to be done in the coming years (within and beyond Con Espressione)? Read our strategic document,

>   The Con Espressione Manifesto

(If you are interested in applying for a research position in the project, please think about how your research ideas and plans would fit in this scheme (or go beyond it, because we may have missed some crucial directions ...).


Want to know who is or has been working on the project and has contributed to the scientific output of thr project? Here's our

>   List of Researchers

Project Publications

Want to know about the scientific results of the project? Here's a page with our

>   Scientific Publications from the Con Espressione project.

Project Presentations

Want to know how the project and its vision is presented to (scientific and general) audiences? Here's a page with links to

>   Invited Conference Talks and Public Presentations on the Con Espressione project.

Media Coverage

Want to know how the (national and international) public media report on the project? Here's a page with links to documentaries/articles on our music expression research in

>   TV, Radio, Internet and Print Media.


We gratefully acknowledge material support for this research (in the form of direct financial support, music, scores, access to musical instruments and performance spaces) from the following institutions:

This project receives funding from the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation programme under grant agreement No 670035.
The Bösendorfer Piano Company, Vienna, Austria
The Royal Concertgebouw Orchestra (RCO), Amsterdam, The Netherlands
Bärenreiter Music Publishers, Kassel, Germany
The Eliette and Herbert von Karajan Institute, Salzburg, Austria

last edited by gw on Jan 8, 2016