Department of Computational Perception

COMPUTATIONAL PERFORMANCE STYLE ANALYSIS FROM AUDIO RECORDINGS

Project Title: Computational Performance Style Analysis from Audio Recordings

Sponsor: Austrian National Science Fund (Fonds zur Förderung der wissenschaftlichen Forschung, FWF),
Project Number: P19349-N15

Duration: 36 months (Feb. 2007 – Jan. 2010)

In cooperation with: Austrian Research Institute for Artificial Intelligence (ÖFAI), Vienna – Machine Learning, Data Mining, and Intelligent Music Processing Group

Persons involved:

Abstract

The project aims at investigating the fascinating, but elusive phenomenon of individual artistic music performance style with quantitative, computational methods. In particular, the goal is to discover and characterise significant patterns and regularities in the way great music performers (classical pianists) shape the music through expressive timing, dynamics, articulation, etc., and express their personal style and artistic intentions.

The starting point is a unique and unprecedented collection of empirical measurement data: recordings of essentially the complete works for solo piano by Frederic Chopin, made by a world-class pianist (Nikita Magaloff) on the Bösendorfer computer-controlled SE290 grand piano. This huge data set, which comprises hundreds of thousands of played notes, gives precise information about how each note was played, including precise onset time, duration, and loudness. State-of-the-art methods of intelligent data analysis and automatic pattern discovery will be applied to these data in order to derive quantitative and predictive models of various aspects of performance, such as expressive timing, dynamic shaping, articulation, etc. This will give new insights into the performance strategies applied by an accomplished concert pianist over a large corpus of music. Moreover, by automatically matching these precisely measured performances against sound recordings by a large number of famous concert pianists, comparative studies will be performed which, for the first time, will permit truly quantitative statements about individual artistic performance style.

All this requires extensive research into new methods for intelligent audio analysis (e.g., extraction of expressive parameters from audio, and precise alignment of different sound recordings) and intelligent data analysis and modelling (e.g., sequential pattern discovery, hierarchical probabilistic models, etc.).

The project can be seen as a continuation and extension of previous work or ours, in which we managed to show that expressive music performance is indeed amenable to computational modelling and analysis, and which has contributed to establishing an international research trend in computational music performance research. An easily readable account of that earlier work can be found in

Widmer, G., Dixon, S., Goebl, W., Pampalk, E., and Tobudic, A. (2003).
In Search of the Horowitz Factor. AI Magazine 24(3), 111-130.

The current project will go beyond earlier work by working with new empirical data of unprecedented size and quality, and by focusing on the very elusive, but fascinating question of the individual style of great artists.

Publicity, Awards, etc.

• cp.jku's expressive performance rendering system YQX won the Rencon Award, the Rencon Technical Award, and the Rencon Murao Award, at the 7th Contest for Performance Rendering Systems (Rencon '08), in Sapporo, Japan. Hear the results.
  
  

Publications

• Goebl, W. and Widmer, G. (2008).
  On the Use of Computational Methods for Expressive Music Performance
  In T. Crawford and L. Gibson (Eds.), Modern Methods for Musicology: Prospects, Proposals and Realities.
  London: Ashgate Publishing.
• G. Widmer, S. Flossmann, M. Grachten (2009).
  YQX plays Chopin.
  In AI Magazine. AAAI Press. (in press).
• M. Grachten, W. Goebl, S. Flossmann, G. Widmer (2009).
  Phase-plane representation and visualization of gestural structure in expressive timing.
  In Journal of New Music Research. 38(2), 183–195.
• S. Flossmann, W. Goebl, and G. Widmer. (2009).
  Maintaining skill across the life span: Magaloff's entire Chopin at age 77.
  International Symposium on Performance Science (ISPS 2009) (15–18 December 2009), Auckland, New Zealand.
  European Association of Conservatoires (AEC), Utrecht, NL, pp. 119–124.
• M. Grachten, G. Widmer (2009).
  Who is who in the end? Recognizing pianists by their final ritardandi.
  In proceedings of the Tenth International Society for Music Information Retrieval Conference. Kobe, Japan.
• M. Grachten, M. Schedl, T. Pohle, and G. Widmer (2009).
  The ISMIR Cloud: A Decade of ISMIR Conferences at Your Fingertips.
  In proceedings of the Tenth International Society for Music Information Retrieval Conference. Kobe, Japan.
• B. Niedermayer (2009).
  Improving Accuracy of Polyphonic Music-to-Score Alignment.
  In proceedings of the Tenth International Society for Music Information Retrieval Conference. Kobe, Japan.
• W. Goebl, S. Flossmann, G. Widmer (2009).
  Computational investigations into between-hand synchronization in piano playing: Magaloff's complete Chopin.
  In Proceedings of the 6th Sound and Music Computing Conference. (SMC 2009), Porto, Portugal.
• M. Grachten, G. Widmer (2009).
  The kinematic rubato model as a means of studying final ritards across pieces and pianists.
  In Proceedings of the 6th Sound and Music Computing Conference. (SMC 2009), Porto, Portugal.
• Niedermayer, B. (2009).
  Towards Audio to Score Alignment in the Symbolic Domain.
  In Proceedings of the 6th Sound and Music Computing Conference. (SMC 2009), Porto, Portugal.
• Flossmann, S., Grachten, M. and Widmer, G. (2009).
  Expressive Performance Rendering: Introducing Performance Context.
  In Proceedings of the 6th Sound and Music Computing Conference. (SMC 2009), Porto, Portugal.
• Niedermayer, B. (2008).
  Non-Negative Matrix Division for the Automatic Transcription of Polyphonic Music.
  In Proceedings of the 9th International Conference on Music Information Retrieval (ISMIR 2008), Philadelphia, PA, USA.
• Arzt, A., Widmer, G., and Dixon, S. (2008).
  Automatic Page Turning for Musicians via Real-Time Machine Listening.
  In Proceedings of the 18th European Conference on Artificial Intelligence (ECAI 2008). Patras, Greece.
• Flossmann, S., Grachten, M., and Widmer, G. (2008).
  Experimentally Investigating the Use of Score Features for Computational Models of Expressive Timing.
  In Proceedings of the 10th International Conference on Music Perception and Cognition (ICMPC10). Sapporo, Japan.
  
• Grachten, M., Goebl, W., Flossmann, S. and Widmer, G. (2008)
  Intuitive visualization of gestures in expressive timing: A case study on the final ritard.
  In Proceedings of the 10th International Conference on Music Perception and Cognition (ICMPC10). Sapporo, Japan.
  
• Grachten, M., Goebl, W., Flossmann, S. and Widmer, G. (2008).
  Phase-plane visualizations of gestural structure in expressive timing.
  In Proceedings of the Fourth Conference on Interdisciplinary Musicology (CIM08). Thessaloniki, Greece.
  
• Grachten, M. and Widmer, G. (2007).
  Towards Phrase Structure Reconstruction from Expressive Performance Data.
  In Proceedings of the International Conference on Music Communication Science (ICOMCS). Sydney, Australia.