Department of Computational Perception

PERSONALIZED MUSIC RETRIEVAL VIA MUSIC CONTENT, MUSIC CONTEXT, AND USER CONTEXT

Project Title: Personalized Music Retrieval via Music Content, Music Context, and User Context

Sponsor: Austrian Science Fund (Fonds zur Förderung der wissenschaftlichen Forschung, FWF)

Project Number: P22856-N23

Duration: 48 months (Nov 2010 – Oct 2014)

Persons involved:

Abstract

One of the biggest challenges for today's users of digital music collections is how to find music that matches his or her taste depending on factors like the music content, the user's general music taste, but also his or her current emotional state, activity, situation, or surrounding. This project aims at supporting the user by modeling such aspects in terms of similarity functions, which will eventually be integrated to build a system for personalized access to music collections.

Aspects of music similarity can be broadly categorized into music content (e.g., the rhythm or timbre of a song), music context (e.g., song lyrics or terms used on an artist's Web page to describe his or her music), and user context (i.e., external human factors that may influence how a listener perceives music, for example, the listener's mood, location, used playback device, or listening situation).

In this project, we will address the third pillar of music similarity, i.e., taking into account the user context when he or she is listening to music. We are thus going to develop novel models to describe user-related aspects of music similarity, novel methods to combine the three broad dimensions of music similarity, and novel strategies to use this combined model for information retrieval (IR) tasks. These IR tasks will include, for example, a personalized music recommendation engine, user-adaptive playlist generation or an adaptive user interface to access music collections.

Facing the challenges described above, we will carry out research to accomplish the following six goals:

• investigate the suitability and acceptance of different data sources to create user profiles
• develop methods to mine the most promising data sources
• create a model that reflects different aspects of the user context
• investigate directions to integrate different similarity measures (content-based, music context-based, and user context-based) 
• develop and thoroughly evaluate integrated models for the three kinds of music similarity
• build a user-adaptive system for music retrieval, taking into account user-related factors
  

• novel ways of accessing music collections
• novel approaches to mine the user context
• a multimodal similarity measure that incorporates aspects of music content, music context, and user context
• music recommendation that is tailored to the user and does transcend the common collaborative filtering approach

Publications

Journals

Music Information Retrieval: Recent Developments and Applications Schedl, M., Gómez, E., and Urbano, J. Foundations and Trends in Information Retrieval, volume 8, number 2-3, pp. 127-261, 2014

A Survey of Music Similarity and Recommendation from Music Context Data Knees, P. and Schedl, M. ACM Transactions on Multimedia Computing, Communications and Applications, 2014.

The Neglected User in Music Information Retrieval Research Schedl, M., Flexer, A., and Urbano, J. Journal of Intelligent Information Systems, 2013.

Evaluation in Music Information Retrieval Urbano, J., Schedl, M., and Serra, X. Journal of Intelligent Information Systems, 2013.

Harvesting Microblogs for Contextual Music Similarity Estimation - A Co-occurrence-based Framework Schedl, M., Hauger, D., and Urbano J. Multimedia Systems, 2013.

Minimal Test Collections for Low-Cost Evaluation of Audio Music Similarity and Retrieval Systems Urbano, J. and Schedl, M. International Journal of Multimedia Information Retrieval: Special Issue on Hybrid Music Information Retrieval, 2013.

Local and Global Scaling Reduce Hubs in Space Schnitzer, D., Flexer, A., Schedl, M., and Widmer, G. Journal of Machine Learning Research,  volume 13, October 2012.

#nowplaying Madonna: A Large-Scale Evaluation on Estimating Similarities Between Music Artists and Between Movies from Microblogs Schedl, M. Information Retrieval, 2012, 10.1007/s10791-012-9187-y.

User-Aware Music Retrieval Schedl, M., Stober, S., Gómez, E., Orio, N., Liem, C.C.S. In Meinard Müller, Masataka Goto, Markus Schedl (eds.), Multimodal Music Processing Schloss Dagstuhl - Leibniz-Zentrum für Informatik, Dagstuhl Publishing, Saarbrücken/Wadern, Germany, 2012.

Exploring the Music Similarity Space on the Web Schedl, M., Pohle, T., Knees, P., and Widmer, G. ACM Transactions on Information Systems, volume 29, number 3, article 14, July 2011.

Peer-Reviewed Conference and Workshop Proceedings

Iron Maiden while jogging, Debussy for dinner? - An analysis of music listening behavior in context Gillhofer, M. and Schedl, M. Proceedings of the 21th International Conference on MultiMedia Modeling (MMM 2015), Sydney, Australia, January 2015.

Genre-based Analysis of Social Media Data on Music Listening Behavior Schedl, M. and Tkalčič, M. Proceedings of the 1st ACM International Workshop on Internet-Scale Multimedia Management (ISMM 2014), Orlando, USA, Novermber 2014.

Impact of Listening Behavior on Music Recommendation Farrahi, K., Schedl, M., Vall, A., Hauger, D., and Tkalčič, M. Proceedings of the 15th International Society for Music Information Retrieval Conference (ISMIR 2014), Taipei, Taiwan, October 2014.

Formant Analysis of Altered Notes in a Diatonic Harmonica Eghbalzadeh, H. and Schedl, M. Proceedings of the International Symposium on Musical Acoustics (ISMA 2014), Le Mans, France, July 2014.

Social Media and Classical Music? - A first analysis within the PHENICX project: “Performances as Highly Enriched aNd Interactive Concert eXperiences” Schedl, M. SoMeRA'14: Proceedings of the First International Workshop on Social Media Retrieval and Analysis

Enhancing Music Recommender Systems with Personality Information and Emotional States: A Proposal Ferwerda, B. and Schedl, M. Proceedings of the 2nd Workshop on Emotions and Personality in Personalized Services (EMPIRE 2014), Aalborg, Denmark, July 2014

Using Social Media Mining for Estimating Theory of Planned Behaviour Parameters Tkalčič, M., Ferwerda, B., Schedl, M., Liem, C. Melenhorst, M., Odić, A., and Košir, A. Proceedings of the 2nd Workshop on Emotions and Personality in Personalized Services (EMPIRE 2014), Aalborg, Denmark, July 2014.

User Geospatial Context for Music Recommendation in Microblogs Schedl, M., Vall, A., and Farrahi, K. Proceedings of the 37th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2014), Gold Coast, Australia, July 2014.

To Post or Not to Post: The Effects of Persuasive Cues and Group Targeting Mechanisms on Posting Behavior Ferwerda, B., Schedl, M., and Tkalčič, M. Proceedings of the 6th ASE International Conference on Social Computing (SocialCom 2014), Stanford, USA, May 2014.

SoMeRA 2014: Social Media Retrieval and Analysis Workshop Schedl, M., Knees, P., and Shen, J. Proceedings of the 37th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2014), Gold Coast, Australia, July 2014.

Benchmarking Violent Scenes Detection in Movies Demarty, C.-H., Ionescu, B., Jiang Y.-G., Quang V.L., Schedl, M., and Penet C. Proceedings of the 12th International Workshop on Content-Based Multimedia Indexing (CBMI 2014), Klagenfurt, Austria, June 2014.

Mobile Music Genius: Reggae at the Beach, Metal on a Friday Night? Schedl, M., Breitschopf, G., and Ionescu, B. Proceedings of the ACM International Conference on Multimedia Retrieval (ICMR 2014), Glasgow, Scotland, April 2014.

Location-Aware Music Artist Recommendation Schedl, M. and Schnitzer, D. Proceedings of the 20th International Conference on MultiMedia Modeling (MMM 2014), Dublin, Ireland, January 2014.

Ameliorating Music Recommendation: Integrating Music Content, Music Context, and User Context for Improved Music Retrieval and Recommendation Schedl, M. Proceedings of the 11th International Conference on Advances in Mobile Computing & Multimedia (MoMM 2013), Vienna, Austria, December 2013

Personalized Music Recommendation in a Mobile Environment Schabetsberger, C. and Schedl, M. Proceedings of the 11th International Conference on Advances in Mobile Computing & Multimedia (MoMM 2013), Vienna, Austria, December 2013.

Location-aware Music Recommendation Using Auto-Tagging and Hybrid Matching Kaminskas, M., Ricci, F., and Schedl, M. Proceedings of the 7th ACM Conference on Recommender Systems (RecSys 2013), Hong Kong, China, October 2013.

The Million Musical Tweet Dataset: What We Can Learn From Microblogs Hauger, D., Schedl, M., Košir, A., and Tkalčič, M. Proceedings of the 14th International Society for Music Information Retrieval Conference (ISMIR 2013), Curitiba, Brazil, November 2013.

Hybrid Retrieval Approaches to Geospatial Music Recommendation Schedl, M. and Schnitzer, D. Proceedings of the 35th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2013), Dublin, Ireland, July-August 2013.

A Professionally Annotated and Enriched Multimodal Data Set on Popular Music Schedl, M., Liem, C.C.S., Peeters, G., and Orio, N. Proceedings of the 4th ACM Multimedia Systems Conference (MMSys 2013), Oslo, Norway, February-March 2013.

Leveraging Microblogs for Spatiotemporal Music Information Retrieval Schedl, M. Proceedings of the 35th European Conference on Information Retrieval (ECIR 2013), Moscow, Russia, March 2013.

Brave New Task: Musiclef Multimodal Music Tagging Liem, C.C.S., Orio, N., Peeters, G., and Schedl, M. Working Notes Proceedings of MediaEval 2012: Multimedia Benchmark Workshop, Pisa, Italy, October 2012.

ARF @ MediaEval 2012: An Uninformed Approach to Violence Detection in Hollywood Movies Schlüter, J., Ionescu, B., Mironica, I., and Schedl, M. Working Notes Proceedings of MediaEval 2012: Multimedia Benchmark Workshop, Pisa, Italy, October 2012.

ARF @ MediaEval 2012: Multimodal Video Classification Ionescu, B., Mironica, I., Seyerlehner, K., Knees, P., Schlüter, J., Schedl, M., Cucu, H., Buzo, A., and Lambert, P. Working Notes Proceedings of MediaEval 2012: Multimedia Benchmark Workshop, Pisa, Italy, October 2012.

Exploring Geospatial Music Listening Patterns in Microblog Data Hauger, D. and Schedl, M. Proceedings of the 10th Workshop on Adaptive Multimedia Retrieval (AMR 2012), Copenhagen, Denmark, October 2012.

From Improved Auto-taggers to Improved Music Similarity Measures Seyerlehner, K., Sonnleitner, R., Schedl, M., Hauger, D., and Ionescu, B. Proceedings of the 10th Workshop on Adaptive Multimedia Retrieval (AMR 2012), Copenhagen, Denmark, October 2012.

Putting the User in the Center of Music Information Retrieval Schedl, M. and Flexer, A. Proceedings of the 13th International Society for Music Information Retrieval Conference (ISMIR 2012), Porto, Portugal, October 2012.

Evaluating the Online Capabilities of Onset Detection Methods Böck, S., Krebs, F., and Schedl, M. Proceedings of the 13th International Society for Music Information Retrieval Conference (ISMIR 2012), Porto, Portugal, October 2012.

How Significant is Statistically Significant? The Case of Audio Music Similarity and Retrieval Urbano, J., Downie, J.S., McFee, B., and Schedl, M. Proceedings of the 13th International Society for Music Information Retrieval Conference (ISMIR 2012), Porto, Portugal, October 2012.

MusiClef: Multimodal Music Tagging Task Orio, N., Liem, C.C.S., Peeters, G., and Schedl, M. Proceedings of the 3rd Conference on Multilingual and Multimodal Information Access Evaluation (CLEF 2012), Rome, Italy, September 2012.

Online Real-time Onset Detection with Recurrent Neural Networks Böck, S., Arzt, A., Krebs, F., and Schedl, M. Proceedings of the 15th International Conference on Digital Audio Effects (DAFx 2012), York, UK, September 2012.

nepDroid: An Intelligent Mobile Music Player Huber, S., Schedl, M., and Knees, P. Proceedings of the ACM International Conference on Multimedia Retrieval (ICMR 2012), Hong Kong, June 2012.

Mining Microblogs to Infer Music Artist Similarity and Cultural Listening Patterns Schedl, M. and Hauger, D. Proceedings of the 21st International World Wide Web Conference (WWW 2012): 4th International Workshop on Advances in Music Information Research (AdMIRe 2012), Lyon, France, April 2012.

Towards Minimal Test Collections for Evaluation of Audio Music Similarity and Retrieval Urbano, J. and Schedl, M. Proceedings of the 21st International World Wide Web Conference (WWW 2012): 4th International Workshop on Advances in Music Information Research (AdMIRe 2012), Lyon, France, April 2012.

A Model for Serendipitous Music Retrieval Schedl, M., Hauger, D., and Schnitzer, D. Proceedings of the 16th International Conference on Intelligent User Interfaces (IUI 2012): 2nd International Workshop on Context-awareness in Retrieval and Recommendation (CaRR 2012), Lisbon, Portugal, February 2012.

Polyphonic Piano Note Transcription with Recurrent Neural Networks Böck, S. and Schedl, M. Proceedings of the 37th International Conference on Acoustics, Speech, and Signal Processing (ICASSP 2012), Kyoto, Japan, March 2012.

Investigating the Similarity Space of Music Artists on the Micro-Blogosphere Schedl, M., Knees, P., and Böck, S. Proceedings of the 12th International Society for Music Information Retrieval Conference (ISMIR 2011), Miami, FL, USA, October 2011.

Using Mutual Proximity to Improve Content-Based Audio Similarity Schnitzer, D., Flexer, A., Schedl, M., and Widmer, G. Proceedings of the 12th International Society for Music Information Retrieval Conference (ISMIR 2011), Miami, FL, USA, October 2011.

MusiCLEF: A Benchmark Activity in Multimodal Music Information Retrieval Orio, N., Rizo, D., Miotto, R., Montecchio, N., Schedl, M., and Lartillot, O. Proceedings of the 12th International Society for Music Information Retrieval Conference (ISMIR 2011), Miami, FL, USA, October 2011.

Personalization in Multimodal Music Retrieval Schedl, M. and Knees, P. Proceedings of the 9th Workshop on Adaptive Multimedia Retrieval (AMR 2011), Barcelona, Spain, July 2011.

Enhanced Beat Tracking with Context-Aware Neural Networks Böck, S. and Schedl, M. Proceedings of the 14th International Conference on Digital Audio Effects (DAFx 2011), Paris, France, September 2011.

Large-Scale Music Exploration in Hierarchically Organized Landscapes Using Prototypicality Information Schedl, M., Höglinger, C., and Knees, P. Proceedings of the ACM International Conference on Multimedia Retrieval (ICMR 2011), Trento, Italy, April 2011.

Books and Book Chapters

Music Similarity and Retrieval Knees, P. and Schedl, M. Springer, expected to be published in 2015.

Music Recommender Systems Schedl, M., Knees, P., McFee, B., Bogdanov, D., and Kaminskas, M. In Francesco Ricci, Lior Rokach, Bracha Shapira (eds.), Recommender Systems Handbook (2nd edition) Springer, expected to be published in 2015.

Mining User-generated Data for Music Information Retrieval Schedl, M., Sordo, M., Koenigstein, N., and Weinsberg, U. In Marie-Francine Moens, Juanzi Li, Tat-Seng Chua (eds.), Mining of User Generated Content and Its Applications CRC Press, January 2014.

Exploiting Social Media for Music Information Retrieval Schedl, M. In Naeem Ramzan, Roelof van Zwol, Jong‐Seok Lee, Kai Clüver, Xian‐Sheng Hua (eds.), Social Media Retrieval Springer, December 2012.

User-Aware Music Retrieval Schedl, M., Stober, S., Gómez, E., Orio, N., Liem, C.C.S. In Meinard Müller, Masataka Goto, Markus Schedl (eds.), Multimodal Music Processing Schloss Dagstuhl - Leibniz-Zentrum für Informatik, Dagstuhl Publishing, Saarbrücken/Wadern, Germany, 2012

Web- and Community-based Music Information Extraction Schedl, M. In Tao Li, Mitsunori Ogihara, George Tzanetakis (eds.), Music Data Mining CRC Press/Chapman Hall, July 2011.

---

last edited by ms on 2015-02-17