Department of Computational Perception

Social Media Mining for Multimodal Music Retrieval

Project Title: Social Media Mining for Multimodal Music Retrieval

Sponsor: Austrian Science Fund (Fonds zur Förderung der wissenschaftlichen Forschung, FWF)

Project Number: P25655

Duration: 48 months (July 2013 – June 2017)

Persons involved:

Abstract

Hence, in this project we propose to comprehensively research the use of a large number of different user-generated data sources for several innovative tasks in MIR research. To this end, we identified three goals that will be thoroughly addressed: 

    1. Inferring Geospatial Listening Patterns from User-generated Data

    2. Multimodal Modeling for Serendipitous Music Access 

    3. Building Intelligent, Adaptive User Interfaces to Music Collections 

    In the first challenge we will investigate whether social media can be used as reliable source to derive music listening patterns on a worldwide perspective. Upon identification of such patterns, we will be able to study geospatial aspects of music consumption and in turn look into methods to predict the popularity and trendiness of music items by fusing data from different sources. We will further research methods to detect novelty (e.g., novel releases, up-and-coming artists). 

    In the second challenge, we will develop multimodal approaches to model the user and the music items. Defining and building user models from his or her digital traces in social media will be enabled by machine learning techniques dedicated to user-generated data. Multimodal representations of music items (such as video clips, song lyrics, cover artwork and audio files) will be generated by elaborating novel feature extractors and fusion techniques. 

    Integrating multimodal descriptors of users and music items, we will then research ways to create serendipitous music access models and systems in the third challenge. Incorporating these different modalities will enable to develop personalized retrieval schemes, paying particular attention to the aspects of similarity, diversification, novelty, and familiarity/popularity. Based on the elaborated retrieval models, we will eventually create prototypical intelligent user interfaces to explore music collections.

Expected results of the realization of the project include: 

• deeper understanding and exploitation of social media for music-related activities
• novel user models dedicated to the music domain
• serendipitous music retrieval and recommendation systems
• novel multimodal models of music items
• highly efficient algorithms to identify music items in user-generated data streams
• understanding the emergence and evolution of music trends
• extending research in user-centric MIR
• intelligent user interfaces to explore music collections

Publications

Journals

On the Interrelation between Listener Characteristics and the Perception of Emotions in Classical Orchestra Music Schedl, M., Gómez, E., Trent, E.S., Tkalčič, M., Eghbal-Zadeh, H., and Martorell, A. IEEE Transactions on Affective Computing, accepted, to appear, 2017.

Investigating Country-specific Music Preferences and Music Recommendation Algorithms with the LFM-1b Dataset Schedl, M. International Journal of Multimedia Information Retrieval, accepted, to appear, 2017.

Introduction to Intelligent Music Systems and Applications Schedl, M., Yang, Y.-H., and Herrera, P. ACM Transactions on Intelligent Systems and Technology, volume 8, issue 2, October 2016.

Music Information Retrieval: Recent Developments and Applications Schedl, M., Gómez, E., and Urbano, J. Foundations and Trends in Information Retrieval, volume 8, number 2-3, pp. 127-261, 2014.

Harvesting Microblogs for Contextual Music Similarity Estimation - A Co-occurrence-based Framework Schedl, M., Hauger, D., and Urbano J. Multimedia Systems, 2013.

Peer-Reviewed Conference and Workshop Proceedings

Towards Predicting the Popularity of Music Artists Jetzinger, F., Huemer, F., and Schedl, M. Proceedings of the 10th International Workshop on Machine Learning and Music (MML), Barcelona, Spain, October 2017.

Artist Preferences and Cultural, Socio-economic Distances Across Countries: A Big Data Perspective Liu, M., Hu, X., and Schedl, M. Proceedings of the 18th International Society for Music Information Retrieval Conference (ISMIR 2017), Suzhou, China, October 2017.

Online Music Listening Culture of Kids and Adolescents: Listening Analysis and Music Recommendation Tailored to the Young Schedl, M. and Bauer, C. Proceedings of the 11th ACM Conference on Recommender Systems (RecSys 2017): International Workshop on Children and Recommender Systems (KIDREC 2017), Como, Italy, August 2017.

Music Playlist Continuation by Learning from Hand-Curated Examples and Song Features Vall, A., Eghbal-zadeh, Dorfer, M., and Schedl, M. Proceedings of the 11th ACM Conference on Recommender Systems (RecSys 2017): Workshop on Deep Learning for Recommender Systems (DLRS 2017), Como, Italy, August 2017.

Personality Traits and Music Genre Preferences: How Music Taste Vary Over Age Groups Ferwerda, B., Tkalcic, M., and Schedl, M. Proceedings of the 11th ACM Conference on Recommender Systems (RecSys 2017): Workshop on Temporal Reasoning in Recommender Systems (TempRRS 2017), Como, Italy, August 2017.

The Importance of Song Context in Music Playlists Vall, A., Quadrana, M., Schedl, M., Widmer, G., and Cremonesi, P. Proceedings of the Poster Track of the 11th ACM Conference on Recommender Systems (RecSys 2017), Como, Italy, August 2017.

New Paths in Music Recommender Systems Research Schedl, M., and Knees, P., and Gouyon, F. Proceedings of the 11th ACM Conference on Recommender Systems (RecSys 2017), Como, Italy, August 2017.

Music Genre Classification Revisited: An In-Depth Examination Guided by Music Experts Pálmason, H., Jónsson, B. Þ., Schedl, M., and Knees, P. Proceedings of the 13th International Symposium on Computer Music Multidisciplinary Research (CMMR 2017), Porto, Portugal, September 2017.

Distance- and Rank-based Music Mainstreaminess Measurement Schedl, M. and Bauer, C. Proceedings of the 25th International Conference on User Modeling, Adaptation and Personalization (UMAP 2017): Workshop on Surprise, Opposition, and Obstruction in Adaptive and Personalized Systems (SOAP 2017), Bratislava, Slovakia, July 2017.

Introducing Surprise and Opposition by Design in Recommender Systems Bauer, C. and Schedl, M. Proceedings of the 25th International Conference on User Modeling, Adaptation and Personalization (UMAP 2017): Workshop on Surprise, Opposition, and Obstruction in Adaptive and Personalized Systems (SOAP 2017), Bratislava, Slovakia, July 2017.

The Effect of Different Video Summarization Models on the Quality of Video Recommendation Based on Low-level Visual Features Deldjoo, Y., Cremonesi, P., Schedl, M., and Quadrana, M. Proceedings of the 15th International Workshop on Content-based Multimedia Indexing (CBMI 2017), Florence, Italy, June 2017.

Prediction of User Demographics from Music Listening Habits Krismayer, T., Schedl, M., Knees, P., and Rabiser, R. Proceedings of the 15th International Workshop on Content-based Multimedia Indexing (CBMI 2017), Florence, Italy, June 2017.

Personality Traits and Music Genres: What Do People Prefer to Listen To? Ferwerda, B., Tkalčič, M., and Schedl, M. Proceedings of the 25th International Conference on User Modeling, Adaptation and Personalization (UMAP 2017), Bratislava, Slovakia, July 2017.

IUI’17 Companion-Workshop Summary for HUMANIZE’17 Graus, M., Ferwerda, B., Schedl, M., Tkalčič, M., Willemsen, M., and Germanakos, P. Companion of the 22nd ACM International Conference on Intelligent User Interfaces (IUI 2017), Limassol, Cyprus, March 2017.

Intelligent User Interfaces for Social Music Discovery and Exploration of Large-scale Music Repositories Schedl, M. Proceedings of the 22nd ACM International Conference on Intelligent User Interfaces (IUI 2017): Workshop on Theory-Informed User Modeling for Tailoring and Personalizing Interfaces (HUMANIZE 2017), Limassol, Cyprus, March 2017.

Predicting Genre Preferences from Cultural and Socio-economic Factors for Music Retrieval Skowron, M., Lemmerich, F., Ferwerda, B., and Schedl, M. Proceedings of the 39th European Conference on Information Retrieval (ECIR 2017), Aberdeen, Scotland, UK, April 2017.

How Item Discovery Enabled by Diversity Leads to Increased Recommendation List Attractiveness Ferwerda, B., Graus, M., Vall, A., Tkalčič, M., and Schedl, M. Proceedings of the 32nd ACM Symposium on Applied Computing (SAC 2017), Marrakesh, Morocco, April 2017.

The Influence of Users' Personality Traits on Satisfaction and Attractiveness of Diversified Recommendation Lists Ferwerda, B., Graus, M., Vall, A., Tkalčič, M., and Schedl, M. Proceedings of the 4th Workshop on Emotions and Personality in Personalized Services (EMPIRE 2016), Boston, USA, September 2016.

Personality-Based User Modeling for Music Recommender Systems Ferwerda, B. and Schedl, M. Proceedings of the European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2016), Riva del Garda, Italy, September 2016.

Investigating the Relationship Between Diversity in Music Consumption Behavior and Cultural Dimensions: A Cross-country Analysis Ferwerda, B. and Schedl, M. Proceedings of the 24th International Conference on User Modeling, Adaptation and Personalization (UMAP 2016): Workshop on Surprise, Opposition, and Obstruction in Adaptive and Personalized Systems (SOAP 2016), Halifax, Canada, July 2016.

Timbral and Semantic Features for Music Playlists Vall, A., Eghbal-Zadeh, H., Dorfer, M., and Schedl, M. International Conference on Machine Learning (ICML 2016): Machine Learning for Music Discovery Workshop, New York, USA, June 2016.

An Analysis of Agreement in Classical Music Perception and Its Relationship to Listener Characteristics Schedl, M., Eghbal-Zadeh, H., Gómez, E., and Tkalčič, M. Proceedings of the 17th International Society for Music Information Retrieval Conference (ISMIR 2016), New York, USA, August 2016.

Music Information Retrieval: Overview, Recent Developments and Future Challenges Gómez, E., Schedl M., Serra X., and Hu X. Proceedings of the 17th International Society for Music Information Retrieval Conference (ISMIR 2016), New York, USA, August 2016.

Exploring Music Diversity Needs Across Countries Ferwerda, B., Vall, A., Tkalčič, M., and Schedl, M. Proceedings of the 24th International Conference on User Modeling, Adaptation and Personalization (UMAP 2016), Halifax, Canada, July 2016.

A Dataset of Multimedia Material About Classical Music: PHENICX-SMM Schedl, M., Hauger, D., Tkalčič, M., Melenhorst, M., and Liem, C.C.S. Proceedings of the 14th International Workshop on Content-Based Multimedia Indexing (CBMI 2016), Bucharest, Romania, June 2016.

Music Tweet Map: A Browsing Interface to Explore the Microblogosphere of Music Hauger, D. and Schedl, M. Proceedings of the 14th International Workshop on Content-Based Multimedia Indexing (CBMI 2016), Bucharest, Romania, June 2016.

The LFM-1b Dataset for Music Retrieval and Recommendation Schedl, M. Proceedings of the ACM International Conference on Multimedia Retrieval (ICMR 2016), New York, USA, April 2016.

Personalized Retrieval and Browsing of Classical Music and Supporting Multimedia Material Tkalčič, M., Schedl, M., Liem, C.C.S., and Melenhorst, M. Proceedings of the ACM International Conference on Multimedia Retrieval (ICMR 2016), New York, USA, April 2016.

A Personality-based Adaptive System for Visualizing Classical Music Performances Schedl, M., Melenhorst, M., Liem, C.C.S., Dominguez, A.M., and Tkalčič, M. Proceedings of the 7th ACM Multimedia Systems Conference (MMSys 2016), Klagenfurt, Austria, May 2016.

Fusing Social Media Cues: Personality Prediction from Twitter and Instagram Skowron, M., Ferwerda, B., Tkalčič, M., and Schedl, M. Proceedings of the 25th International World Wide Web Conference (WWW 2016), Montreal, Canada, April 2016.

Personality Traits and the Relationship with (Non-)Disclosure Behavior on Facebook Ferwerda, B., Schedl, M., and Tkalčič, M. Companion of the 25th International World Wide Web Conference (WWW 2016): 7th International Workshop on Modeling Social Media (MSM 2016), Montreal, Canada, April 2016.

Fusing Web and Audio Predictors to Localize the Origin of Music Pieces for Geospatial Retrieval Schedl, M. and Zhou, F. Proceedings of the 38th European Conference on Information Retrieval (ECIR 2016), Padova, Italy, March 2016.

Using Instagram Picture Features to Predict Users' Personality Ferwerda, B., Schedl, M., and Tkalčič, M. Proceedings of the 22nd International Conference on MultiMedia Modeling (MMM 2016), Miami, USA, January 2016.

Towards Personalizing Classical Music Recommendations Schedl, M. Proceedings of the 2nd International Workshop on Social Media Retrieval and Analysis (SoMeRA 2015), Atlantic City, USA, November 2015.

RFA at MediaEval 2015 Affective Impact of Movies Task: A Multimodal Approach Mironică, I., Ionescu, B., Sjöberg, M., Schedl, M., and Skowron, M. Working Notes Proceedings of MediaEval 2015: Multimedia Benchmark Workshop, Wurzen, Germany, September 2015.

The MediaEval 2015 Affective Impact of Movies Task Sjöberg, M., Baveye, Y., Wang, H., Quang V.L., Ionescu, B., Dellandréa, E., Schedl, M., Demarty, C.-H., and Chen, L. Working Notes Proceedings of MediaEval 2015: Multimedia Benchmark Workshop, Wurzen, Germany, September 2015.

Listener-aware Music Search and Recommendation Schedl, M. Proceedings of the 6th International Workshop on Mining Ubiquitous and Social Environments (MUSE 2015), Porto, Portugal, September 2015.

Improving Music Recommendations with a Weighted Factorization of the Tagging Activity Vall, A., Skowron, M., Knees, P., and Schedl, M. Proceedings of the 16th International Society for Music Information Retrieval Conference (ISMIR 2015), Malaga, Spain, October 2015.

Timbral Modeling for Music Artist Recognition Using I-vectors Eghbal-zadeh, H., Schedl, M., and Widmer, G. Proceedings of the 23rd European Signal Processing Conference (EUSIPCO 2015), Nice, France, August-September 2015.

Listener-aware Music Recommendation from Sensor and Social Media Data Schedl, M. Proceedings of the European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2015), Porto, Portugal, September 2015.

I-Vectors for Timbre-Based Music Similarity and Classification Eghbal-zadeh, H., Lehner, B., Schedl, M., and Widmer, G. Proceedings of the 16th International Society for Music Information Retrieval Conference (ISMIR 2015), Malaga, Spain, October 2015.

Music Retrieval and Recommendation - A Tutorial Overview Knees, P. and Schedl, M. Proceedings of the 38th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2015), Santiago, Chile, August 2015.

Tailoring Music Recommendations to Users by Considering Diversity, Mainstreaminess, and Novelty Schedl, M., and Hauger, D. Proceedings of the 38th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2015), Santiago, Chile, August 2015.

Personality & Emotional States: Understanding Users' Music Listening Needs Ferwerda, B., Schedl, M., and Tkalčič, M. Extended Proceedings of the 22nd International Conference on User Modeling, Adaptation and Personalization (UMAP 2015), Dublin, Ireland, June-July 2015.

Personality Correlates for Digital Concert Program Notes Tkalčič, M., Ferwerda, B., Hauger, D., and Schedl, M. Proceedings of the 22nd International Conference on User Modeling, Adaptation and Personalization (UMAP 2015), Dublin, Ireland, June-July 2015.

Personality Traits Predict Music Taxonomy Preferences Ferwerda, B., Yang, E., Schedl, M., and Tkalčič, M. ACM CHI '15 Extended Abstracts on Human Factors in Computing Systems (CHI 2015), Seoul, Republic of Korea, April 2015.

On the Influence of User Characteristics on Music Recommendation Schedl, M., Hauger, D., Farrahi, K., and Tkalčič, M. Proceedings of the Proceedings of the 37th European Conference on Information Retrieval (ECIR 2015), Vienna, Austria, March-April 2015.

Iron Maiden While Jogging, Debussy for Dinner? - An Analysis of Music Listening Behavior in Context Gillhofer, M. and Schedl, M. Proceedings of the 21st International Conference on MultiMedia Modeling (MMM 2015), Sydney, Australia, January 2015.

Genre-based Analysis of Social Media Data on Music Listening Behavior Schedl, M. and Tkalčič, M. Proceedings of the 1st ACM International Workshop on Internet-Scale Multimedia Management (ISMM 2014), Orlando, USA, November 2014.

Impact of Listening Behavior on Music Recommendation Farrahi, K., Schedl, M., Vall, A., Hauger, D., and Tkalčič, M. Proceedings of the 15th International Society for Music Information Retrieval Conference (ISMIR 2014), Taipei, Taiwan, October 2014.

Social Media and Classical Music? - A first analysis within the PHENICX project: “Performances as Highly Enriched aNd Interactive Concert eXperiences” Schedl, M. SoMeRA'14: Proceedings of the First International Workshop on Social Media Retrieval and Analysis, Gold Coast, Australia, July 2014.

Enhancing Music Recommender Systems with Personality Information and Emotional States: A Proposal Ferwerda, B. and Schedl, M. Proceedings of the 2nd Workshop on Emotions and Personality in Personalized Services (EMPIRE 2014), Aalborg, Denmark, July 2014.

Using Social Media Mining for Estimating Theory of Planned Behaviour Parameters Tkalčič, M., Ferwerda, B., Schedl, M., Liem, C. Melenhorst, M., Odić, A., and Košir, A. Proceedings of the 2nd Workshop on Emotions and Personality in Personalized Services (EMPIRE 2014), Aalborg, Denmark, July 2014.

To Post or Not to Post: The Effects of Persuasive Cues and Group Targeting Mechanisms on Posting Behavior Ferwerda, B., Schedl, M., and Tkalčič, M. Proceedings of the 6th ASE International Conference on Social Computing (SocialCom 2014), Stanford, USA, May 2014.

User Geospatial Context for Music Recommendation in Microblogs Schedl, M., Vall, A., and Farrahi, K. Proceedings of the 37th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2014), Gold Coast, Australia, July 2014.

SoMeRA 2014: Social Media Retrieval and Analysis Workshop Schedl, M., Knees, P., and Shen, J. Proceedings of the 37th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2014), Gold Coast, Australia, July 2014.

Mobile Music Genius: Reggae at the Beach, Metal on a Friday Night? Schedl, M., Breitschopf, G., and Ionescu, B. Proceedings of the ACM International Conference on Multimedia Retrieval (ICMR 2014), Glasgow, Scotland, April 2014.

Location-Aware Music Artist Recommendation Schedl, M. and Schnitzer, D. Proceedings of the 20th International Conference on MultiMedia Modeling (MMM 2014), Dublin, Ireland, January 2014.

Location-aware Music Recommendation Using Auto-Tagging and Hybrid Matching Kaminskas, M., Ricci, F., and Schedl, M. Proceedings of the 7th ACM Conference on Recommender Systems (RecSys 2013), Hong Kong, China, October 2013.

Ameliorating Music Recommendation: Integrating Music Content, Music Context, and User Context for Improved Music Retrieval and Recommendation Schedl, M. Proceedings of the 11th International Conference on Advances in Mobile Computing & Multimedia (MoMM 2013), Vienna, Austria, December 2013

Personalized Music Recommendation in a Mobile Environment Schabetsberger, C. and Schedl, M. Proceedings of the 11th International Conference on Advances in Mobile Computing & Multimedia (MoMM 2013), Vienna, Austria, December 2013.

The Million Musical Tweet Dataset: What We Can Learn From Microblogs Hauger, D., Schedl, M., Košir, A., and Tkalčič, M. Proceedings of the 14th International Society for Music Information Retrieval Conference (ISMIR 2013), Curitiba, Brazil, November 2013.