Department of Computational Perception
Department of
Computational Perception
Johannes Kepler Universitšt Linz


Home  –  Mission  –  Teaching  –  People  –  Research  –  Media  –  Impressum


The nepTune Interface

nepTune is an innovative user interface to music repositories. Given an arbitrary collection of digital music files, nepTune creates a virtual landscape which allows the user to freely navigate in this collection. This is accomplished by automatically extracting features from the audio signal and clustering the music pieces. The clustering is used to generate a 3D island landscape in which the user can freely navigate and hear the closest sounds with respect to his/her current position via a surround sound system. Additionally, knowledge extracted automatically from the Web is incorporated to enrich the landscape with semantic information. More precisely, nepTune displays words that describe the heard music and related images on the landscape to support the exploration.

General Idea

Despite the rapid changes in the way music is used, the way music collections are organized on computers and music players has basically remained the same: owners of digital music collections traditionally organize their thousands of audio tracks in hierarchical directories, often structured according to the common scheme (genre --) artist -- album -- track. Indeed, they do not have much of a choice, given the options offered by current music players and computers. The general philosophy underlying nepTune is that music collections should be structured (automatically, by the computer) and presented according to intuitive musical criteria, and that music interfaces should permit and encourage the creative exploration of music repositories, and new ways of discovering hidden treasures in large collections.

Description

The nepTune interface offers an original opportunity to playfully explore music by creating an immersive virtual reality that is founded in the sounds of a user's digital audio collection. The interface should be fun to use and engage people. The basic ingredients are as follows: Using intelligent audio analysis, the pieces of music are clustered according to sound similarity. Based on this clustering, a three≠dimensional island landscape is created that contains the pieces (see screenshot). Hence, in the resulting landscape, similar sounding pieces are grouped together. The more similar pieces the user owns, the higher is the terrain in the corresponding region. The user can move through the virtual landscape and explore his/her collection by using a game pad. Each music collection creates a characteristic and unique landscape. In addition to seeing the music pieces in the landscape, the listener hears the pieces closest to his/her current position. Thus, the user gets an auditory impression of the musical style in the surrounding region, via a 5.1 surround sound system.

Furthermore, the landscape can be enriched with semantic and visual information by using Web retrieval techniques. Instead of displaying song title and performing artist on the landscape, the user can also choose to see words that describe the heard music or images that are related to this content. Thus, besides a purely audio≠based structuring, nepTune also offers more contextual information that may trigger new associations in the listener/viewer, thus making the experience more interesting and rewarding.




Above are four screenshots from the same scene in four different exploration modes. The upper left depicts the plain landscape. In the upper right mode, artist and song name are displayed. Since this island contains Rap music, we find tracks of artists like Busta Rhymes and NaS. The mode depicted in the lower left shows typical words that describe the music, such as Rap, Gangsta, West Coast, lyrical, or Mainstream. In the lower right screenshot, related images from the Web are presented on the landscape. In this case, these images show Rap artists (Busta Rhymes, 2Pac, etc.), as well as related artwork.

Beside the exploration mode, nepTune now offers also the possibility to create playlists on-the-fly. By selecting a start and an end track, nepTune will automatically find a path along the landscape that connects various tracks and allows for continous playback with smooth transitions. A resulting playlist is displayed as a red line leading through the landscape. During playback, this path is followed automatically, making the playlist progress visually more interesting than common music player interfaces.

Download

Unfortunately we don't offer to download nepTune at the moment. Please watch this space for updates or contact Peter Knees to get notified when nepTune will be made available. For the moment, to get a better impression, you can watch a small (and low-quality) demo video.

References

Knees, P., Schedl, M., Pohle, T., and Widmer, G. (2007).
Exploring Music Collections in Virtual Landscapes. IEEE MultiMedia 14(3), 46-54.

Knees, P., Schedl, M., Pohle, T., and Widmer, G. (2006).
An Innovative Three-Dimensional User Interface for Exploring Music Collections Enriched with Meta-Information from the Web. In Proceedings of the ACM Multimedia 2006, Santa Barbara, California, USA, October 2006

Media Coverage


Related Activities




last edited by pk at 2008-02-12