nepTune is an innovative user interface to music repositories. Given an arbitrary collection of digital music files, nepTune creates a virtual landscape which allows the user to freely navigate in this collection. This is accomplished by automatically extracting features from the audio signal and clustering the music pieces. The clustering is used to generate a 3D island landscape in which the user can freely navigate and hear the closest sounds with respect to his/her current position via a surround sound system. Additionally, knowledge extracted automatically from the Web is incorporated to enrich the landscape with semantic information. More precisely, nepTune displays words that describe the heard music and related images on the landscape to support the exploration.

Despite the rapid changes in the way music is used, the way music collections are organized on computers and music players has basically remained the same: owners of digital music collections traditionally organize their thousands of audio tracks in hierarchical directories, often structured according to the common scheme (genre --) artist -- album -- track. Indeed, they do not have much of a choice, given the options offered by current music players and computers. The general philosophy underlying nepTune is that music collections should be structured (automatically, by the computer) and presented according to intuitive musical criteria, and that music interfaces should permit and encourage the creative exploration of music repositories, and new ways of discovering hidden treasures in large collections.

Description

The nepTune interface offers an original opportunity to playfully explore music by creating an immersive virtual reality that is founded in the sounds of a user's digital audio collection. The interface should be fun to use and engage people. The basic ingredients are as follows: Using intelligent audio analysis, the pieces of music are clustered according to sound similarity. Based on this clustering, a three­dimensional island landscape is created that contains the pieces (see screenshot). Hence, in the resulting landscape, similar sounding pieces are grouped together. The more similar pieces the user owns, the higher is the terrain in the corresponding region. The user can move through the virtual landscape and explore his/her collection by using a game pad. Each music collection creates a characteristic and unique landscape. In addition to seeing the music pieces in the landscape, the listener hears the pieces closest to his/her current position. Thus, the user gets an auditory impression of the musical style in the surrounding region, via a 5.1 surround sound system.

Furthermore, the landscape can be enriched with semantic and visual information by using Web retrieval techniques. Instead of displaying song title and performing artist on the landscape, the user can also choose to see words that describe the heard music or images that are related to this content. Thus, besides a purely audio­based structuring, nepTune also offers more contextual information that may trigger new associations in the listener/viewer, thus making the experience more interesting and rewarding.

Above are four screenshots from the same scene in four different exploration modes. The upper left depicts the plain landscape. In the upper right mode, artist and song name are displayed. Since this island contains Rap music, we find tracks of artists like Busta Rhymes and NaS. The mode depicted in the lower left shows typical words that describe the music, such as Rap, Gangsta, West Coast, lyrical, or Mainstream. In the lower right screenshot, related images from the Web are presented on the landscape. In this case, these images show Rap artists (Busta Rhymes, 2Pac, etc.), as well as related artwork.