The nepTune Interface
nepTune is an innovative user interface to music repositories. Given an
arbitrary collection of digital music files, nepTune creates a virtual
landscape which allows the user to freely navigate in this collection.
This is accomplished by automatically extracting features from the
audio signal and clustering the music pieces. The clustering is used to
generate a 3D island landscape in which the user can freely navigate
and hear the closest sounds with respect to his/her current position
via a surround sound system.
Additionally, knowledge extracted automatically from the Web is
incorporated to enrich the landscape with semantic information. More
precisely, nepTune displays words that describe the heard music and
related images on the landscape to support the exploration.
rapid changes in the way music is used,
the way music
collections are organized on computers and music players has basically
remained the same: owners of digital music collections traditionally
organize their thousands of audio tracks in hierarchical directories,
often structured according to the common scheme (genre --) artist --
album -- track. Indeed, they do not have much of a choice, given the
options offered by current music players and computers. The general
philosophy underlying nepTune is that music collections should be
structured (automatically, by the computer) and presented according to
intuitive musical criteria, and that music interfaces should permit and
encourage the creative exploration of music repositories, and new ways
discovering hidden treasures in large collections.
nepTune interface offers an original opportunity to playfully explore
music by creating an immersive virtual reality that is founded in the
sounds of a user's digital audio collection. The interface should be
fun to use and engage people. The basic ingredients are as follows:
Using intelligent audio analysis, the pieces of music are clustered
according to sound similarity. Based on this clustering, a
three≠dimensional island landscape is created that contains the pieces
(see screenshot). Hence, in the resulting landscape, similar sounding
pieces are grouped together. The more similar pieces the user owns, the
higher is the terrain in the corresponding region. The user can move
through the virtual landscape and explore his/her collection by using a game pad. Each
music collection creates a characteristic and unique landscape. In
addition to seeing the music pieces in the landscape, the listener
hears the pieces closest to his/her current position. Thus, the user
gets an auditory impression of the musical style in the surrounding
region, via a 5.1 surround sound system. |
landscape can be enriched with semantic and visual information by using
Web retrieval techniques. Instead of displaying song title and
performing artist on the landscape, the user can also choose to see
words that describe the heard music or images that are related to this
content. Thus, besides a purely audio≠based structuring, nepTune also
offers more contextual information that may trigger new associations in
the listener/viewer, thus making the experience more interesting and
are four screenshots
from the same
scene in four different exploration modes. The
upper left depicts the plain landscape. In the upper right
mode, artist and song name are displayed. Since this island
music, we find tracks of artists like Busta Rhymes and NaS. The mode
depicted in the lower left shows typical words that describe the music,
such as Rap, Gangsta, West Coast, lyrical, or Mainstream. In the lower
right screenshot, related images from the Web are presented on
landscape. In this case, these images show Rap artists (Busta Rhymes,
2Pac, etc.), as well as related artwork.
|Beside the exploration mode, nepTune now offers also the possibility to create playlists on-the-fly.
By selecting a start and an end track, nepTune will automatically find
a path along the landscape that connects various tracks and allows for
continous playback with smooth transitions. A resulting playlist is
displayed as a red line leading through the landscape. During playback,
this path is followed automatically, making the playlist progress
visually more interesting than common music player interfaces.|
DownloadUnfortunately we don't offer to download nepTune at the
moment. Please watch this space for updates or contact Peter Knees
to get notified when nepTune will be made available. For the moment, to
get a better impression, you can watch a small (and low-quality) demo video.
P., Schedl, M., Pohle, T., and Widmer, G. (2007).
Exploring Music Collections in Virtual Landscapes.
Knees, P., Schedl, M.,
Pohle, T., and Widmer, G.
Three-Dimensional User Interface for Exploring Music Collections
Enriched with Meta-Information from the Web.
In Proceedings of the ACM Multimedia 2006,
Barbara, California, USA, October 2006