SpectAR


Hi Ableton! :) When I was chatting with Nathan a couple weeks back, I had the idea of using Augmented Reality to display a track’s stereo mix around me. So  I decided to build a quick concept that showcases this as an iPad App syncing with Live 10 via Link. 


The idea at its core is simple. When using the App, the user is able to sit within the stereo mix that is surrounding her or him in an 180° arc. By moving the iPad around horizontally, the user can observe the different panned instruments and their respective positions in the stereo mix.

It is also possible to see the frequencies of those instruments by slightly shifting the iPad vertically. When wearing headphones, the user hears the instruments in front of her or him only in a narrow cone, making it possible to precisely explore individual areas of the stereo mix.

Because it somehow merges AR elements with a Spectrum, I went for SpectAR as the working title. 👻 








Top Section
Apart from the application name, the user finds two buttons here. Using the ‘Focus’ button, the user can toggle Focus Mode on or off (see below). The ‘Link’ element indicates if the app is connected via Live Link. If not, tapping here offers options to establish it. 


Main Section
In this view, the user sees the color-coded waveforms and the corresponding clip names below. By shifting the iPad on the x and y axis, the user can explore the stereo mix and the frequency bands. Pan and Frequency labels mark help with orientation. When Focus Mode is active (see above), the user only hears the clips in the area in front of her or him (marked in red). Otherwise, the whole mix is played. 


Bottom Section
Clips with no Pan defined will be displayed here. 


Possible Future Functionalities

In the current state, this app would only display the mix, but offer no option to manipulate it. In a future iteration, it could be possible to move the position (read: pan) of the clips and to toggle if they are mute or not. Additionally, it would be very helpful to have means to adjust how narrow the ‘listening cone’ defined by the Focus Mode is. Also, taking a step towards VR support for smaller mobile devices would not be far fetched. 


Prototype
To demonstrate the idea, I began building a prototype with Aframe. But because of the restrictions that both iOS and Android have in place for manipulating sound source volumes on mobile browsers, I hit a roadblock I was unable to overcome. Nevertheless the prototype illustrates the basic principle of the concept. Try the SpectAR Prototype︎, best on a mobile browser. 


Thank you for taking your time to check out the concept. Let me know what you think or head over to my website︎ for more info and projects!