B-Audio
Summary
This project is an exploration of contactless full-body audio-visual interactions and investigations on how to enable interaction using a web-browser. The design of the interface is motivated by cross-modal investigations of ‘stability’ or ‘balance’. Conceptual representations of human posture and music perception are co-related and empirically evaluated to inform a posture-to-sound mapping. A scalable browser-compatible sonic interface is developed by leveraging machine learning in conjunction with the recent additions to the MAX-MSP audio synthesis environment, namely Node for MAX.
Background
Digital Musical Instruments, Sonic Interfaces and live installations have explored the idea of contactless body-based interaction with audio and visuals. At the core of these studies is an attempt to increase accessibility, facilitate intuitive learning and support engagement. This has been done largely by reviving the focus on aesthetics and undertaking a participant sourced design process. With this intent, recent studies have adopted embodied design methods from Human-Computer Interaction (HCI) and electroacoustic music practice.
In addition to adopting empirical and design-based approaches, the grammar of electroacoustic music analysis has been studied for its strong embodied basis. This project aimed to explore the embodied basis of sound by focussing on a design-specific evaluation and thereby construct a bridge between abstract concepts within the context of cross-modal and interactive audio-visual interfaces.
Design
The design of the system adopted a bottom-up ‘conceptual blending’ approach. This deals with how multiple conceptual domains arising from different sensory stimulus can blend to create an emergent experience. Spectromorphology literature was consulted to extract conceptual descriptors for sound. These were mapped to balance and spatiality schemas relating to the body. Participant sourced empirical data was collected and results were analysed to identify prevalent sound-to-body conceptual relationships.
System development was carried out incrementally by integrating machine learning, client-server architecture and audio-synthesis in stages. Ml5js and P5js were used for pose recognition, data processing and visuals. Node for MAX, Nodejs, Express, socket.io was used for communication and MAX / MSP for audio-synthesis.
Instrument Models (Percolate), Ecological Sound design toolkit (SDT) and BEAP modular synthesizer were utilised for sound design. All mapping strategies were implemented by applying conceptual relations identified in the design phase. Each of the mapping strategies was then individually evaluated via an Étude.
Further details are available on the Design page.
Prominent references for this project include - Spectromorphology: Explaining Sound-Shapes (Dennis Smalley), Designing with Blends (Imaz and Benyon) and Embodied Sonification (Stephen Roddy)