Synaesthetic Syntax – Investigations into gestural and touch interaction with anisotropic metaphors for audiovisual virtual instruments

Having a background in classical and digital audiovisual arts, since some time I have been interested in the potentials of anisotropic metaphors for ‘audiovisual virtual instruments’.

Anisotropy is a concept describing orientation and direction dependent properties of materials and space, used in physics, chemistry, cosmology and is used as a metaphor to develop derived metaphors for interfaces that can use (but also extend the limits of) Cartesian coordinate system.

This presentation introduces main concepts and practical investigations using several input devices ranging from graphic tablets, ‘5D touch’ MIDI keyboard, VR controllers and motion-, brain bio-feedback sensors, integrated in spatial concepts of interface, usable in immersive media performance. I wanted to explore the capacities of combined control (audiovisual and haptic/gestural) as dynamic, multi-dimensional relationship models that can be experienced in spatial representation (stereoscopic image and multi-channel audio).

This research aims to expand the range of interface implementations, offering expressivity dimensions from expert level of complex and subtle manipulations to first-time encounter and possible accommodation of ludic approach or even brutality.

The work is situated in the cognate fields of Immersive Analytics (Skarbez, Polys, Ogle, North, Bowman, 2019) – integrating recent developments in visualization, auditory displays, computing and machine learning and neuroaesthetics.

The work aims to build upon various initiatives of integration of audiovisual expressivity of such as ‘AVUI’ (Correia & Tanaka, 2021) and reflect on the meaning of gesture within concepts of ‘mediation of self’ (eg. Stiegler 2021) or within the social construct of ‘dispositif’ (Foucault 1977).