Synaesthetic Syntax – Interactive Multimedia Generated by Rubbing/Tactile Interfaces – Biofeedback Effects for Wellness Entertainment

This research aims the Wellness Entertainment with biofeedback especially tactile interfaces. A Japanese company “RT” developed the small “PAW sensor” which is covered with Urethane Foam and detects four individual values of rubbing controls by fingers optically, and reacts natural elastic repulsions. I have developed four generations system with this unique sensor, and I will report and discuss the possibility of human tactile interaction with a point of view of wellness entertainment. As is well known, there are so many sensors/interfaces are used in the interactive systems, however, almost these interfaces miss the soft feel and sensation of gentle manipulation. The pressure sensor or strain gauge detects the “maximum” value like a physical fitness test. Of course the “CV” sensors have no physical reaction because they are separated with people.

I developed a new musical instrument called “Multi Rubbing Tactile Instrument”. This had ten sensors on “Egg” shaped acrylic sphere to be held in both hands. All PAW sensor information were continuous value, so the were converted to the parameters for realtime “voice-like” generating system (like the meow of a cat or similar animal). Corresponding to the abstract nature of the auditory output of the system, I employed an algorithm that generated fractal images in real time, drawing beautiful two-dimensional fractal images in grayscale for each of the two modes, and changing their state in real time in response to 32-channel parameters coming from the sensor.

The demonstrations as an interactive installation work were surprisingly well received. I had two chances of demonstration: “Sketching” conference in Arizona and Science Museum in Singapore. The researchers/engineers and children began their experience by touching this unknown toy with trepidation at first. To my surprise, I discovered that they all “smiled” without realizing it as they enjoyed the strange world of sounds and animations generated and the sensation of their fingers being gently returned to them.

This experience led me to to develop a new 8-channel tactile sensor “PAW-eight” with my collaborator, a rehabilitation specialist. This system has eight PAW sensor for both hands to wrap up, total 32 channels information reach the system. From the beginning, the objective of this system was set as “MCI prevention” (activating the brain through the sense of touch at the fingertips in conjunction with hearing and vision). This system is as a “Serious Game”, people must keep all eight fingers pushing PAW sensor with the “half” level and softly (4 channel signals will be almost equal) to move the 3D pointer into the “black hall” in the center of the space. The pitch of sound become low when the pointer is inside of the “black whole”, so the aim of the game is to make the space will be silent with gentle control of all eight fingers. Care and rehabilitation professionals have evaluated the system and found it to be effective in leading to “wellness”.

I would like to pursue multimedia that effectively stimulates the emotions of touch and wellness.