Toward Music Communication with Supplemental Visual Information

概要

論文の詳細を見る
We recognized that college students who are deaf and hard of hearing are also interested inmusic and have been working toward creating an environment in which both people with andwithout hearing impairments can enjoy music together. Visual assistance raises one possibilityof enhancing information that music implies. We have developed a system music with motionpictures (MPM), which generates motion pictures from drawings. MPM animates a drawingwith sound data, thus providing a motion picture that is synchronized with the sound data.We generated a set of motion pictures using a musical performance and a drawing that wereintended to convey the same specific emotion. Then, we conducted an experiment to investigatewhether a musical performer's emotional intention was better conveyed to listeners whenmotion pictures were provided with the music. While the results matched our expectations inthat the motion pictures improved the recognition of the emotion in the music, we observedconflict of recognition when combining information from several media and generating motionpictures that yielded a better recognition of emotion.