Statistical multimodal integration for audio-visual speech processing

@article{citeulike:1182334, abstract = {Sensory information is indispensable for living things. It is also important for living things to integrate multiple types of senses to understand their surroundings. In human communications, human beings must further integrate the multimodal senses of audition and vision to understand intention. In this paper, we describe speech related modalities since speech is the most important media to transmit human intention. To date, there have been a lot of studies concerning technologies in speech communications, but performance levels still have room for improvement. For instance, although speech recognition has achieved remarkable progress, the speech recognition performance still seriously degrades in acoustically adverse environments. On the other hand, perceptual research has proved the existence of the complementary integration of audio speech and visual face movements in human perception mechanisms. Such research has stimulated attempts to apply visual face information to speech recognition and synthesis. This paper introduces works on audio-visual speech recognition, speech to lip movement mapping for audio-visual speech synthesis, and audio-visual speech translation.}, author = {Nakamura, S. }, citeulike-article-id = {1182334}, doi = {10.1109/TNN.2002.1021886}, journal = {Neural Networks, IEEE Transactions on}, keywords = {avsr, multimodal-integration, review}, number = {4}, pages = {854--866}, posted-at = {2007-03-24 06:12:50}, priority = {0}, title = {Statistical multimodal integration for audio-visual speech processing}, url = {http://dx.doi.org/10.1109/TNN.2002.1021886}, volume = {13}, year = {2002} }

TY - JOUR ID - citeulike:1182334 L3 - citeulike-article-id:1182334 TI - Statistical multimodal integration for audio-visual speech processing JF - Neural Networks, IEEE Transactions on VL - 13 IS - 4 SP - 854 EP - 866 N2 - Sensory information is indispensable for living things. It is also important for living things to integrate multiple types of senses to understand their surroundings. In human communications, human beings must further integrate the multimodal senses of audition and vision to understand intention. In this paper, we describe speech related modalities since speech is the most important media to transmit human intention. To date, there have been a lot of studies concerning technologies in speech communications, but performance levels still have room for improvement. For instance, although speech recognition has achieved remarkable progress, the speech recognition performance still seriously degrades in acoustically adverse environments. On the other hand, perceptual research has proved the existence of the complementary integration of audio speech and visual face movements in human perception mechanisms. Such research has stimulated attempts to apply visual face information to speech recognition and synthesis. This paper introduces works on audio-visual speech recognition, speech to lip movement mapping for audio-visual speech synthesis, and audio-visual speech translation. KW - avsr KW - multimodal-integration KW - review AU - Nakamura, S PY - 2002/// UR - http://dx.doi.org/10.1109/TNN.2002.1021886 ER -