Chapter 4 A Background Perspective on Touch as a Multimodal (and Multisensor) Construct
Ken Hinckley
4.2 The Duality of Sensors and Modalities
4.3 A Model of Foreground and Background Interaction
4.4 Seven Views of Touch Interaction
4.5 Summary and Discussion
Focus Questions
References
Chapter 5 Understanding and Supporting Modality Choices
Anthony Jameson, Per Ola Kristensson
5.2 Synthesis of Research on Modality Choices
5.3 Brief Introduction to the ASPECT and ARCADE Models
5.4 Consequence-Based Choice
5.5 Trial-and-Error-Based Choice
5.6 Policy-Based Choice
5.7 Experience-Based Choice
5.8 Other Choice Patterns
5.9 Recapitulation and Ideas for Future Research
Focus Questions
References
Stefan Kopp, Kirsten Bergmann
6.1 Introduction
6.2 Multimodal Communication with Speech and Gesture
6.3 Models of Speech and Gesture Production
6.4 A Computational Cognitive Model of Speech and Gesture Production
6.5 Simulation-based Testing
6.6 Summary
Focus Questions
References
Chapter 7 Multimodal Feedback in HCI: Haptics, Non-Speech Audio, and Their Applications
Euan Freeman, Graham Wilson, Dong-Bach Vo, Alex Ng, Ioannis Politis, Stephen Brewster
7.1 Overview of Non-Visual Feedback Modalities
7.2 Applications of Multimodal Feedback: Accessibility and Mobility
7.3 Conclusions and Future Directions
Focus Questions
References
Chapter 8 Multimodal Technologies for Seniors: Challenges and Opportunities
Cosmin Munteanu, Albert Ali Salah
8.2 Senior Users and Challenges
8.3 Specific Application Areas
8.4 Available Multimodal-Multisensor Technologies
8.5 Multimodal Interaction for Older Adults—Usability, Design, and Adoption Challenges
8.6 Conclusions
Focus Questions
References
PART III COMMON MODALITY COMBINATIONS
Chapter 9 Gaze-Informed Multimodal Interaction
Pernilla Qvarfordt
9.2 Eye Movements and Eye Tracking Data Analysis
9.3 Eye Movements in Relation to Other Modalities
9.4 Gaze in Multimodal Interaction and Systems
9.5 Conclusion and Outlook
Focus Questions
References
Chapter 10 Multimodal Speech and Pen Interfaces
Philip R. Cohen, Sharon Oviatt
10.2 Empirical Research on Multimodal Speech and Pen Interaction
10.3 Design Prototyping and Data Collection
10.4 Flow of Signal and Information Processing
10.5 Distributed Architectural Components
10.6 Multimodal Fusion and Semantic Integration Architectures
10.7 Multimodal Speech and Pen Systems
10.8 Conclusion and Future Directions
Focus Questions
References
Chapter 11 Multimodal Gesture Recognition
Athanasios Katsamanis, Vassilis Pitsikalis, Stavros Theodorakis, Petros Maragos
11.1 Introduction
11.2 Multimodal Communication and Gestures
11.3 Recognizing Speech and Gestures
11.4 A System in Detail
11.5 Conclusions and Outlook
Focus Questions
References
Chapter 12 Audio and Visual Modality Combination in Speech Processing Applications
Gerasimos Potamianos, Etienne Marcheret, Youssef Mroueh, Vaibhava Goel, Alexandros Koumbaroulis, Argyrios Vartholomaios, Spyridon Thermos
12.1 Introduction
12.2 Bimodality in Perception and Production of Human Speech
12.3 AVASR Applications and Resources
12.4 The Visual Front-End
12.5 Audio-Visual