Current interfaces on a mobile phone are based on alpha-numeric keys and touch technologies, which expect a human being to operate over a tiny area. These interfaces are extremely sub-optimal and restricted for literate/illiterate and differently-abled. Our proposal aims to overcome this limitation of and build a rich interface using voice and gesture based technologies for a multi-modal mobile interaction and computing.
The project would like to address how to seamlessly integrate voice and gesture technologies for building a rich multi-modal conversational interface, in order to understand the issues of fusion and interplay of voice and gesture technologies on a mobile device, so that literate, illiterate, and differently abled persons can use this rich multimodal interface for better communication, information access for services related to health care and entertainment.
The specific research issues of interest include:
- Multimodal grammars integrating voice and gestures for task completion.
- Active learning of pronunciation and acoustic models for scalability and adaptability,and
- Exploitation of syllabic and postpositional nature of Indian languages for speech recognition and synthesis.
The Proposed Deliverables of this Project
|Task||0-6 months||7-12 months||2nd Year||3rd Year|
|Data collection||Conversational||Hand-written||Multimodal data|
|Differently-abled datasets||socionity datasets|
|Virtual assistant||Voice mode||Multimodal||Tuning|
|Dr. Kishore Prahallad, IIIT Hyderabademail@example.com||Link to homepage|
|Dr. Y. Padma Sai, VNRVGIETfirstname.lastname@example.org||Link to homepage|
|Dr. A. Jagan, BVRITemail@example.com||Link to homepage|
For more details, check the project website.
|Number of Workshop/Training(s)||0|