"Eesti Teadusfondi uurimistoetus" projekt ETF8958
ETF8958 "Multimodaalne suhtlus - videoandmete kogumise, analüüsi ja kasutamise kultuuridevahelised ja tehnoloogilised aspektid (1.01.2011−31.12.2015)", Päivi Kristiina Jokinen, Tartu Ülikool, Matemaatika-informaatikateaduskond.
ETF8958
Multimodaalne suhtlus - videoandmete kogumise, analüüsi ja kasutamise kultuuridevahelised ja tehnoloogilised aspektid
Multimodal Interaction - intercultural and technological aspects of videodata collection, analysis, and use
1.01.2011
31.12.2015
Eesti Teadusfondi uurimistoetus
ValdkondAlamvaldkondCERCS erialaFrascati Manual’i erialaProtsent
2. Ühiskonnateadused ja kultuur2.6. Filoloogia ja lingvistikaH360 Rakenduslingvistika, võõrkeelte õpetamine, sotsiolingvistika 6.2. Keeleteadus ja kirjandus50,0
4. Loodusteadused ja tehnika4.6. ArvutiteadusedP176 Tehisintellekt 1.1. Matemaatika ja arvutiteadus (matemaatika ja teised sellega seotud teadused: arvutiteadus ja sellega seotud teadused (ainult tarkvaraarendus, riistvara arendus kuulub tehnikavaldkonda)50,0
PerioodSumma
01.01.2011−31.12.20112 760,01 EUR
01.01.2012−31.12.201211 040,00 EUR
01.01.2013−31.12.201311 040,00 EUR
01.01.2014−31.12.201411 040,00 EUR
01.01.2015−31.12.201511 040,00 EUR
46 920,01 EUR

Projekti eesmärk on multimodaalse suhtluse uurimine vestluse kontekstis ning tagasiside ja vooruvahetuse arvutusmudelite väljatöötamine. Selleks leitakse andmetes seosed verbaalsete lausungite ja multimodaalsete väljendite vahel, nagu seda on käeliigutused, näoilmed ja kehaasendid. Projektis kogutakse, märgendatakse ja analüüsitakse mitme osalejaga videosuhtluse andmeid. Mudelite ehitamiseks kasutatavad meetodid on nii uurijatepoolsed vaatlused kui ka masinõppe tehnikad. Projektis võrreldakse eestikeelse suhtlemise strateegiaid nii Põhjamaades kui ka mujal kasutatavatega. Projekt põhineb taotleja varasemale tööle multimodaalse korpuse kogumisel ja analüüsimisel ning on seotud Põhjamaade ja rahvusvahelise projektiga.
The project aims at investigating multimodal communication in conversational contexts and at providing computational models for feedback and turn-taking processes based on the correlations found between verbal utterances and multimodal expressions such as hand gestures, facial expressions, and body posture. The project will collect, annotate and analyse multiparty videodata, and its methodology is anchored in both human observations and machine-learning techniques to build models. The project also compares Estonian interaction strategies with those used in the neighbouring Nordic countries, and internationally. The project builds on the previous work on multimodal corpus collection and analysis, and relates to the Nordic and international project.

Vastutav täitja (1)

IsikKraadTöökoht ja ametCVOsalemise periood
Päivi Kristiina JokinendoktorikraadTartu Ülikool, Matemaatika-informaatikateaduskond, Arvutiteaduse instituut, külalisprofessor (0,50)EST / ENG01.01.2011−31.12.2015

Põhitäitjad (5)

IsikKraadTöökoht ja ametCVOsalemise periood
Sven LaaterTartu Ülikool, Matemaatika-informaatikateaduskond, Arvutiteaduse instituut, Tarkvarasüsteemide õppetool, Projekti täitja (1,00) Tartu Ülikool, Matemaatika-informaatikateaduskond, Arvutiteaduse instituut, Tarkvarasüsteemide õppetool, Projekti täitja (1,00)EST / ENG01.01.2012−30.06.2012
Siiri PärksondoktorikraadEST / ENG01.01.2011−31.12.2014
Silvi TenjesdoktorikraadDotsentEST / ENG01.01.2011−31.12.2014
Margus Treumuthdoktorikraadkeeleressursside keskuse analüütikEST / ENG01.01.2011−31.12.2011
Martin VelsmagistrikraadEST / ENG01.10.2013−31.12.2015
Projektid
Projekt
EKKTT09-57; "Intelligentne kasutajaliides andmebaasidele"; Mare Koit;
ETF5519; "Žestide kommunikatiivsed ja sotsiokultuurilised aspektid"; Silvi Tenjes;
ETF7503; "Suhtlusstrateegiad suhtlusmudelis: eestikeelse dialoogi modelleerimine arvutil"; Mare Koit;
SF0180078s08; "Loomulike keelte arvutitöötluse formalismide ja efektiivsete algoritmide väljatöötamine ning eesti keelele rakendamine"; Mare Koit;
Publikatsioonid
Publikatsioonid
Joknien, Kristiina; Wilcock, Graham (2013). User Experience in Human-Robot Interactions. Proceedings of The Fourth Workshop on Perceptual Quality of Systems (PQS 2013). Ed. R. Schatz; T. Hossfeld. Vienna, Austria: FTW-TECHREPORT- 128, FTW,.
Jokinen, Kristiina; Wilcock, Graham (2013). Open-Domain Information Access with Talking Robots. Proceedings of the 14th Annual SIGdial Meeting on Discourse and Dialogue. Metz, France: Association for Computational Linguistics (ACL), 360−362.
(2013). Proceedings of the 4th Nordic Symposium on Multimodal Communication 2013. Gothenburg, Sweden, November 15-16. Linköping: Linköping University Electronic Press.
Jokinen, K. (2013). Multimodal interactions – from corpus collection to conversational systems. Invited academic talk. In: .Young Researchers’ Roundtable on Spoken Dialogue Systems (YRRSDS). Metz, France.
Jokinen, K. (2013). Studying multimodal communication with eye-­tracking. Invited talk. In: .the First European Symposium on Multimodal Communication. University of Malta, Malta.
Jokinen, K.; Wilcock, G. (2013). Towards Cloud-based Speech Interfaces for Open-Domain CogInfoCom Systems. Proceedings of the 3rd IEEE International Conference on Cognitive Infocommunications (CogInfoCom). Budapest, Hungary,.
Jokinen, K. (2013). Multimodal Interaction (MINT) Research and Teaching at the University of Tartu. Erasmus Exchange Talk. In: ..
Paggio, P.; Allwood, J.; Ahlsen, E.; Navarretta, C.; Jokinen, K. (2013). Proceedings of The 1st European Symposium on Multimodal Communication. In: .The 1st European Multimodal Symposium. [ilmumas].
Jokinen, K. (2013). Visit by Professor Bipin Indurkhya: Perceptual Similarity, Visual Metaphor and Creativity. In: ..
Jokinen, K. (2013). Erasmus Visit by Professor Minker: Spoken Dialogue Systems. In: ..
Jokinen, K. (2013). Interview of K.Jokinen: Emotsioone moistvad arvutid voiksid opetama ja vanureidhooldama.
Jokinen, Kristiina; Tenjes, Silvi; Rummo, Ingrid (2013). Embodied interaction and semiotic categorization: communicative gestures of a girl with Patau syndrome. In: C. Paradis; J. Hudson; U. Magnusson (Ed.). The Construal of Spatial Meaning: Windows into Conceptual Space (74−97).. Oxford: Oxford University Press. (Explorations in Language and Space; 07).
Jokinen, Kristiina; Wilcock, Graham (2014). Towards Automatic Annotation of Communicative Gesturing. Proceedings of the COLING Workshop Vision and Language (VL’14). Dublin, Ireland,.
Jokinen, Kristiina; Wilcock, Graham (2014). Modelling User Experience in Human-Robot Interactions. In: Ronald Böck, Francesca Bonin, Nick Campbell, Ronald Poppe (Ed.). Multimodal Analyses enabling Artificial Agents in Human-Machine Interaction (45−56).The 2nd Interspeech Satellite Workshop on Multimodal Analyses Enabling Artificial Agents in Human-Machine Interaction (MA3HMI 2014). Singapore: Springer Verlag. (LNAI).
Jokinen, Kristiina; Wilcock, Graham (2014). Automatic and Manual Annotations in First Encounter Dialogues. Human Language Technologies – The Baltic Perspective, 268: The Sixth Baltic Conference Human Language Technologies (HLT) - the Baltic Perspective. Ed. Andrius Utka, Gintarė Grigonytė, Jurgita Kapočiūtė-Dzikienė, Jurgita Vaičenonienė. Kaunas, Lithuania: IOS Press,. ( Frontiers in Artificial Intelligence and Applications).
Särg, Dage; Jokinen, Kristiina (2015). Nodding in Estonian First Encounters. The 2nd European and the 5th Nordic Symposium on Multimodal Communication. Linköping University Electronic Press; Linköpings universitet, 87−95. (Linköping Electronic Conference Proceedings).
Vels, Martin; Jokinen, Kristiina (2015). Recognition of Human Body Movements for Studying Engagement in Conversational Video Files. The 2nd European and the 5th Nordic Symposium on Multimodal Communication. Ed. Jokinen, K.; Vels, M. Linköping: Linköping University Electronic Press; Linköpings universitet, 97−105.
Saatmann, Pihel; Jokinen, Kristiina (2015). Experiments With Hand-tracking Algorithm in Video Conversations. The 2nd European and the 5th Nordic Symposium on Multimodal Communication. Linköping University Electronic Press; Linköpings universitet, 81−86.
Jokinen, Kristiina; Vels, Martin (2015). Proceedings from the 2nd European and the 5th Nordic Symposium on Multimodal Communication. Linköping University Electronic Press; Linköpings universitet.
Jokinen, K. (2014). Multimodal Interaction in the Context of Foreign Language Learning. In: Raud, N. and Burdakova, O. (Ed.). Culture, Languages, History: Integration and Border-Crossing. (182−195).. Narva. (Studia Humaniora et Paedagogica Collegii Narovensis).
Birgitta Ojamaa, Kristiina Jokinen and Kadri Muischenk (2015). Sentiment Analysis on Conversational Texts. 23: Nodalida 2015 http://www2.lingfil.uu.se/nodalida2015/. Ed. Beata Megyesi. 233−238. (NEALT Proceedings Series).
Vels, Martin; Jokinen, Kristiina (2015). Detecting Body, Head, and Speech Signals for Conversational Engagement. The IVA (Interactive Virtual Agents) Conference Workshop 2: Engagement in Social Intelligent Virtual Agents http://iva2015.tudelft.nl/?q=node/11. Delft, The Neatherlands,.
Wilcock, Graham; Jokinen, Kristiina (2015). Multilingual WikiTalk: Wikipedia-based talking robots that switch languages. The 16th Annual SIGdial Meeting on Discourse and Dialogue (SIGDIAL 2015) http://www.sigdial.org/workshops/conference16/. Prague, Czech Republic,.
Jokinen, Kristiina (2015). Multimodal engagement in the WikiTalk robot application. A Satellite workshop of INTERSPEECH 2015 International Workshop on Speech Robotics http://iwsr2015.org/. Dresden, Germany: Tartu Ülikool, x.
Saatmann, Pihel; Jokinen, Kristiina (2015). Experiments With Hand-tracking Algorithm in Video Conversations. The 2nd European and the 5th Nordic Symposium on Multimodal Communication. Linköping University Electronic Press; Linköpings universitet, 81−86.
Jokinen, Kristiina (2015). Multimodal engagement in the WikiTalk robot application. A Satellite workshop of INTERSPEECH 2015 International Workshop on Speech Robotics http://iwsr2015.org/. Dresden, Germany: Tartu Ülikool, x.
Vels, Martin; Jokinen, Kristiina (2015). Recognition of Human Body Movements for Studying Engagement in Conversational Video Files. The 2nd European and the 5th Nordic Symposium on Multimodal Communication. Ed. Jokinen, K.; Vels, M. Linköping: Linköping University Electronic Press; Linköpings universitet, 97−105.
Vels, Martin; Jokinen, Kristiina (2015). Detecting Body, Head, and Speech Signals for Conversational Engagement. The IVA (Interactive Virtual Agents) Conference Workshop 2: Engagement in Social Intelligent Virtual Agents http://iva2015.tudelft.nl/?q=node/11. Delft, The Neatherlands,.
Särg, Dage; Jokinen, Kristiina (2015). Nodding in Estonian First Encounters. The 2nd European and the 5th Nordic Symposium on Multimodal Communication. Linköping University Electronic Press; Linköpings universitet, 87−95. (Linköping Electronic Conference Proceedings).
Birgitta Ojamaa, Kristiina Jokinen and Kadri Muischenk (2015). Sentiment Analysis on Conversational Texts. 23: Nodalida 2015 http://www2.lingfil.uu.se/nodalida2015/. Ed. Beata Megyesi. 233−238. (NEALT Proceedings Series).
Jokinen, Päivi Kristiina; Vels, Martin (2015). Proceedings of the 2nd European and the 5th Nordic Symposium on Multimodal Communication, August 6-8, 2014, Tartu, Estonia. Linköping University Electronic Press, Linköpings universitet.