COGNITIVE-PRAGMATIC MOTIVATION OF RECURRENT CONTACT-ESTABLISHING GESTURES IN MULTIMODAL DIALOGUE*

COGNITIVE-PRAGMATIC MOTIVATION OF RECURRENT CONTACT-ESTABLISHING GESTURES IN MULTIMODAL DIALOGUE*

Author:  M.I. Kiose1, A.V. Leonteva2, O.V. Agafonova2, A.A. Petrov 2

Affiliation:  1Institute of Linguistics RAS, 2Moscow State Linguistic University

Abstract:  The study elaborates a cognitive-pragmatic approach towards establishing contact manifested in speech, gesture and gaze in expository dialogue. We presume that cognitive dimension of contact-establishing gestures expressed in mimetic schemas underlies the pragmatic dimension demonstrating communicative and interactional potential. To verify the hypothesis, we conduct the multimodal experiment to collect the expository dialogical behavior of participants using contact-establishing gestures. We explored as cognitive coordinates two frequently appearing mimetic schema groups, SHOW in palm-up-open-hand gestures and RESTRAIN in palm-down-open-hand gestures. Communicative potential of gestures is assessed via a contingent speech mode in the communicative moves: Request, Response, Novel and Common Topic elaboration, while interactional potential is explored via contingent gaze behavior in Face-oriented and Face-averted gaze.
In terms of communicative potential, the study identifies two communicative moves, Novel and Common Topic elaboration which are significantly more frequently mediated by SHOW schemas. Additionally, we reveal several single mimetic schemas which show different distribution with communicative moves, Request vs. Response, Novel vs. Common Topic elaboration. The interactional potential of mimetic schemas expressed in contact-establishing gestures is revealed in the direction of interaction (Face-oriented vs. Face-averted gaze) and it is higher with SHOW schema group; however, we did not find the differences in single mimetic schemas, which means that irrespective of mimetic schema group, gaze behavior is mostly mediated by the interactive regime of dialogue. Consequently, the results reveal that while communicative potential of contact-establishing gestures is largely dependent on their cognitive nature, the interactional potential is less directly dependent on it, being mediated by the communicative nature of gaze behavior during expository dialogue.

Keywords:  cognitive pragmatics, contact-establishing, multimodal dialogue, gesture, gaze, speech, mimetic schemas, multimodal experiment.

References:  Abner N., Cooperrider K., Goldin‐Meadow S. Gesture for Linguists: A Handy Primer // Language and Linguistics Compass. 2015. Vol. 9. No. 11. P. 437-451.
Alibali M.W., Spencer R.C., Knox L., Kita S. Spontaneous Gestures Influence Strategy Choices in Problem Solving // Psychological Science. 2011. Vol. 22. No. 9. P. 1138-1144.
Alibali M.W., Yeo A., Hostetter A.B., Kita S. Representational gestures help speakers package information for speaking // Gesture Studies / eds. R.B. Church, M.W. Alibali, S.D. Kelly. Amsterdam: John Benjamins Publishing Company, 2017. Vol. 7. P. 15-37.
Apostolopoulos J.G., Chou P.A., Culbertson B., Kalker T., Trott M.D., Wee S. The Road to Immersive Communication // Proceedings of the IEEE. 2012. Vol. 100. No. 4. P. 974-990.
Blakemore D. Relevance and Linguistic Meaning: The Semantics and Pragmatics of Discourse Markers. 1. Cambridge University Press, 2002.
Bressem J., Ladewig S.H. Rethinking gesture phases: Articulatory features of gestural movement? // Semiotica. 2011. Vol. 2011. No. 184.
Brône G., Oben B., Jehoul A., Vranjes J., Feyaerts K. Eye gaze and viewpoint in multimodal interaction management // Cognitive Linguistics. 2017. Vol. 28. No. 3. P. 449-483.
Carlson L. Discourse Grammar // Dialogue Games: An Approach to Discourse Analysis / ed. L. Carlson. Dordrecht: Springer Netherlands, 1983. P. 146-151.
Chafe W.L. Discourse, consciousness, and time: The flow and displacement of conscious experience in speaking and writing. Chicago, IL, US: University of Chicago Press, 1994.
Cienki A. From the finger lift to the palm-up open hand when presenting a point: A methodological exploration of forms and functions // Languages and Modalities. 2021. Vol. 1 P. 17-30.
Cienki A. Questions about mental imagery, gesture, and image schemas // Journal of Mental Imagery. 2002. Vol. 26. No. 1-2. P. 43-46.
Cienki A. Some Properties and Groupings of Image Schemas // Current Issues in Linguistic Theory / eds. M. Verspoor, K.D. Lee, E. Sweetser. Amsterdam: John Benjamins Publishing Company, 1997. Vol. 150. P. 3.
Cienki A. Ten Lectures on Spoken Language and Gesture from the Perspective of Cognitive Linguistics: Issues of Dynamicity and Multimodality. Leiden: Brill, 2017. P. 215.
Cienki A. The study of gesture in cognitive linguistics: How it could inform and inspire other research in cognitive science // WIREs Cognitive Science. 2022. Vol. 13. No. 6. P. 1623.
Cienki A., Müller C. Metaphor, Gesture, and Thought // The Cambridge Handbook of Metaphor and Thought: Cambridge Handbooks in Psychology / ed. Jr. Gibbs Raymond W. Cambridge: Cambridge University Press, 2008. P. 483-501.
Clark H.H., Brennan S.E. Grounding in communication // Perspectives on socially shared cognition. Washington, DC, US: American Psychological Association, 1991. P. 127-149.
Cooperrider K., Abner N., Goldin-Meadow S. The Palm-Up Puzzle: Meanings and Origins of a Widespread Form in Gesture and Sign // Frontiers in Communication. 2018. Vol. 3. P. 23.
Dodge E., Lakoff G. Image schemas: From linguistic analysis to neural grounding // From Perception to Meaning / eds. B. Hampe, J.E. Grady. Mouton de Gruyter, 2005. P. 57-92.
Fauconnier G. Pragmatics and Cognitive Linguistics // The Handbook of Pragmatics / eds. L.R. Horn, G. Ward. Wiley, 2006. P. 657-674.
Fritz I., Kita S., Littlemore J., Krott A. Information packaging in speech shapes information packaging in gesture: The role of speech planning units in the coordination of speech-gesture production // Journal of Memory and Language. 2019. Vol. 104. P. 56-69.
Gibbs R.W., Colston H.L. The cognitive psychological reality of image schemas and their transformations // Cognitive Linguistics. 1995. Vol. 6. No. 4. P. 347-378.
Givón T. Beyond foreground and background // Coherence and Grounding in Discourse: Outcome of a Symposium, Eugene, Oregon, June 1984: Typological Studies in Language. Amsterdam: John Benjamins Publishing Company, 1987. Vol. 11.
Göksun T., Goldin-Meadow S., Newcombe N., Shipley T. Individual differences in mental rotation: what does gesture tell us? // Cognitive Processing. 2013. Vol. 14. No. 2. P. 153-162.
Grady J.E. Image schemas and perception: Refining a definition // From Perception to Meaning / eds. B. Hampe, J.E. Grady. Mouton de Gruyter, 2005. P. 35-56.
Gullberg M., Holmqvist K. What speakers do and what addressees look at: Visual attention to gestures in human interaction live and on video // Pragmatics & Cognition. 2006. Vol. 14 No. 1. P. 53-82.
Horn L.R., Ward G.L. The handbook of pragmatics. Malden, MA; Oxford: Blackwell, 2006.
Iriskhanova O., Kiose M., Leonteva A., Agafonova O., Petrov A. Multimodal Collaboration in Expository Discourse: Verbal and Nonverbal Moves Alignment // Speech and Computer: Lecture Notes in Computer Science / eds. A. Karpov, K. Samudravijaya, K.T. Deepak, R.M. Hegde, S.S. Agrawal, S.R.M. Prasanna. Cham: Springer Nature Switzerland, 2023. Vol. 14338. P. 350-363.
Iriskhanova O.K., Nikolayeva A.I. Cognitive basis for the variation of recurrent gestures in explanatory discourse: palm-up-open-hand and palm-down-open-hand gestures // Voprosy Kognitivnoy Lingvistiki. 2023. No. 4. P. 26-34.
Johnson M. The body in the mind: The bodily basis of meaning, imagination, and reason. Chicago, IL, US: University of Chicago Press, 1987.
Kang S., Tversky B., Black J.B. Coordinating Gesture, Word, and Diagram: Explanations for Experts and Novices // Spatial Cognition & Computation. 2015. Vol. 15. No. 1. P. 1-26.
Kiose M.I., Leonteva A.V., Agafonova O.V., Petrov A.A. Multimodal Communicative Moves in Expositive Dialogue: Common and Novel Topic Elaboration // RUDN Journal of Language Studies, Semiotics and Semantics. 2023. Vol. 14. No. 4. P. 1013-1035.
Kita S. How representational gestures help speaking // Language and Gesture / ed. D. McNeill. Cambridge University Press, 2000. P. 162-185.
Kok K.I., Cienki A. Cognitive Grammar and gesture: Points of convergence, advances and challenges // Cognitive Linguistics. 2016. Vol. 27. No. 1. P. 67-100.
Korotaev N.A. Collaborative constructions in Russian conversations: A multichannel perspective // “Computational linguistics and intellectual technologies” International conference on Computational Linguistics and Intellectual Technologies. Moscow: RSUH, 2023.
Ladewig S.H. Recurrent gestures // Handbücher zur Sprach- und Kommunikationswissenschaft / Handbooks of Linguistics and Communication Science (HSK) 38/2 / eds. C. Müller, A. Cienki, E. Fricke, S. Ladewig, D. McNeill, J. Bressem. De Gruyter, 2014. P. 1558-1574.
Lakoff G. Women, fire, and dangerous things: What categories reveal about the mind. Chicago, IL, US: University of Chicago Press, 1987.
Lundine J.P. The Language of Learning: Expository Discourse and the Influences of Cognition and Language / J.P. Lundine. The Ohio State University, 2016.
Mandler J.M. How to build a baby: II. Conceptual primitives. // Psychological Review. 1992. Vol. 99. No. 4. P. 587-604.
Mandler J.M., Pagán Cánovas C. On defining image schemas // Language and Cognition. 2014. Vol. 6. No. 4. P. 510-532.
Oben B., Brône G. What you see is what you do: on the relationship between gaze and gesture in multimodal alignment // Language and Cognition. 2015. Vol. 7. No. 4. P. 546-562.
Rauscher F.H., Krauss R.M., Chen Y. Gesture, speech, and lexical access: The role of lexical movements in speech production. // Psychological Science. 1996. Vol. 7. No. 4. P. 226-231.
Rodero E. Effectiveness, Attractiveness, and Emotional Response to Voice Pitch and Hand Gestures in Public Speaking // Frontiers in Communication. 2022. Vol. 7. P. 869084.
Schiffrin D. Discourse Markers. Cambridge University Press, 1987.
Yasui E. Collaborative idea construction: Repetition of gestures and talk in joint brainstorming // Journal of Pragmatics. 2013. Vol. 46. No. 1. P. 157-172.
Zlatev J. Image schemas, mimetic schemas and children’s gestures // Cognitive Semiotics. 2014. Vol. 7. No. 1. P. 3-29.
Zlatev J. What´s in a schema? Bodily mimesis and the grounding of language // From Perception to Meaning / eds. B. Hampe, J.E. Grady. Mouton de Gruyter, 2005. P. 313-342.
Баранов А.Н., Крейдлин Г.Е. Структура диалогического текста: лексические показатели минимальных диалогов // Вопросы языкознания. 1992. № 3. С. 84-93. [Baranov A.N., Krejdlin G.E. Struktura dialogicheskogo teksta: leksicheskie pokazateli minimal'nyh dialogov // Voprosy jazykoznanija. 1992. №. 3.
P. 84-93.]
Шведова Н.Ю. Русский семантический словарь. Толковый словарь, систематизированный по классам слов и значений / РАН. Ин-т рус. яз.; М.: Азбуковник, 1998. [Shvedova N.Ju. Russkij semanticheskij slovar'. Tolkovyj slovar', sistematizirovannyj po klassam slov i znachenij / RAN. In-t rus. jaz.; M.: Azbukovnik, 1998.]

For citation:  Kiose, M. I., Leonteva, A. V., Agafonova, O. V., & Petrov, A. A. (2024). Cognitive-pragmatic motivation of recurrent contact-establishing gestures in multimodal dialogue. Voprosy Kognitivnoy Lingvistiki, 4, 74-90.

Pages:  74-90