Grammatical workspace sharing during language production and language comprehension Evidence from grammatical multitasking.pdf

grammatical workspace sharing during language production and language comprehension evidence from grammatical multitasking gerard kempen1 2 nomi olsthoorn3 and simone sprenger4 1max planck institute for psycholinguistics nijmegen the netherlands 2cognitive psychology unit leiden university leiden the netherlands 3amsterdam centre for language and communication university of amsterdam amsterdam netherlands 4department of psychology university of groningen groningen the netherlands grammatical encoding and grammatical decoding in sentence production and comprehension respectively are often portrayed as independent modalities of grammatical performance that only share declarative resources lexicon and grammar the processing resources subserving these modalities are supposed to be distinct in particular one assumes the existence of two workspaces where grammatical structures are assembled and temporarily maintained one for each modality an alternative theory holds that the two modalities share many of their processing resources and postulates a single mechanism for the online assemblage and short term storage of grammatical structures a shared workspace we report two experiments with a novel grammatical multi tasking paradigm the participants had to read i e decode and to paraphrase encode sentences presented in fragments responding to each correspondenceshouldbeaddressedtogerardkempen maxplanckinstitute for psycholinguistics po box 310 6500 ah nijmegen the netherlands e mail gerard kem pen mpi nl the experiments reported herein were conducted at the cognitive psychology unit of leiden university experiment 1 as part of the second author s phd project and at the max planck institute for psycholinguistics experiment 2 as part of a postdoc project awarded to the third author we thank three anonymous reviewers for comments on earlier versions of the text and for helpful suggestions language and cognitive processes 0000 00 00 1 36 2011 psychology press an imprint of the taylor language comprehension grammatical en coding grammatical decoding parsing human sentence processing speech shadowing self monitoring self repair simultaneous interpreting simulta neous translation multitasking spoken dialogue introduction in our daily lives we often find ourselves producing language and comprehending language at the same time involved in a conversation and listening to what our dialogue partner is saying we sometimes produce a comment or reply already before the other person finishes speaking over lapping dialogue turns sacks schegloff stivers et al 2009 listening in to a news broadcast we produce spoken comments on an exciting item while talking we listen to our own speech in order to spot and quickly rectify erroneous or infelicitous utterances self monitoring levelt 1983 1989 if we are fluent in more than one spoken language we can translate between these languages online simultaneous interpreting christoffels de groot and some of us accomplish this feat with much shorter ear voice spans of less than a third of a second marslen wilson 1973 1985 language production and comprehension in parallel is commonplace when dealing with written and sign language as well taking notes during a lecture we produce condensed written versions of the ongoing spoken discourse while reading out loud a newspaper headline or a telegram style message to another person we can insert the missing little words on the fly during proofreading we rephrase incorrect or infelicitous fragments of the text we are reading interpreters for the deaf listen and sign at the same time language production and language comprehension both include cognitive processes that deal with grammatical form in this paper we focus on these processes called grammaticalencoding during language produc tion and grammaticaldecoding during language comprehension 2kempen olsthoorn sprenger grammatical encoding and grammatical decoding both perform mappings between meaning and form more precisely mappings between nonlinguistic communicative intentions and lemma1strings that underlie spoken written or signed linguistic expressions of such intentions grammatical encoding deals with mappings from communicative intentions to linguistic expressions grammatical decoding with mappings from linguistic expressions to com municative intentions in both directions the mappings are usually assumed to include submappings between communicative intentions andconcep tualstructures on the one hand and between conceptual andsyntactic structures on the other communicative intentions include representations of states state changes events etc in the extralinguistic context together with their relationships e g locative temporal causal in the psycholinguistic community the two modalities of grammatical performance tend to be studied separately witness the fact that they are typically addressed in different textbook chapters in unconnected research groups and in separate conferences the grammatical encoding and decoding tasks are supposed to be subserved by distinct dedicated processing resources which presumably do not have much more in common than access to the same declarative resources the lexicon and the grammar 2 among the cognitive resources subserving grammatical processing is a workspace for the assemblage and short term storage of syntactic and conceptual structures 3the assumption of dedicated processing resources then suggests there are two workspaces one subserving each modality an alternative theory of grammatical encoding and decoding holds that the two modalities execute the mappings between communicative intention and linguistic expressions in very similar manners notwithstanding the 1lemmas are syntactic words their lexical entries specify the sentential environments in which they are allowed to occur the morphological and phonological information associated with words is specified in another type of lexical entry called word forms or lexemes 2it is usually acknowledged that developmental pathological and performance factors may lead to differential accessibility of the lexical or grammatical information needed for the two tasks 3the term workspace should not be interpreted with emphasis on work in casu the assemblage of key elements of grammatical structures the assembly process presupposes temporary storage of partial assembled structures working memory but we do not wish to imply that the cognitive resources underlying grammatical assembly processes are distinct from the memory resources enabling temporary storage of the products the unification space model of human syntactic parsing by vosse and kempen 2000 2009 exemplifies a style of neurocognitive modelling that makes do without such a division of labour the distinction tends to be absent from brain style modelling generally the grammatical workspace intended here should also be distinguished from the short term buffer where strings of words more precisely strings of phonological forms or lexemes are temporarily stored this buffer can be occupied by the output of the word recognition mechanism during language comprehension and by the output of the morpho phonological encoding process during language production grammatical multitasking3 directional difference see kempen 2000 for a list of similarities hence grammatical encoding and decoding could be accomplished byshared processing resources by a single exemplar of all or all important parts of the cognitive resources in the psycholinguistic literature we find a limited number of theoretical proposals in this direction suggesting that the mechanism responsible for executing a certain grammatical subtask within one modality could also be deployed in service of the other modality and perform there the same or a similar subtask garrett 2000 pickering townsend hence in each trial the two modalities were active alternatingly task switching the participants reacted to each fragment with a spoken response as quickly as possible a voice key measured the latency of every response in the course of processing an entire sentence the participants switched modality repeatedly in each trial a decoding episode was followed by an encoding episode on the dedicated workspaces hypothesis the currently active modality always works with its own representation of the sentence stored in its own workspace either the original input or the output paraphrase the shared workspace hypothesis predicts that the currently grammatical multitasking5 active modality works with the sentence representation that the other modality has left behind in the shared workspace all experimental stimuli contained a quoted main clause direct speech with the first person personal pronoun as subject np e g 1a however in some trials the subordinating conjunction that was presented in a visually salient manner indicating that from then onward thedirect speech clause had to be realised as grammatically correctindirectspeech e g as 1b see figure 1 the experimental stimuli contained either a grammatically correct reflexive pronoun myself like in 1a or an incorrect one himself as in 1c the critical measurements were the response onset latencies rts to the correct output reflexive pronoun i e himself nb the example figure 1 sentence fragments left and vocal responses right during the paraphrasing task example similar to the materials of experiment 2 the grey rectangles represent the computer screen the enclosed white rectangles the frames where input could appear 6kempen olsthoorn sprenger sentences in this section are in english but the experiments were carried out in dutch 1 a the angry headmaster complained i have seen a nasty cartoon of myself in the hall b the angry headmaster complained that he had seen a nasty cartoon of himself in the hall c the angry headmaster complained i have seen a nasty cartoon of himself in the hall on the dedicated workspaces hypothesis once the decoder has parsed the pronoun i the first person value remains intact throughout the input sentence and is used when the reflexive pronoun is decoded later on the fact that in the meantime the encoder has produced a structure with a third person subject np does not affect the content of the decoding workspace stated differently each modality of grammatical performance works with its own grammatical form under construction and switching between modality entails switching between associated forms on the shared workspace hypothesis however the two modalities work with the same token identical grammatical form under construction hence the encoding process over writes the value of the subject s person feature to third person and the new value is used when decoding the reflexive pronoun because the latter has to agree with the subject in number gender and person the decoding process spots a mismatch this suggests that corrupting an input sentence like 1a by changing the correct myself to incorrect himself as in 1c will not be noticed by the decoding process this prediction was tested in the experiments below 4 in addition to the paraphrasing task the experiments included a very similar control task dubbed proofreading again the participants read sentences with either a correct or an incorrect input reflexive all they had to do was to repeat the input fragments out loud immediately correcting any morpho syntactically incorrect input words for proofreading trials with input sentences like 1c we predict on the dedicated workspaces as well as the shared workspace hypothesis a response delay due to the combination of an expectancy violation incorrect input reflexive and a repair action whereas no delay is predicted for well formed input sentences like 1a 4an informal demonstration of the sensitivity of the decoding process to parallel events in the encoding process can proceed as follows one asks participants who have performed a paraphrasing trial with an ungrammatical input sentence like 1c to re read the sentence without paraphrasing according to the first author s experience during lectures and in classrooms since 2003 while re reading nearly all serious participants express surprise that the sentence is ill formed and that they had failed to notice this during the paraphrasing trial grammatical multitasking7 baseline performance the proofreading task provides an independent measurement in the absence of paraphrasing operations but under otherwise identical conditions of the time needed to deal with an expectancy violation followed by carrying out the corresponding repair 5 as the overt response to an input reflexive pronoun that does not fit the output sentence is determined by sequentially executing decoding and encoding steps any processing delays during these steps add up the decoder may incur a delay due toviolationof a grammatical expectation the encoder is delayed when it has to execute a repair setting the person number gender features of the reflexive equal to those of the subject np and retrieving the corresponding lexeme from the mental lexicon 6expectancy violations will often be followed by repairs but on the dedicated workspaces hypothesis this does not always hold for the paraphrasing task on this hypothesis the encoding and the decoding processes are able to work with distinct grammatical structures concurrently hence situations may arise where the decoding process runs into a violated expectation whereas the encoding process due to having embarked upon a modified output sentence need not launch a repair simply because the input word that caused the expectation violation actually happens to fit syntactically into the output sentence under construction this holds for paraphrasing trials with sentence 1c as input in the opposite case exemplified by paraphrasing trials with sentence 1a as input the decoding process can proceed unhampered by any expectancy violations but the encoding process needs to initiate a repair in order to come up with the reflexive pronoun that fits into the correct output sentence 1b according to theshared workspacehypothesis which entails that the decoding process always works with the same token identical grammatical structure as the encoding process an expectancy violation and a repair are either both present causing a double delay violation repair or both absent no delay baseline performance applied to upcoming reflexive pronouns this means that such a pronoun either violates an expectation engendered by the decoding process thereby indirectly triggering a repair 5incidentally we assume that at every input fragment proofreading also requires a switch from grammatical decoding to grammatical encoding during an encoding episode the conceptual content derived from the current input fragment is grammatically encoded by using as much as possible the lexical material of the input fragment under this characterisation the proofreading task is neutral between the two architectural hypotheses under investigation 6one may object that the second source of rt delays is merely a stroop effect insofar as the speaker has to withhold the primary response to the stimulus word and to produce another response instead however the precise response selection mechanism underlying the rt delays is irrelevant we only assume that the rts reflect the syntactic expectancies prevailing at the moment the speakers prepare their response to the reflexive pronoun these serve as a window on the object of our inquiry the grammatical structure s currently under assembly 8kempen olsthoorn sprenger within the encoding process e g in paraphrasing trials with 1a as input or in proofreading trials with 1c as input or does not violate any expectation hence obviating the need for a repair yielding baseline performance e g in paraphrasing trials with 1c or in proofreading trials with 1a as input thededicated workspacesmodel in contrast predicts a different rt pattern for paraphrasing trials with input sentences 1a and 1c the rts should lie in between those predicted for baseline conditions in paraphrasing and proofreading no delay on the one hand and the condition that triggers a double delay on the other violation repair proofreading incorrect input like sentence 1c during paraphrasing after well formed input 1a no violation will be spotted by the decoder but the encoder needs to make a repair selection and production of a new reflexive after ill formed input 1c there is an expectancy violation but no repair in proofreading incorrect input 1c gives rise to a double violation both on the dedicated workspaces and the shared workspace model see also the derivation of the predictions for the two experiments and the tables in the appendix in the next two sections we describe two paraphrasing proofreading experiments in experiment 1 the proofreading and the paraphrasing tasks were performed by different groups of participants i e task was a between subjects factor we preferred this over a within subjects design because of fear that the participants might mix up the two tasks both groups of participants produced indirect speech output sentences where third person reflexive pronouns were correct experiment 2 used a within subject design all participants were tested on both tasks one block of proofreading trials and another block of paraphrasing trials with order of blocks counterbalanced moreover after these blocks the participants executed a silent self paced reading spr task with the same sentences direct speech versions the spr data served to verify that the ungramma tical reflexives indeed cause comprehension problems when no spoken responses are required experiment 1 in experiment 1 the two tasks proofreading and paraphrasing were administered to different groups of dutch speaking participants both groups had to read input sentences fragment by fragment responding to each fragment as fast as possible with a spoken response either a repetition of the input fragment or a morpho syntactic variant thereof depending on the task