Anaphoric dependencies : A window into the architecture of the language system Sergey Avrutin Eric Reuland Frank Wijnen Olga Khomitsevitch Arnout Koornneef Natalia Slioussar Nada Vasic

Goals of the course Explain how language structure and neurocognitive organization can meet Provide some background on current approaches Present a number of issues on which current discussions focus

Overview Fundamentals of Linguistics against a neurocognitive background The grammatical encoding of anaphoric dependencies Linguistic architecture and cognitive architecture: –Language processing –Language impairment Acquired Congenital

What is Language? Language: Systematic relation between Forms: events in an (external) medium (sound, gesture, ink on paper) & Interpretations: change in information state of the mind

Fundamental issues 1.How did language evolve? 2.What is the structure of language? 3.How is language represented in the brain? 4.How is language acquired? Focus on the relation between 2 and 3

How is language represented in the brain? Reflects general issue of the division of labour between brain areas –Modularity –Functionality –Plasticity

Theoretical approach: Minimalist Program The (minimal) language system: Sensori-motor system | PF-interface | Computational system of Human Language (C HL ) (+Lexicon) | Conceptual-Intentional Interface (C-I interface) | System of thought

Theoretical approach: Minimalist Program The minimal language system PF interfaceC-I interface Sensori- C HL Interpretation Motor system system (system of thought) Lexicon - dedicated +dedicated(?) -dedicated

Levelt : Speaking (1989)

Task: find map between linguistic operations and neurocognitive processes PF-interface | Computational system of Human Language (C HL ) (+Lexicon) | Conceptual-Intentional Interface (C-I interface) ?

The triangle of cognitive neuroscience (Hagoort 2003) Computational model Cognitive Archtecture Neural Architecture Neuro- physiology Neuro anatomy Behaviour

A note on method The brain is doing a lot at the same time In whatever you try to measure, you will find a lot of noise no escape from forming precise (and falsifiable) hypotheses: a theory is your eyes - without it you are blind

On the relation between linguistics and psycholinguistics "The split between linguistics and psycholinguistics in the 1970s has been interpreted as being a retreat by linguists from the notion that every operation of the grammar is a mental operation that a speaker must perform in speaking and understanding language. But, putting history aside for the moment, we as linguists cannot take the position that there is another way to construct mental representations of sentences other than the machinery of grammar.....There is no retreat from the strictest possible interpretation of grammatical operations as the only way to construct linguistic representations" (Alec Marantz, lecture notes 2000)

Correspondence Thesis Differences between operations within (major) modules of the grammatical system correspond with differences in processes at the neural level and vice versa. (Reuland 2003)

In plain language We have to figure out what the brain does in order to be able to figure out how the brain does it We have to figure out how the brain does things in order to figure out how it can do what it does the main danger is not being precise enough on either side

Tensions between requirements on Linguistic descriptions What do we need for easy description? What do we need for explanation? Compare - Quantum physics - Newtonian mechanics For understanding planetary motion For understanding why there are no white holes

An example: how local are the dependencies we (can) compute? What did John see? What did John see – What did John [ [see - ] ] Issues of this type may occasionally seem abstract but are crucial for our understanding

The level of correpondence

Current views on modularity Is there a division of labour between brain areas? Answer: There is specialisation Lateralisation: left-right asymmetry Specialized areas of the cortex: - Motor-cortex - Visual cortex - Auditory cortex Etc.

Specialisation within areas Example: (Kandel et al. 2000, Principles of Neural Science; Ch 28): Visual system: specific neurons for: Black/white detection Colour detection Form detection Depth detection Movement detection Facial recognition

FUNCTIONS & functions Kosslyn & König (1995) The Wet Mind: FUNCTIONS (Vision, Hearing, etc. ) v.s. functions (movement detection, depth detection, etc.) Binding problem: How do the different functions lead to one unified perception?

Language : FUNCTION vs functions Is there one language system? Or: Are there different subsystems that contribute to the FUNCTION Language? If so, what are the functions subserving language? How elementary are these functions? Are there any functions dedicated to language?

Summary of the task Precise analysis of the operations needed to capture the structure of language Match these operations with real time processes in the brain Identify brain areas involved in these processes

Methods 1.Grammar: Precise modeling of structure and interpretation 2.Studying Experiments of nature: language impairment (genetic, acquired) 3.Behavioral studies: complexity, processing resources 4.Eye-tracking 5.ERP 6.fMRI 7.PET

Language: basic structure Components: Phonology Syntax Lexicon Semantics Discourse

Required for explanation What do we minimally need to account for language structure? What do we minimally need to assume is dedicated to language? Behind these questions: –What kind of elements and what type of properties can be plausibly represented in the brain?

Time to step back and think How can linguistic knowledge be mentally represented? –Lexical items –Instructions for procedures Ullman (2001/2004): Contributions of Brain Memory Circuits to Language: The procedural/declarative model

The minimalist program Start out assuming: What has to be the case by conceptual necessity (and no more) Add no enrichment to the system unless empirically unavoidable But be as precise as possible

Consequences Treat with suspicion: Categories Labels Indices Traces

Lexicon Lexicon: Atomic form-meaning combinations (morphemes) Each morpheme contains -phonological information how to pronounce (phonology: Sound system will not be discussed here) - grammatical information category (Noun, Verb, Adj., Preposition, etc.), features (person, gender, number, Case, etc.) -semantic information concept, instructions for computation (quantifiers: every, a, some, etc.)

The basic combinatory process BINARY OPERATION Merge: a, b a b But: Not all combinations of morphemes are possible words *real, real-hood, real-able, real-ish,... *work, work-ish, work-hood,... *hold, holded, up-holded | hold, held, up-held *boy, boy-en | boy, boys Morphemes select what they combine with

Aside Fundamental question: How are word classes represented in the mental lexicon? Or: How are the zillions of Nouns all characterized as Nouns, Verbs as Verbs, etc.? Options: - categorial network - computationally: [ N n, ] (functional n-head, v-head, etc.) - intrinsically: inspect the concept (Vinokurova 2005)

Merge Binary merge asymmetry The effect of this asymmetry is pervasive in the computation of dependencies

Hierarchical relations: Tree structures A NA boy-ish N A AAN un- happy ness [ N [ A un A - [ A happy]] –ness N ]

Hierarchy and Asymmetry Asymmetries determine interpretation abcabc [top [squadron commander]] [[top squadron] commander] [California [history teacher]] [[California history] teacher]

Syntax: the computational system Basic operation: Merge a,b (= combine a,b) Each combination has a head represented in structure [ A grey] + [ N mice] [ NP [ A grey] + [ N mice] ] the N mice is the head of the Noun Phrase grey mice [ V feed] + [ NP [ A grey] [ N mice] ] [ VP [ V feed] [ NP [ A grey] [ N mice] ]] the V feed is the head of the Verb Phrase

An elementary Tree structure V(P) N(P) V AN feed grey mice

Basic clause structure 1 Three types of information 1.What happened to whom? Core Predicate -(I saw) Mary feed the cat 2.When did it happen? Tense/Mood -Mary will feed the cat 3.Force: Assertion, Question, Command -(I saw) that Mary fed the cat -(I wondered) if Mary fed the cat -(I wondered) who Mary fed

Basic clause structure 2 Force in root clauses Ø Mary will feed the cat Will Mary __ feed the cat Who will Mary __ feed __ Forming questions requires dislocation

Some more examples love V, Bonzo N VP love V Bonzo N John, [ VP love V Bonzo N ] VP V John N love V Bonzo N V indicates that the construction of the verb phrase continues Terms: Head, complement, specifier

Selection Selection restricts possible combinations Syntactic selection: D selects NP, T selects VP, C selects TP Semantic selection: John loves Mary ??The brick loves Mary John opened the lock/the key opened the lock ??Serenity opened the lock

Principles of structure building Working space: Lexicon, assembly line i) access (a head) a from the lexicon; ii) access b from the lexicon or assembly line that is selected for by a; iii) merge a and b as an a-category; (b is a complement of a) iv) put ab back on the assembly line; or: v) access c that is selected for by a vi) merge ab and c as an a-category; (c is a specifier of a) vii) put abc back, etc. : recursion

Limitations on operations Limited possibilities to disassemble what has been put together Locality of linguistic operations Cycles and Phases Surprise: Constructing a language works best bottom up

Basic pattern of linguistic structure: X' structure Lexical categoriesFunctional categories [ VP b[ V' V a]] [ TP b[ T' T a]] [ NP b[ N' N a]] [ DP b[ D' D a]] [ AP b[ A' A a]] [ CP b[ C' C a]] [ PP b[ P' P a]]etc. XP Specifier X' X 0 Complement

X-structure: a matter of convenience Lexical categoriesFunctional categories [ V b[ V V a]] [ T b[ T T a]] [ N b[ N N a]] [ D b[ D D a]] [ A b[ A A a]] [ C b[ C C a]] [ P b[ P P a]]etc. X Specifier X XComplement

X-structure: a matter of convenience Merge a, b {a, {a,b}} substitution Merge a,b { {a,b}} adjunction {X, {Y, {X, {X,W}}} Specifier:Y {X, {X,W}} XComp: W

Dependencies A fundamental property of all human languages: Dependency Relations. Local: Semantic roles, Case, agreement, category selection (functional-lexical: D-NP, T-VP) Non-local: dislocation, anaphors, pronominals Dependencies are always constrained must be obeyed in putting expressions together

Semantic roles 1 - agent, e.g. John in John hit the ball - instrument, e.g. a knife in John cut the salami with a knife - cause, e.g. the wind in the wind opened the door - experiencer, e.g. John in John worried about his health, - goal (sometimes benefactor) e.g. Mary, in John gave Mary a book, Boston, in John went to Boston - patient, e.g. the cat in John kicked the cat, or the couch in John moved the couch - source, e.g. the police, from prison in Max escaped the police /from prison Agents and experiencers are animate; Causes, instruments, etc. need not be.

Semantic roles 2 Reinhart (2002): The computational system can only see themativc information that has passed through a limited channel: Two binary formal features: [+/- m, +/- c] m: mental involvement c: causation

Local dependencies Selection: conceptual (semantic-roles) ??Sincerity admires John Subcategorization: formal/arbitrary John loves Mary| Jan houdt van Marie Case He saw her again| *her saw he again Agreement You love(*s) Fluffy| These/*this boys

Verbs show role-alternations: Passive John discovered *(Mary) Mary was discovered (by John) John fed the cat The cat was fed by John John gave (Mary) *(a book) Mary was given a book (by John) Systematic combination of three factors: i) the verb is in participial form ii) there is a form of to be as a passive auxiliary iii) the object shows up in subject position dislocation (a general phenomenon in language)

Putting expressions together (I saw) John feed Fluffy (bare VP) (I expect) John to feed Fluffy (to + VP but!! mismatch) John will feed Fluffy (T+VP, T takes over, but!! mismatch) John feeds Fluffy (T+VP, but !! mismatch) TP T' TVP will/toV' JohnV feedFluffy

Rearranging elements (I saw) [John [feed Fluffy]] (bare VP) [John [to [(John) feed Fluffy]]] (to+VP+rearrangement) [John [will [(John) feed Fluffy]]] ([ T will]+VP+rearrangement) [John [(-s) [ (John) feeds Fluffy]]] ([ T –s] +VP+ rearr.) TP JohnT' TVP to/will/-sV' (John)V feedFluffy

Dislocation 1 Dislocation: Mismatch between positions of interpretation and position of realization Metaphorical term: Movement Dislocation/Movement expresses Double Duty: Essence: One and the same element is active in two (or more) positions and realized in only one position.

Dislocation 2 The specifier of T must be filled: it will rain there arrived a man Dual use: re-use an element from the structure TP HeT' TVP willV' (he)V loveMary

Adding Force: CP 1 (I thought) [that [ TP John would love *(her)]] ( )CPdeclarative marker: that ----C' CTP thatT' JohnTVP wouldV' (John) love V her

Expressing questions: CP 2 (Mary wondered) [ CP if C [ TP John would love her]] ( )CPQuestion marker added ----C' CTP ifT' JohnTVP wouldV' (John) love V her

Expressing Questions: CP 3 (Mary wondered) [whom [ TP John would love]] ( )CP whomC' CTP -T' JohnTVP wouldV' (John) love V (whom)

How to express dislocation? (Mary wondered) [whom i [ TP John would love - ]] ( )CP whom i C' CTP -T' JohnTVP wouldV' (John) love V -

The canonical trace notation (Mary wondered) [whom i [ TP John would love t i ]] ( )CP whom i C' CTP -T' JohnTVP wouldV' (John) love V t i

The status of traces What do traces represent? What kind of elements are they? Are they needed? If so, why? Answer in Minimalist Program: Double duty can be expressed without an additional element in the theory Copies can do the same job Merge: Internal/External traces only for convenience

Questions in root clauses Whom did [John love t] CP whom h C' CTP did j John i T' TVP t j V' t i Vt h love

Clausal layers Predicational core: verb + arguments Tense/mood layer: coordinates for evaluation Force layer (C): assertion, question, command Movement enables one and the same element to be used in more than one layer Whom i did [John love t i ] Whom: argument of love in predicational core; signals question in Force domain Did: carrier Tense in Tense/mood layer; identifies C in Force domain

Dislocation: General Formally encoded by requirements of feature checking A head (such as C, T, etc.) looks down into the structure to which it has been attached, and probes for a goal - an element that carries a feature matching its requirements – and attracts it Empirical questions : –Is all dislocation triggered by probe-goal relations? Requirements of information structure may suffice. –Do all probe-goal relations result in dislocation? The existence of a probe-goal relation (=AGREE) is necessary, but not sufficient

Wh-movement: Question formation Instruction: Merge a question word (Wh-word) in the position of which you wish to elicit the value, and link it to the Force layer of the clause by moving it there A very similar operation works in relatives

Wh-movement as an interpretive dependency The interpreter must crucially know: i) a wh-element up front of the clause is part of the Force layer, and must therefore be interpreted as signalling a question; ii) a wh-element up front must be related to a gap (a trace, silent copy, etc.) and his computational system must be able to figure out where that gap is.

Some questions and relatives Wh-movement: Movement to a Force position (non-argument: no semantic role, no Case) Question formation and relativization I wonder [ CP which man i [ t i read the book]] I wonder [ CP which book i [the man read t i ]] Subject versus object relatives: I admired the man [ CP who i [t i wrote the book]] I admired the book [ CP that i [the man wrote t i ]]

Wh-movement: illustrations a.(John was wondering) whom he loved b.(John was wondering) [ --- [he loved whom] ] c.(John was wondering) [whom i [he loved t i ] Possible over an unbounded domain: Whom i did you say that Bill told Mary that he was willing to bet a million bucks that she never considered to promise Cindy she would leave t i alone?

Structure and processing What would you predict about the representation of –functional structure versus –core predication by subjects with reduced processing capacity? They will be selective: functional structure affected

Language comprehension (Cutler & Clifton)

Common denominator of processing models Modularity –language results from a number of specialized components responsible for different aspects of language representation/processing Major divisions –form (syntax) vs. meaning (semantics) vs. use (discourse) Hypothesis –Syntax, meaning, and use are subserved by different types of processes Investigation tool –Dissociability of processing mechanisms

Evidence: Neurolinguistics The study of brain – language relationships through neurological deficits Prime example: aphasia –A deficit in producing and understanding spoken and written language due to focal brain damage in persons who have gone through normal language development. (Prins & Bastiaanse 1997) –Incidence approx new cases per year in NL approx patients in NL Founding fathers of aphasiology: –Paul Broca –Carl Wernicke

Paul Broca ( ) 1861: Broca discovers in a post mortem study (Monsieur Tan) that speech/language (production) is associated with the foot of the 3 rd convolution of the frontal lobe –Brodmanns areas 44 & 45 today, we call this Brocas area

Mr. Tans brain

Carl Wernicke ( ) 1874: Wernicke discovers a second cortical area connected to language: the posterior part of the uppermost temporal gyrus (STG), right behind the primary auditory cortex –Brodmanns area 22 today: Wernickes area

Brodmanns areas Korbinian Brodmann ( )

Aphasia: syndromes

Wernicke-Lichtheim-Geschwindt motor images concepts word images

Agrammatism and the CP-layer Question production in agrammatism: The Tree pruning hypothesis, Naama Friedman, Tel Aviv University, Brain and Language 80, Patients: Hebrew versus English speakers with similar brain injuries Variable: Wh-questions versus Yes-no Questions English: both involve C Hebrew: only Wh-questions involve C

Observations and results Wh-questions (similar for Hebrew and Arabic) (1)H: Miri mecaryeret portret E: Miri draws a portrait (2)H: Ma i Miri mecayeret t i : Wh moves to C E: What (does) Miri paint: Wh moves to C Yes/no questions Usually differ from declarative sentences in intonation only no involvement of C (3)H: Miri mecaryeret portret E: Does Miri draw a portrait: T moves to C Result: In Hebrew only Wh-questions were affected; in English both were structure is reflected in pathology

Explanation Tree pruning hypothesis: The highest nodes of the syntactic tree are inaccessible in agrammatism: CP wh h C' CTP T j Miri i T' TVP t j V' t i Vt h draw VP DPV' V DP

Dependencies: Passive Movement into the subject position T' TVP wasV' V ' DP chased the mouse Passive morphology: -no semantic role to the subject -no case for the object - double use of the object requirement to move into the Tense system

Dependencies: Passive 2 Movement into the T-system: TP the mouse i T' TVP wasV' V ' DP chased t i Passive morphology to interpreter: -do not assign standard semantic role to the subject -look for a gap -assign to subject the role that otherwise would go to the gap

Reversible and non-reversible passives Non-reversible The apple was eaten by John Reversible The dog was chased by the cat The ability to process passive morphology is a prerequisite for the correct interpretation of reversible passives

Passives and the language system The prerequisite of being able to accurately process morphology is naturally satisfied in the mature and intact language system, but it need not be met in an immature or impaired system. Both for young children and for patients with certain types of language impairment this condition may not be met, and hence such speakers may have considerable problems with reversible passives.

Passives and agrammatism A Restrictive Theory of Agrammatic Comprehension, Yosef Grodzinsky, Tel Aviv University and Aphasia Research Center, Boston University School of Medicine, Brain and Language 50, 27-51

Observations (1)Above chance performance The girl pushed the boy (2)Chance performance The boy was pushed by the girl Hypothesis: syntactic movement yields problems More precisely: - traces are invisible to semantic role assignment What do subjects do?

Strategy The agent role is the most prominent role in a hierarchy of semantic roles Subjects use this for an auxiliary strategy: Assign the agent role to the leftmost NP of the clause as a default role

Result TP T' NP i TVP the boywas V'PP V pushed t i by the girl The default strategy + correctly interpreting the by- phrase ill-formed interpretation guessing

A below-chance performance Agrammatic role assignmentNormal assignm. a. The man i is pushing the woman | | agenttheme b. The woman i is pushed t i by the man | | *agent agent c. The man i is hated t i by the woman | | *agent experiencer c) yields below chance performance, since agent wins over experiencer

Reversibility in wh-movement The ball that [the boy is kicking t] is red The cat that [the dog is chasing t] is black The latter also presents problems for Brocas aphasics Again the trace deletion hypothesis (Grodzinsky et al.) can be adduced: Brocas aphasics have problems processing traces they use a default strategy to interpret sentences with traces

Some Caveats Lesion in Brocas area neither sufficient nor necessary to induce syntactic deficits: –Brocas area is not always lesioned in a clinically significant Brocas aphasia; –Brocas area can be affected in patients who do not display a Broca syndrome; most of these patients are mildly anomic. Severity of morphosyntactic problems in aphasia is correlated with the extent of damage in BA 22. Also semantic deficits in Brocas aphasia.

Disadvantages of patient studies damage to neural tissue may not be well delineated (in functional terms) –rather, depends on histological properties or on structure of vascular system possibility of compensation/adaptation aphasic symptoms evolve over time (post onset) unclear which symptoms (and hence: processes) are linked to which neural networks

Healthy subjects: Imaging studies PET fMRI

Is Brocas area the syntax center? Kaan & Swaab 2002, review (Trends in Cognitive Sciences) perception/comprehension studies; subtraction method paradigms: 1.complex sentences vs. simple sentences 2.sentences vs. word lists 3.jabberwocky/syntactic prose vs. word lists/normal sts 4.syntactic violations vs. correct sentences

4. Syntactic violations trees can grew vs trees can eat assumptions –more work in violation cases results –syntactic anomalies NO activation of Brocas area –sometimes: more superior frontal activity (also for semantic violations) –more frontal than temporal activation in with syntactic errors caveat –syntactic violations may have semantic consequences

Syntactic violations

Wheres syntax in the brain? Overview: Kaan & Swaab 2002, review (TICS) Brocas area –NOT necessarily involved in syntactic processing –more activation with more working memory demands other areas associated with syntax –anterior temporal lobe –anterior parts of BA 21, 22 –superior & middle temporal gyri –not only Left Hemisphere!

Brocas area … … is an excitable piece of tissue! (David Poeppel, p.c.) activated by (i.a.) –word/syllable lists (memory) –semantic tasks –phonological tasks –music perception

Suggestions (K&S 2002) Middle/superior temporal lobe –lexical processing (activating semantic/syntactic, phonological features of words) Anterior temporal lobe –combining activated information Brocas area –storing non-integrated materials Right hemisphere –prosody –ambiguity –discourse –error detection

What is syntactic processing? structure building –grouping words into phrases, phrases into sentences determining dependencies –what goes with what?

dislocation/movement In structural terms: –an element is doing double duty by having two copies in the structure – only one of these is spelled out phonetically (ie., has an audible form) in processing terms: –the processor has to recognize an empty spot at the location of the object NP and infer which noun phrase can be connected to it.

Electro-magnetic signals EEG/ERP (MEG/ERF) 1.Different signatures for syntactic and semantic processing? 2.Autonomy of syntactic processing?

EEG

ERP (Bressler 2002) The physiological basis of the cortical ERP: Fields of potential generated by interacting neurons. Field potentials result from the summed extracellular currents generated by electromotive forces (EMFs) in the dendrites of synchronously active cortical neurons. The EMFs, arising from synaptic activation of postsynaptic ion channels, circulate current in closed loops across the cell membrane and through the intracellular and extracellular spaces. Summed closed-loop currents generated by an ensemble of neighboring neurons flow across the external resistance to form the local ensemble mean field potential. The event-related potential (ERP): Neural signal that reflects coordinated neural network activity. The cortical ERP provides a window onto the dynamics of network activity in relation to a variety of different cognitive processes at both mesoscopic and macroscopic levels on a time scale comparable to that of single-neuron activity. Good: Temporal resolution Bad: Spatial resolution

Event-Related Potentials ERP

ERP & language: N400 Kutas & Hillyard 1980

N400 negative(-going) component peak latency around 400ms bi-lateral slightly posterior distribution N400 effect (amplitude modulation): –(mis)match of word meaning with preceding context –semantic priming

P600/SPS Osterhout & Holcomb 1993; Hagoort, Brown & Groothusen 1993

P600 positive(-going) deflection peak latency around 600ms bilateral, centro-parietal distribution grammatical anomalies ambiguities that are resolved in a dispreferred way long-distance dependencies

ERP: Early Left-Anterior Negativity add picture

ELAN negative(-going) deflection peak latency around 200ms left anterior distribution grammatical violations, e.g. –phrase structure –inflection, function words

lexical vs. functional categories lexical: N, V, A, (P) functional: Infl, Det, Comp agrammatism child language ERP (ter Keurs et al.)

lexical vs. functional categories negative peak earlier for functional categories

What do ERPs signify? physiologically: synchronous post-synaptic activation of several hundreds of thousands of radially oriented pyramidal cells functionally: No idea! –the brain (groups of neurons) responds in a particular, consistent way to particular stimuli

Interim conclusion wrt ERPs N400, P600 and (E)LAN differ in –polarity –latency –distribution –eliciting conditions P600, (E)LAN – syntactic problems N400 – semantic problems different generators, i.e., different neural processors dealing with syntax and semantics.

The big picture? (Friederici et al)

Relating Neurocognition and Linguistic Architecture A neuro-cognitive contrast between linguistic computational mechanisms and the lexicon ((Ullman 2001): the computational system: procedural memory the lexicon: declarative memory

Linguistic Theory and Neural Activity What can we expect? Very abstractly: Match between properties of derivations and processes in production or comprehension? Match at the architectural level - differences between modules involved in a mental computation reflect differences in neural activity