Incremental probabilistic models of human linguistic performance
Cascaded Markov Models

Matthew Crocker, Thorsten Brants

Universität des Saarlandes
Computational Linguistics
P.O.Box 151150, D-66041 Saarbrücken, Germany

In Proceedings of AMLAP-99
September 23 - 25, 1999, Edinburgh


Models of human language processing increasingly advocate probabilistic mechanisms for parsing and disambiguation (e.g. Jurafsky, 1996; MacDonald et al 1994; Crocker and Corley; to appear). These models resolve local syntactic and lexical ambiguity by promoting the analysis which has the greatest probability of being correct. In this talk we will outline a new probabilistic parsing model which is a generalisation of the Hidden Markov Models which have previously been defended as pschological models of lexical category disambiguation (Corley and Crocker, in press). The model uses layered, or cascaded, markov models (CMMs) to build up a syntactic analysis (Brants, 1999).

In contrast with many probabilisic parsing models, CMMs can easily be implemented to parse incrementally. Incremental CMMs have the property of generating partial structures including hypothetical continuations after receiving each new word in the input. New material is incorporated into the existing structure and ambiguities are resolved based on local context. Alternative hypotheses are assigned probabilities which are used for ranking, and only a bounded number of parallel alternatives are pursued. Simple bounds on the model straightforwardly predict the recency effects often attributed only to connectionist-based models (Stevenson, 1994; Macdonald et al, 1884; Kempen and Vosse, 1987).

In contrast with several current models, the combination of weights in CMMs is motivated directly by probability theory. The parameters of the model are acquired automatically from a corpus, and there are relatively few stipulations about how probabilities are combined (contra Jurafsky, 1996; Tanenhaus et al, in press). An important cognitive parameter concerns the number of analyses which are maintained in parallel. We will present results of experiments which evaluate the performance of the model for both general language processing, and on several critical ambiguities where human performance is well understood.

The model is a first step in exploring the role of optimal models of human linguistic performance, as motivated by Chater, Crocker, and Pickering (1998). Recently, Pickering, Traxler and Crocker (to appear) have provided experimental evidence which challenges a pure maximum likelihood model of syntactic ambiguity resolution. As an alternative, they propose a measure, termed Informativity, which they derive from a rational analysis of the parsing and interpretation problem. In the final part of the talk we will outline how the presented model can be adapted to implement Informativity, which combines probability with a newly proposed measure termed Specificity.


Brants, T. (1999). Cascaded Markov Models. In: Proceedings of 9th Conference of the European Chapter of the Association for Computational Linguistics (EACL-99), Bergen, Norway.

Chater, N., Crocker, M. & Pickering, M. (1998). The Rational Analysis of Inquiry: The Case for Parsing. In: Chater & Oaksford (eds),Rational Models of Cognition, pp. 441-468, Oxford University Press, Oxford, UK.

Corley, S. & Crocker, M.W. (in press). The Modular Statistical Hypothesis: Exploring Lexical Category Ambiguity. In: Crocker, Pickering & Clifton (eds), Architectures and Mechanisms for Language Processing, CUP, England.

Crocker, M. & Corley. S. (to appear). Modular Architectures and Statistical Mechanisms: The Case from Lexical Category Disambiguation. In: Merlo & Stevenson (eds), The Lexical Basis of Sentence Processing, John Benjamins, Amsterdam.

Jurafsky, D.A (1996). Probabilistic Model of Lexical and Syntactic Access and Disambiguation, Cognitive Science, 20, 137-194.

MacDonald, M.C., Pearlmutter, N.J., & Seidenberg, M.S. (1994). The lexical nature of syntactic ambiguity resolution. Psychological Review, 10(4), 676-703.

Pickering, M., Traxler, M. & Crocker, M. (submittd). Ambiguity Resolution in Sentence Processing: Evidence Against Likelihood.

Tanenhaus, M.K., Spivey-Knowlton, M.J., & Hanna, J.E. (in press). Modelling Discourse Context Effects: A Multiple Constraints Approach. In Crocker, Pickering & Clifton (eds.) Architectures and Mechanisms for Language Processing, Cambridge University Press, Cambridge, UK.