Ulf Hermjakob, Abdessamad Echihabi, Daniel Marcu. 2002.
Natural Language Based Reformulation Resource and Web Exploitation for Question Answering. Proceedings of the TREC-2002 Conference. Get paper in pdf. Get paper in gzipped PostScript.

We describe and evaluate how a generalized natural language based reformulation resource in our TextMap question answering system improves web exploitation and answer pinpointing. The reformulation resource, which can be viewed as a clausal extension of WordNet, supports high-precision syntactic and semantic reformulations of questions and other sentences, as well as inferencing and answer generation. The paper shows in some detail how these reformulations can be used to overcome challenges and benefit from the advantages of using the Web.

Ulf Hermjakob, Eduard Hovy and Chin-Yew Lin. 2002.
Automated Question Answering in Webclopedia - A Demonstration. Proceedings of the ACL 2002 Conference in Philadelphia, demo section. Get paper in pdf. Get paper in gzipped PostScript.

In this demonstration we present Webclopedia, a semantics-based question answering system accessible via the web (Hovy et al. 2002, 2001, 2000). Through a live interface, users can type in their questions or select a predefined question. The system returns its top 5 candidate answers, drawn from NIST's TREC corpus, a collection of 1 million newspaper texts.

U. Hermjakob. 2001.
Parsing and Question Classification for Question Answering. Proceedings of the Workshop on Open-Domain Question Answering at the ACL-01 Conference. Toulouse, France. Get paper in pdf. Get paper in gzipped PostScript.

This paper describes machine learning based parsing and question classification for question answering. We demonstrate that for this type of application, parse trees have to be semantically richer and structurally more oriented towards semantics than what most treebanks offer. We empirically show how question parsing dramatically improves when augmenting a semantically enriched Penn treebank training corpus with an additional question treebank.

E.H. Hovy, L. Gerber, U. Hermjakob, C.-Y. Lin, D. Ravichandran. 2001.
Toward Semantics-Based Answer Pinpointing. Proceedings of the DARPA Human Language Technology conference (HLT). San Diego, CA. Get paper in pdf.

This paper describes the overall structure of Webclopedia, and its use of the CONTEX parser to provide analyses of the question and candidate answers, which are then compared for answer pinpointing.

Hovy, E.H., A. Philpot, J.-L. Ambite, Y. Arens, J.L. Klavans, W. Bourne, and D. Saroz. 2001.
Data Acquisition and Integration in the DGRC's Energy Data Collection Project. Proceedings of the dg.o 2001 Conference. Los Angeles, CA. Get paper in pdf.

The EDC project is developing new methods to make data that has been represented in disparate ways and stored in heterogeneous forms available to users in an integrated, manageable, and understandable way. Our approach is to represent the structure and types of data in the disparate collections in a standard format (called a domain model) and then to embed the domain model(s) into a large overarching taxonomy of terms (the ontology). Once thus standardized, the data collections can be found and retrieved via a single interface and access planning system. A major bottleneck, however, is the rapid inclusion of new data collections into this system. This paper describes some recent research in developing methods to automate the acquisition and inclusion processes.

E. Filatova and E.H. Hovy. 2001.
Assigning Time-Stamps to Event-Clauses. Proceedings of the Workshop on Temporal and Spatial Reasoning at the ACL-01 Conference. Toulouse, France. Get paper in pdf.

We describe a procedure for arranging into a time-line the contents of news stories describing the development of some situation. We describe the parts of the system that deal with 1. breaking sentences into event-clauses and 2. resolving both explicit and implicit temporal references.

Yamada, K. and Knight, K. 2001
A Syntax-Based Statistical Translation Model. Proceedings of ACL-01. Toulouse, France. Get paper in PostScript.

We present a syntax-based statistical translation model. Our model transforms a source-language parse tree into a target-language string by applying stochastic operations at each node. These operations capture linguistic differences such as word order and case marking. Model parameters are estimated in polynomial time using an EM algorithm. The model produces word alignments that are better than those produced by IBM Model 5.

Germann, U., Jahr, M., Knight, K., Marcu, D., and Yamada, K. 2001
Fast Decoding and Optimal Decoding for Machine Translation. Proceedings of ACL-01. Toulouse, France. Get paper in PDF.

A good decoding algorithm is critical to the success of any statistical machine translation system. The decoder's job is to find the translation that is most likely according to set of previously learned parameters. Since the space of possible translations is extremely large, typical decoding algorithms are only able to examine a portion of it, thus risking missing good solutions. In this paper, we compare the speed and output quality of a traditional stack-based decoding algorithm with two new decoders: a fast greedy decoder and a slow but optimal decoder that treats decoding as an integer-programming optimization problem.

Marcu, D. 2001
Towards a Unified Approach to Memory- and Statistical-Based Machine Translation. Proceedings of ACL-01. Toulouse, France. Get paper in PDF.

We present a set of algorithms that enable us to translate natural language sentences by exploiting both a translation memory and a statistical-based translation model. Our results show that an automatically derived translation memory can be used within a statistical framework to often find translations of higher probability than those found using solely a statistical model. The translations produced using both the translation memory and the statistical model are significantly better than translations produced by two commercial systems: our hybrid system translated perfectly 58% of the 505 sentences in a test collection, while the commercial systems translated perfectly only 40-42% of them.

Germann, U. 2001
Building a Statistical Machine Translation System from Scratch: How Much Bang Can We Expect for the Buck. Proceedings of the Data-Driven MT Workshop of ACL-01. Toulouse, France. Get paper in PDF.

We report on our experience with building a statistical MT system from scratch, including the creation of a small parallel Tamil-English corpus, and the results of a task-based pilot evaluation of statistical MT systems trained on sets of ca. 1300 and ca. 5000 parallel sentences of Tamil and English data. Our results show that even with apparently incomprehensible system output, humans without any knowledge of Tamil can achieve performance rates as high as 86% accuracy for topic identification, 93% recall for document retrieval, and 64% recall on question answering (plus an additional 14% partially correct answers).

Koehn, P. and Knight, K. 2001
Knowledge Sources for Word-Level Translation Models. Empirical Methods in Natural Language Processing conference (EMNLP'01). Get paper in PostScript. Get paper in PDF.

We present various methods to train word-level translation models for statistical machine translation systems that use widely different knowledge sources ranging from parallel corpora and a bilingual lexicon to only monolingual corpora in two languages. Some novel methods are presented and previously published methods are reviewed. Also, a common evaluation metric enables the first quantitative comparison of these approaches.

Ambite, J.-L., Y. Arens, E.H. Hovy, A. Philpot, L. Gravano, V. Hatzivassiloglou, J.L. Klavans. 2001.
Simplifying Data Access: The Energy Data Collection Project. IEEE Computer 34(2), February. Get paper in pdf.

The massive amount of statistical and text data available from government agencies has created a set of daunting challenges to both research and analysis communities. These problems include heterogeneity, size, distribution, and control of terminology. At the Digital Government Research Center we are investigating solutions to these key problems. In this paper we focus on (1) ontological mappings for terminology standardization, (2) data integration across data bases with high speed query processing, and (3) interfaces for query input and presentation of results. This collaboration between researchers from Columbia University and the Information Sciences Institute of the University of Southern California employs technology developed at both locations, in particular the SENSUS ontology, the SIMS multi-database access planner, the LKB automated dictionary and terminology analysis system, and others. The pilot application targets gasoline data from the Bureau of Labor Statistics, the Energy Information Administration of the Department of Energy, the Census Bureau, and other government agencies.

Hermjakob, U. 2000.
Rapid Parser Development: A Machine Learning Approach for Korean. Association for Computational Linguistics conference, North American chapter (NAACL'00). Get paper in pdf. Get paper in gzipped PostScript.

This paper demonstrates that machine learning is a suitable approach for rapid parser development. From 1000 newly treebanked Korean sentences we generate a deterministic shift-reduce parser. The quality of the treebank, particularly crucial given its small size, is supported by a consistency checker.

E.H. Hovy, L. Gerber, U. Hermjakob, M. Junk, C.-Y. Lin. 2000.
Question Answering in Webclopedia. Proceedings of the TREC-9 conference. NIST, Gaithersburg, MD. Get paper in pdf.

The paper provides a high-level overview of Webclopedia, focusing on the performance of the information retrieval and text segmentation components, with a description of the performance in the TREC-9 QA competition.

Langkilde, I. 2000.
Forest-Based Statistical Sentence Generation. Association for Computational Linguistics conference, North American chapter (NAACL'00). Get paper in PostScript.

This paper presents a new approach to statistical sentence generation in which alternative phrases are represented as packed sets of trees, or forests, and then ranked statistically to choose the best one. This representation offers advantages in compactness and in the ability to represent syntactic information. It also facilitates more efficient statistical ranking than a previous approach to statistical generation. An efficient ranking algorithm is described, together with experimental results showing significant improvements over simple enumeration or a lattice-based approach.

Walker, M., Langkilde, I., Wright, J., Gorin, A., Litman, D. 2000.
Learning to Recognize Probabilistic Situations in a Spoken Dialogue System. Association for Computational Linguistics conference, North American chapter (NAACL'00).

Marcu, D., Carlson, L., Watanabe, M. 2000.
The Automatic Translation of Discourse Structures. Association for Computational Linguistics conference, North American chapter (NAACL'00).

Al-Onaizan, Y., Germann, U., Hermjakob, U., Knight, K., Koehn, K., Marcu, D., Yamada, K. 2000.
Translating with Scarce Resources. American Association for Artificial Intelligence conference (AAAI'00). Get paper in PostScript.

Current corpus-based machine translation techniques do not work very well when given scarce linguistic resources. To examine the gap between human and machine translators, we created an experiment in which human beings were asked to translate an unknown language into English on the sole basis of a very small bilingual text. Participants performed quite well, and debriefings revealed a number of valuable strategies. We discuss these strategies and apply some of them to a statistical translation system.

Knight, K. and Langkilde, I. 2000.
Preserving Ambiguities in Generation via Automata Intersection. American Association for Artificial Intelligence conference (AAAI'00). Get paper in PostScript.

We discuss the problem of generating text that preserves certain ambiguities, a capability that is useful in applications such as machine translation. We show that it is relatively simple to extend a hybrid symbolic/statistical generator to do ambiguity preservation. The paper gives algorithms and examples, and it discusses practical linguistic difficulties that arise in ambiguity preservation.

Koehn, P., and Knight, K. 2000.
Estimating Word Translation Probabilities from Unrelated Monolingual Corpora Using the EM Algorithm. American Association for Artificial Intelligence conference (AAAI'00).

Selecting the right word translation among several options in the lexicon is a core problem for machine translation. We present a novel approach to this problem that can be trained using only unrelated monolingual corpora and a lexicon. By estimating word translation probabilities using the EM algorithm, we extend upon target language modeling. We construct a word translation model for 3830 German and 6147 English noun tokens, with very promising results.

Knight, K. and Marcu, D. 2000.
Statistics-Based Summarization -- Step One: Sentence Compression. American Association for Artificial Intelligence conference (AAAI'00). Get paper in PostScript.

When human beings produce summaries of documents, they do not simply extract sentences and concatenate them. Rather, they create new sentences that are grammatical, that cohere with one another, and that capture the most salient pieces of information in the original document. Given that large collections of text/abstract pairs are available online, it is now possible to envision algorithms that are trained to mimic this process. In this paper, we focus on sentence compression, a simpler version of this larger challenge. We aim to achieve two goals simultaneously: our compressions should be grammatical, and they should retain the most important pieces of information. These two goals can conflict. We devise both noisy-channel and decision-tree approaches to the problem, and we evaluate results against manual compressions and a simple baseline.

Knight, K. 1999.
Mining Online Text. Communications of the ACM, 42(11).

Al-Onaizan, Y., Curin, J., Jahr, M., Knight, K., Lafferty, J., Melamed, D., Och, F.-J., Purdy, D., Smith, N. A., and Yarowsky, D. 1999.
Statistical Machine Translation, Final Report, JHU Workshop 1999. Technical Report, CLSP/JHU. Get paper in postscript. See also workshop site.

Knight, K. 1999.
Decoding Complexity in Word-Replacement Translation Models. Computational Linguistics, 25(4).

Statistical machine translation is a relatively new approach to the longstanding problem of translating human languages by computer. Current statistical techniques uncover translation rules from bilingual training texts and use those rules to translate new texts. The general architecture is the source-channel model: an English string is statistically generated (source), then statistically transformed into French (channel). In order to translate (or "decode") a French string, we look for the most likely English source. We show that for the simplest form of statistical models, this problem is NP-complete, i.e., probably exponential in the length of the observed sentence. We trace this complexity to factors not present in other decoding problems.

Knight, K. 1999.
A Statistical MT Tutorial Workbook. Ms, August 1999. Get paper in Word.

Knight, K. and K. Yamada. 1999.
A Computational Approach to Deciphering Unknown Scripts. Proceedings of the ACL Workshop on Unsupervised Learning in Natural Language Processing. Get paper in PostScript.

We propose and evaluate computational techniques for deciphering unknown scripts. We focus on the case in which an unfamiliar script encodes a known language. The decipherment of a brief document or inscription is driven by data about the spoken language. We consider which scripts are easy or hard to decipher, how much data is required, and whether the techniques are robust against language change over time.

Germann, U. 1998.
Making Semantic Interpretation Parser-Independent. Proceedings of the 4th AMTA Conference.

We present an approach to semantic interpretation of syntactially parsed Japanese sentences that works largely parser-indepedent. The approach relies on a standardized parse tree format that restricts the number of syntactic configurations that the semantic interpretation rules have to anticipate. All parse trees are converted to this format to semantic intpretation. This setup allows us not only to apply the same set of semantic interpretation rules from different parsers, but also to independently develop parsers and semantic interpretation rules.

Knight, K. and Al-Onaizan, Y. 1998.
Translation with Finite-State Devices. Proceedings of the 4th AMTA Conference.

Statistical models have recently been applied to machine translation with interesting results. Algorithms for processing these models have not received wide circulation, however. By contrast, general finite-state transduction algorithms have been applied in a variety of tasks. This paper gives a finite-state reconstruction of statistical translation and demonstrates the use of standard tools to compute statistically likely translations. Ours is the first translation algorithm for "fertility/permutation" statistical models to be described in replicable detail.

Stalls, B. and Knight, K. 1998.
Translating Names and Technical Terms in Arabic Text. COLING/ACL Workshop on Computational Approaches to Semitic Languages. Montreal, Quebéc.
Get paper in PostScript.

It is challenging to translate names and technical terms from English into Arabic. Translation is usually done phonetically: different alphabets and sound inventories force various compromises. For example, Peter Streams may come out as "bytr strymz". This process is called transliteration. We address here the reverse problem: given a foreign name or loanword in Arabic text, we want to recover the original in Roman script. For example, an input like "bytr strymz" should yield an output like Peter Streams. Arabic presents special challenges due to unwritten vowels and phonetic-context effects. We present results and examples of use in an Arabic-to-English machine translator.

Germann, U. 1998.
Visualization of Protocols of the Parsing and Semantic Interpretation Steps in a Machine Translation System. COLING-ACL Workshop on Content Visualization and Intermedia Representations. Montreal, Quebéc.
Get paper in PostScript.

In this paper, we describe a tool for the visualization of process protocols produced by the parsing and semantic interpretation modules in a complex machine translation system. These protocols tend to reach considerable sizes, and error tracking in them is tedious and time-consuming. We show how the data in the protocols can be made more easily accessible by extracting a procedural trace, by splitting the protocols into a collection of cross-linked hypertext files, by indexing the files, and by using simple text formatting and sorting of structural elements.

Langkilde, I. and Knight, K. 1998.
The Practical Value of N-Grams in Generation. Proceedings of the International Natural Language Generation Workshop. Niagra-on-the-Lake, Ontario.
Get paper in PostScript.

We examine the practical synergy between symbolic and statistical language processing in a generator called Nitrogen. The analysis provides insight into the kinds of linguistic decisions that bigram frequency statistics can make, and how it improves scalability. We also discuss the limits of bigram statistical knowledge. We focus on specific examples of Nitrogen's output.

Langkilde, I. and Knight, K. 1998.
Generation that Exploits Corpus-based Statistical Knowledge. Proceedings of the ACL/COLING-98. Montreal, Quebéc.
Get paper in PostScript.

We describe novel aspects of a new natural language generator called Nitrogen. This generator has a highly flexible input representation that allows a spectrum of input from syntactic to semantic depth, and shifts the burden of many linguistic decisions to the statistical post-processor. The generation algorithm is compositional, making it efficient, yet it also handles non-compositional aspects of language. Nitrogen's design makes it robust and scalable, operating with lexicons and knowledge bases of one hundred thousand entities.

Knight, K. 1997.
Automating Knowledge Acquisition for Machine Translation. AI Magazine 18(4).
Get paper in PostScript.

This article surveys some of the recent literature in corpus-based approaches to machine translation.

Knight, K. and J. Graehl. 1997.
Machine Transliteration. Proceedings of the ACL-97. Madrid, Spain.
Get paper in PostScript.

It is challenging to translate names and technical terms across languages with different alphabets and sound inventories. These items are commonly transliterated, i.e., replaced with approximate phonetic equivalents. For example, "computer" in English comes out as "konpyuutaa" in Japanese. Translating such items from Japanese back to English is even more challenging, and of practical interest, since transliterated items make up the bulk of text phrases not found in bilingual dictionaries. We describe and evaluate a method for performing backwards transliterations by machine. This method uses a generative model, incorporating several distinct stages in the transliteration process.

Yamada, K. 1996.
A Controlled Skip Parser. Proceedings of the 2nd AMTA Conference. Montreal, Quebéc.
Get paper in PostScript.

Real-world natural language sentences are long and complex, and always contain unexpected grammatical constructions. It even includes noise and ungrammaticality. This paper describes the Controlled Skip Parser, a program that parses such real-world sentences by skipping some of the words in the sentence. The new feature of this parser is that it can control its behavior to find out which words to skip, without using domain-specific knowledge. Statistical information (N-grams), which is a generalized approximation of the grammar learned from past successful experiences, is used for the controlled skip. Experiments on real newspaper articles are shown, and our experience with this parser in a machine translation system is described.

Knight, K. 1996.
Learning Word Meanings by Instruction. Proceedings of the American Association of Artificial Intelligence AAAI-96. Portland, OR.
Get paper in PostScript.

We develop techniques for learning the meanings of unknown words in context. Working within a compositional semantics framework, we write down equations in which a sentence's meaning is some combination function of the meaning of its words. When one of the words is unknown, we ask for a paraphrase of the sentence. We then compute the meaning of the unknown word by inverting parts of the semantic combination function. This technique can be used to learn word-concept mappings, decomposed meanings, and mappings between syntactic and semantic roles. It works for all parts of speech.

Knight, K., I. Chander, M. Haines, V. Hatzivassiloglou, E.H. Hovy, M. Iida, S.K. Luk, R.A. Whitney, and K. Yamada. 1995
Filling Knowledge Gaps in a Broad-Coverage MT System. Proceedings of the 14th IJCAI Conference. Montreal, Quebéc.
Get paper in PostScript.

Knowledge-based machine translation (KBMT) techniques yield high quality in domains with detailed semantic models, limited vocabulary, and controlled input grammar. Scaling up along these dimensions means acquiring large knowledge resources. It also means behaving reasonably when definitive knowledge is not yet available. This paper describes how we can fill various KBMT knowledge gaps, often using robust statistical techniques. We describe quantitative and qualitative results from JAPANGLOSS, a broad-coverage Japanese-English MT system.

Knight, K. and V. Hatzivassiloglou. 1995.
Two-Level, Many-Paths Generation. Proceedings of the ACL-95. Cambridge, MA.
Get paper in PostScript.

Large-scale natural language generation requires the integration of vast amounts of knowledge: lexical, grammatical, and conceptual. A robust generator must be able to operate well even when pieces of knowledge are missing. It must also be robust against incomplete or inaccurate inputs. To attack these problems, we have built a hybrid generator, in which gaps in symbolic knowledge are filled by statistical methods. We describe algorithms and show experimental results. We also discuss how the hybrid generation model can be used to simplify current generators and enhance their portability, even when perfect knowledge is in principle obtainable.

Hatzivassiloglou, V. and K. Knight. 1995.
Unification-Based Glossing. Proceedings of the 14th IJCAI Conference. Montreal, Quebéc.
Get paper in compressed PostScript.

We present an approach to syntax-based machine translation that combines unification-style interpretation with statistical processing. This approach enables us to translate any Japanese newspaper article into English, with quality far better than a word-for-word translation. Novel ideas include the use of feature structures to encode word lattices and the use of unification to compose and manipulate lattices. Unification also allows us to specify abstract features that delay target-language synthesis until enough source-language information is assembled. Our statistical component enables us to search efficiently among competing translations and locate those with high English fluency.

Knight, K., I. Chander, M. Haines, V. Hatzivassiloglou, E.H. Hovy, M. Iida, S.K. Luk, A. Okumura, R.A. Whitney, and K. Yamada. 1994.
Integrating Knowledge Bases and Statistics in MT. Proceedings of the 1st AMTA Conference. Columbia, MD.
Get paper in PostScript.

We summarize recent machine translation (MT) research at the Information Sciences Institute of USC, and we describe its application to the development of a Japanese-English newspaper MT system. Our work aims at scaling up grammar-based, knowledge-based MT techniques. This scale-up involves the use of statistical methods, both in acquiring effective knowledge resources and in making reasonable linguistic choices in the face of knowledge gaps.

Knight, K. and S. Luk. 1994.
Building a Large-Scale Knowledge Base for Machine Translation. Proceedings of the American Association of Artificial Intelligence AAAI-94. Seattle, WA.
Get paper in PostScript.

Knowledge-based machine translation (KBMT) systems have achieved excellent results in constrained domains, but have not yet scaled up to newspaper text. The reason is that knowledge resources (lexicons, grammar rules, world models) must be painstakingly handcrafted from scratch. One of the hypotheses being tested in the PANGLOSS machine translation project is whether or not these resources can be semi-automatically acquired on a very large scale. This paper focuses on the construction of a large ontology (or knowledge base, or world model) for supporting KBMT. It contains representations for some 70,000 commonly encountered objects, processes, qualities, and relations. The ontology was constructed by merging various online dictionaries, semantic networks, and bilingual resources, through semi-automatic methods. Some of these methods (e.g., conceptual matching of semantic taxonomies) are broadly applicable to problems of importing/exporting knowledge from one KB to another. Other methods (e.g., bilingual matching) allow a knowledge engineer to build up an index to a KB in a second language, such as Spanish or Japanese.

Knight, K. and I. Chander. 1994.
Automated Postediting of Documents. Proceedings of the American Association of Artificial Intelligence AAAI-94. Seattle, WA.
Get paper in PostScript.

Large amounts of low- to medium-quality English texts are now being produced by machine translation (MT) systems, optical character readers (OCR), and non-native speakers of English. Most of this text must be postedited by hand before it sees the light of day. Improving text quality is tedious work, but its automation has not received much research attention. Anyone who has postedited a technical report or thesis written by a non-native speaker of English knows the potential of an automated postediting system. For the case of MT-generated text, we argue for the construction of postediting modules that are portable across MT systems, as an alternative to hardcoding improvements inside any one system. As an example, we have built a complete self-contained postediting module for the task of article selection (a, an, the) for English noun phrases. This is a notoriously difficult problem for Japanese-English MT. Our system contains over 200,000 rules derived automatically from online text resources. We report on learning algorithms, accuracy, and comparisons with human performance.

Okumura, A. and E.H. Hovy. 1994.
Lexicon-to-Ontology Concept Association Using a Bilingual Dictionary. Proceedings of the 1st AMTA Conference. Columbia, MD.
Get paper in PostScript.

This paper describes a semi-automatic method for associating a Japanese lexicon with a semantic concept taxonomy called an ontology, using a Japanese-English bilingual dictionary as a "bridge". The ontology supports semantic processing in a knowledge-based machine translation system by providing a set of language-neutral symbols with semantic information. To put the ontology to use, lexical items of each language of interest must be linked to appropriate ontology items. The association of ontology items with lexical items of various languages is a process fraught with difficulty: since much of this work depends on the subjective decisions of human workers, large MT dictionaries tend to be subject to some dispersion and inconsistency. The problem we focus on here is how to associate concepts in the ontology with Japanese lexical entities by automatic methods, since it is too difficult to define adequately many concepts manually. We have designed three algorithms to associate a Japanese lexicon with the concepts of the ontology: the equivalent-word match, the argument match, and the example match.

Arens, Y. and E.H. Hovy. 1995.
The Design of a Model-Based Multimedia Interaction Manager. AI Review 8(3) Special Issue on Natural Language and Vision.
Get paper in PostScript.

We describe the conceptual design of Cicero, an application-independent human-computer interaction manager that performs run-time media coordination and allocation, so as to adapt dynamically to the presentation context; knows what it is presenting, so as to maintain coherent extended human-machine dialogues; and is plug-in compatible with host information resources such as "briefing associate" workstations, expert systems, databases, etc., as well as with multiple media such as natural language, graphics, etc. The system's design calls for two linked mutually activating planners that coordinate the actions of the system's media and information sources. To enable presentational flexibility, the capabilities of each medium and the nature of the contents of each information source are semantically modeled as Virtual Devices -- abstract representations of device I/O capabilities -- and abstract information types respectively in a single uniform knowledge representation framework. These models facilitate extensibility by supporting the specification of new interaction behaviors and the inclusion of new media and information sources.

Arens, Y., E.H. Hovy, and S. Van Mulken. 1993a.
Structure and Rules in Automated Multimedia Presentation Planning. Proceedings of the International Joint Conference on Artificial Intelligence IJCAI-93. Chambéry, France.
Get paper in PostScript.

During the planning of multimedia presentations, at least two distinct processes are required: planning the underlying discourse structure (that is, ordering and interrelating the information to be presented) and allocating the media (that is, delimiting the portions to be displayed by each individual medium). The former process has been the topic of several studies in the area of text planning, but numerous questions remain for the latter, including: What is the nature of the allocation process -- what does it start with and what does it produce? What information does it depend on? How should the two processes be performed -- sequentially, interleaved, or simultaneously? In this paper, we define Discourse Structure and Presentation Structure and outline the kinds of information that media allocation rules must depend on, including, centrally, information about the discourse structure. We describe a prototype planning system that performs the information-to-media allocation, arguing that since media allocation rules depend on the characteristics of the information to be presented, they can only be applied once the overall discourse structure has been essentially planned out and the individual portions of information have become apparent.

Arens, Y., E.H. Hovy, and M. Vossers. 1993b.
Describing the Presentational Knowledge Underlying Multimedia Instruction Manuals. In M. Maybury (ed), Intelligent Multimedia Interfaces. Cambridge: MIT Press.

Get paper in PostScript.

We address one of the problems at the heart of automated multimedia presentation production and interpretation. The media allocation problem can be stated as follows: how does the producer of a presentation determine which information to allocate to which medium, and how does a perceiver recognize the function of each part as displayed in the presentation and integrate them into a coherent whole? What knowledge is used, and what processes? We describe the four major types of knowledge that play a role in the allocation problem as well as interdependencies that hold among them. We discuss two formalisms that can be used to represent this knowledge and, using examples, describe the kinds of processing required for the media allocation problem.

Arens, Y. and E.H. Hovy. 1993c.
The Planning Paradigm Required for Automated Multimedia Presentation Planning. Presented at the AAAI Fall Symposium on Human-Computer Interfaces, Raleigh, NC.

Get paper in PostScript.

In this paper we argue that the planning of multimedia presentations requires at least two distinct (though interrelated) independent reactive planning processes: one to plan the underlying discourse structure (that is, to order and interrelate the information to be presented) and the other to allocate the media (that is, to delimit the portions to be displayed by each individual medium). The former process has been the topic of several studies in the area of automated text planning, in which the traditional methods of constructing tree-like plans in deliberative, top-down, planning mode have been applied with varying amounts of success. The latter process remains less clear, in part (we believe) because the deliberative planning mode is even less appropriate for it. We outline in this paper the reasoning behind our belief that neither planning process can be a simple deliberative top-down one and describe the kind of interplay between the two processes.

NLG overview | Project Members | Projects| Demonstrations