Authoring Procedures Using Demonstrations and Experiments

This page covers my dissertation, which is called

LEARNING WHAT TO INSTRUCT: ACQUIRING KNOWLEDGE FROM DEMONSTRATIONS AND FOCUSSED EXPERIMENTATION.

Abstract

One way that people can learn procedural tasks (e.g. machine maintenance) is by performing them in a simulation of the domain. Simulation-based training not only allows repetition, but also allows exposure to situations that would be dangerous or expensive in the real-world.

However, for simulation-based training system to teach students more effectively, the training system needs to know the procedures that students are learning. Unfortunately, it has been difficult to acquire this type of knowledge.

This dissertation looks at using machine learning techniques to acquire procedures that can be used for tutoring human students. The work focuses on exploiting access to a simulation of the domain. Because we assume that an automated tutoring program will already know general principles for teaching, we focus on learning ``what'' to teach rather than ``how.'' The approach learns general-purpose operators and outputs knowledge of what to teach in the form of hierarchical partially ordered plans.

The approach is implemented by a system called Diligent. Procedures are specified by having human authors demonstrate them. To demonstrate, a human performs the procedure by directly manipulating a graphical representation of the simulation. This technique could be used by domain experts, who may not be programmers or expert knowledge engineers. However, there is a problem: a single demonstration may be insufficient for understanding the causal relationships between a procedure's steps, while requiring many demonstrations would take too much time and effort.

Diligent attempts to overcome this problem by using a novel approach for understanding demonstrations. Although Diligent may start with no domain knowledge, it can reduce the requisite number of demonstrations by performing autonomous experiments that exploit the domain knowledge embedded in the simulation. These experiments identify the causal dependencies between a demonstration's steps by removing a step from the demonstration and observing how this affects later steps. These experiments are augmented by heuristics that focus on the dependencies between steps.

We performed an evaluation on human subjects that tested the benefits of our approach. As dependent variables, we measured the amount of work performed and the number of errors. The results suggest that using demonstrations and experiments together is better than using just demonstrations. The results also suggest that demonstrating a procedure is better than using an editor to declaratively specify it. The benefits of both demonstrations and experiments appeared to be greater on more complicated procedures.


Download

Because the thesis is long, it is available in several pieces. (The document's main body is 250 pages, and there are 100 pages of appendices.)