3.1 Multiple-level learning design

Figure 2 shows a typical case-based diagnosis exercise in which students are presented with a simulated patient in a clinical setting. In the role of physicians, students are able to perform a variety of actions on the simulated patient; they may ask questions about medical history, perform a physical examination, order diagnostic tests, and make diagnoses. Adele monitors the student's actions and provides feedback accordingly. Depending upon the instructional goals, Adele may highlight aspects of the case, suggest correct actions, provide hints and rationales for particular actions, reference relevant background material, and provide contextual assessment.

A clinical case presents a rich context for learning at many levels. Adele can emphasize the procedure, namely, the best practice approach to a particular diagnosis, or she can concentrate on the related learning materials. Exercises can focus on different aspects of a case: in the medical domain, we may have pre-clinical exercises, which stress fundamental facts; clinical exercises, which stress clinical diagnosis; or resident-level exercises, which stress treatment and management. One case can support all three levels, although the pedagogical emphasis of the learning task changes for each level.

3.2 Computer-based learning design

The pedagogical emphasis also changes with respect to the learning content, in order to exploit the benefits, or mitigate the drawbacks, of computer-based learning. For example, discovery by palpation and percussion, two forms of touch, cannot be simulated on a computer. Teaching listening skills is problematic, too, because it is difficult to record high fidelity body sounds without special equipment. The best an application can do is to provide a textual description of what something feels like or sounds like.

On the other hand, the visual bias of the computer makes it especially good for teaching observation skills, something that is often under-emphasized in clinical settings, where students are apt to miss important visual cues in their rush to complete an examination. We can exploit visual opportunities further: for example, after the simulation response, "The throat is not inflamed.", when examining a patient's throat, Adele will display and compare images that show both normal and inflamed throats.

If Adele is to be adopted as part a system-wide curriculum for case-based medical education, the issue of authoring the task representation and simulation logic for each case becomes very important. This section discusses issues in large-scale case authoring, and tools to assist in this process.

Adele's interactive cases are developed by layering domain, simulation, and pedagogical data. Domain knowledge, such as reusable case and disease data can be combined to produce a best practice task model. This task model specifies an ordering of steps to be taken, places constraints on the steps, and relates the steps to the diagnosis. Rationales and hints for the steps are added here. The amount of case material that is used to build the task model depends on the instructional level and objective. Once the task model is completed, the author supplies the media assets needed for the simulation. Media include text for Adele's responses, and text, image, video and audio files for the patient and clinical findings. The pedagogical model, which consists of contextual comments, reference materials, and quizzes for assessment, is then layered over the resulting task model and simulation.

4.1 Building an authoring tool

Currently, case authors enter data for new cases via a form that sits on top of a database. A translator takes the output and parses it into an object-oriented case representation scheme from which both the task representation as well as the simulation logic for the case are generated automatically. The translation process performs a variety of syntactic consistency checks on the input provided by the author and also saves the author the burden of matching simulation events with the actions in the plan. The current tool isn't ideal; authoring a non-sequential path through a case can be confusing, and it does not yet provide for a persistent repository so that data can be easily reused across cases. We are actively designing an authoring tool with these qualities that will meet the needs of both instructors and developers.

In their paper on authoring via task-specific tools, Bell and Zirkel [1997] argue that many ITS authoring tools sacrifice pedagogical power for content flexibility but that developing an authoring tool for a specialized category of instructional applications has advantages even so. We agree that while targeted authoring tools do indeed sacrifice flexibility, it need not be for pedagogical power. Power is afforded to authors who understand the types of tasks they want to teach, and the constraints that are inherent within those tasks. From our experiences with one-on-one authoring between physicians and developers, we propose that a successful authoring tool must be

1. Domain-intuitive: Medical professionals understand how clinical cases are constructed and presented, and their knowledge of the domain can be accounted for to help tailor a system for them, rather than one for the programmer, or architecture;
2. Reusable: Productivity and ease of use are increased for authors who can select from a repository of previously-authored steps and rationales, etc., instead of supplying new information each time. Standard ontologies [19] can also serve as reusable knowledge bases from which knowledge for cases is drawn as needed;
3. Testable: Case authors need feedback on the cases they author and familiarization with the system they are authoring for will make them better authors. Adele's system may be a good candidate for an authoring tool that utilizes Programming By Demonstration (PBL). Diligent [Angros et. al 1998], is a good example of a PBL system in which users author tasks within the tutoring environment; it has been shown that instructors more accurately do in context, than explain out of context.

In this section we describe the initial evaluation of Adele. The evaluation was given to a class of second year medical students during the third week of November 1998 and was based on a new case on Lung Cancer that the students had studied in class. The students were unmonitored and worked through the case on their own time. Over one hundred students used the system, although only twenty-five of them completed the final questionnaire. Two face-to-face evaluations were also conducted.

Our goals for the evaluation were twofold: to discover how the students would react to Adele, and to confirm that the system could support the students. Did Adele and the concept of Web-based exercises have potential in a medical school environment? Was the Adele system easy and robust enough to be used in such an asynchronous setting? The final questionnaire contained thirty questions in six categories: system use, system components, rationales, Adele, and learning. They addressed both specific elements of the tutoring system, such as the interface and rationales, and the general reaction of the students to Adele and the concept of the system. Answers were scored on a Likert scale.

5.1 Analysis of the results

Overwhelmingly, students thought the system was easy to use, yet some found it difficult to figure out what to do next, and how to do it. The latter finding indicates a need for more guidance than the system currently provides. We are looking to Adele to fill this role; to suggest actions at a system level, as opposed to a task level, when a user becomes confused, and to point out the importance of interface elements if they are not utilized. Individual comments ran the gamut, from frustrated students who were unable to use the system to enthusiastic users whose comments sound like testimonials. As researchers and initial users of the systems we develop, we are inclined to bias the usability of the system in favor of users like ourselves, in other words, sophisticated users. Before any system can be successfully deployed on a large scale, this bias must be remedied. If there was a consensus among the students, it was that they wanted more cases to work on. In the words of one participant, "the more practice, the better."

The majority of students thought Adele would be a good distance education tool, and useful as a classroom preparation tool, but would not suffice as a replacement for a class lecture. They did not find Adele believable as an attending physician, a feeling that conflicted with a statement made by a fourth year student who had actually worked with attending physicians. It is also not clear if students prefer the persona to a text-only tutor. Though the numbers would indicate otherwise, the students provided favorable impressions of Adele in their general textual comments. Not surprisingly, students would have preferred a real voice to a synthesized one. Surprisingly, however, they did not mind that Adele's lips and voice were not synchronized. Adele's lips and voice move at the same time but are not yet phonetically synchronized and some people find this phenomenon disturbing. It may not be as important as we once thought.

Almost all students agreed that Adele's hints were helpful and her rationales useful. They had mixed feelings, though, about when they wanted to hear the rationales given. We noticed during a one-on-one session that the student was not asking "Why?", and therefore missing much of the authored knowledge, so we decided to have Adele give some of the rationales automatically, whether a student asks for them or not. We implemented three variations: 1) give a rationale only when asked, 2) give it automatically after a hint, and 3) give it automatically after a user takes a step, and then asked the students which variation they preferred. Most students answered that they prefer to hear a rationale only when they ask for it, although our experience suggests that they would not ask at all if given a choice. We continue to explore these issues further.

Adele and other pedagogical agents like her have lessons to offer regarding autonomous agent design. Researchers in believable agents argue that the audience's perception of an agent's competence is more important than the competence itself [Sengers 1998], and that competence is but one of many factors that agent authors should take into account [Reilly 1997]. A key question is to what extent these claims are true for pedagogical agents. User feedback from Adele suggests that while students find her advice and feedback useful, and think she is a good learning supplement, they don't think her knowledge is equal to that of a lecturer's or attending physician's.

Adele's task representation is being enhanced to explicitly reason with hypotheses and their likelihoods as in GUIDON [Clancey 1983]. This will allow Adele to present rationales in a more flexible manner and also automatically generate quizzes to verify the student's knowledge of the hypotheses underlying their actions. From an authoring perspective, the explicit representation of hypotheses and their relationships to findings could allow for automated generation of rationale without requiring the author to provide the rationale for every diagnostic step in a case. These limitations notwithstanding, the approach of orienting Adele's run-time knowledge base to individual cases appears to be workable, and will continue to be followed as the library of cases is developed.

We are encouraged by the acceptance of Adele by the medical students and the positive impression she made on the medical faculty and staff, whose cooperation and support was crucial. We will be continuing our efforts in curriculum innovation in the medical domain with a larger and more formal effort beginning in the summer of 1999, which will include the development and evaluation of complete course modules aimed at multiple organ systems and student expertise levels. Meanwhile, we continue to make steps toward the adoption of the technology in other domains. In collaboration with the USC School of Dentistry, we have implemented a case for a field trial in April 1999 and are planning to author more cases for a course in the fall. We are also developing a new non-clinical simulation that will allow Adele to be employed in more general settings.

CARTE staff members Kate Labore and Dr. Jeff Rickel, 'A' Team members Ami Adler, Andrew Marshal, Anna Romero, and graduate student Chon Yi all contributed to the work presented here. Dr. Allan Abbott led the evaluation at the medical school and he and medical student Michael Hasler authored the case. Our collaborators, Drs. Demetrio Demetriatis, William La, Wesley Naritoku, Sidney Ontai, and Beverly Wood, and Angela Atencio and Leah Flodin at the USC School of Medicine provided indispensable assistance. This work was supported by an internal research and development grant from the USC Information Sciences Institute.