Summary of Users Feedback after TKCP

July 20, 2001 was the last day of the Textbook Challenge Problem (TKCP) for Year 1 of RKF. We gave our users a final questionnaire to get detailed feedback about their experience with SHAKEN. Below is a summary of their answers .

The users filled out first a detailed questionnaire, and I asked them for additional comments for some of their answers. The feedback from our users was positive overall:

• They really liked the graphical interface that enables them to create Cmaps. They had some suggestions to improve it, but overall they were really pleased with it.
• They seemed quite happy with the component library, and very happy with the pumppriming knowledge. They had some requests and suggestions, but overall they thought they were appropriate.
• They did not seem to think that the system showed reall intelligence and knowledge, since it seemed to mostly be able to repeat the specific things that they had entered. This may be the most serious drawback of our system with respect to TKCP performance.
• Their plea to us is to add a delete button. It drove them crazy to have to re-enter everything from scratch when they made a mistake. It also made them reluctant to run the Test knowledge (KANAL) facility because every time it found an error it meant that they had to restart building the Cmap from scratch.

Users pointed out several important defficiencies in the tool:

• specifying more complex event structures, such as overlapping events and repetition. This was the most common request.
• specifying detailed information about specific events (e.g., production amounts and rates and duration) as well as calculations based on those
• represent/specify/manage different views and abstractions on a concept (e.g, DNA molecule)
• handling a wider variety of properties about entities, including spatial knowledge
• specifying negative facts (e.g., to answer question 7.1.3.89) and opposite situations
• additional kinds of questions

After discussing the questionnaire, I took some videotape interviews where I asked them to discuss the main issues they mentioned in their anwers to the questionnaires.

At the end of the video tape interview, I asked them a surprise question: "Next week SHAKEN will be asked questions and answer them using the knowledge you entered, and based on that it will be given a grade. Do you think it will be a passing grade?". I truly did not know what they would answer, and I thought it would be a good question to gauge how happy they had been with SHAKEN despite all the limitations they had pointed out.

All three SMEs reponded quite confidently, saying things like "definitely" and "oh yes". Then I asked "What kind of grade? Maybe a B?".

Two SMEs answered "A-" . The third said B.

An interesting comment from one of the users was:

"the approach and way Shaken understands and interprets knowledge has changed my way of looking at the biology"

This user explained that when reading a biology paper the immediate reaction is to think of how the knowledge would look in a Cmap.

Please contact me (gil@isi.edu) if you have any additional questions or comments.

Back to Final SME Feedback on RKF TKCP Summer 2001 Evaluations