======================================================================= Summary of T1 SME testing - Yolanda Gil A detailed summary of T1 SME testing was reported to the team during a telecon held March 5, 2001 where the following topics were discussed: - what we learned about training SMEs and our training/test material - what we learned about SMEs - duration of training and testing - what worked really well - what we did not explain well enough (or not at all) and should - biology-related comments on the scenarios we used - comments and suggestions for improvements - status of the implementation for T1 (what they could and could not do in this version of SHAKEN) - what they really liked - SME comments and suggestions for improvements (bigger things) - nitty gritty comments on GUI interaction (smaller things) - comments on the components (maybe only UT wants to hear these) - bugs in the system (maybe only SRI wants to hear these) - what I learned about the big picture - miscellaneous - the best ten quotes ======================================================================= Date: Thu, 01 Mar 2001 19:35:20 -0800 To: rkf-tkcp@AI.SRI.COM From: Yolanda Gil Subject: [rkf-tkcp] T1 SME testing: brief report on day 1 Dear all: We had a very interesting and productive day today with two biologists learning about SHAKEN and testing it. I will not be able to spend much time writing a report probably until Monday, but wanted to make sure to report on how things are going. We spent time today training them to use SHAKEN, with slides and with hands on practice. We explained virus invasion, and they entered the scenario not once but 4 and 5 times from scratch, and seemed to never get tired to ask questions to SHAKEN. They understood the tutorial material, and had some hard biology questions that I was somehow able to address thanks to previous Art and Bruce's input (Bruce: I had to use your adenoviridae schematic twice). This is significant, since we successfully hid all the underlying technical terms from them. As you can see in the slides, they never learn what is a concept, or a relation, or an instance, or what unification means, and so on. We never showed them or described or mentioned KM code. So I think our essential approach to designing this interface seems to be a good one. That is the first good result that I am taking from this so far. One of the main questions that we had in coming to this is whether the SMEs understand the basic idea of knowledge entry by composition. I wondered if they would "get it", if we as engineers found this notion completely natural but they would not. They seem to be fine with this so far, and Of course, tomorrow they will have to show that they can actually do the transcription scenario from scratch, which unlike virus invasion they will not have seen how we do in SHAKEN beforehand. And that will be a more challenging matter. Based on today, I am quite confident that they will do fine. We will try to send a report tomorrow evening, even if short. There are many other things and details to report, which will follow next week. I am happy to do a telecon if there is interest and people would like a interactive discussion. Oh, and surely there were glitches in the tool and bugs and things that didn't quite work, but they were very understanding and this did not confuse them. They are very good SMEs, some of the best I've worked with. They made a lot of suggestions, and we are taking, of course many detailed notes. I was very impressed with the way everyone has worked together to make this happen, from the software implementation to the scenarios to the training material. The last two weeks have been the most fun ones I at least have had in the project, mainly because I got to work with and learn from many of you. Thanks everyone for your great effort and good teamwork to pull this together! I will say that I am also very happy that the basic scientific goals of the project seem to be modestly validated by what the SMEs did today: easy/accessible interfaces and entry by composition. This is just a tiny step given the long road ahead with many challenging research issues (scale, component design, alternative entry modes, etc etc etc), but I am personally very excited and reassured by today's experience that I am putting my bets in the right places. Reporting live from SRI :) Yolanda (and Vinay!) ======================================================================= Date: Sat, 03 Mar 2001 01:52:14 -0800 To: rkf-tkcp@AI.SRI.COM From: Yolanda Gil Subject: [rkf-tkcp] T1 SME testing: brief report on day 2 Hi everyone, I'm back reporting from the studio (not live at SRI anymore), after a second quite intense day with biologists testing SHAKEN. I have learned so much more in the past two days than over the last few months of working in RKF! Thanks again to everyone in the team that provided just what we needed to make this happen. In conducting this exercise, I interacted directly with 13 of you to make T1 happen! I have compiled 50 pages of notes, and could have gotten many many more. I am not planning to generate a report from my notes, it would be a lot of work and probably only 1 or 2 people would read it all anyway... But please ask me any details, any time though, or call me if you have specific questions or are curious about anything I don't mention. My notes contain two things: a list of items to report to the team, and the transcriptions of what one of the subjects did during the test (Vinay has notes for the other subject). I reviewed my notes and divided the items to report in several categories: - what we learned about training SMEs and our training/test material - what we learned about SMEs - duration of training and testing - what worked really well - what we did not explain well enough (or not at all) and should - biology-related comments on the scenarios we used - comments and suggestions for improvements - status of the implementation for T1 (what they could and could not do in this version of SHAKEN) - what they really liked - SME comments and suggestions for improvements (bigger things) - nitty gritty comments on GUI interaction (smaller things) - comments on the components (maybe only UT wants to hear these) - bugs in the system (maybe only SRI wants to hear these) - what I learned about the big picture - miscellaneous - the best ten quotes :) I will report as much as I can on Monday's telecon with Vinay's help, since he was there the whole time. Thanks to Bruce for setting that up. I will try to keep the telecon as short as possible, so I propose that we make it more a "journalism-style" report of what happened rather than getting into discussions of what we could do about it in the future. I estimate it will take 60 to 90 minutes to go through things, and I'll be happy to change the order above and leave items that are not of general interest for the end. As far as the second thing captured in my notes, i.e., the transcriptions of how the subjects implemented the test scenario (which, by the way I found absolutely positively fascinating), I will need a few days to put together some distilled nuggets that will communicate efficiently to all of you what was going on there. Perhaps we could plan on a telecon the following Monday at the same time, I will send out slides beforehand and I can talk over the pictures in the slides. Last, but not least, I would like to discuss with those interested what we should do in the next three planned practice SME tests (T2, T3, T4) based on what we learned from T1. Unless there are any objections, I would suggest that we invite Bob Schrag (hi Bob, I know you are listening) to join in these two telecons. I would understand if any or most of these things are not of interest to some of you. I'll be happy to look into other ways to report to the team that will be more useful to everyone. Personally, I felt like Fleming looking at the Petri dish :) We have tested many versions of many tools with SMEs in the last few years, and each time I am humbled by the challenges of the knowledge acquisition problem and by the greatness of the human mind in its capability of acquiring new knowledge so seemingly easy... I will do my best to share that excitement with all of you on Monday. Yolanda