Design and Procedure

The study was conducted with the students as a group, although each participant was allowed to advance through the texts at their own pace. In particular, each participant was given the stories, one at a time. After reading the story, the text was collected. The questions were then distributed to the participants. No time limit was imposed on answering the questions. The second story was then given and so on. After the three stories were finished, the students filled out a short questionnaire asking for information concerning the participants' educational background, interest in reading and   science fiction, and familiarity with the stories (or similar ones) used in the study. The shortest period taken to complete the three stories and the questionnaire was 80 minutes; the longest was 150 minutes. Most participants took approximately 100 minutes to finish the study.

In order to test the ISAAC system on the same questions, a number of precautions were taken to lessen researcher bias. The ISAAC system was developed to the competence level which was judged necessary to read and comprehend the stories. The model was then ``frozen'' at that level of development. This occurred prior to the questions being solicited from the evaluators. Thus, there was little chance that the researcher would unintentionally bias the outcome by tailoring the computer model to the specific questions being asked. Then, ISAAC read each story in the order in which they were given to the human participants. After each story was completed, ISAAC was asked the questions. The questions were given to the ISAAC system in their original English forms. ISAAC possesses an external question answering task which interprets questions as a mixture of memory requests and the need to build connections between items which are retrieved. As ISAAC possesses no English generation capability at this time, the answers produced were in conceptual form. These forms were translated into English and written on a questionnaire sheet. A more complete description of the process by which ISAAC was evaluated, including examples of conceptual representations and the resulting English translations can be found in Appendix C.

The human responses and the ISAAC responses were then given to the four evaluators, with the identities of each participant hidden via a simple code number. Each evaluator was responsible for grading the questions which they had developed. No instructions were given to the judges as to what criteria should be used in the evaluation process; thus, the harshness of their standard of performance was completely under their control. In addition to grading the answers, each evaluator was asked to judge which participant they believed to be the ISAAC system and why.