Prototyping at the Museum of Science
Formative evaluation, or prototyping has become a way of life for the exhibits department. This process first began in the late eighties with the renovation of the Habitats exhibit. Dioramas were enhanced to create a more accessible environment for visitors. Label changes as well as the placement of new exhibits near older ones led to a better understanding of the exhibit for all visitors. The prototyping was conducted in situ and the evaluation was done in conjunction with an outside consultant.
In house evaluation began during the development of the first activity center, The Observatory, which opened in 1991. For the first time, the visitor was part of the development team. Everyone on the team from designer, planner and even technical designer became a visitor advocate. Each member evaluated different aspects of exhibit components be it content, design elements or technical aspects including durability. Much of the evaluation of prototypes is completed by Exhibit Planners. Most often the planner of the exhibit is also the evaluator. On some occasions, planners will serve as evaluators for other projects within the Exhibits Department.
As part of this project, the Test Tube was opened to the public. Test Tube use has increased dramatically over the last five years. This past year six projects used the Test Tube concurrently. Recently, even temporary and traveling exhibits experience some level of prototyping. During the development of the Leonardo da Vinci: Scientist, Inventor, Artist exhibit in 1997, all interactives developed by the Museum were prototyped in the Test Tube.
By 1999, all exhibit projects utilized some type of formative evaluation. In the spring of 2000, a new Test Tube will be designed to meet the Museum's growing prototyping needs.
The Test Tube 
Prototyping occurs primarily in a designated area called the Test Tube. This 1900 square foot space consists of an enclosed room with two entrances. The front wall is made of glass. A curtain can be opened or closed to allow visitors to see what is occurring inside. The other walls are painted but bare to allow for a variety of treatments from label copy to mock-up dioramas. Electrical sockets are available on each wall for components requiring power. A stationary video camera and VCR are often used for videotaping visitor interactions with components.
While new exhibits are being developed, the Test Tube is open for family groups on weekend days and during school vacation weeks. During these two to three hour sessions the room operates as a free choice setting. Families may enter and leave the room at will and can use any of the interactives that interest them. A staff member often greets visitors at the door and welcomes them to this unique space. A sign outside the door indicated which exhibits are currently being tested, the hours of operation and if videotaping is occurring.
During the week, school groups are also invited into the Test Tube, however, these visits are more structured and serve as focus groups for developers.
Three stages of Prototyping in and out of the Test Tube
Each component to be evaluated passes through three stages of development. During the first stage of prototyping, a rough table-top version of an exhibit is created. The activity is facilitated by exhibit developers. This stage of prototyping occurs both in the Test Tube and on the exhibit halls. Developers observe and interview visitors while assisting them with the activity. If a component cannot meet it’s educational goal while being interpreted, it is unlikely that it can function as a stand alone exhibit. Therefore the exhibit is put aside to be assessed as a potential element for the Program’s department. School programs, demonstrations, and interpretations are often developed to accompany the final exhibit.
If the component is successful in meeting it’s educational goals during this first stage of prototyping, a "stand alone" version of the activity is created. Most second stage prototyping occurs in the Test Tube. During this second stage of prototyping a variety of evaluation techniques are utilized. Evaluators primarily observe and interview visitors. Based on their interaction with a component, modifications are made. Changes can include new instructions or additional labeling. The layout of the interactive elements as well as the mechanical, electrical or computer software may also change to enhance the experience. Design elements are also incorporated into this stage of prototyping. It is important to determine a final look and feel for each component. If the exhibit is prototyped as a tabletop activity and then is changed into something more elaborate with a new layout, visitors often have a different educational experience.
During the final stage of prototyping, the exhibit component is nearly complete. Final changes to label copy are implemented. The housing is often the finished piece of cabinetry . This shake down period allows the developers to put any final touches on the component. During the final development of an exhibit, many of the components in this stage are prototyped in situ. Although this takes additional time (and, in reality, many components do not make it to this final stage) this type of testing can provide valuable information.
During second stage prototyping of the Investigate! Exhibit, the Motion Match was tested in the front area of the Test Tube behind a large glass window. Visitors outside of the room could easily watch other visitors participate in this activity which involved walking back and forth in front of a computer screen. There were no other activities immediately to the left or right of the exhibit.
Visitors actively participated with this component and with minor modifications to the screen size and time of screen changes, the component was successful. Once installed in the exhibit space, visitors were often seen pulling a stool from a neighboring exhibit and sitting in front of the computer screen. This baffled developers as there was no indication that one should sit at this exhibit. With a number of modifications including life-sized images of people on the adjacent wall moving in front of the screen, a large eye to enhance the photodetector which "sees" movement, footsteps on the carpet to simulate movement, as well as a sign that says no stools, visitors seem to understand the concept of the exhibit. In situ prototyping might have saved some of this remedial work.
A classic example of an exhibit component which was developed using all three stages of prototyping is the Temperature Investigation Station, now in the Investigate! exhibit. The first stage simply included a water cooler, thermometers, paper and Styrofoam cups and a clock. Developers assisted visitors with a variety of activities associated with temperature and determined which experiments appealed to visitors.
Once visitor interest was established, a more elaborate second stage was created. Temperature probes replaced the thermometers. These were connected to a computer running the data acquisition software LabView. Visitors could now view the results of their temperature experiments via a two colored graph on the monitor. Fans were added as a results of visitors feedback. The exhibit was still in prototype form. Label copy was handwritten on a white board. Visitors used a keyboard and mouse to control the computer and the probes were not very durable.
During this stage, evaluators observed visitors interacting with the component. Family groups spent an average of four minutes at this component. Visitor comments included: Can we break a record? Can we get it to 100? Look, the lines are converging, and Let's try the water. Several interesting behaviors were also observed at this station. Visitors were seen measuring and comparing temperature of two visitors’ hands using probes, measuring and comparing temperature of two hands (same person) using probes, measuring and comparing temperature of water in two types of cups, using fans to change temperature of water or probes, rubbing the probes to generate heat, reading and interpreting temperature graphs and interacting with others in group and with other groups.
These positive interactions with the component led to the development of the third stage prototype. This prototype, which became the final component, allowed for final changes in label copy. The computer was hardened and a paper towel dispenser was added. Evaluators observed large quantities of water being used and dumped by many young visitors. Therefore the final version included a system where a specific amount was provided with a push of a button.
Many exhibit ideas do not survive the three stage process. One favorite example involves a component called Where’s the Prize. This exhibit, similar to the Lets Make A Deal game show, allowed visitors play a game of chance. Visitors were provided with three doors. A "prize" was placed randomly behind one of three "doors." The visitor could choose one of the doors, but didn't get to open it. Then the game host (another visitor or a computer) opened one of the two remaining doors — one that didn't have a prize — and said, "It's not behind this door. Would you like to stay with your original choice, or do you want to switch to the remaining unopened door?"
Visitors were asked to come up with the best strategy for winning, either to stay with their original choice, or to switch. Space was provided to record their predictions. Visitors could play the game for many rounds and see a running record of their wins, and of all previous visitors' wins, for both strategies (staying or switching).
The first stage of prototyping was very successful. As a facilitated activity, it was not difficult to keep an accurate running total of all visitors’ games. Visitors were interested in the game and in developing strategies for winning. Several versions of a stand alone second stage prototype were created. An elaborate electronic version was developed. Doors automatically opened to reveal the prize. Visitors still needed to play the part of host and hid the prize behind one of the doors. This prototype had many flaws as did a computerized version. After many weeks of prototyping, and too many versions of this activity, it was determined that is was best suited as an interactive demonstration - similar to the first stage prototype.
Focus Groups in the Test Tube
It is primarily during the second stage of prototyping that various focus groups are invited to interact with the components in the Test Tube. Focus groups allow for in-depth observations and interviews with selected groups. These can include teachers, students, special needs groups, community groups, as well as volunteers and staff within the museum.
Teachers and School groups
During the creation of the Investigate! exhibit, developers involved a number of minority groups and students at risk. These focus groups were used to ensure broad-based appeal, relevance, and effectiveness of exhibits. A broad coalition of students and community groups were utilized in developing and evaluating the prototypes. Students from a local inner city high school participated in evaluation studies, as well as community groups from around the Boston area. After school programs, church and library groups were targeted.
A Teachers Advisory Board was established to assist in the development of the Investigate! exhibit. These teachers were part of the Massachusetts Educational Reform project. They assessed prototypes and provided a teacher's perspective on improving and selecting exhibit activities. Students of Teacher Advisory Board members also prototyped activities in the Test Tube.
Teachers participating in week long sabbaticals at the Museum of Science also evaluated prototype ideas and activities and provide feedback to the exhibit developers.
Camp-Ins
The Camp-In program at the Museum of Science was developed to promote scientific literacy and curiosity in young women. This program provides a unique prototyping opportunity. During prototyping for the Investigate! exhibit, a number of girls were observed and interviewed as they interacted with the prototypes. In particular, girls were used to test for gender inequity in the Solar Car activity. It was determined that girls in single sex groups more actively participated in this activity than girls in mixed groups. In mixed group settings, girls more often served as recorders and data collectors. These girls also offered suggestions such as the addition of scenery to the track to make the component more appealing to them.
Accessibility advocates
Click here for Universal Design (Accessibility) section.
During the development of the Investigate! exhibit, children in physical therapy at Children's Hospital came to the Museum of Science with their families to try out many of the exhibit components. With their help, the development team was able to determine appropriate tabletop heights of exhibits. The children also helped to determine which activities were most accessible to wheelchairs and how minor changes to these components would greatly enhance the experience of a visitor in a wheelchairs.
For example, the Motion Match exhibit allows visitors to move back and forth in front of a motion detector (usually walking, skipping jumping or running). Developers weren't sure if a visitor in a wheelchair could use this interactive. The children demonstrated that they could easily use it. They also came up with new creative ways to use this interactive. The children provided the idea of having the words ready, set, go appear on the computer screen before the activity gets started. This brief delay gives visitors in wheelchairs a few extra seconds to get into a good position before the motion detector begins to read movements. As is the case in many components, what is good for some is good for all. Standing visitors, it turns out also benefit from having few seconds to get ready to start the activity.
Students with limited mobility from a nearby school also came to the Museum to help prototype some of the Investigate! exhibit components. These students helped with placement of buttons on individual components. They also assisted in designing wording on directional label copy and gave feedback on appropriate lengths of time for screens to appear on computer interactives.
Blind students came to the Museum to help the exhibit development team write audio labels for several components. Several versions of label copy were tested as these students participated in each activity. The students explained what information was still needed for blind visitors to successfully interact with the exhibits. Again, what is good for some is good for all. Most visitors, sighted and blind, use the audio labels.
Blind adults were consulted and gave suggestions to improve the interactives. These visitors helped determine placement of tactile labels and the most appropriate use of audio labels.
Strategies used in Prototyping
Observation
Informal observations are performed by many members of the Exhibits and Programs staff. More formal observations are used to define the expected behaviors for specific components. These assist the developers in modifying units in ways to optimize appropriate behaviors. Observation sheets are used to estimate the time needed to fully participate in each component. Modifications are based on these observations in order to increase elapsed time at individual components. Questions such as do visitors read labels, and do they manipulate materials appropriately are addressed. Family interactions are also observed using criteria described by Minda Borun’s characteristics of successful family exhibits.
copy of observation sheet
Interviews
Both formal and informal interviewing techniques are utilized. During the prototyping of The Observatory and Investigate!, formal interviews were used to assess visitor’s use of specific experimental skills at single components. Additionally, visitors were interviewed after viewing clusters of activities to determine what themes visitors perceived. Informal interviews were conducted after visitors were observed at a specific component. Questions were guided by what the visitors were observed doing at the activity.
copy of interview sheet
Comment Cards and Feedback Forms
Because it is impossible to interview all visitors in the Test Tube during a given session, comment cards and feedback forms are often used to gather unsolicited data regarding components.
copy of comment card/feedback form
Phone Line
During very busy prototyping sessions it is difficult to get comments from many of the visitors who interact with exhibits in the Test Tube. A "hot line" was installed in the Test Tube for visitors to leave comments for developers via an answering machine.
Ant Trails
Tracking visitors through a particular groups of activities is used when exhibits are sequential or participation in a specific order is necessary for content acquisition.
Word Pair Surveys
These are used to identify affective attitudes toward activities.
(copy of word survey pair)
Video Taping in the Test Tube
