The Classical Model of Outcome Evaluation

The logic of outcome evaluation rests on a series of five fundamentalassumptions that are seldom stated explicitly. They are:

1. Pre-existence of purpose

Outcome evaluation is rooted in the idea that people have purposes(or goals) that exist prior to their action. They take actionsto achieve those pre-existent purposes. It thus makes sense toevaluate the usefulness of those actions in terms of whether ornot those purposes were achieved.

2. Technological certainty

This assumption holds that action can lead to the achievement ofgoals in ways that are both intentional and understandable. If anaction is understood as a technology for achieving a specifiedgoal, then the "certainty" of the technology depends on twothings. First, the actor must understand the link between thetechnology and the expected outcome. Second, the actor mustintentionally choose to initiate such action because he or shewants to achieve those specific goals.

3. Perceptibility of purpose

To evaluate outcomes, you have to be able to observe them.  For purposes of evaluation, outcomes that can't be observed don'treally exist. We usually think of this in terms of measurement realoutcomes are ones that can be measured in some objectiveway. But measurement is simply one way of observing. The keypoint is that outcomes have to be observable by some valid andreliable method.

4. Interpretability of outcomes

Measuring something doesn't necessarily mean that you understandit. Evaluation assumes that if you observe an outcome, youcan understand whether it actually constitutes achievement ofthe pre-existing goal(s).

5. Usefulness of findings

The final assumption holds that if success is limited, you can usethe findings to alter the technology, thus making it more effective.In other words, you have to be able to understand theimplications of the findings for future practice.

To sum up, the classical model of summative, or outcome, evaluation assumes that:

• you have goals and understand what actions can achieve those goals
• after taking the actions you can accurately observe the outcomes and can interpret whether those outcomes truly constitute achievement of the goals
• if success is limited, you understand how to use your findings to alter the technology, making it more effective in achieving the goals.

By its own logic, evaluation is only worth doing if its finding are"useful," and that usefulness is achievable only given the certaintyof the technology and the perceptibility and interpretability ofthe outcomes.

Sometimes all of these assumptions are fulfilled, and in suchcircumstances classical evaluation is justified. But today's museumsseldom find themselves in such circumstances. More frequently, one or more of the model's five

assumptions remainunfulfilled, rendering classical evaluation inappropriate.

A full review of the circumstances affecting all five assumptionswould be far beyond the scope of this article. Instead, I willexamine six situations that challenge the assumptions of the preexistence of purpose and the link between purpose and intentionalaction. Each is a circumstance we frequently find ourselvesin as developers of museum exhibits and education programs (orfor that matter, of any other kind of organizational activity).

1. You have goals you can articulate, but you can’t measure whether you’ve achieved them.

Sometimes legitimate goals are inherently not measurable. Acentral goal of museums and other educational institutions is "tohelp produce highly educated human beings." This assertionmakes intuitive sense to all of us who believe in the transformativepower of museum experiences, even though we don't know exactlywhat being "highly educated" means. But classical evaluationmethodology states that such an assertion is not a proper goal,because there is no way to measure success in its achievement.Evaluation methodology involves breaking down such "improper"goals into smaller, more concrete "objectives" that are susceptibleto measurement. The abstract state of "being educated," whichcannot be measured, is replaced with a series of concrete actionsthat can: learning certain facts, passing a test, completing a curriculum, and so forth. The validity of this methodology, of course, depends upon the accuracy with which the concretely specifiedobjectives capture the essence of the original goal-the degree towhich such objectives as "passing a test" faithfully measure theessence of "being educated."

There is no doubt that much research conducted in this traditionhas led to interesting and useful findings about museums andtheir visitors. Sometimes, though, such efforts at "goal clarification"lead to absurdity. McLean argues that "cognitive learning goals articulated with the reductionism of a multiple-choice test have begun to drive the exhibition-development process." She cites the case of a natural history exhibit for which a key goal wasstated as "Visitors will be able to name three different organismson display in the hall and a fact about each one" (1999,88, 104). Our common sense recoils from such an impoverishedconcept of education. We may not be able to define education, butwe know when we don't see it.

Certainly some goals can be validly measured, but in other cases such efforts lead to "the error of hypostatization," definedas "treating that which cannot be denoted as if it could" (Weickand Westley 1996, 441). Some highly abstract goals (such as "creating highly educated people") properly and accurately state values that cannot be reduced to a series of concrete objectives without losing something of their essence. As seasoned museum professionals, we should trust our intuition when it tells us that a given list of "facts to be learned" is missing something vital to the educational value of the museum experience. As Sheldon Annis asserted, "Educational tests are verbal, but museums are not. Their power is to expand upon imagery, not to reduce it" (1986, 171).

Classical evaluation can drive us to forget our real goals and tolearn to live with inferior ones simply because the latter are susceptible to measurement. In a sense, evaluation-or, at least, thedemands of the evaluation methodology-becomes the goal thatdrives our choices. But we all know better, and we all know that thegoal of education ought to be more important than the goal ofevaluation. Yet exhibits and programs regularly are built aroundthe opposite set of priorities. When we are tempted to start believingin the reductionism of evaluation methodology, a useful antidotemay be found in John Steinbeck's meditation on the problems of measuring a fish:

It is good to know what you are doing. The man withhis pickled fish has set down one truth and recorded in hisexperience many lies. The fish is not that color, that texture,that dead, nor does he smell that way (Steinbeck1941, 2-3).

2. You have goals you can articulate, but they are complex character and all the aspects can’t be measured simultaneously.

Outside carefully controlled situations in the research laboratory, actions seldom do only one thing at a time. Actions may serve many purposes simultaneously, which all may be important andsignificant. A single exhibit or program might simultaneously buildself-esteem and self-awareness; teach concrete knowledge or skills;stimulate visitors to personal meaning-making; generate revenuesfor the organization; conserve important artistic or historicresources; raise the cultural tone of the community; increase thetotal "social capital" of the community; strengthen democracy;promote truth, justice, and the American way; and so on. In fact,exhibits are more likely to be widely supported when they have thismultiplex character, since such exhibits are able to tap the interestsof diverse constituencies.

Traditional evaluation, though, tends to obscure the multiplexnature of goals. Resources are seldom available to evaluate all of theoutcomes of a complex exhibit or program, even where they are allexplicitly recognized (and often some are not). Even more fundamentally, evaluation seems to be subject to its own version ofHeisenberg's famous "uncertainty principle": choices made to measureone type of outcome tend to hide other types of outcomes.These trade-offs are not random. Classic evaluation methodologyproduces systematic biases in what it reveals and what it conceals.Empirical observation of evaluation research in a widevariety of disciplines suggests that the methodology:

• favors strong effects over weak ones, since they're easier to detect;
• favors effects on small things over effects on big things, since a given amount of force produces more measurable effects on smaller things;
• favors close effects over those far away;
• favors short-term effects over long-term effects;
• favors effects that are easily measured;
• favors effects that are predictable over those that are surprising;
• favors effects produced by understandable portions of the technology rather than those produced by aspects of the technology that are ambiguous;
• favors effects that are easily interpretable over those that are mysterious;
• and, as the final and paradoxical result of the preceding biases, favors production of findings that are useless over those that might have real impact on future practice. The methodology tends to produce findings that belong to the realm of "that which everyone already knew" or that relate to the simplest and most trivial aspects of the program's technology.

These biases might be acceptable if there were some objectivereason for believing that the things that end up getting measuredare more important than those that do not. But this is almost certainly not the case. Kenneth Boulding (1979, 29) noted that manyprograms consist of "the process of solving one problem by creatingworse problems elsewhere." These effects, he argued, areobscured by "numerology," his term for the over-reliance onmeasures of things that can be conveniently measured-a biasthat tends to hide those pernicious effects that occur "elsewhere."According to March and Olsen (1979), most organizations haveto make choices based on highly ambiguous technologies, so thatevaluations are likely to deal solely with the most trivial of manyoutcomes, such as remembering the names of three organismsdisplayed in an exhibit. At the 2000 AAM conference in Baltimore,John Falk and Michael Spock argued that some of the mostimportant outcomes of exhibit experiences only appear long afterthe visit to the museum, when they would be invisible to traditionalevaluation methodologies.

3. You have goals, but you can’t prove that they’re good ones.

To demonstrate our worthiness of public support, it is not enough to prove that museums are achieving their goals. We also must demonstrate that those goals are, in fact, worth achieving. But themethodology of classical evaluation begins with the assumption of pre-existent goals or purposes and so does not include procedures for questioning the value of the goals.

Prominent scientists have observed that scientific research can address questions of fact, but not questions of value. Einstein (1950, 26) said that "Science can tell us what is but not what ought to be."Since classical evaluation is modeled after scientific research, it has the same limitations. Evaluation may document that your museum did something, but it can never prove that that something was worth doing.

As Steinbeck said, it's good to know what you are doing. Count the spines on your programmatic fish if you must, but don't assume that such counting will demonstrate that your museum is a valuablemember of society. Such demonstrations of value are certainly necessary, but they belong to an entirely different realmof social process.

4. You have goals, but they are, by their nature, mismatched to evaluation methodology.

There are at least two ways in which goals might be inconsistent with the assumptions of classical evaluation. In the first, goals might be invisible to evaluation. In the second, goals might be unrelated tooutcomes and, therefore, mismatched to a methodology that works specifically by measuring outcomes.

It is striking that evaluators still use words and numbers as the only forms of goals for museum exhibits and programs, at a time when the field has made so much of Howard Gardner's (1983) concept of"multiple intelligences" for improving our understanding of visitors. "Multiple intelligences" asserts that different people relate to the world in very different ways, through the use of distinctively differentfaculties, and that each of these different ways is of equal validity. In most cases, museums have used this concept to devise strategies for selling verbal ideas to visitors for whom linguistic intelligence is not the dominant or preferred form of intelligence. The experiences may be designed to appeal to a variety of intelligences, but they all do so in the service of goals that are defined through linguistic intelligence.

There is no reason why possession of "goals" should be a property only of linguistic or logico-mathematical intelligence. Gardner's "kinesthetic" intelligence, for instance, might generate goalsof a type completely incompatible with classical evaluation methodology. For example, the goal of many of our physicalactivities is simply the pleasure of skilled physical performance-as in tossing and catching a ball, which for most of us is entirely devoid of utilitarian outcomes. The "goal" of playing "catch" is simply the body's pleasure in mastery of the task.

A traditional evaluator might argue that this assertion is mistaken and that the "real" goals of playing catch are to "improve physical conditioning" or "cement social relationships," etc. We can all come up with such rationalizations when pressed to do so, but that certainly does not mean they ought to be believed. Rather, they should be understood as a form of "social desirability artifact," thetendency of respondents to tell a researcher what they think he wants to hear or what they think will make them look good; e.g., "I came to the museum today because I value learning" (Rounds1977). Our ability to produce rationalizations on demand does not prove that they were what motivated the action in the first place.

Like Gardner, we should refuse to privilege a single intelligence as themaster and so should refuse to insist that all goals for all exhibits derive solely from linguistic intelligence. Since the classical model of summative evaluation is based entirely on words and numbers, it is ill suited for observing outcomes in terms of goals derived from Gardner's other intelligences. Relying on the classical model is like relying on an astronomy based only on the portion of the spectrumvisible to the human eye-blind to a wealth of data that might reveal extraordinary new insights.

A second mismatch between goals and evaluation methodology occurs when the goals are unrelated to the types of outcomesevaluation knows how to measure. James G. March (1994, 1996), the leading scholar of decision-making, has argued that "rational" models of decision-making (including classical evaluation research) are rooted in a "logic of consequence" - the assumption that we take actionsbecause their consequences will serve our pre-existent goals. March contrasts this with a "logic of appropriateness," in which we take actions because they're "the right thing to do," or because we perceive such actions as consistent with our sense of self. He cites the case of Don Quixote, who, when asked to account forhis actions, responded simply that "I know who I am" (March 1994, 12). Thus. we sometimes do things (such as visit museums) not because of any expectation of specific benefits, but simply because"that's the type of person I am." Humans make choices and take actions to maintain consistency with their sense of personal identity. The logic of consequence (and thus the classical model of summative evaluation) is irrelevant to such case because the goal is satisfied in the taking of action itself, regardless of the eventual outcomes. Sometimes, for instance, we create exhibits simply because we know”that such an exhibit ought to exist – a knowledge that comes from consulting our deeply held sense of "rightness rather than from the prior specification goals for outcomes.

The dominant value on rationality in modern society, March notes, leads us to denigrate the logic of appropriateness as "irrational." However, he also notes that the conditions necessary to actuallybehave "rationally" seldom occur in the real world. Our daily behavior is more often conducted in terms of appropriateness. Therefore, evaluating consequences in terms of goals does not apply to muchof human behavior. And there is much reason to suspect that the value a museum visit has more to do with identity than with  consequences. Sheldon Annis (1986, 170) argued that for most museum visitors "nothing is more interesting than acting out and within the social role their own lives."

More recently, Mihaly Csikszentmihalyi (1993), in an article entitled “Why We Need Things,” argued that a principal function of our creation and interaction with things is to stave off “psychic entropy,” the deterioration of our identity that threatens to set in whenever we relax the activities that continually reassert a sense of structure and continuity in our lives. Interacting with objects in a museum might thus be motivated by this need maintain our sense of identity-inshort, by the logic of appropriateness.

Evaluation projects often begin with an exercise in "goal clarification," particularly when the participants seem unable articulate their goals clearly. The classical model assumes that we haven't yet done the hard thinking necessary for communicating our pre-existent goalsclearly and in a measurable form. Like a scientist designing an experiment, we shouldn't act until we've gotten the goals straight in our heads.

Certainly, this is often true. But sometimes the goals are not sitting in there, waiting to be teased out with a good talk. Instead, we may sense that goals will emerge if we act, but we really don't know what those goals might be.

When we assert that people have pre-existent goals, we are asserting that they know how they will feel about particular outcomes of action and that those feelings will be positive. However, it is often extremely difficult to predict how we will feel about some future event or potential possession. Even when we think we knowhow we will value a future outcome, we're often wrong. And, much of the time, we don't even think we know. Many of the difficult decisions in life, the ones that keep us dithering on and on, arise precisely because we're not certain how we will value an outcome, particularly when there are multiple alternatives for action.

This common situation is further exacerbated by technological ambiguity uncertainty over exactly what outcomes will result from a given course of action. Then we're not quite sure what will happen,or how we'll feel about any outcome that does happen.

Unfortunately, much of life consists of exactly this situation. As soon as we step away from a highly predictable routine repeating the same actions in stable circumstances - we're almost always facing the double whammy of ambiguous goals and uncertain outcomes. In these situations, problems can be solved only by taking action and then interpreting the result. We act not because we believe that will achieve our pre-existent purposes, but because the action itself is appealing and because we have a vague, intuitive feeling that something good might come of it. Sometimes we have an exhibit idea that we know is good; we just don't know what it is good for. And we won't ever find out if we don't build it. The ambiguity is inherent in the situation, and can only be resolved through acting.

Much of museum project management is like this. Goals can only bebrought into clarity through reflective action. But classical evaluation demands that we specify goals first and then evaluate action only in terms of those goals. This damages us in two ways. First, it may make it impossible for us to act at all and so prevents the learning that might have occurred. Second, it may force us into action with a set of goals that was "what we could come up with," that substantially suboptimizes the true potential in the situation. Since we then evaluate only the effects specified by this impoverished set of goals, evaluation may mask the most important type of learning that might be gained.

6. You need to discover new goals.

March points out that we tend to hold two distinctly different ideas about goals, one for children and one for adults: For children, "we emphasize choices as leading to experiences that develop the child's scope, his complexity, his awareness of the world. As parents, or psychologists, we try to lead the child to do things that are inconsistent with his present goals because we know (or believe)that he can only develop into an interesting person by coming to appreciate aspects of experience that he initially rejects" (1979,73). In contrast, we assume that adults have fully developed goals thatwill remain stable over time and that all they have to do is make choices that will maximize achievement of those pre-existinggoals. In other words, children need to seek new goals, but adults do not.

Nonetheless, like everything else, goals sometimes do wear out and need to be replaced. Goals that served us well in the past may lose their power to motivate or satisfy us.We sense that we have exhausted the potential in our existing goals and face diminishing returns on their pursuit. Or conditions change in our environments, and past goals now appear outmoded, irrelevant. Or we simply outgrow our goals and find that they now seem naive or callow, ready to be put aside.

Thus, we sometimes find that our problem lies not in achieving our current goals, but in discovering new goals that can revitalize us or help us relate better to the rest of the world. This is as true fororganizations as it is for individuals, and it seems particularly true for museums in our current, challenging times. The field needs creative, radical thinking about the roles that museums play in society, yieldingnew ways of defining our goals (Rounds 1999).

March argues that we need an entirely different type of technology to lead us to the discovery of new goals. He suggests that this must be a technology of playfulness or "foolishness" -a way of doing things for which we presently have no good reason, but which might help us to create new purposes. We must, he argues, learn to "treat action as a way of creating interesting goals at the same time as we treat goals as a way of justifying action" (1979, 75).

Change is inherently hard. John Dewey (1984,31) observed that "Old ideas give way slowly; for they are more than abstract logical forms and categories. They are habits, predispositions, deeply engrained attitudes of aversion and preference." Old goals are hard to get rid of, even when we have started flirting with new ones. Classical evaluation threatens our ability to break their hold because it insists that we define our actions explicitly in terms of those old goals, and so constrains our ability to experiment -for instance, to create compelling exhibit experiences for which we cannot articulaterational goals. Classical summative evaluation actively inhibits the type of learning that can lead to discovery of the new goals that museums need for new times. It is a technology mismatched tothe needs of the present day.

Conclusion

We hear endless variations on the exhortation that "If you don't know where you're going, you'll end up somewhere else." A simple and obvious idea. And yet, a literal adherence to that prescription cancripple the effectiveness and potential of our museums. Sometimes "somewhere else" is precisely where we need to end up.

It is hard to resist demands for outcome evaluations, particularly when they are attached to the funding that is so badly needed. Short-term considerations push us toward acquiescing to thesedemands. But the long-term health of our museums is at stake and demands that we resist backward-looking pressures and make the case for better ways of understanding and assessing our work. Whenfaced with the standard rhetoric about goal clarification and achievement, we must remember that, in reality, pre-existinggoals are fundamentally problematic, often ambiguous, and subject to continuing change.

No doubt classical outcome evaluation is justified in some circumstances. When you are confident of your explicit goals,understand your technology, can really measure and interpret your outcomes, and know how to use the findings, then by all means do that kind of evaluation. But be hard headed in assessing whetherthose conditions are truly fulfilled.

Often, you will find that they are not. Instead, you will find that your work is less a solution for achieving pre-existing goals than a process for discovering new goals and creating opportunities for visitors todiscover theirs. We need to focus our efforts on forms of evaluation that facilitate that process, rather than retard it.

References

Annis, Sheldon. 1986. "The Museum as a StagingGround for Symbolic Action." Museum151: 168-171.

Ansbacher, Ted. 1999. "Experience, Inquiry,and Making Meaning." Exhibitionist 18(2): 22 -26.

Boulding, Kenneth. 1979. "In Praise of Inefficieney."Graduate Woman, July/August 1979:28-30.

Csikszentmihalyi, Mihaly. 1993. "Why WeNeed Things." In History from Things: Essayson Material Culture. Steven Lubar and W.David Kingery, editors. Washington: SmithsonianInstitution Press. pp. 20-29.

Dewey, John. 1984. "The Influence of Darwinismon Philosophy." In Martin Gardner, ed.,The Sacred Beetle and Other Great Essays in Science.Buffalo: Prometheus Books.

Einstein, Albert. 1950. Out of My Later Years.New York: Philosophical Library.

Gardner, Howard. 1983. Frames of Mind: TheTheory of Multiple Intelligences. New York:Basic Books.

Guba, Egon and Yvonna Lincoln. 1989. FourthGeneration Evaluation. Newbury Park: SagePublications.

March, James G. 1979. "The Technology ofFoolishness." In March and Olsen 1979, pp.69-81.

-. 1994. A Primer on Decision Making:How Decisions Happen. New York: The FreePress.

-.1996. "A Scholar's Quest." StanfordBusiness School Magazine, June 1996, pp. 11-13.

March, James G., and Johan P. Olsen. 1979.Ambiguity and Choice in Organizations.Bergen, Norway: Universitetsforlaget.

McLean, Kathleen. 1999. "Museum Exhibitionsand the Dynamics of Dialogue." Daedalus 128(3): 83-108.

Rounds, Jay. 1977. "Social Desirability andMachiavellianism Artifact in the CaliforniaSelf-Esteem Measure." Journal of Research inCrime and Delinquency 14(1):84-87.

- 1999. "Creating Creative Exhibits."Exhibitionist 18(1): 35-40.

Spock, Michael. 1999. "The Stories We TellAbout Meaning Making." Exhibitionist 18(2):30-34.

Steinbeck, John. 1941. The Log from the Sea ofCortez. NewYork: Viking.

Watkins, Charles. 1994. "Are Museums StillNecessary?" Curator 37(1): 25-35.

Weick, Karl and Frances Westley. 1996 "OrganizationLearning: Affirming an Oxymoron." In Stewart Clegg, Cynthia Hardy, and Walter Nord, eds., Handbook of Organization Studies,pp.440-458. London: Sage.