By Stephen E. Weil
This article was published in Museum News November/December 2003.
When Stephen Jay Gould, the great science writer and evolutionary theorist, passed away in May 2002,the obituaries in both the New York Times andthe Washington Post noted that his interest in what would become hislife’s work had been kindled by a visit with his father to the American Museum of Natural History when he was 5 years old. “I dreamed of becoming a scientist, in general, and a paleontologist, in particular,” he subsequently wrote, “ever since the Tyrannosaurus skeleton [both] awed and scared me.”
Four things seem worth noting here. First, if the museum is a “learning” experience, what was it that Gould learned as a 5-year-old museum visitor? Was it only something the curator hoped to communicate about dinosaurs (they’re big, they’re awesome)? Or did he also learn something about himself, about what genuinely caught his interest and stirred his imagination?
Second, it seems striking that neither of the newspapers reported this anecdote as something bizarre or exceptional. While museums struggle over how intended learning might compare with actual learning, the outside world seems perfectly relaxed with the notion that there might be some connection between an early museum experience and a later life or career choice.
Third, assume that the museum had done an exit interview or some other form of visitor research on the day young Gould made his first visit. Would the full impact of that visit have been recognized?
Fourth, how could you adequately describe the outcome of that visit? Was it simply the impact on Gould himself that mattered, or did the visit ultimately have important consequences for his students at Harvard or his hundreds of thousands of readers throughout the world? Is Gould’s case unique, or can we generalize that the impact of a museum may frequently extend well beyond those who enter its doors and ripple throughout the community or even the world in ever-widening circles?
In other words, what can a museum learn from its visitors’ experiences—both inside and outside its walls? In evaluating a museum’s worthiness, the starting point must be the positive and intended differences that it makes in the lives of the individuals and communities that constitute its target audience. The critical issue is not how those differences are measured but that such differences must become and remain an institution’s central focus.
As a prelude to a discussion of outcome-based evaluation, consider the following overarching and relatively theoretical propositions about museums in general and about learning in museums.
First, today there is widespread agreement, at least in the United States, that the measure of a museum’s worthiness is no longer an internal assessment of what it might possess—a superior collection, a talented staff, splendid facilities, a hefty endowment—but an external consideration of the benefits it provides to the individuals and communities it seeks to serve.
Second, to echo the maxim that was so forcefully set out in AAM’s 1991 publication Excellence and Equity: Education and the Public Dimension of Museums, it is “education—in the broadest sense of that word—[that lies] at the heart of [every museum’s] public service role.”
Third, donors and grant-makers are decreasingly willing to accept simply as a matter of faith that recipient museums have achieved their educational or other intended outcomes. More and more, they are asking for some proof of performance. Daunting as the prospect may be, it will be essential for this country’s museums to develop credible systems of feedback that will allow them to monitor the impact of their efforts and make adjustments when necessary.
The Bottom Line
Let us start with the first proposition, that in determining the worthiness of a museum, the primary measure is the extent to which it can produce positive outcomes for the individuals and communities it seeks to serve. Two questions arise immediately: What do we mean by “outcomes”? And how do these “outcomes” fit into the overall conception of what a museum is and does? Concerning the first question, the best answer to my mind was offered in the mid-1990s by the United Way of America, a pioneer in the use of outcome-based evaluation to determine the allocation of its grant funds. To be successful, a United Way applicant must be able to demonstrate its ability to bring about some beneficial changes for individuals and/or populations that participate in its programs. The areas in which such changes may occur, though, are remarkably broad. They may “. . . relate to behavior, skills, knowledge, attitudes, values, condition, status, or other attributes. They are what participants know, think, or can do; or how they behave; or what their condition is, that is different following the program.”
These are almost precisely the kind of differences—most particularly, differences in skills, knowledge, attitudes, and values—that museums aspire to make through their educational activities. How, though, do these outcomes fit into an overall conception of the museum?
To answer that question, I’ll refer to the “social enterprise model” that J. Gregory Dees, a professor at Stanford’s Graduate School of Business, first proposed some years ago when he was teaching at the Harvard Business School. Let’s begin by envisioning a large box; that’s the button factory. To get it started, investors will pour money in through the top; the factory then purchases the equipment and raw materials it needs to make buttons. It also hires mechanics, designers, machine operators, and clerks to design, produce, and keep track of the buttons. Now, from the bottom of the box, we can see the factory’s output: millions and millions of buttons. Do those buttons constitute the factory’s real bottom line? Are they the outcome that the investors were hoping for when they made their investments? Of course not. One more thing has to happen. Somebody has to sell those buttons and convert them into money.
That money—that’s the bottom line, the outcome everyone wanted, the reason the whole enterprise was constructed in the first place. The buttons were only an output, an internal manifestation, something wholly within the factory’s capacity to produce. To produce the outcome—the money—one more critical ingredient was required: what economists call “demand.” A market had to want the buttons, to think they had value. Assuming the factory finds such a market and is able to sell enough buttons to be successful, some of that bottom-line button money can go back to the investors in the form of dividends. Most of it, however, will recirculate back to the top so the factory can purchase more materials, upgrade its equipment, retain its workers, and start a new cycle of button-making. In the best of all scenarios, it may even become a perpetual motion button factory, able to continue these cycles indefinitely with no further infusion of capital.
Let us now envision a second box; this one is a museum. Again, money will pour in through the top, this time as contributions, grants, and other public and private support. Some of that money will be used to acquire raw materials: objects, library books, information technology. The rest may be spent to hire workers—in this case, such specialists as curators, conservators, exhibit designers, and educators—to process the raw materials into such valued products as exhibitions, public lectures, catalogues, gallery guides, and specialized tours.
These valued products eventually will emerge from the second box as publicly accessible programs. But are these programs, these outputs, the museum’s bottom line? Are they the outcomes that its donors and grant-makers sought when they gave money to the institution? Again, the answer is “of course not.” Something more has to happen. The programs need to reach and have a positive impact on an audience. Unless they make some kind of a difference to that audience, they might just as well never have happened.
For the museum, that difference is the bottom line, the desired outcome, the reason the whole enterprise was constructed in the first place. Just as the button factory could only convert its output (buttons) into its desired bottom-line outcome (money) through a market that values those buttons, the museum can only convert its output (programs) into its desired bottom-line outcome (a beneficial impact) if a potential audience thinks its offerings have value.
That museums and other social enterprises have so much in common with business enterprises does not mean that the business model is now dominant and other kinds of organizations are under pressure to follow its example. In fact, in recent years we have come to recognize that business, social, and governmental organizations have long been structured and operate in startlingly similar ways. Common to all of them is the degree to which they commandeer and exercise dominion over the scarce resources the larger society has available. Common as well is the growing expectation that these organizations, in exchange for their temporary appropriation of those scarce resources, must return a commensurate quantum of value back to that larger society. Commenting on the case of the business enterprise, management expert Peter Drucker has observed: “A business that does not show a profit at least equal to its cost of capital is socially irresponsible. It wastes society’s resources. Economic performance is the basis; without it a business cannot discharge any other responsibilities, cannot be a good employer, a good citizen, a good neighbor.”
As profits are to a business, so outcomes are to a museum. The museum that does not provide an outcome to its community is as socially irresponsible as the business that fails to show a profit. It wastes society’s resources.
An Opportunity to Learn
How, then, is the museum to justify itself? With that question, we come to the second of our three propositions, i.e., that the museum justifies itself through the educational activities that lie at the very heart of its public service role. “Education,” though, is a remarkably spacious concept. It includes both the notion of teaching or imparting knowledge (as in “to educate”) and the not always reciprocal notion of receiving or acquiring knowledge (as in “to be educated”). In seeking to understand itself (or explain itself to others), a museum must think about how it intends to be educational—like the police academy or a vocational/career school, like a liberal arts college or a great research university, or like a public library?
A museum also must think about where it positions itself on the continuum that runs from providing formal instruction at one end to serving as a site for self-directed learning at the other end. In a recent draft of a strategic plan for learning in museums, the United Kingdom’s Council for Museums, Archives and Libraries seems distinctly inclined toward the self-directed learning end of this continuum: “Definitions of and approaches to learning have changed considerably. . . . Learning is no longer seen simply as being at the receiving end of the transmission of knowledge and information: rather, it is a process which requires the participation of the learner, which people approach in a variety of different ways, and which is linked to improving the quality of people’s lives.”
That the public itself may be increasingly sensitive to this distinction is suggested by some focus group research conducted in 2001 by students at the Winterthur Program in Early American Culture. In their study, focus group members reacted in sharply different ways to the notion that either a “teaching” or a “learning” component might be included as an integral part of a hypothetical museum visit. The prospect of being taught elicited an almost wholly negative response. But the prospect that visitors might be given an opportunity to learn was considered highly positive.
In that same Winterthur study, a second pair of words—“entertainment” and “recreation”—provoked a similar set of conflicting responses. “Entertainment” was viewed in negative terms; “recreation” was seen as correspondingly positive. Common to both of sets of words was the difference between being a passive museum visitor to whom something is done—you are taught, you are entertained—and being an active one who takes the opportunity to learn for herself and/or pursue her own recreation.
Whether a museum should primarily be a place for teaching or one for learning is something about which even its own staff may disagree. In my own experience, curators, particularly those with the heaviest academic credentials, incline toward what my Smithsonian colleague Zahava Doering has called the “baby-bird” theory of museum education. That’s an approach in which the museum, as an all-knowing mother bird, carefully chews up appropriate helpings of fact and opinion and then, beakful by beakful, doles them out to the fledgling visitor. Educators occupy a middle ground; for them, the museum is a place where a variety of teaching methodologies and free-choice learning opportunities can be fruitfully combined. At the other end of the continuum, the manager of visitor services cares only that some aspect of a visit—whether the shop, restaurant, parking, rest rooms, or even the objects on exhibit—makes a visitor happy and eager to return.
Let’s imagine three groups of visitors to an exhibition of Greek and Roman pottery. Members of the first group, though they found their museum visit generally pleasurable, had little or no interest in this particular exhibition. They walked through it quickly and paid scant attention. In consolation, think about what Indiana University’s Lois Silverman and three of her research colleagues wrote some years ago in the fall 1996 Journal of Museum Education: “Although visitors say they come to museums to learn things, more often than not the social agenda takes precedence. Quality family time, a date, something to do with out-of-town guests, a place to hang out with friends: those are some of the primary reasons people choose to go to museums.”
Our second group of visitors is prepared to invest time looking carefully at the objects and reading the labels. They may wait for a docent’s tour, pick up and study the gallery handouts, and have animated discussions with each other. Given the nice symmetrical fit between their interests and those of the curator, we can expect that they’ll enjoy their visits. In carrying out its educational role, the museum clearly will have served these visitors as a teacher.
My central concern, however, is with the third group, who include:
an elderly immigrant from Greece for whom the exhibition triggers an enormous outpouring of pride in her ancestors’ achievements;
two brothers whose younger sister has recently died and who find a degree of solace in the notion that, the potential brevity of life notwithstanding, art and beauty can somehow endure;
an amateur potter who regularly scavenges through ceramics exhibitions in search of forms that she might adapt to her own use;
a tourist for whom the exhibition evokes pleasurable memories of his recent trip to Rome;
and a pair of young lovers who amuse themselves by concocting improbable mythologies to explain the amorous and other activities depicted on the pottery.
Let’s hypothesize further that this third group did not do particularly well at learning to tell the difference between a Greek pot and a Roman one. But just because their agendas are not congruent with the curator’s, should we classify their visits as failures? Or should we be open to the possibility that the museum is a place in which some visitors may learn little or nothing and still find pleasure, others will learn more or less what the museum is trying to teach, and still others may experience something of value, even if it isn’t what the museum intended? In a great research university or even a police academy, this casual hodgepodge of the intentional and the accidental might not be tolerable. Daydreaming at police academy might even be dangerous. In the museum, though, we must remain sensitive to those peculiarly unstructured and frequently unexpected aspects of the visit that can make it such a different and idiosyncratic experience.
For example, Mike Wallace, a professor at the City University of New York, has suggested that the most important outcome for history museums might be in “helping visitors [to] develop their historical sensibilities, strengthening their ability to locate themselves in time, and enhancing their capacity as citizens to be historically informed makers of history.” The underlying notion is that one genuinely important outcome of a history museum’s exhibition program might be to enhance the capacity of citizens to govern themselves in a better informed way.
That notion could certainly be extended to other types of museums as well. Consider the Hall of Biodiversity at the American Museum of Natural History, which reflects the museum’s efforts—and these are the staff’s own words—“to alert the public to the critical roles biodiversity plays in sustaining life as we know it and to the ecological crisis we now face.” This is not just about contemplation or passive understanding but, rather, is a call to social and political action.
An art museum might help its visitors to hone still other skills. In a memorable address delivered at AAM’s 1980 annual meeting in Boston, the late Harvard philosopher Nelson Goodman argued that rather than simply cultivate an appreciation for art or engender an interest in art history, an art museum should function as an “institution for the prevention and cure of blindness.” The works of art displayed in such a museum contribute to that function, he said, when “. . . by stimulating inquisitive looking, sharpening perceptions, raising visual intelligence, widening perspectives, bringing out new connections and contrasts, and marking off neglected significant kinds, they participate in the organization and reorganization of experience, and thus in the making and remaking of our worlds.”
It Takes More than Good Intentions
Let us move on to the third proposition: that museums, faced with ever-more explicit demands for accountability, must develop credible systems of feedback that will allow them to monitor and report on the impact of their educational efforts. As they develop these systems, it seems to me that they must take into account the widest range of educational experiences possible in a museum.
In the case of our second hypothetical group, most of whom did learn how to distinguish a Greek pot from a Roman one, that should be relatively easy. Far tougher to deal with are the cases in which the museum’s intentions and its visitors’ experiences do not overlap quite so neatly. Like those wayward daydreamers in our hypothetical third group. Like the young Stephen Jay Gould whose life work was rooted in a childhood museum visit. In fact, most of us can remember a visit that indelibly marked the museum as the kind of place where we might some day like to work.
One caution should be noted: a museum might be better off with no system in place for gauging its impact than with a bad one that uses an inappropriate set of measures. This point was forcefully made in a May 30, 2002, Chronicle of Philanthropy article by Peter Frumkin, assistant professor of public policy at the John F. Kennedy School of Government. Writing about nonprofit organizations’ widespread and continuing failure to develop more meaningful measures for documenting the effectiveness of their “programmatic bottom line,” Frumkin warns that there may be a growing tendency to substitute financial measures instead. The great advantage of financial measures is that they can readily be connected to such familiar quantitative benchmarks as return on endowment, year-to-year fluctuations in membership dues, or the relationship of annual revenue to annual expense. “The problem that this creates in the nonprofit world is clear,” Frumkin writes:
From foundations and universities to hospitals and museums, nonprofit groups of all kinds, but particularly large institutions, are understandably led to focus on financial measures of performance because they are so much more concrete and robust than programmatic ones. They are also what outsiders can observe easily and compare quickly in sizing up one organization’s management compared with another.
Consider just how preposterous such financial measures would be if they were used to evaluate the Armenian Genocide Museum and Memorial, a new museum scheduled to open in downtown Washington, D.C., between 2007 and 2010. Today, historians (Turkish ones aside) accept that more than 1 million Armenians were killed by the Ottoman Empire during a 1915-16 campaign to uproot them from the territory that would ultimately become Turkey. Tens of thousands more were forced into exile, many coming to the United States. Descendants of this latter group have been primarily responsible for organizing the Genocide Museum and raising its estimated $100-million cost. Among the museum’s goals will be to make a historical record of the 1915-16 massacre, to develop a genocide-prevention component, to tell the story of the Armenian diaspora, and to celebrate the rebirth of the Armenian community in the United States and elsewhere. Its future location, a former bank building located two blocks from the White House, makes it clear that the audience for that story is intended to be far larger than the anticipated annual attendance of 250,000.
On what basis will supporters of the Genocide Museum measure its success or failure? Certainly by whether its programs make their intended educational impact on the public, and not by whether its revenues consistently exceed its expenses. Of course, a museum, like any organization, must be attentive to its revenues and expenses. What makes the museum so different from the button factory, though, is the inadequacy of those terms to define its success or failure. For the museum, the bottom line must be a programmatic outcome, not a fiscal one.
The case of the Armenian Genocide Museum also suggests that it may be simpler to establish a credible approach to evaluation when an institution’s goals are clearly defined and it can articulate with some precision what sort of an educational impact it hopes to make. Such clearly defined goals are by no means universal in the field. For older museums that trace their roots back to that time when a museum might, indeed, have been its own excuse for being—institutions established decades before the phrase “mission statement” had even been coined—the first steps in constructing a system of program evaluation may well involve the clarification of its institutional purpose. If evaluation is nothing more than a means to determine whether an organization is achieving its desired outcomes, a first step must be to ensure that all the important players—trustees, staff, and volunteers inside the museum; donors and visitors outside the museum—share some basic understanding of what those outcomes are.
One financial aspect must, however, be addressed in even the most program-based system of evaluation. That’s the question of cost effectiveness. Suppose a regional museum in rural Texas proposes to do an exhibition about tornadoes. Its educational objectives are clear. It wants visitors to learn how to anticipate a tornado and prepare their homes and look after their family’s safety in the event one occurs. In seeking support, though, the museum may be asked not only about its intended educational impact, but also whether an exhibition is the most cost-effective way to impart that knowledge. What if, at a fraction of the exhibition’s cost, somebody else could teach as much to as many people, in as memorable a way, through an illustrated publication, a videotape, or a television special?
In the past, when museums and other nonprofits were focused on outputs rather than outcomes, that was less of a problem. Museums could point to a unique output—the exhibition—that nobody else could match. Today, however, organizations with the most varied of outputs might pursue a common outcome. The downside for museums is that they may face competition from less costly providers. But the upside, which to my mind is far more than compensatory, is that a focus on out-comes may open the door for collaboration. If several community groups have a common and concurrent interest in tornado awareness, for example, perhaps their collaborative impact will far exceed the sum of their separate achievements. In the magical world of collaboration, two plus two can frequently make five.
Concerning Frumkin’s observation that not-for-profit organizations have failed to develop meaningful measures to document the impact of their programs, that certainly has not been due, at least among museums, to any lack of serious effort. For a variety of reasons, though, museums have turned out to be difficult cases when it comes to evaluation. One problem was pointed out by Jay Rounds of the University of Missouri-St. Louis in the July/August 2001 issue of Museum News. There is, he noted, an almost complete disjunction between evaluation techniques that are essentially verbal and numerical and museum-going experiences that may be primarily visual.
Another problem relates to timing. The impact of museums on their visitors tends to be cumulative over time, subtle rather than obvious, indirect rather than direct, and more often than not deeply entangled with the impact made by a myriad of other community organizations. The most apt analogy might be to a liberal arts college. Determining the ultimate impact of a museum visit through an exit interview is like determining on a student’s graduation day what her liberal arts education might contribute to society. It may take, in fact, the better part of a lifetime for that contribution to become evident. Think again about the 5-year-old Stephen Jay Gould. Think about your own experiences.
To add to this complexity, the variety of objectives that are pursued by museums today is astonishingly diverse, not only between museums of different disciplines but even between museums within a single discipline. One art museum may aspire to induce in its visitors a purely aesthetic response; another might envision itself as a teaching institution. A third art museum might center itself around creativity; a fourth might seek to interest visitors in the interplay of such aesthetic concepts as complexity, unity, and intensity. And a fifth might aspire to Nelson Goodman’s ideal: to be an institution “for the prevention and cure of blindness.” Add to these the equally diverse agendas of history and natural history museums, science centers and children’s museums, and you will begin to grasp part of this complexity.
To grasp the full complexity of museum evaluation, though, requires multiplying those institutional agendas by the equally diverse personal agendas of museum visitors. Some come to browse pleasurably, to be with friends or family, or to enjoy one of the few community spaces where they can safely share the company of strangers. Some come specifically to be taught. And some come because the museum offers them a particularly stimulating environment that invites them to learn about life, about stuff, about themselves.
What this complexity suggests is that, over time, the museum field will need to develop a vast arsenal of richer and more persuasive ways to document and/or demonstrate the myriad and beneficial outcomes that may occur for their individual visitors and have impact on the community beyond. Some of these ways may be quantitative but, to the horror of some social scientists, a great many may be anecdotal or qualitative. What is critical is that these evaluation techniques fit the real complexity of what museums actually do. It would be a calamity beyond telling—the equivalent of the “teaching to the test” phenomenon in our public schools—if museums were to dumb down their work to squeeze into the straightjacket of whatever assessment tools happen to be available.
Where does that leave us? Hopefully, with this understanding: that outcome-based evaluation, at least at this stage, is not about the use of any specific technique, or metric, or the imposition of any particular management style. It is, rather, a way of thinking, what Institute for Learning Innovation Director John Falk referred to as “state of mind” at the 2002 AAM Annual Meeting in Dallas. From this point of view, the primary measure used to evaluate a museum’s worthiness is the positive and intended differences that it makes in the lives of the individuals and communities that constitute its target audiences. That is the outcome, the achievement, toward which every activity undertaken in the museum should be aligned.
The critical point is not how this achievement should be measured but, rather, that striving after such an achievement becomes and remains the museum’s central focus. A museum may have a championship-caliber staff, a splendid building, superb collections, great management, great programs, great everything. But if it makes no difference to anybody, if it has no impact, if no good outcomes follow from what it does, then all it can be is a great so what?: a gorgeous and resplendent wheel spinning prettily in the air.
As recently as two generations ago, organizations like museums were primarily about good intentions. The vocabulary used to describe them centered on such words as philanthropic or charitable or benevolent. Bene volent—to wish somebody good! Those days are gone. Museums are no longer their own excuse for being. As the resources they require have become greater and greater, so, too, have the expectations of those called upon to provide those resources. What is demanded today is that organizations perform, deliver, and demonstrate their effectiveness. Good intentions may still be where museums begin but, today, positive impact is where they have to finish.
Stephen E. Weil is scholar emeritus, Smithsonian Center for Education and Museum Studies, Washington, D.C. His most recent essays are collected in Making Museums Matter (Washington, D.C.: Smithsonian Institution Press, 2002). This article is adapted from his presentation at AAM’s 2002 Learning in Museums seminar in Nashville, Tenn.
