This article originally appeared in the May/June 2019 issue of Museum magazine, a benefit of AAM membership.
If you are reading this magazine article, you know that competition for museum resources is increasingly tough. A relatively small group of funders is flooded with requests from many worthy endeavors. To help in the decision process, many funders now require museums to demonstrate relevance and impact with evidence-based data.
When museums talk about making an impact, that often covers the educational, economic, and social realms. Of the three, social impact is perhaps the most difficult to measure with data-based evidence. Americans for the Arts and the UK’s Museums Association have done extensive work gathering research and case studies that document the social impact of arts and museums. And in August 2018, the Institute of Museum and Library Services announced a project to measure, on a national level, the conditions under which museums and libraries contribute to quality of life and well-being in the communities they serve.
Demonstrating social impact is similarly on the minds of museum professionals in Utah. In 2017, Utah Division of Arts & Museums (UDAM) partnered with Thanksgiving Point, a multi-museum complex in Lehi, Utah, to conduct a pilot study to determine what social impact the state’s museums have on the communities they serve. The goal of the study, which is still in progress, is to develop a tool that measures how attending Utah’s museums can improve family and personal well-being, health, relationships, empathy toward others, knowledge, and understanding of one’s community and others in it.
The Study’s Catalyst
Strong partnerships are valuable in a major undertaking such as a social impact study. UDAM has tapped into the experience of Thanksgiving Point, a farm, garden, and museum complex along Utah’s rapidly growing Wasatch Front.
In 2015, inspired by the success of the Cincinnati Museum Center’s 2014 economic impact study, Thanksgiving Point retained an outside consulting firm to conduct a similar study. The firm also
recommended a social impact study to paint a holistic picture of Thanksgiving Point’s impact in the community.
Later in 2015 and into 2016, Thanksgiving Point partnered with a professor and graduate students from Brigham Young University’s Public Administration program to conduct the social impact
study. The students worked with Thanksgiving Point to develop a logic model, a measurement tool, and a plan for carrying out the study. The study was designed to measure short-term outcomes
that research has shown lead to long-term outcomes such as stronger families, lower drug use, higher educational attainment, and a greater sense of health and wellbeing. These are, in turn, indicators of strong and healthy communities, which policymakers, legislators, and other grantors want. Thanksgiving Point hoped to share positive findings with these stakeholders and garner their support.
Thanksgiving Point was pleased with the study’s findings, which demonstrated that it was indeed making a positive social impact with its visitors. Thanksgiving Point used these findings alongside the results of the economic impact study in a pamphlet that was given to Utah state legislators during both the 2016 and 2017 legislative sessions. Over those two years, Thanksgiving Point received a total of $1.5 million in state funding to support the construction of a fifth venue, the Butterfly Biosphere.
In 2016, UDAM and Thanksgiving Point began discussions on partnering on a similar but larger social impact study that would involve museums across the state. Early in 2017,with funding solidified, UDAM and Thanksgiving Point formed a partnership to help Utah’s museums show their social impact.
Getting to Work
The first phase of the study involved conducting a pilot with a small, representative group of museums from Utah. The goal of the first phase was threefold:
- demonstrate the social impact these museums have in the community;
- test the validity of the social impact measurement tool;
- and discover the time, effort, and money required by UDAM and participating museums to conduct this kind of study.
The second phase, anticipated to begin in summer 2019, will use findings from the pilot to determine the best ways to expand this project throughout the state and ultimately create a tool for museums—from our small, volunteer-run historical societies to our large, accredited museums—to use independently.
One of the first steps UDAM took in customizing Thanksgiving Point’s study was to re-create the logic model governing the study. Thanksgiving Point has a very nuanced and thoughtful understanding of its own audience and mission, and the logic model from its earlier study reflected that. This project needed a logic model that reflected the varied audiences and goals of the museums throughout the state that would be invited to participate in the pilot. Together, UDAM and Thanksgiving Point drafted a logic model to share when recruiting museums to participate as host sites for the study.
In early 2018, UDAM staff began reaching out to museums across Utah to gauge their interest in participating in the study as host sites. The final cohort of eight museums reflects those UDAM classifies as both small and large (ranging from 1.5 to more than 50 employees). They are mostly located along the Wasatch Front (the most populous area of Utah) and include art, history, and life science museums; an aviary; a garden; and a planetarium.
In June 2018, UDAM led an introductory training for the point people from each host site. Our research team worked with them to refine the logic model and corresponding end-of-experience survey (more on that later). We also provided training for each participating museum on how to recruit participants. The museums were allowed to recruit both individuals who had attended their museums and those who had not. Participants would be allowed to visit their host museum at least three times, free of charge, within three months.
A minimum of 40 survey responses were needed to perform tests of statistical significance for an individual site. We suggested that each host museum aim to recruit 100 participants to allow for attrition. We also discussed the logistics of monitoring the participants’ progress in a centralized database. In September, UDAM staff conducted on-site training at each of the host sites for the front-of-house staff who would be interacting with participants. This training covered questions they might field, the free admission for participants, and using the shared data-tracking systems.
Beginning in the fall of 2018, participants started visiting their host museums. Each participant received a survey at the end of the three-month test period, after their three visits to their host museum. The survey measured more than 100 indicators of social impact categorized and connected to the long-term impacts (see logic model on p. 29). Using a retrospective post-then-pre-survey design—which allowed participants to mark how they felt after visiting the museum and compare that to how they felt before visiting the museum—these indicators measured if there had been a statistically significant change in the intermediate outcomes listed on the logic model. These intermediate outcomes, according to research, lead to the long-term outcomes/impacts on the logic model. The survey also included open-ended questions regarding the participants’
experiences and what they learned.
Considering the self-reported data and absence of a control group, we do not intend to claim causation with our findings. However, showing correlational change is still a strong argument. The second phase of the study will explore how to scale the study process to fit a variety of museum sizes.
Lessons Learned So Far
Conducting a social impact study with museums of various sizes has had its challenges. Following are a few of the things we have learned along the way. Partnerships can fill in the gaps. Partnering with Thanksgiving Point, with its research experience and prior similar study, strengthened the validity of UDAM’s work.
Communication is key—and time-consuming! The research team knew that communicating with and organizing the eight host museums and their study participants would be challenging. UDAM was responsible for coordinating training, developing marketing materials, and managing participant contact lists. Additionally, UDAM was responsible for all direct communication with study participants. This included thousands of emails related to accepting participants, encouraging them to attend, and reminding them to complete the survey following their visits.
Every museum has its own motivations. Some museums hope for audience data to help them determine future programming. Others want data about the impacts of their programming to help them make stronger arguments for funding. Still others want to investigate their place in their community and reflect on their mission in a data-driven way. It is important to understand these motivations and try Utah Division of Arts & Museums to accommodate them, while also being clear about reasonable expectations from the data.
Museum size makes a difference. The research team designed the project so that UDAM would administer the study to make the project manageable for small museums with limited staff. Still, some smaller or more rural sites struggled to recruit participants. It is possible that to develop a universal tool, larger museums will need to do more of the early work.
Understanding and, more importantly, measuring an organization’s impact is an increasingly important topic across the cultural sector. Many funders, including UDAM, give preference to organizations that can illustrate not only their economic but also their social impact in the community. Developing a tool that all museums can use to do this is an increasingly valuable—and necessary—endeavor.
Comments