As the national Measurement of Museum Social Impact (MOMSI) study comes to a close, come learn how study sites navigated visitor participation in the project. Hear from three different museums – a rural museum, an art museum, and a zoo – on their successes and challenges in all steps of the project, from recruitment to participant retention. No study is perfect, and the recommendations we share just might help you on your next evaluation project. This webinar is co-hosted by the Visitor Studies Association.

Transcript

Ann Atwood:

Hello, and welcome everyone to this CARE-apy session on how to measure museum social impact from participant recruitment to retention. This webinar is co-produced by the Committee on Audience Research and Evaluation, one of the American Alliance of Museums Professional Networks, and the Visitors Studies Association. This webinar is intended to connect our members and share professional expertise. My name is Ann Atwood. I use the pronouns she/her, and I work with the Museum of Science, Boston. Today, we’ll be talking about how research and evaluation studies allow museums to engage their communities, and particularly strategies used by teams from the Measurement of Museum Social Impact study to do this. Focusing on recruitment through retention of participants. This conversation builds on 2:00 AM blog posts, which are being linked in the chat right now if you’d like to read those later.

Today, we’ll be talking with Michelle Mileham, the project manager for the Measurement of Museum Social Impact study. Alice Anderson from the Minneapolis Institute of Art. Sabre Moore from the Carter County Museum, and Amy Niedbalski from the Saint Louis Zoo. A few quick housekeeping items before we start. We’ll be recording today’s session, and we’ll have time for question/answer at the end. So, in the meantime, I’ll keep you all on mute. If you have questions as we’re going in the conversation, please go ahead and drop those in the chat and we’ll answer them as best we can during the conversation, as well as after in the Q&A session. We do plan to make this webinar available on AM’s YouTube, and it should normally be available within three to six weeks. The webinar is kept free of charge, so we’d like to encourage you to consider membership to the American Alliance of Museums and the Visitor Studies Association. With that, I’ll turn it over to Michelle. Thanks, Michelle.

Michelle Mileham:

Thank you so much for the introduction, Ann, and welcome everyone. We are so glad you could join us today. My name is Michelle Mileham. As Ann said, I use she/her pronouns, and I am the project manager for the Measurement of Museum Social Impact study. Just an introduction slide here. As Ann mentioned, and as Amanda has dropped in the chat, we are building this webinar off of two previously published blog posts. So, we’re going to talk a lot today about participant recruitment and retention, and of course, we’re framing this conversation for this particular study, but hope that whatever kind of social science research or evaluation you may be doing at the moment or project to do, that this information might be helpful to you in those circumstances.

So, I am joined today by three staff from three of our host museums and I’m going to let them do a brief introduction of themselves and their sites, but I just want to recognize a few things. This is recognizing three of our 38 host museums. I’m going to go a little bit more into depth in that in a future slide, but these represent all free museums. We actually did not intend this when we planned this webinar, but when we started talking through what we would present, recognized that all three of these museums happen to be a free admission. They also represent both urban and rural populations and are in some different geographic areas. So, we’re really excited to have them here and sharing their experience with us. I think, Amy, I’m going to turn it to you first.

Amy Niedbalski:

Sure. We decided we wanted to give a little context about each of our institutions just in case anyone is not very familiar with us. Saint Louis Zoo was founded in 1910. We were founded after the 1904 World’s Fair took place in Forest Park, and there were some remnants from that that ended up developing into the Saint Louis Zoo. We are one of three large association of zoo and aquarium zoos that are free to the public. That is in addition to Lincoln Park Zoo and National Zoo in Washington DC. Lincoln Park is in Chicago. We do have several fee-based attractions, so it is not a completely free visit. Parking does cost money, but you are able to park on the street for free and bring in your own food and snacks, and drinks.

So, free visits are possible. We did boast and host 3 million guests annually pre-COVID. We had one 3.5 million year, which we were very proud of. It was a very nice weather year. We have been back up to 2.3 million in 2021 and in 2022, and we have 50,000 plus member households, which is interesting because we’re free, so what is the benefit of being a member? That’s a whole nother study. We do have our public support and it’s very heavily expressed in our social media channels, in our local media publications, and television. We are supported via property tax in the city and county since the 1970s, and that is along with our sister institutions, art museum, history museum, science center, and botanical garden, which are all located in or around Forest Park.

The sales tax, just as an example of our ardent public support, in the climate that was in Missouri in 2018, every single precinct in St. Louis County voted majority vote, a positive vote, to pay a sales tax directly to the Saint Louis Zoo with every purchase that they make. So, that was amazing. Those taxes make up about 35% of our revenue, and contributions, philanthropic donations, also make up about 30%. The additional come from guest revenue and investment revenue. I think I was going to say also that our economic impact is $200 million to the region annually.

Michelle Mileham:

Great. Thank you, Amy.

Amy Niedbalski:

Mm-hmm.

Alice Anderson:

Just up north in Minneapolis, I’m Alice Anderson. I work at MIA as the manager of audience research and impact. That’s my full title. Similar to Amy, as I was looking at her slides, I was like, “Well, there’s a lot of similarities here.” We were founded in 1915, so we had a very exciting birthday. 100 years in 2015 with a lot of fun events. We are also free admission and have been most of that time. We do have one fee-based special exhibition space, and that is one where we drive a lot of attention and activity towards. Pre-COVID, we were getting 700,000 guests there, or visits I should say. Obviously, much less in the past couple of years. Our member households, Amy, similarly, I think membership to us, we do give discounts at the store and our cafe. Any programming, there’s free member days, but that sort of your supporting an institution that you care about is more of the, I think, sell for us. Similarly, we get tax revenue and philanthropic donations, of course. I personally work a lot with our grants funding exhibition or program development and evaluation.

Michelle Mileham:

Thanks. All right, Sabre?

Sabre Moore:

Hi everyone, my name is Sabre Moore. My pronouns are she/her. I’m the executive director of the Carter County Museum in Ekalaka, Montana. If you look at our logo in the bottom right, Ekalaka is just above the toe of the triceratops in that bottom right-hand corner. We always have to say that because Ekalaka is a rather unknown community in the state. There’s only 400 people here, and until 2010, we were literally at the end of the highway. So we say, “All roads lead to Ekalaka.” Works out pretty well. You don’t happen here by accident, it’s definitely a destination. The Carter County Museum was founded in 1936. It was the first county museum in Montana and the first to display dinosaurs. The reason that we’re the first county museum, is that our director, he was at the state senate and passed legislation in order to get county museums funded in Montana. So as soon as that legislation passed and was signed into law, he raced home, filed the paperwork, and here we are.

We have free admission. Prior to COVID, we welcomed 6,000 guests annually, which is a very big deal in a community of 400, and in a county of only 1,300 people. We’re the primary tours and driver here. In 2022, we started climbing back up after the COVID downturn, to 5,302 people. We reached an additional 100,000 people through educational programs and traveling exhibitions. Those programs vary anywhere from virtual to in-person. I travel all over the state. I go west a lot and drive seven hours to Helena and things to make sure that people are hearing about Ekalaka, and then coming and visiting us. We have ardent public support as well. We’re supported via property tax in the county through that wonderful bill that I mentioned earlier, as well as philanthropic donations from our 80-member household, still pretty good, and the grants that we get that are mostly project-based.

Michelle Mileham:

Great. Thank you all. So, we’re going to shift just a little bit into some project background, so a little bit more about the study itself. The Measurement of Museum Social Impact study, or as we like to call her, MOMSI, it’s a mouthful otherwise, is built upon two Utah-based pilot studies. MOMSI is funded by an IMLS National Leadership grant, it’s a research grant. The goal of the project is to scale up and test a museum social impact survey tool that was designed here in Utah and see if that is applicable at a national level. So as I mentioned, we have three of our host museums here. This slide shows our whole cohort of 38 museums, and you’ll recognize some names on here hopefully, some of you all as well. But what we tried to do, was really get regional representation. That was a main goal, as well as content area diversity. So, you’ll see on here we have children’s museums, science center, zoos and aquariums, art museums, historic houses. If they were in our applicant pool, we really wanted to make sure we had that kind of diversity of types of museums.

We were also really aware of size, and so that was really both in the size of the museum itself, how many staff they had. So some of these had pre-COVID, 400 or more staff members. Some are entirely run by volunteers, and that was a really important aspect for us to take into consideration. Again, as we’re just scaling up and testing this kind of study at a national level, is this something that can even be done at museums of different content areas of different sizes with different budgets and staffing resources? So those are all things we wanted to take into consideration. Once we had these host museums, we did have specific asks, that as a study we requested them to meet as part of their participation. So, they did have to be open to the public. We did not want virtual experiences to count as part of the study. That was because, again, we’re building off of two previous pilot studies that were pre-COVID that required visitors to be in-person. For consistency, we wanted to measure and use the survey under the same circumstances, so they did have to be open to the public.

They did have to recruit at least 100 participants, and you’ll hear more about that from our three host museums. Then, they did have to allow those participants, and at least one guest, free admission to their museum for three separate visits. So of course, and I think our host museum folks will talk a little bit about this, they’re already free, so that was not a hurdle for them. But for our other host museums that did have a paid admission fee, this was something, and a revenue source especially coming out of the pandemic, that they were giving up as participants and host museums in this study. I just want to share a little bit about our timeline. We recruited host museums at the start of 2021. As many of you can imagine, there was a lot of uncertainty in the museum field about closures due to COVID. They were ongoing. Some states or cities still had mandates in place about what museums could do and how they could operate at that time. So, we were really fortunate that we ended up with a pool of about 70 museums who applied to participate in the study.

We selected about half of them into our cohort. All of the museums that were selected underwent a pretty short half-day training just to understand the expectations of the study, ask us questions, things like that. Once they completed that training, we opened a survey for host museums to recruit visitors, which became the participants in the study. So, the recruitment process was really managed by the host museums themselves, but we managed the survey that people applied through and kept the list and rosters of the applicants of participants. We also provided logos, images, language, anything we could to at least help museums with the recruitment effort. Once participants were selected into the study, they had anywhere from about six to eight months to complete their three visits. We didn’t have any kind of system in place, they could complete those visits at their leisure. Then, after they completed those visits, our project team, again, was responsible for sending the social impact survey. So, that was all done via email, again, from our project team, not the host museums.

That survey was taken by participants, and then our project team is analyzing that data and then writing up reports and sharing it with our host museums. So, part of why I wanted to show this timeline, is that this is a very complex study. We have 38 museums. There are a lot of moving pieces, and I just want to recognize that this timeline was not what we set out for. We ended up having to spend a lot more time on recruiting visitors than we had initially planned for in our grant. We extended that recruitment period, I think, by about three months. Again, just trying to understand public’s willingness to come back into indoor spaces. Then, we had to open the study and have them complete their visits for a few extra months. Recognizing we had some outdoor museums in cold climates. Having people visit in winter months was not ideal. So, keeping the study open through some summer warmer months was really the best option.

This is really just to say that, you can have the best intentions of what your study might look like, but to be really flexible and understand where you might have some wiggle room to make exceptions undertaking a large research study like this. We understand that there’s a lot of ways to define social impact. For the purpose of MOMSI, we just put a stake in the ground, and we define social impact as the effect of an activity on the social fabric of a community, and the well-being of the individuals and families who live there. We measure this through four long-term outcomes. A continued learning and engagement, health and well-being, strengthened relationships, and valuing diverse communities. I’m just going to give a little preview of what this looks like from a visitor and survey participant. Again, for each of these four long-term outcomes, we have indicator statements, and I’m just going to show a little sample of some of these indicator statements that participants are responding to.

So, for continued learning and engagement, there are statements like, “I wonder about how things work. I incorporate recently learned information into my day-to-day life.” For increased health and well-being, statements include, “I am able to bounce back from adversity. I am open to new ideas. I am confident in contributing my opinion to a conversation.” Strengthened relationship indicator statements include, “It is easy for me to develop social relationships. I make it a point to spend time with my family and/or friends. I build strong and supportive relationships with a variety of people.” Finally, valuing diverse communities include indicator statements like, “I learn new things from people who are different than me. I understand how cultures are similar and different. I am aware of challenges faced by others with backgrounds different than my own.” I’ll leave this QR code up on the screen for just a second if you want to snatch a phone and take a shot of this. You can see more indicator statements. It’s still not all of them, but a few more examples, as well as the six open-ended questions that we ask on the survey if you scan that QR code.

If you miss it now, we’ll put it up again at the end of our presentation. Like I said, our project team is responsible for all of the data analysis, and we’ve been working hard at this for the last several months. I just want to really briefly share what we’re finding from the aggregate data, so the information from across all 38 of our host museums. We’re happy to say that all four long-term outcome areas had statistically significant positive change. So, you can see the number of indicators for each of those four long-term outcomes. Of course, it’s a really large sample. This is about 2,000 survey responses, so we know that this is not what we’re going to see for each individual museum. We are seeing some changes in some of these long-term outcomes based on sites specifically. For the qualitative data analysis, one of the open-ended questions that’s on the survey was, how does this museum benefit your community?

I wanted to share just a little bit about this one, because what we were seeing when we’re analyzing this data, and we are using a bottom-up approach, and so we are just letting the data speak for itself, grouping these into like-themes and content, and we started seeing our four long-term outcomes emerge from this data. So in the responses, we are seeing participants saying health and well-being is a benefit. “It’s a safe space to take a relaxing stroll.” We saw continued learning come up a lot in this question and in responses. “It’s educative and interactive, pointing out scientific questions related to daily life.” Then, valuing diverse communities. “It offers an opportunity to learn about events of the past and about people different than our own.” So, that was just really exciting for us as we started to dive into the qualitative data. Those are of some examples and themes that we’ve pulled out from that. I’m going to turn it over to our three host museums now as little case studies. Amy, I think you’re first and just let me know when you want to advance.

Amy Niedbalski:

Sure.

Michelle Mileham:

For participants, if you have any questions now or throughout their presentations, feel free to drop them in the chat.

Amy Niedbalski:

Yeah, if you could go ahead and hit the text onto the slide? I don’t think I animated anything. As I stated before, we are quite a large institution with quite large attendance, and we very much wanted to try to recruit individuals who were not typical zoo goers. It was difficult for us. I meant to mention that, I do have capacity of three full-time staff on the research and evaluation team at the zoo, but we do community studies annually. I’m representative of St. Louis City and County, and we find that probably 95 plus percent of individual residents have been to the zoo before. So, it’s really hard to find someone who has never been, but over half of St. Louis residents come at least annually. Again, it’s kind of hard to get to those people who aren’t zoo goers, right? One of the challenges we had was that, what could we offer as an incentive? We put in our proposal or application to be part of the study in this same year that we were conducting the study, so I hadn’t rebudgeted for any type of incentive.

We usually budget a year in advance, so the only thing that we had to offer were free memberships. The membership was not able to give 200 plus free ones away, so we did have a drawing. I really wish that we had been able to offer a better incentive, or an incentive for at least everyone participating, so that is something that I was not happy with. However, we were able to still recruit. Again, trying to do that recruit and trying to get some folks who don’t typically come to the zoo, our approach was working with partners at St. Louis City and County libraries who we partner with frequently. We did utilize Michelle’s… The flyers and the information and the logos that she gave us for from her media kit. We did do flyers in the libraries in Spanish language and in English language. I think some of the librarians very actively tried to recruit, while others just put it on their peg board and let people see. After a few weeks of that, we generated very few responses, so we decided, “Oh, we need to more actively recruit from contacts that we do have.”

Being a free institution, there is no admission, so people don’t have to stop at the gate. We don’t have information about our visitors, besides from our exit studies, so we do annual year round exit studies for satisfaction ratings, things like that, and for our demographics. But it means the people that I can communicate with online are our members. So, definitely a convenient sample and definitely people who are committed to the zoo. However, during this time of COVID, we had introduced what we called, a reservation system. In order to control capacity in our crowds, we only allowed a certain number of people to come in to the zoo each hour to allow for social distancing. That was a really difficult procedural process to put into the institution, but we ended up changing that over the timeframe to block visitation, so you can sign up to come in the afternoon or in the morning, rather than hourly. People really seemed to like that because there’s a lot lower crowds.

Our regular visitors really, really liked the lower crowds, and so we didn’t take it away, we didn’t take the reservation system away. We reopened in June 2020, and kept the reservation system through 2022. We’ll get to that. Anyway, so I was able to utilize those email addresses from the zoo reservation. The individual visitors could select whether they wanted to receive communications from the zoo or not, so with their permission we were able to recruit using those email addresses. So, it wasn’t only a sample of Zoom members, it was a representative sample of St. Louis residential visitors that we recruited from. That made for a fairly easy recruit. I think we got our 200 from sending out just a few email blasts fairly quickly. I think we started later than most people because we knew it wouldn’t take us that long once we realized that we had this database to work from. We can go ahead to the next slide. So, retention. We have so many staff. We were really concerned about people checking-in to our two entrances, our welcome desk.

We just thought it didn’t feel like it was going to be easy or convenient or efficient tracking method. So, very kindly, Michelle took it upon herself to… I would send her all of our guest email addresses, so no personally identifying information, and that was covered by our IRB to be able to do that, but she was matching participants with many of the tens of thousands visitors that we had. Every three weeks I think we did it, Michelle. It was quite cumbersome, I believe, but it allowed us to track to be sure that that participants were actually coming and fulfilling the visits. Michelle and her team did… I don’t know how frequently you did email communications, but they did very nice. They would get on our website and find events and say, “Come to the zoo for this,” as just an incentive, a reminder. Say, “Hey, we’re here. You’re part of that study. Come do this.” Reminders of the incentive and reminders that this helps the zoo. I utilize that strategy a lot and it really works for our visitors and members.

As we were going through this study, one day I was at a meeting and zoo leadership said, “Let’s stop the reservation system. Pandemic’s getting over. Everyone should be able to come to the zoo that wants to come to the zoo.” That was surprising to me, because I thought we had committed to longer and to keeping it in place. We had four days, right, Michelle? We had to completely change our tracking methodology, so very quickly I worked with our creative services team, and we made, you can see in the picture here, these signs with a QR code on it so participants could log their visit through the QR code. That was probably easier for Michelle. She emailed everyone and let them know that change, but that could have definitely caused some issues with our numbers. People could have still come and not done that. Anyway, that was our only frantic, really difficult thing that happened. We had great retention. I believe the numbers are at the end of the presentation, but we did have 81% who completed their first visit and were sent the survey. Next steps, I believe, is the next slide.

Michelle Mileham:

If I can find my mouse. There we go.

Amy Niedbalski:

I just got preliminary results a few days ago, so it’s really exciting. It’s not all the results, but it’s just the percent of indicators that they saw a statistically significant difference from pre to post. So, we just set up a meeting for Michelle and her team to present to the Saint Louis Zoo staff in a few weeks. Yeah, I think it’s in April that we’re doing that. I really was thinking about those indicator categories of health and well-being, valuing diverse communities, education and engagement, strengthening relationship, and trying to decide who this initial presentation should be to. I have a few departments there that will be included and invited, but I’m typically welcome to present wherever I want to present data. So, I will be doing presentations of our results to our leadership team, which is director level at the zoo and above, our strategic operations, to our vice president level and above, and also likely to our boards.

I really want to get the information out to staff as well. Sometimes we do what we call, brown bag luncheons, where zookeepers or other folks who have very specific lunch timeframes, to be able to get away during the day. You bring your lunch to the brown bag and we can do a presentation there to let them know about the study and our results. We also have a quarterly meeting for all staff of the zoo, so very likely we’ll be able to present there as well. I do not know how we will share with the community, but we will give it a whirl and make those decisions likely again through all these internal discussions, and figure out how we can best do that. It might be through a press release. A news story may be the best way to share it. But, what to do next? We’ve never measured social impact before.

We do a community study where we ask general, again, St. Louis residents about different activities that a modern zoo does, such as, educating visitors, saving endangered species, offering wonderful care to the animals in our care, things like that. We ask importance of the various measures, and we ask how well you think we do at it. We get some interesting perceptions there from our community about what they think we should be doing and how well we are doing it. We’ve based some strategic decisions based off of that data, so I foresee us doing something similar in the same way utilizing this data. I just don’t know what that looks like yet because we haven’t done it. So, I’m hopeful that people are really excited. I mean, it definitely shows really positive impact, but what to do beyond just showing your positive impact, is continue to improve it. We will see, maybe that’ll be a webinar next year. Okay, that’s all I’ve got. Thank you. Are we stopping for questions in between?

Michelle Mileham:

We won’t pause, but if anyone has questions, feel free to drop them in the chat. We can either come back to them or Amy can respond directly there as well.

Amy Niedbalski:

Thank you.

Michelle Mileham:

Thanks, Amy.

Alice Anderson:

I think I’m next at MIA. Michelle, I think I made those slides with some animation from our old presentation. Basically, what this was trying to show is that, when we started thinking about recruitment with our staff who does both community engagement, as well as marketing outreach, PR, we had some major staff turnover. So, the kind of more grassroots, like working with community partners, to recruit folks who maybe hadn’t been to the museum or isn’t a very high frequent visitor, just weren’t possible the ways that we would normally do that. I should say that, one of our goals was to get 50% of participants who had never been to the museum and 50% of participants who are regular visitors. This opportunity to participate in MOMSI came on the heels of a neighborhood study we had done where we had mailed out surveys to our neighbors, our neighborhood, just get a sense of their awareness, their usage of the museum, and their attitudes towards MIA. What we learned from that survey was that, over 90% of people that responded to the survey already knew about MIA and had visited at least once.

So, we have very high awareness, we think within the Twin cities broadly. I’ve been in focus groups with folks who have been like, “Yeah, I guess I’ve just never been here. I’ve lived in the Twin Cities my whole life.” So, we’re really hoping to reach more people, but it just wasn’t possible. I think out of the 110, we ended up recruiting… Only 10 folks said that they hadn’t visited before. What we did instead was, we did a Facebook ad, both in English and Spanish, and set a geo target around 50 to 70 mile radius. This had really good visibility, as they say. It reached 160,000 people. Let’s see. We also put a post on Nextdoor, the social media platform for what’s happening in your neighborhood, and we also emailed a portion of our email list, as well, to recruit. We gave a $50 Visa gift card as an incentive, and that is typical for what we do when we do focus groups or whenever we’re doing visitor research that we say takes longer than 10 minutes. We’re going to give you something.

So, if we do an interview with you, we might give you a $40 gift card. Focus groups, we give out $75 gift cards. So, we felt like $50 was about appropriate for something that most people would be willing to do already. Come to the museum a few different times, but spend the 20 to 30 minutes filling out their survey. I think we can go next. We have a really wonderful visitor experience staff. This is a picture of our entrance, so if you walk right in, you sort of see this white cafe bar kind of thing, and that’s where a visitor experience staff sit or stand, but you don’t have to interact with anyone to enter the museum. You don’t have to show a ticket or check-in. We needed a way to let MOMSI participants know that they had to check-in. So, in their confirmation email, we sent them them instructions about, “Approach the desk.” We also sent some little punch cards that Michelle had helped us with, and that was really great. Our participants came between January and August, which overlapped two of our special exhibitions.

We had a couple moments to remind them of, “Hey, something new is happening at the museum.” Michelle really helped with communicating with the participants. You see at the bottom, folks who… We recruited a little over 100 people, but only half of those ended up filling out the survey. That survey was sent to anyone who had visited at least once. I don’t know, it’s interesting to see the number of folks as it decreases over who visited once, twice, and three times. I think I’m ready for the next slide. Okay, so what are we going to do with our data? We get to meet with Michelle next week. I have a colleague in data insights and strategy, named Rachel Wolf. Many of you may know, she is our next PN director chair. I forgot what her title is. She and I are just going to meet with Michelle to think about, “Okay, how do we want to position what we’re learning here,” and ask all those nitty gritty data questions before we share it. But then we want to present to our leadership team, as well as an all staff meeting.

We’re thinking about framing it in two related things. One, is that we’ve also been a part of a field wide study around the concept of belonging that’s been led by folks at the Museum of Science and Industry. We have data back from that as well. We think these concepts are related, because it’s all about, how do I feel I can show up? To what extent do I feel like I can be myself in this space? Therefore, what does that allow me to experience as part of the visit to the museum? Therefore, what impact might this have on me and my community? The other piece, is around strategic plans. So again, when we applied for the MOMSI participation, we were in the end of our previous strategic plan, which had, one pillar was around engaging communities. So, this was a really high priority, that’s why we did this study of our neighbors, and we were really invested in trying out different community outreach and partnership approaches.

Because of the pandemic and because of closure, and we have a new leadership team and a new director, our new strategic plan center is around a rebuilding effort of bringing museum visitors back. Our three pillars now are, inclusion, excellence, and sustainability. So, we want to think about how this study around social impact fits within that, because it didn’t come directly out of where we’re at right now. Next, we’ll probably similarly do a brown bag lunch for folks who like to dig into specific questions with us. We know we work with a lot of curious folks, so we’re always happy to provide some critical thinking. Then finally, this is one thing we’re thinking about, which is to bring together a cross-functional team that’s focused on that audience growth and development focus, because we have had some staff turnover and we’re rebuilding those teams. We want to offer this survey, and the data we received as a way of thinking about, “Huh, we did this survey, we got these results, why do we think we got that as results?”

I was thinking, and Sabre, Amy, and Michelle, we were talking about the different evaluation techniques that you can use when you have some data but you don’t totally know why you got there. Actually, I came across this concept called, the causal systems analysis, which lets you look at a system that might be changing, and dynamic, and you’re not doing a strategic plan of input, to output, to outcome. Lots of things are happening at once, so I want to use this data as a way of sparking conversations with a lot of our new staff to say like, “Huh, why do we think we might have gotten these outcomes? Huh, what was happening during the time that people visited us?” Which, honestly wasn’t a lot. We didn’t have a lot of public programming or engagement at that time. I’d like to use it to push us then to think about, “Okay, if we wanted to increase this factor, or this area of social impact, how might we do that?” So, to be continued. I saw Laureen and Megan in the chat asking about how you use the data. I would be happy to talk about that when we figure it out. Thank you.

Michelle Mileham:

Yeah, future, future webinar. Thank you so much, Alice. Because Amy and Alice both mentioned it, I’m just going to drop a little context about the emails and why we were responsible for sending all of those, and how often those reminders went out. It was pretty often with 6,000 participants across our study, that there was a lot of emailing. All right, Sabre?

Sabre Moore:

Okay, so we are also a free museum, as everyone else has mentioned. The first thing that we came up with, was a 10% gift discount in our gift shop. Then, anybody that filled out the whole survey would also receive entry to win gift cards. We had 24 gift cards, ranging anywhere from $10 all the way up to $100. That provided pretty good incentive actually for people. We also had flyers in the lobby and a custom QR code that was made with a link specific to our institution where people could sign up, as well as a custom QR code for people to actually sign in with once they became participants. That custom QR code link to sign up that was made for the Carter County Museum, we were very concerned about accessibility. A lot of our potential recruitment, our members who do not have computers, are not interested in computers, and do not want to sign up, something like that. So, we removed that barrier by allowing our staff to sign up people as well.

Then, we gave an option for our older members to meet with us in the museum and we would dictate the survey to them, and they would tell us what they wanted to put in there. That actually worked out pretty well for some of our senior members. We shared it through the newsletter, and staff were encouraged to help people sign up at the front desk. We also attended a couple of events. Right after we started recruiting, we had our annual holiday bazaar, which takes place in the high school gym. There’s lots of vendors that come from all over, and they get a lot of people into Ekalaka on that day in November. I had my laptop there and had our signup sheet, and were ready to go. In that way, we actually were able to recruit a lot of people from the local community that had not been in the museum in 30 years. For reference, 30 years ago we were in the basement of the old high school. We’ve now had three new high schools built in that time and we’re in our own building. So, the museum’s changed quite a bit since then.

Our visitation from local was only 31% prior to this study, but it’s actually bumped up into the mid 40 percentage right now, thanks to the mom MOMSI study, which is pretty cool. We also emailed our field expedition participants that come as part of the annual Dino shindig every year. I was one of the museums that was like, “Please, please, please, Michelle. Please let us go all the way until August,” because our biggest event of the year is in July, and we knew we would have a lot of people coming for that whole month that would be able to be making their visits. We recruited heavily at our monthly meetings, so every first Thursday at 7:00 PM we have a meeting at the museum where we present on subjects of arts, science, and history. I would also not only talk about MOMSI in depth, but I would also say how our recruitment was going, how many more we needed. As a rural museum and a very small town, we were allowed to have a minimum of 60 people if those people also filled out the survey.

So, I was very motivated in getting those people signed up. We recruited 68, and I’m happy to report that we had 63 visits fill out the survey, and life is good. So, next slide. For our retention, I mentioned the monthly meeting reports on progress, I would see people there. Everybody on the staff became very quickly trained. I’m the only full-time, but there are three part-time staff members, and we all knew everybody that participated in the survey by site, even those that we hadn’t met previously to signing up. So, we made sure to develop those relationships with those individuals. Even if they didn’t check-in, we would remind them and we would punch their little punch card like you get at a coffee shop. That worked out really well. I also sent reminder emails and updates in our monthly newsletter, but also personally to every participant pretty regularly. Whenever something was coming out I would say, “Hey, we have a new exhibit, we have a new display, the shindig’s coming up. This would be a good time to make your visit happen.”

Our gift cards, again, were only for those who completed the survey, and that included your museum discount in that gift card. We had 99% complete their first visit, that was 67 people out of the 68 that registered. 63 surveys were completed, and we gave out 24 gift cards to our museum gift shop. So, that worked out well for us as well. That’s a 93% completion rate, which we are very proud of. We also had all of our gift cards spent within the year that we gave them out, so that was also really nice not to have that liability hanging out there. People were delighted to be a part of the study, and to feel like they were part of that helping out their local museum. So, next slide. We’re going to jump right into the deep end with our data. We are sharing our results with the public at our March 2nd meeting. We had a article go out in the paper today that talked about that meeting, what to expect.

As I mentioned, a lot of people local to Ekalaka did participate in this study and they’ve been asking me constantly about, “What happened with that study? What are the results? Can you tell me more about it?” Now I can say, “Hey, come on March 2nd.” You can also sign up virtually, so this is a hybrid meeting where we’ll have Michelle join us and present our results. It will be also available on Zoom so that people can hop on. It’ll go out in our membership newsletter. Some of our members are as far away as Japan and Australia, and some of our participants too actually from Japan and Australia joined. So, they’ll be able to see the results of the study that way. I’ve already sent out some of the quotes from the study, as well as some of what we’ve learned, to our various grant partners and some of the people that we are working with on fundraising. We’re working on building a new museum expansion, and this social impact has helped quite a bit in positioning us as an institution that not only gets tax money from the community, but gives back quite a bit, both economically and socially, which was really important for our commissioners to hear.

I’ll be meeting with them, and they all get personal invites to the March 2nd event. I am currently a PhD candidate at Montana State University, and my dissertation is all about rural museums and their contributions to community vitality, so I’m directly using our results in that conversation. We’re using them internally at the museum for our strategic planning and areas for improvement. I mentioned this to Alice yesterday, but we had 10 out of the 12 strengthening relationship indicators, and so that of course is an opportunity for a little bit more improvements. There were also a few areas of health and well-being that we didn’t hit, just a couple. I’m pretty proud that we were over 10 in all of the categories. We’re going to increase our offerings of some of our events, like yoga at the museum. Right now it’s once a week, but maybe we’ll do it twice a week throughout the summer. Bring back our art classes that we do with our artists and residency program. Then, external community planning.

We’re involved in a citywide community plan for Ekalaka right now as a project, and I’ve already given the result to the architects that are creating the community plan for us in order to put that in there, about not only the museum and its expansion, but where we’re positioned with the community. Again, just reiterating the fact that the Carter County Museum and other rural museums, are really big contributors to the vitality of their communities. So are urban museums, but we can definitely position ourselves better. Again, taking that rural museum model, I’m a humanities Montana speaker and will be speaking on the contributions of rural museums across the state this year, as well as at our conferences. The next one that comes up is the Museum’s Association of Montana Conference where I’m giving a keynote about this data and what people can look for in the toolkit to come. So, I hope that lots of museums in Montana pick it up as soon as that toolkit is out there. Be ready, Michelle.

Michelle Mileham:

We will. All right, thank you so much Sabre, and Alice, and Amy. We want to turn it over to any of you listening in. If you have questions, we can just take a moment to pause here and you’re welcome to put them in the chat or feel free to unmute yourself and ask it live if you’re comfortable with that. Okay, we had brainstormed a few things. I just want to point out a couple of things as I was listening to the presenters that came to mind. So, I put in the chat about emailing participants, but I also wanted to mention the translation services that we provided. That was not actually something we had thought of when we wrote the grant.

My home agency, which is a state agency here in Utah, actually works with some businesses that are on contract with the state that provide translation services. On the request of our host museums, we ended up actually translating materials into, I think, three or four different languages based on the needs of their community. So, I think that’s something to keep in mind. Then, of course, we then got responses back to those open-ended questions and having to translate back into English for the purpose of our analysis. That was a service we were able to provide, and were really responsive to based on our host museums. Amy, Alice, or Sabre, do any of you have questions for each other or something you want to hear more about?

Amy Niedbalski:

I would just love to keep in touch and see how the reporting out to the community goes to see how results are implemented and strategic planning, et cetera. So, maybe we can do this again. There are some great resources discussion happening in the chat if anyone is not peeking at that.

Michelle Mileham:

Yeah, and thank you. Someone dropped in a thought of disentangling the impacts of the museum visit versus impacts of participating in the study itself. This is something we’ve thought about a lot, as we recognize a lot of the participants signing up for the study were probably regular museum-goers. We did have a few sites that really focused and achieved getting a high percentage of first-time visitors to their particular site, not to say those participants don’t visit other museums in their cities. So, that is definitely something we are thinking about. Like, “What is this?” And all of the other experiences that they’re having at the same time that this study is running as well.

Alice Anderson:

Can I add onto that, Michelle?

Michelle Mileham:

Yes, please, Alice.

Alice Anderson:

I think it’s true that whenever we ask people, “Tell me more about your experience,” in whatever format at a museum, we are asking them to be reflective about what they just experienced, which is a support for learning. So, I think participating in a study, of course, will have some impact on the data that we received because they’ve been asked to really think about it. Another way to use that idea then would be to say, “What could we take from the survey that we could embed within some of our exhibitions or programs or experiences?” So, in what way might we embed a, “How am I feeling here today,” question at the beginning of a tour? Our visitor experience staff are always telling me that as folks leave, they’re always saying, “How was your experience today?” Or, “Did you find the thing you were looking for?” I just think there’s ways to embed evaluative or research-based questions also into the educational or experiences people have at the museum too. But of course, yeah.

Michelle Mileham:

Thank you, Alice.

Alice Anderson:

Mm-hmm.

Michelle Mileham:

Amy or Sabre, anything else?

Sabre Moore:

I was looking last night at some of the qualitative answers to the question about whether the study had impacted people. A lot of our participants said, “Yeah, I looked at the museum differently. I had never really thought about the museum being an agent of social impact before.” So, I’m interested to continue those conversations as Amy and Alice both mentioned, and to have those in a wider array of places like, “Hey, commissioners, did you know that museums have social impact?” I know I’m always talking about the commissioners, but I think it’s important, especially for our economic stakeholders, to have that knowledge that it’s not just us bringing tourism dollars into the community, but it’s having that really interesting connection with our community members as well, not just in continued learning and engagement, but of course, in health and well-being.

Michelle Mileham:

Amy, did you want to add on?

Amy Niedbalski:

I realized how late it was, so I wasn’t going to. I was going to just say, as all of us are free, heavily tax-supported institution, I meant this justification of impact on where your tax dollars go and what it does for a community, we’ve never had before. So, that’s awesome. I’m excited to see how we can utilize it.

Alice Anderson:

I think it’s a really interesting moment to do this survey in our field too because I think when I took my job at the museum, which was about five years ago, they had previously done some evaluation workshops and a lot of deep diving into visitor motivation and considered the museum a very audience centered place. So, really there already had been a lot of support for the idea that museum visitors come for lots of different reasons, but often around learning about the artists and the history. I think now our field, and other arts and cultural organizations, are learning more about the ways in which the impact can be both rippled out in other ways within communities, but individually how this idea of well-being is so interesting to many of my colleagues and to our field.

I don’t know, I think it’s just such a fascinating moment in time to do this, because it seems like, at least I would expect, that many people would say, “Yes,” for the respect and interest in diverse communities, and learning, and engagement. But, the well-being and social interactive may be lagging just ’cause it’s like we’ve never really put that out there, that this is absolutely what we’re here for. To your point, Sabre, about, how do we want to position this with our fellow stakeholders about what value our institutions bring? Maybe we can do it again in 10 years, Michelle, and see where the field is then in our culture.

Michelle Mileham:

Yes, I would love that. I want to plug this, I saw some questions in the chat. Sabre had mentioned, we are working on the toolkit, so watch for that in June 2023. A lot of that comes from the perspectives and what these host museums have gone through. I also want to say, if this is not a feasible thing for your museum to pick up and do, even once we publish that toolkit with resources, we are going to post a report of the aggregate data so you can at least use this across the US data to help support your museum. So, that will be a public resource that can help shape your narrative. Then, I saw some folks were looking at how to stay connected.

Please, please do. We have our museumsocialimpact.org website. You can find my contact information on there. The QR code on the left side of the screen is to subscribe to a newsletter. So, we’ll send that out occasionally and then obviously when the toolkit comes out. So, it’s just a great way to get pinged with updates from the project. Then, if you miss the social impact surveys, that QR code is back up there on the right-hand side of the screen. Thank you so much for joining us today. I think we’re at time. Thank you Ann, and AM, and VSA for hosting this webinar, and follow up with any other questions.

Amy Niedbalski:

Thank you for having us.

Michelle Mileham:

Yeah, thank you, Amy, Alice, and Sabre.

Ann Atwood:

Thank you everyone for joining, and I hope you have a great rest of your day.