This is a recorded session from the 2020 AAM Virtual Annual Meeting and MuseumExpo.

How will the COVID‐19 pandemic impact the ways we conduct research and evaluation? Join evaluators in discussing this question and thinking about how we may need to change our data collection efforts at our museums after our doors reopen.

The session is hosted by Committee on Audience Response and Evaluation.

Transcript

Sarah Cohn: Hi all welcome so much for joining us this afternoon or morning or evening, wherever you are. We are members of a committee on audience research and evaluation, and we welcome you to the session, the future of museum evaluation after we’re excited to be here with you to have an open reflective conversation among colleagues about the future of museum evaluation post COVID

We appreciate you taking time today. This Week in the midst of massive change and hopefully community building and reimagining at the system’s level.

I’m Sarah Cohn principal consultant at Aurora consulting in Minneapolis, Minnesota, and former evaluation staff at the Science Museum of Minnesota.

I’m excited to be here as moderator and a chat conversationalist

The panelists will be keeping things pretty open and flexible. There are no side slides. There’ll be working off of notes and questions that they have previously identified to move us through a conversation about what questions ideas and newly identified barriers and opportunities they see for evaluation as our museums move to being open once again for our communities.

We’d love for you to participate and share your thoughts about what the panelists are discussing in an open conversation in the chat window.

If you have any questions that you’d specifically like presenters to respond to or address through the conversation. Please post those in the Q&A window. And both of those are found at the bottom of the screen.

As I hand over the microphone to Michelle and the panelists, please take a minute to respond to this first poll regarding your evaluation background.

And I’ll hand it over to Michelle.

Michelle Mileham: Thank you, Sarah, for that introduction. And thank you all for joining us today. We’re really excited to be here and to share our opinions and thoughts with you and hope this is collaborative

My name is Michelle, I’m coming to you from Salt Lake City, Utah, where I’m the Director of Education at Tracy aviary and Botanical Gardens. And Tracy aviary is an accredited zoo. We just house birds in our collection. So, I’m excited that you all are joining us today and I’m going to pass it to Emily.

Emily Craig: Hi, my name is Emily, Craig. I’m coming to you from LA where I oversee the docent Council and the school and public tour programs at LACMA the LA County Museum of Art, and I’m very excited to be part of this discussion. So, I will pass it along to Lori.

Laureen Trainer: Hello everybody, my name is Lorraine trainer, I am principle of trainer evaluation actually is actually remote we’re kind of West Coast Mountain states, I’m coming from Denver, Colorado, and I’m an external evaluator working primarily with museums and focusing on informal education.

And it looks like we have stopped their first poll and we’ve got about 54% of you. So, about half of the group has some experience.

Anybody should run, which is fantastic. And then a little bit more in the novice and then going towards a lot of experience and expert so we should have a pretty varied group here in terms of where people are coming from.

I hope geographically, I hope, in terms of institution type and also their level of experience with evaluation. So that’s excellent. We actually do have one more poll that we’re going to launch. We’re just trying to get a sense of who’s in the room today as we begin our conversation. So, if y’all wouldn’t mind taking a look at this next one.

Is really trying to understand as of today, we all, we all know there’s a lot of question. When is your museum scheduled to reopen to visitors?

And I would say even since we met, I don’t know, like three days ago, I feel like the world has flipped upside down all over again.

So, I think that could also just be a great part of our discussion of we just tried to navigate this change. And what does that mean

And it looks like most of you are unsure as well.

So, we’ve got about 45% of you are coming in on. Sure. 24 are looking up in July, August, and just 4% of people are currently open right now.

Excellent, thank you everyone for participating. That just kind of helps everybody understand who’s part of this conversation right now.

Alright, Michelle, do you wanna do you want to kick off the conversation. And just a reminder to everybody that this is a truly just a conversation that we’re having. We just thought it’d be great to just bounce some ideas off of each other and then hopefully you all will have a chance to bounce some ideas off of the chat function and we’ll be trying to reminding you to like prompt different conversation prompts in the chat feature that Sarah will be monitoring for us.

Michelle Mileham: Yes. And it’s great to see we have some so many kind of novice and just some experience. Hopefully some ideas that we share and will be very fruitful for you all. But knowing that we also have some search joining us to also be chiming in in that chat box to help some people who haven’t had as much experience grow and learn as well. So as Sarah mentioned, we as a group identified a handful of themes that we thought were particularly relevant for right now and kind of what we’re thinking as evaluators in this field and the first one is really just our place and our role as evaluators and this quote unquote new normal. So, I’m thinking about who all is doing evaluation right now and what role does the evaluator serve, while a museum is closed and is looking to reopen and certainly once it is reopen and I know so many of you said you’re unsure about your reopening and for the seven of you on this call who responded on that you’re open. I am also coming to this conversation.

From a place of my facility being open to the public. We have been since May so I’m obviously a lot of kind of what I’m contributing is coming from both our closure state and this reopening phase as well.

And I think for me, now more than ever. I can really justify the role of evaluation in my organization as someone who oversees education. I do evaluation for the organization, kind of just as part of a lot of my other duties and responsibilities.

And during our closure that became a large portion of what I was doing, reaching out to our volunteer corps and which we can also see kind of through the lens of long-term visitors.

And what their needs are and making sure they’re all okay and staying connected to us as an organization, and our mission.

Certainly, with visitors as we thought about our reopening and measuring that intent to visit and what people were needing and wanting from us as an organization through that phase. And certainly, now that we’re reopen um

And I have to say, and I don’t know if our other panelists and are thinking about this, but I also encourage all of you to as well to think about your staff.

And the role of us an evaluator kind of making sure and measuring that staff comfort level two. So, for us, it’s like this role really became more of a priority, I would say than it had been prior to closing for COVID so I’m going to let Emily chime in. Next, just to kind of see where she’s coming from

Emily Craig: Yeah, so, um, LACMA we do not have a confirmed opening date yet but we, you know, Michelle, like you said, I think we’re thinking about our audiences as someone who oversees the docents we are starting to think about what you know what kind of information we need to gather from them as we approach reopening and I think your point about staff is a great one. I know, there are senior leadership conversations taking place and we as an education department received a survey from our Director of Education asking us about some of our comfort levels and some of the things that we would like to see in place.

Not just when the institution reopens but when the staff offices, we reopen, we have an open floor plan, and you can’t put the same number of people in a space anymore.

I think also we’ve reached out and I know several of our colleagues in the LA area have also reached out to our teacher cohorts.

A lot of my thinking is around tours. So, thinking a lot about what does that specific audience need they are confronting a lot of new ways of teaching. They aren’t entirely sure what the classroom is going to look like in the fall. And so how can we support that not just through the obvious answer of producing programming, but what can we do to produce programming that is useful to the teachers.

So, I think that that is also part of it. And I think now more than ever.

It’s so important for evaluators or anyone who has, you know, like Michelle I am not a full-time evaluator at my institution. I do this as part of something else.

But really serving in that audience advocate role and reminding staff, you know, making sure those surveys are going out, making sure you’re talking to people talking to your members.

Talking to your volunteers in your staff to really bring that audience voice.

Into the into the room, I think, in one of our previous conversations and planning this Michelle said it so beautifully. The or possibly. It was Larry and I’m sorry, I don’t remember.

It’s not that the visitors aren’t just in the room when we’re having those conversations. They’re not even allowed in the building anymore. And so, it’s so important that we you know we remind people that we can’t just make plans that makes sense for us as an organization, it’s important to understand the community we’re serving.

And Lorraine, I would be interested in how this looks. You know, you are an external evaluator. So, I’m sure you’re dealing with very different thoughts and considerations right now. Yeah.

Laureen Trainer: Thanks, and I would love just to encourage people in the chat feature to just we would we want to know what is the role of evaluation, as you all are thinking and if it’s not, that’s, that’s fair to. We want to make sure that’s part of the conversation. So, if you all can kind of share with each other in the chat section.

What has been the role of evaluation as you are making your plans for reopening and that includes staff that includes audience that includes volunteer. So really thinking about the entire spectrum of the community and not just visitor so

And we look forward to seeing some of your ideas in there. And I guess I get to be a little bit of it to be the person that may be because I’m not working internally.

I could be the person from the outside that says we need more than ever to be the visitor advocate voice.

And I do actually think that I’d be real curious to hear people’s reactions to that. I think evaluators need to take on more of an activist role.

I think many people would disagree with me. And so, I’d love to see that conversation about where is the line between advocate and activist for your community and for your visitors.

This whole idea that I heard a lot of during the shutdown was this was going to be a chance to rethink our institutions, right, we could resets we could rethink we could rethink our equity we could rethink who columns, who doesn’t come to our community is and I think in part of the rush to reopen and I truly do understand the rush trio Ben, I understand the economics.

And what that means for your staff and for their families and for the community and for everybody that relies on all the money that comes out of institutions economics is a huge part of it.

But we need to be thinking really seriously, especially now with the overlay of the social justice protests that are going on the social inequities that have been brought to the fore in a way that I feel like society is not ignoring at the moment.

Really understanding some of the policies that we put into reopening, and I’ll go with touch entry as one thing that I’ve heard a lot of people doing everything’s online. Everything’s based on your phone. Everything’s based on app. Everything’s based on credit cards.

That’s not exactly equitable and it might be a way to reopen your institution but is it a way to reopen your institution in an equitable fashion.

And I think that’s something where evaluator can come at it from the community advocate and keep saying keep pushing back. Okay, fine. If we need to do this to reopen because we have to pay bills and our staff needs to be paid. And I truly do understand that what conversations. Do we need to be having in our community?

And what does the evaluator bring back to the institution to consider constantly free bringing up these questions around to our audience is who are we serving. Are we staying true to our mission on ourselves and what we say we’re doing in terms, if we say we’re an equitable institution? What does that look like? So that’s a huge haul a lot to put out there VI and, you know, we should survey people and we should talk to the, you know, teachers and I absolutely agree with all of that. I just feel like there’s a new overlay that has happened in this past week that we can’t ignore and also needs to become part of, I think what the evaluator can bring to the institution moving forward. And that’s the changing role. I think that’s a new role. I think it’s a stronger role. I think it’s potentially more controversial role, but we’d love to kind of hear what people think about that.

And if there’s anything Sarah that you heard or saw in the chat that we can kind of draw out

Sarah Cohn: You three. Keep talking. I’m getting through it. I’ll call

Laureen Trainer: It. Yeah, I see it scrolling out of the corner of my eye. So, I’m so glad you’re paying attention.

Michelle Mileham: Appreciate that you’re trying to look at that, Lauren. It’s a lot. I see. I’m kind of popping up on my screen here and appreciate that.

I think to one thing I’ve heard in several presentations that I’ve attended throughout this week.

Is when we do think about this equity and diversity, how often that falls on one person at our institution or somebody who has that kind of title and that diversity officer or equity officer community officer, whatever that may look like and that it’s not yet integrated through all levels of our organization, but it is everyone’s job. It is all of our responsibility to make sure we’re speaking up for all voices and making sure our full community is being represented and, in a lot, the same way. I think the same is true for evaluation that this shouldn’t really just be the responsibility of somebody who have this in their title, but, um, you know, our roles, maybe helping others understand the power of evaluation and what it can do for an organization to keep growing and thinking forward.

And making sure that all of these voices are being heard and integrated and in meaningful ways at that so one thing I was really thinking about and thinking about the role of evaluation in museums, as we reopen is actually us training more of our staff and this field so that you know when we look at signs that we’re putting out when we consider one way pathways that we are equipping everyone who’s out on our floor and observing our visitors.

And kind of in the best practices of how to collect that how to report it back and certainly to identify what may be missing from all of that, so that we can be equitable in a lot of ways.

And for me, I was actually filtering through data right before I joined this presentation that we’ve been collecting and have gotten feedback that the spray paint. We used for one-way pathways is too light. It’s not really visible for a lot of folks.

And that’s certainly something we did not think about before taking that action, but it’s certainly something we can rectify, and we’ve also had feedback about the difficulty of one-way pathways for strollers and people in wheelchairs.

So, you know, thinking through and collecting all of that and knowing that you can then do better with that information. So, I think a lot of our role to is really equipping everybody with the skills to be able to make our organizations better based on what we’re hearing from our visitors and our community at large.

Laureen Trainer: Yeah, I think that’s a good point. I mean, I think there’s a huge role on couple France. And one is that there is continuous evaluation mean things are changing, you know, what happened, a month ago. Is it feels like a year ago at this point? And so being sure that just once we open. We don’t say we’re reopen but that we continuously reach out and just like you’re talking about Michelle, making sure that we’re checking in with our audiences and doing some observations on you know flow and signage and social distancing and what’s working for people but I think one of the one of the roles, also for evaluation and feel going forward is a large portion of see Wilkening slack that I was listening to the other days just about the level of grief and trauma that people are going to be bringing with them into their institutions.

That was largely code related. And I do believe there is now another overlay of social justice trauma, that’s going to come as part of this.

And so really trying to understand how that affects our audiences and then does that affect our programming our messaging.

How do we, how do we understand that in terms of what our what our visitors are bringing on the inside into our institutions as well as some of the more logistical things like whoa. signage and distancing one for in the galleries.

Emily Craig: Yeah Lorene. I think that’s a really good point. Um, I think, sorry. Sorry. I didn’t mean to cut you off.

With that, with the audience input, but I think that’s a really good point and I think it’s something that in you know you had talked before about this push to reopen and it’s an understandable push and I know we’ve been doing a lot of thinking about programming and what kind of programming, we can do and you want to keep your staff involved and I want to keep my volunteers involved in active, but I think, you know, there is this underlying or not. So, the underlying feeling of trauma and just massive change in anxiety.

And so is returning to the same old proclaiming formats are just putting things online virtually really working. And so, I think as evaluators, or even just you know i right now, I am very much a practitioner who conducts evaluation as part of her position I what are those questions and I don’t have a good answer right now but figuring out what those questions are that we can we can be very upfront with visitors and say we’re putting these things online we’re designing these programs.

Is this meeting your needs. Is this not meeting your needs. Are there other lenses that we need to be looking through. Are there other methodologies, you know it’s not always a PowerPoint presentation with someone talking about art for 15 minutes what other things can we be doing to meet those needs at this time. So, I think you know it’s all very interconnected but it’s important that you aren’t simply saying, how’s this program going but looking more broadly at is this program meeting the needs that you’re bringing to the table right.

Laureen Trainer: Yeah. And I think, again, part of it is understanding the experiences. They’re going to have in the gallery. So is being in an indoor spaces. Other people is that going to be so anxiety inducing that it actually doesn’t allow the visitor to enjoy the space or to be able to take advantage of it. Or conversely if your area doesn’t require masks, but you’re wearing a mask at in the museum and you see other people, not exactly. And, distract you so much that you like. You have to leave the building. So I’m just trying to try to understand all these different protocols that we’re putting into place social distancing mask drain cleaning. What effect is that going to have on the expense that people are happening. And so, I think that’s also something for evaluators to be thinking about as we reopen and as we are thinking about some of those next questions.

We did reopen people came to the building. But now what’s happening, what’s happening on the inside because that’s going to be very different experience than, you know, two months ago when we closed our doors. So, another aspect that I think for evaluators to be thinking about.

Sarah Cohn: And there’s been a lot of questions and chat around how do you find that balance between sort of responding to this.

Current crisis with questions and monitoring and sort of how do we become more responsive rather than kind of reactive in this moment with

Longer term questions bigger picture questions about where are we headed. How do you find that balance within the the data collection that you do with either with staff or with visitors?

Is a question that’s coming up as I am hearing you all talk about a bit

And then there’s also a specific question about how do you shift. What does this process look like? So, I don’t know if you want to talk to an answer Lorraine’s question that she just threw out of what does this experience do to us as we change data collection and what are you, Michelle. What are you trying right now or what do you, you know, testing out in small ways?

Michelle Mileham: Yeah, I think this is a good shift and to, you know, what does evaluation look like and data collection look like as we’re reopening

And I think to Laurens point with the trauma, I have to say, like, this is a probably say in the last week and through all of our conversations, the most

I’ve really thought about that trauma connection and like what I’m kind of comfort zone, we can provide as museums, like I’m really fortunate to be at a place that is primarily outdoors.

Our mission is very much connection to nature and research shows that does help people overcome trauma. So now what spinning in my head is like, how do I make that happen and measure that at our organization, you know, like, I think this is opening up a lot of possibilities.

And as far as data collection. I do think like, you know, we’re going to have to take a lot of novel approaches moving forward. And what this looks like. And I think it won’t work right from the get-go. I think it’s going to be a lot of trial and error. I will say we tested out right after our reopening putting up a QR code for an exit survey that was intended to measure their comfort level while visiting the aviary and you know, in some ways that have a lot of positives, people could use their own devices to access a QR code.

And we didn’t necessarily need staff there to direct people. So, we thought, um, but it also came with a lot of challenges.

And one being, you know, sign fatigue. We now have a lot more signs on grounds to consider distancing and to keep everyone safe.

And so maybe by the time they exit that one more sign, asking them to do something was just too much.

And so that’s really been something we’ve been piloting and trying to figure out is the location, right, do we still need an evaluator or a staff member or someone saying thank you for visiting please, you know, scan this QR code and kind of doing that ask like we would in with paper and pencil surveys or with tablet based surveys and but we’re also building in the assumption that all of our visitors come with the technology to be able to take that survey as they exit that they have a smartphone that they understand what a QR code is and how it works with their phone. So, I think there’s a lot of assumptions. We also have to consider as we move forward with some of these newer approaches.

Emily, do you want to

Emily Craig: chime in. Yeah, I think, you know, as you mentioned, thinking about different ways of enhancing that you know of gathering data. You also have different people with different comfort levels.

And so how are you balancing, you know, is the expectation. Now, if we want to be able to get feedback from as many people as possible.

Do we need to offer a QR code so that someone can take a survey on their phone as well as an iPad that someone might feel comfortable taking that you know you have a person and you can hand that to them and you can disinfect it when they’re done, and also have paper and pencil and have some sort of system for separating the you know the non-used or the disinfected pencils from the used pencils. I know you know i was at a doctor’s office recently and they have the new pens cup, and they have the user pens cup.

And but then is, that’s obviously a lot more output than we have done in the past. And are you having one staff member, man. All of those outputs. So, someone pointing to the sign for the QR code someone HANDLING THE iPads AND SOMEONE handling the pencil and paper.

And if your staff member is wearing a mask. How does someone feel being approached by. I mean, you know, and that’s just to hand them something to take a survey.

I think masks also open up a whole realm of considerations when it comes to things like intercept interviews and is you know, is someone going to feel comfortable with someone that they don’t know who is affiliated with the institution walking up to them and saying, hey, we would like your feedback about whatever it is, and when you’re doing that interview.

How are you handling recording what they say, can you pick that up on a recording device can you if you’re standing six feet apart. Can you hear what that other person is saying? Does that person feel comfortable sharing their thoughts when they have to speak it aloud. I mean, it allowed her volume when I when I conduct surveys, you’re standing very close to that other person. And so, they aren’t speaking in a loud voice in order to be heard. And does that distance impact what they’re willing to tell you how long they’re willing to talk to you how comfortable they feel with someone walking up to them and saying, Gee, we’d like your feedback.

So, I think, you know, those are all i don’t have good answers to those questions, but I think that they’re all things.

That I’m thinking through in terms of if I was tasked with doing something that included on site interviews, how would I handle that and what are the questions and the considerations that we need to put in place in order to make sure that visitors will participate, but also will feel comfortable participating.

Laureen Trainer: Yeah, and I think it like Michelle said. And again, I see a few people starting to kind of offer what they’ve tried to what they’re thinking about trying. So again, I would offer that as our next kind of chat prompt of if you are open. What has worked. What can you share with us? If you haven’t, what are you thinking about I know I have heard a lot about the idea of capturing emails and then you send a survey, especially because a lot of institutions are going to purchasing tickets online you can capture emails as part of that they’re obviously all sorts of limitations around that it’s generally one person purchasing for the group.

And so, you’re not capturing everybody’s emails, but it’s still it is a point and it is a visitor and it might be another way to do it.

I just wonder, again it just how practical. Can we be of, you know, is there a person in a mask with a clipboard and a phone and a tablet and you know try like data collectors already have a bad rap.

You see somebody walking toward you and all of a sudden, your, you know, your phone is the most interesting thing in the world, or there’s a call. Yes. Okay, so what does that now look like you know when somebody doesn’t put you in a mask.

And so, what is actually this also realistic for an institution or for the staff member or volunteers that you’re going to put in charge of the US, you know how much of a burden on our own ass.

To we have to put on them and putting them in the position of being that person in a mask. I know that has to go and chat with people and really, it’s going to be one of those crazy like split second decisions of trying to gauge what a person is comfortable with.

I was I was telling Michelle and Emily the other day I was shocked. I left the grocery store and there were two signature gatherers, like right outside the entrance and this was this was a month ago this was actually pretty like deep into when we were all sheltered in place.

And they were sitting there with masks, but they had their clipboards and pens intercepting people handing clipboards and one pen I watched them do it and people took it and signed it. I was blown away because there’s no way I would have liked. Anybody approached me at that point or taking something so everybody does have different comfort levels.

And how do we react and judge, you know, judge that in a quick manner? I think it’s something we’re all gonna have to figure out, and I guess. In the meantime, how can we figure out how to use tax. How can we figure out how to use email to maybe get around some of that person-to-person contact or having to, you know, hand something take something?

Sarah Cohn: And there’s a question, Lori and you’re in

Laureen Trainer: Colorado. Yeah.

Sarah Cohn: So, Kelly that happened in Denver.

Laureen Trainer: I’ve seen it now several times. So, it wasn’t a one-time thing.

Sarah Cohn: Yeah, and I’ve seen a couple times in the chat. And I think it’s important to sort of point to right here is the importance of context and the fact that we are in the middle of culture, you know, culture change at a lot of different levels but coming to recognize that some of these questions around masks and comfort may shift with cultural expectations, whether it’s state city regulations or just coming to begin recognizing that I wear a mask to protect you, not just to protect myself and, you know, things like that. And some folks posted there.

So thinking a lot about what is happening, sort of culturally and what’s happening within your communities around new forms of sort of expectation social expectations.

And then, yeah, really understanding. So, the context and were part of what Lauren said at the beginning of like what are people carrying with them. What are they expecting from you as they walk into your institutions? If I walk into an institution and no one’s wearing masks. I’m walking back out of there.

So just thinking about like how are we again balancing these various pieces and recognizing that this is everything is in flux, because we’re all trying to figure it out together, including your visitors.

Laureen Trainer: Yeah, I would just say, I mean I think masks have turned into the one of the trickiest things that I never would have said this would be an issue. It’s now a political issue.

Different regions of the country have different reactions. I was just reading the other day that there’s a restaurant that won’t let you in. If you’re wearing a mask of the political statements.

And again, that varies. It all depends on where you are regional, but it has become a political issue. So, it’s now safety and politics all rolled up into we’re just trying to open our doors and get people to come into our institutions and figure out how to do this. So, it is. It’s definitely taken on a new layer that I’m not, I’m not really quite sure how to navigate. But I think something that each institution is going to have to kind of figure out what is most comfortable. Well, there’s also local regulations, but then beyond that was most comfortable to them.

Michelle Mileham: Yeah, and I can speak to this as an institution that’s been open our state’s never really kind of put out language saying it was required and county and city level either. And so, when we reopened and we made it just strongly recommended

We are again primarily an outdoor facility. And so, you know, we were kind of weighing this information that’s been coming in about you know, being outdoors and that air and flow and all that versus an indoor space. And so, I know that’s something a lot of other museums are considering when your gallery spaces are only indoors that you know they’re requiring a mask. Maybe much more of a stipulation for you to reopen and it’s been interesting seeing feedback. I mean, it’s kind of like a response to everything just from one end of the spectrum to the other that we’ve had people you know, say they won’t visit if they see other people, not in masks. But, you know, we can’t enforce it. We can’t even, you know, because of our situation in the city and with funding, things like that where, you know, not every place can even require it if they wanted to.

And then we’ve had people saying like, I can’t believe people are walking around wearing masks and it’s like, well, that’s you know their own personal preference and everything kind of in between and for us as educators to like thinking of this through the lens of education, we’re really thinking about it in the equity of, you know, when we do when we are on grounds and we talked to guess.

We have not yet found a clear mask, if that’s something anyone has found pretty sure

Laureen Trainer: They are out there.

Michelle Mileham: They are

Laureen Trainer: Hot out for your next

Sarah Cohn: One in the chat. There’s a lot

Michelle Mileham: That’s something we want to think about. I think that’s also, you know, it’s something the organization pays for to keep your staff and open and being able to be accessible to all coming through your door, who may need to read lips and as part of their visit, so you know that’s even just something to kind of move into this mass conversation as well.

Emily, did you have something it’s I felt like I cut someone off.

Emily Craig: No, I had noticed that someone had asked the question about lip reading in the chat. It just happened to pause on my screen long enough to read it.

And I you know I think that that’s a really good point and thinking through clear masks and not just, I mean, honestly, not just for evaluation purposes, but thinking about what your frontline staff should be wearing so that they can engage with those people who come up, you know, are you saying that there is a lack mode. They sit behind Plexiglas. It’s the design of the box office. That’s just how it has been. I know that’s not the case at every museum.

And so, we probably don’t have to put additional barriers in that way.

And so, I don’t know if the plan is for our frontline staff to not wear masks and just use that as a protect you know they have the barrier.

But if you aren’t in that situation and you aren’t putting up plexiglass or something like that. Are you, you know, you want to make sure that when your visitors arrive, they don’t feel like the first thing they encounter is a sort of a roadblock, and they can’t see what the person is saying to them? So, I think those clear masks are such a good idea not just you know I know we’re talking about evaluation, but it goes beyond that, I think a lot of the things that we’re considering right now and that we’ve been talking about.

Really require thinking from other departments, so that ties back to what you were saying before, Michelle I can no longer be you know, if I over I, you know, I worked on evaluation in the education department, and we have marketing staff that runs an audience research study and like that’s it. We’re done. It really is becoming something that so many more people in the institution need to think about getting visitor feedback and incorporating these kinds of things into our daily practice as museum practitioners.

Laureen Trainer: And Emily. I love that idea of, you know, should frontline staff be wearing clear masks. I hadn’t even considered that. But I think that would be a I think personally that would be huge welcoming the actually like see a human file as you walk out to this person and, especially, you know, for the first couple times when you’re coming back into an institution and feeling a little uncertain.

In might be very helpful also think part of it. And I’ve seen it mentioned a little bit in chat and I’ve heard it another call is this idea of, you know, reusing stylist says, and can we use that. So, you know, they don’t have to touch the screen with as its stylists, and I think that’s a great idea. But I asked. I think it goes to, I think, Michelle, you and I were chatting about this of do, are we creating a huge amount of waste in maybe some of the different options that we are thinking about implementing which in itself is just a consideration and especially if it’s your mission as an institution.

To be a connection to nature and you know to conserve that may be ordering you know hundred thousand, you know, plastic reusable stylish says might address one aspect but opens up a whole nother can of worms with your institution and visitors.

To just because, again, there can be no easy answer. So, we need to problem with ties everything, but I think part of what it comes down to is every time we come up with a potential solution and like, what’s the, what’s the conduit. Like, what other aspects to consider.

Meeting part of that. I would love to hear about some Emily AND MICHELLE Your thoughts or Sarah or anybody else’s prototyping and you know a lot of prototyping. It’s meant to be hands on people are supposed to play with something experienced something you know, they’re normally materials that can be clean all the time you know it’s cardboard. It’s paper. We can’t really disinfect that we’d have to throw it out so what does prototyping look like as we kind of continue down this COVID path.

Michelle Mileham: I think this is where I would really love to see people who are dealing with this right now pop in some suggestions in the chat box.

And we generally don’t do a lot of prototyping in the first place. And I can’t see us moving very quickly. And in that direction. Now that we’re kind of reopen and we’re navigating so many other aspects of our work. So, I know that there’s been a lot of other conversations and other platforms happening about, you know, how do you make that possible. And I don’t know if that is something that can be done digitally with how you can move pieces around on a screen and then screen capture that.

You know, does it become using materials are thinking in new ways of laminating so that you could clean and disinfect or are you just going to again maybe produce a lot of waste by having to make so many kind of individual sets that people would use.

To be doing that, but certainly if people have been thinking about that prototyping piece. Drop a comment in the chat box and maybe Sarah can chime in with that.

Sarah Cohn: Yeah, and thanks for the Joanna just shared there’s a upcoming webinar on remote formative evaluation, but Emily is if you have thoughts on this as you’re sharing just talk about what prototyping is because we’re again like a pretty broad variety of folks in here. And then also, I think, wrapping into this. If you’re using social media or other online platforms to do data collection. I know we’ve had a good conversation in the chat around that. But what is online collection of different kinds look like

Emily Craig: Yeah, I mean I so as an art museum, we do not do a lot of prototyping, so trying out different iterations of an interactive or an exhibit installation or something like that. I know we’ve done some focus groups, getting information from visitors as we’re starting a planning process, but we do not have a lot of interactive elements at the museum. So, I feel like we aren’t doing as much iterative testing as other organizations might do.

I don’t know what I said that got my Alexa thinking I was talking. Sorry about that.

But I and I would be interested. So, Sarah. I don’t know what you’re reading in the chat. We are not to the best of my knowledge, using a lot of social media to collect data, but I love that as a as a way to interact with people because I know we are posting. I know we’re sending out weekly emails with materials and all those things and I don’t know that at that level, we have thought about aside from using one of our teacher emails to embed the teacher survey that we did.

I don’t know that we’re using a lot of social media to collect data. So, you know, that sounds like a really great conversation that’s happening in the chat.

Sarah Cohn: Yeah. So previously, folks, we’re talking about using zoom as a platform. And then, including surveys or responses at the end, both for online programs and then also a way to gather feedback from their staff, it’s back towards the very beginning.

And then we’ve had a little bit going on around the use of just gathering information from your members and hearing from them as like one first sort of low bar way of gathering information, but I haven’t seen a whole lot. Yet, in terms of social media platform in particular. Some folks are sharing stuff now.

Laureen Trainer: Yeah, I mean, obviously, social media zoom Facebook email. It’s all a great way to deal with that. We don’t have to be face to face, which can take a lot of those question marks around it and then you could do.

A synchronous interviews over email where it takes a little while but you prompt a question they respond to a question and you just create a running conversation essentially as part of that which is one way of reaching out where people don’t have to be like at a computer at a certain time with video capability, it’s a it’s an email platform. And so, that’s certainly one way I know just even posting questions like on Facebook on Instagram of, you know, how would you feel about or what would you think or even just send like a B testing like which sign with, you know, resonate and kind of do some polls that way. There’s definitely some pros to that. You can also get a lot more reach than just the people that are in your institution. So, it’s not just your members. It’s not just your visitors. It’s Jenna would have a broader reach your kind of reaching people where they are on the social media platforms, I think.

From some of the studies I’ve seen people are on social media a lot more maybe it’s tailed off, but I know in the beginning it was social media all the time.

And then, yeah, clearly zoom Google needs Google Hangouts. I forget which are now calling it is a great interviewing tool Zoom is actually great because you can record which is fantastic, but they don’t come without because, again, of course, nothing could just come without any issue.

So particularly resume. I’ve seen this conversation pop up in some of the online evaluation groups. I think any jack Nelson started for us. What is, what are the equity concerns around zoom. I mean, you all are looking into our homes right now.

That might not be entirely comfortable for some people it might not be appropriate.

In some institutions to look into somebody’s home. I know you can do virtual background. So, if people don’t understand how to do that you’d actually even have to walk them through. Do you do it with no video and just audio back and forth.

There’s some ethical issues. So not only equity, but like ethical of, you know, if you see something happening in the home. Do you have to report it? I know that sounds really drastic, but you never know what you’re going to see, when somebody turns a video on I think there’s all issues of implicit bias from the evaluators point of view, again, once you kind of get that glimpse into somebody’s home. I think that might change a little bit.

I you know I don’t have an example of that. But I have to imagine that could also be an issue. And then, of course, always issues of kids and trying to do any interviews with that and all of the red flags that come along with it so zoom I think is probably going to be making its way into IRB is more than it has ever had before, because we’re just changing the way that we are reaching out to people and so I think they’re most likely will become language that hopefully the field will be able to use and kind of agree upon to guide us through this, but I don’t think we’re quite there yet a conversation that we still need to have them.

Emily Craig: Also, Lauren, you, you mentioned IRB, the Institutional Review Board for you know protections around data collection and human subjects.

Study and that is not something that most museums are used to taking advantage of. They don’t necessarily have a connection to that the process of going through the IRB approval can be a little tense and time consuming and expensive. And so, if you are adding that in, especially if you’re a smaller institution or if you don’t have even if you’re a big institution, you don’t have a large portion of your budget targeted toward evaluation.

What you know i think i wouldn’t feel comfortable doing those kinds of interviews without having that protection of IRB and knowing what all of those things are. But are there small steps we can take to make sure that we are being careful and thoughtful and ethical around all of that without going through that process does that you know is that acceptable to people is that an institution-by-institution decision. Um, but, you know, that’s that changes a lot of the ways institutions have to think about doing Evaluation and Research because it’s not something that is typical protocol that their staff would know to think about.

Michelle Mileham: I think that’s such a good point. Emily and I was going to say the same thing is just what kind of new protocols.

We need to have in place. And if that is the direction of using zoom calls or video features, anything like that to collect data.

You know, if you’re not using IRB. What kind of protocols. Do you have internally that still center around that ethics piece.

To make sure you are at least following those standards and thinking about it, even if it’s not, you know, fully approached and accepted by your be that you as an organization are kind of making that move forward.

And I think it kind of brings us full circle here as we’re nearing the end of our hour back to this equity piece as well. Um, you know, just with the access of you know, technology in people’s homes and how many people do you have personal computers and Wi Fi or Internet access in their homes to begin with and you know, when I think of this through the lens of COVID certainly here in Salt Lake City are low-income neighborhoods and communities are very hard hit. And that’s still, you know, on the rise.

You know here in our communities and thinking about, you know, now that’s maybe the voice. We wanted to collect, and we still need to make that a priority, but is that like is now the time you know to be doing this, like their needs, or maybe something so different from what we are thinking, you know, and how maybe as museums, we reframe how we’re reaching out to those communities and maybe just being supportive and other ways. I’m trying to think of getting them through our gates or, you know, pulling this data and information from them. How can we maybe support those communities? Now in different ways to because, you know, again, just kind of turning this on that equity piece.

Laureen Trainer: And I think one of the things to just, again, think about is wrapping up is also just being true to your, to your mission and we all are not there are a lot of exams and they all do not have the same mission. They all are not feeling the same niche within their community.

And so really focusing the evaluation within the lens of your mission and your community and really trying to understand what’s right within those two filters, as you think about this.

But that said, always making sure that evaluation is the question that goes through those two filters.

And that with these huge, monumental decisions that we’re making about reopening and doing huge system changes within our institutions and evaluation needs to be part of that change at every level.

Within the mission.

Michelle Mileham: So, with that Sarah is going to see if you wanted to chime in, to wrap up any of the chat or any unanswered questions, we may be gotten

Sarah Cohn: Yeah, I think most of these with the open questions down in your Q&A boxes. I think we’ve touched on a bunch of them in different ways.

And given our last six minutes. I’m just wondering about Lauren just shared more for perspective on sort of the importance of this but you all and in the chat. We’ve raised a lot of problems and a lot of questions and even and there’s a question here even known of some museums don’t have funding or haven’t really put a priority on evaluation before the crises that we are in in this chain state of change that we are in. So how do we make the argument to add something more now. So, if you all have pieces.

Or stories that have worked or things that you’ve really seen your institutions or your, your client museums have changed in these last few minutes, that would be, I think, really great.

Emily Craig: So, I, I have not seen a lot of change. Yet I think one of the big things I’m thinking about right now. In addition to the fact that now is probably it’s a challenging time because we all know we’re entering a period when budgets are going to be very tight and push making that push, whether it’s to hire an external evaluator to take on a larger project or you know, you have all this, this reason behind you to push for an internal evaluator and say this is an important role that should be represented within our museum staff. Why does no one on staff have this title.

And I recognize that that I’m not helping with solution, Sarah. But you know, I recognize that that’s a balance that you have to strike.

I think one of the things that I’m really thinking about right now as I you know I talked to my colleagues about what we’re doing and who were reaching out to you also want to avoid. We, we need to think about creative ways. And when that that data that we’re collecting will be meaningful and will be something that we can implement so that we don’t just send out survey after survey after survey to our volunteers and our staff and our visitors and our members.

And either the information isn’t designed in such a way that it aligns and you can actually take all of those voices and make intelligent decisions from it or people just get burnt out because they feel like all you want to do is ask us questions and it’s some, you know, I think that that survey paralysis can be very real and you don’t want to turn this into a time of just saying, let’s ask everybody everything and gather all that data. So, I think I’m thinking about this from obviously there’s a lot of information we want and a lot of information we need but making sure you’re taking those steps in a meaningful way.

And so that from our from my perspective, maybe then a year down the line when the budget isn’t quite as scary anymore. You can say remember when we were in that really challenging time and we asked these questions, and we got these answers.

If you had someone on staff full time as an evaluator. If you set aside money to hire an external evaluator, you could do this. So, you know, sort of setting using this as a time to be thoughtful and cautious.

In terms of how much feedback, you’re trying to get and then really using that as a jumping off point to push a year down the line. Two years down the line and say, find the money in the budget. This is something we need.

Michelle Mileham: And I can also say evaluation is not something that is necessarily funded at my organization and is a very small part of my job and is mostly so because I see the value in it and I’m really lucky that I can say, I think this is worthwhile to pursue and my bosses usually supportive of us doing so and kind of justifying rearranging staff time to do that. So, I think it’s also just making sure that what you’re asking. And what you’re hoping to evaluate really comes from a place that it’s going to be useful to you. And I think that’s the strongest argument you can make is like right now.

We know we all have to make changes to our practices and a lot of different ways. And so, the way you can say like evaluation is going to help us decide what those changes need to be and that will help you justify it, but I know again like that’s not a good fast solution. But I do think, you know, keeping it really simple and looking, you know, just that one question or one thing that you want to change right now.

Sarah Cohn: Yeah. And I’ll just add to that, like, I’m thinking about the role of evaluation beyond call actually collecting information from the smaller number of people who choose to walk through your front doors in the first month of reopening even if you can’t make the argument to start implementing structured data collection press practices at like all of these different points in your various cycles of brainstorming and developing experiences, thinking about how can you implement some of the evaluation.

The ways of thinking around evaluation in terms of reflection and questioning is this decision to move everything online. Who does that automatically cut out from being able to connect with us? What are the, what are the all of the possible outcomes or impacts of a decision that we make to change something and what perhaps what barriers were perhaps there with the process that we had before the structure that we had before. So just thinking, bigger than just the data that you can gather, but the conversations that you can start having internally.

Pass it back to another panelists to close.

Laureen Trainer: I think we actually have hit time. Oh.

Thank you all for joining us. Thanks for my fellow panelists there for helping about and hopefully we raise questions that will spark new ideas, creative ideas innovative ideas as we continue to move forward, and we look forward to seeing what everybody does. In this new space.

Thanks so much. Thanks everyone.