Essential Evaluators seeks to gather evaluators in a common space to dialogue, reflect, and support each other in a world upended by COVID-19 and the Black Lives Matter protest movement. This is a time of uncertain and unknown expectations in our professions, in our institutions, and in our communities. We invite you to join us as we rethink, revision, and ultimately redefine our roles as evaluators and our place in museums. #EssentialEvaluators

Other posts in this series:

• A New Era Emerges

---

During the past month, a few brave voices have emerged from the corridors of the museum world to boldly and defiantly confront institutional racism. Some of these voices have recounted painful personal stories of being ignored or blatantly discounted when raising issues of social injustice. Others have joined forces anonymously to create a cacophonic narrative. Still others have taken a tongue-in-cheek approach to parody scripted responses from leadership.

One of the most striking, though largely overlooked, aspects of these collective narratives is the degree to which fear is used to silence and suppress our voices—fear of retaliation, of termination, of not being promoted, of not being able to find another job in the post-COVID crisis. It’s a pernicious fear that has long been used to preserve the status quo, even as most institutions would not claim to promote it. Yet museums have tacitly managed to sidestep real-world issues like racism, sexism, and classism through a carefully scripted choreographed dance. They have moved in accordance to a well-rehearsed pattern of rhythmic steps—shallow listening, hushed conversations, rhetorical musings, and token appointments—that essentially serve to keep us all in line, stepping to the same beat and moving in the same pattern.

Skip over related stories to continue reading article

In the evaluation field specifically, this fear can manifest in looking for “good numbers” instead of those that reveal uncomfortable truths, avoiding transparency in data collection, and reaching out only to those respondents who want to talk with us.

The pressure for museums to diversify audiences can compel evaluators to “prove” that institutions are already making strides, rather than pointing to where they need to improve. The dance between remaining relevant and needed in our institutions and providing the “right” answers can drive evaluators to solutions that may reflect well on an institution or program but require the evaluator to manipulate data in ways that challenge our rigor and professionalism.

This pattern can look like an example witnessed by our co-author Andréa:

“While visiting with an evaluation team at a science museum, they walked me through their visitor demographic information. I was really interested in how they had managed to serve Latino/a/x visitors in much higher proportions than their city and surrounding area. They explained that members of their team would stand on a bridge above the entrance and ‘eyeball’ Latino/a/x visitors as they came in on special event days. I was shocked; it was a well-respected museum and evaluation department. While this is an extreme example of unethical and unprofessional evaluation practices, it is not the only experience I have had over the years related to data collection efforts that seek to bolster diversity, access, and inclusion efforts—efforts that keep the dominant narrative in-line and alive.”

In another example, a children’s museum gathered visitor demographics on select days throughout the year, including all of their free days and nights, which had a higher proportion of diverse visitors. The result of their sampling strategy was that free days and nights were overrepresented in their data, thus providing a picture that their overall audiences were more diverse.

---

If real change is to finally and fully emerge in museums, it is critical that the field join forces to disrupt this rhythm. We need a new dance, one where fear and silence are no longer part of the beat, where data is data—neither “good” nor “bad,” just more information and opportunity to help reflect, learn, pivot, and plan. To spur evaluators on in this, we want to highlight some examples where our field has pushed for—and normalized—inclusion, equity, and accessibility:

Language—From the Collaboration for Ongoing Visitor Experience Studies

“From its inception, COVES has encouraged collective growth and change among our participating institutions; sometimes survey questions pushed our participating institutions to ask things they would not have otherwise considered, and sometimes our participating institutions pushed us to change our questions to make them more inclusive. As a collaborative project that includes over 30 museums, we know that we have the power to shape the trajectory of visitor experience research, and the questions we ask matter.

One example from our time as a collaborative is when our partners at the Exploratorium translated the bulk of the core COVES survey into Spanish, and others at the Science Museum of Minnesota and the Museum of Science, Boston updated those translations as we made changes to core questions. This ensured we did not systematically exclude our Spanish-speaking visitors from our research, and we continue to work to incorporate more languages.

More recently, we changed a scale question where the high point was the word “Superior” to use the word “Outstanding” instead. Members of our survey review team pointed out that word “superior” is associated with pseudoscientific claims about racial hierarchies, and a partner institution shared that they had indeed encountered visitors who felt uncomfortable with the word.”

Inclusion—From the Denver Museum of Nature & Science

“After years of community partnership, the Museum was interested in gathering information about if respondents identified as part of the LGBTQ+ community. We worked with LGBTQ+ community members, whom we had developed strong relationships with, to help us examine whether gathering visitor demographics on these communities would be helpful or not. Through this partnership of trust and conversation, both internal and external stakeholders felt that a question on the visitor experience survey would help identify any gains in community visitation as well as ensure LGBTQ+ community members felt welcomed, important, and seen.

The Museum’s LGBTQ Core Team worked with community members to craft a question and responses that felt inclusive and comfortable. These stakeholders also helped change the language of our gender-based question, which initially included man, woman, transgender, and gender non-conforming. Based on their feedback, we changed gender non-conforming to non-binary and added ‘prefer to self-describe.’ The community then challenged us to think critically about why we needed to collect data on gender at all. By staying open to that critical reflection, we were able to examine how we use that data and have since removed the gender question from our standard list of demographic questions on our visitor experience survey all together.”

Accessibility—From the Denver Museum of Nature & Science

“To help focus the work of our Accessibility & Relationships initiative team, the evaluation team decided to ask the question: Would accommodations in any of the following categories have benefited anyone in your group? We worked with community members to provide us with guidance and feedback about appropriate language for the options. Together we decided upon these options: physical/mobility, hearing, developmental, intellectual, autism spectrum, dementia/memory loss, and none of the above. Moving into data collection in the time of COVID-19, we added “Compromised Immune System” as another choice.

This example showed the evaluation team that survey-writers impact not only the types of data collected but ideas of welcoming, representation, and belonging; important factors in creating spaces that are truly diverse and inclusive. As we navigate the ‘new normal’ of COVID-19, which feels ever-changing, we must continue to be flexible, open, and reflective about the questions we are asking, how we ask them, who crafts the question and how these decisions impact the kind of data we receive.”

Staff—From a not-to-be-named “Public library with historical collections in an urban setting”

“As one example of using evaluation to correct a problem related to bias and racism, I looked at current staff and reviewed personnel and anecdotal records from the prior 10 years to confirm that, as a personnel director once said in a seminar, ‘there was a distinct pattern of staffing with people who look like me.’ In a community that was 60/40 African American/white, the staff was almost entirely white, and overwhelmingly female. I also recommended the development of policies around anti-bullying and anti-harassment (this was about 10 years ago) with enforceable penalties and also the development of hiring guidelines that called for broader posting of vacancy notices and of making hiring decisions based on qualifications including education and experience. Within 18 months, the changes in staffing and in behavior were apparent and well-received by the community; visitors and other staff alike. Staff who did not appreciate the changes, for the most part, self-selected out of the agencies.”

---

As these examples show, evaluators have instigated change; however, we need to do more. We must disrupt the pattern of fear and silence. We must be liaisons for both audience and community engagement, keepers of empirical and objective data, and stewards for institutional ethics and integrity—all absolutely essential for revisioning and rebuilding museum space to be inclusive, equitable, transparent, and accountable.

We believe that it is critical for all evaluators—employed, unemployed, part-time, internal, and external to join together and share their voices and experiences so that we can help build a robust and complete portrait of evaluation in museums. For many of us this is a new role and will most likely be an uncomfortable one, but we need to take a look at our place in this new landscape—what has changed? What questions do we need to be asking? What should be asked of us? How do we use our position to advocate for our communities? How do we embrace the social justice directive to become visible and viable? How do we speak up, stand up, and demand a seat at the decision-making table?

As part of this call to action, we ask you to send us your stories and examples. When have evaluators challenged institutional norms and injustices? To what end? When have you seen evaluators bring the voice of diversity, equity, inclusion, and accessibility to the table? We want to shine a light on the work that has been done in the field to show others what can happen when we dance our new dance. We also want to show how much work needs to be done. Where have we not stepped up and challenged social injustice? Where do we need to do better? Tell us, either in the comments or at trainerevaluation@gmail.com!