This visual Data Story is based on findings from the 2024 Annual Survey of Museum-Goers, a national survey of American museum visitors from AAM and Wilkening Consulting. Every year, the survey partners with individual museums to research their audiences and yield insights about their behaviors and preferences, both on an institutional and national level. Interested in joining the 2025 edition on the themes of repeat visitation, social connection, and community trust and responsibility? Sign up by February 28, 2025, for a special early bird rate.
Sometimes data challenges us. Analysis often reveals surprising, and sometimes not so surprising, patterns. As we dig deeper and learn more, the findings start to make sense.
Skip over related stories to continue reading articleBut when the data just doesn’t make sense, like that time a children’s museum survey showed the majority of respondents were men without children, we knew immediately what had happened…
They got botted.
Each year, the Annual Survey of Museum-Goers is distributed by museums mostly via their contact lists, as well as social media. Social media platforms allow museums to reach respondents that their contact lists may lack, but this can open up the potential for “botting.”
What’s a “bot”?
Unfortunately, sometimes malicious web-crawlers comb social media for online surveys, spamming them with fake responses. These fake respondents or “bots” use automated tools to complete surveys many, many times and often very quickly.
Why spam a museum survey?
Honestly, we’re not entirely sure. A few years ago a large percentage of bots appeared to be coming from Russia, In that case, we assumed it was intended to indiscriminately mess with any U.S. based survey they could find.
Now, the raw data says most bots are coming from within the United States. That could be the case, or it could also be evidence of more sophisticated tactics of the people behind the bots.
We have found that the small prize drawing incentive offered for completing the survey attracts bots, just like bees to honey. And this is what web-crawlers are hoping to benefit from when they spam a survey on a large scale. Essentially, they’re trying to stack the odds in their favor!
Either way, bots are more than a nuisance…and they can skew your data!
So how do we know which responses are legit or not?
Here are some of the red flags we look for when we suspect bots have infiltrated a survey:
- A cluster of surveys that were started and/or completed at the same time or within seconds of each other
- A large number of responses from the same IP address
- A larger than normal number of responses from out of state and/or country
- Email addresses that consist of jumbled letters and numbers or have names/numbers in a pattern
- Inconsistent and/or contradictory answers to questions
- Ratio of completed surveys and partially completed surveys is out of normal range (it is normal to have between 15%-20% of your surveys only partially completed)
- Repetitive, patterned, nonsensical, irrelevant, and/or less than genuine responses to open-ended questions
- Results that don’t make sense for some museum types (like our children’s museum example above!)
- Surveys that were taken very quickly, sometimes in seconds
Some of these potential red flags on their own may be normal and not necessarily a bot, which is why we conduct thorough data cleaning.
Because participation in the survey is voluntary and questions are not required, data cleaning becomes a subjective process. Therefore, we use our brains (and not computers!) to pre-sort suspicious responses and to clean data. That’s because our brains are now trained in the nuances of responses to make the final decision on the quality of each response and how many red flags is enough to identify the response as a bot.
While we do implement many automatic strategies (sorry, we’re keeping those confidential because we don’t want botters to know our secrets!) to help stop bots from attacking, our experience suggests that they will still find ways to sneak into a survey.
But rest assured, we’ll continue to diligently comb through the data to spot bots—looking for those red flags and then cleaning up the data—to make sure that collected responses are reliable and accurate!
Annual Survey of Museum-Goers Data Stories are created by Wilkening Consulting on behalf of the American Alliance of Museums. Sources include:
• 2017 – 2024 Annual Survey of Museum-Goers
*Data Stories share research about both frequent museum-goers (typically visit multiple museums each year) and the broader population (including casual, sporadic, and non-visitors to museums).
More Data Stories can be found at wilkeningconsulting.com/data-stories.
Comments