You’ve figured out your evaluation questions and methods and collected your data, but now what? Join us for a CARE-apy discussion about what happens after your evaluation data have been collected. We will talk about considerations and processes for analyzing and ways to share data that anyone can implement to make sure their data are used. We will also show examples from various projects to help you think about these processes.

Transcript

Ann Atwood:

Hello and welcome everyone to this therapy webinar on analyzing data for action. This webinar is produced by the Committee on Audience Research and Evaluation. One of American Alliance of Museums’ professional networks. Therapy webinars are intended to connect our members and share professional expertise. My name is Ann Atwood with the Museum of Science in Boston, Massachusetts. Today, we’ll be talking about analyzing data and sharing evaluation findings with Sheila Brommel from the Minnesota Historical Society in St. Paul, Minnesota, and Liz Kollmann from the Museum of Science Boston.

This is the third and final webinar in our Getting Started with Evaluation Series, but please stay tuned for future offerings about other topics later this year. I’d just like to share a few quick housekeeping items before we start. We are recording today’s session and we will have time for Q&A at the end, and interspersed through the presentation. In the meantime, I’ll ask that we all stay on mute. If you have any questions during the presentation, please enter them into the chat and we’ll address them during and after the conversation. We’ll plan to make this webinar available on AAM’s YouTube, and it should be available in three to six weeks. These webinars are kept free of charge, so we’d like to encourage you to consider membership with the American Alliance of Museums. With that, I’ll turn it over to Sheila and Liz.

Liz Kollmann:

Thank you, Ann. And welcome everyone. We’re excited to see you at our session today. I don’t know if any of you all have been at the previous sessions for this series, but similar to the last session, we have slides that we’ve created in Google slide that you can look at on your own as we’re kind of going through the presentation. You should be able to view it. You probably… You shouldn’t be able to change it, but there are links to some of the bigger reports around the examples that we’re going to be providing that are in there that you can look at at your own time. We’ll keep sharing this throughout, so let us know if you do not have access to the presentation. Sheila and I are very excited to meet with you today. If you go in the second slide of the presentation, this is just our contact information. As Ann said, my name is Liz Kollmann. I’m the Director of Research and Evaluation at the Museum of Science in Boston. I’ve been working as an evaluator for, I think 16, almost 17 years now. I’m very excited to talk with all of you. Do you want to go ahead, Sheila?

Sheila Brommel:

Sure. Hi everybody. I’m Sheila Brommel. I am Evaluation Specialist at the Minnesota Historical Society in St. Paul, Minnesota. I have been there for 10 years, doing primarily evaluation across our network of 23 sites throughout the state. I too, am excited to be talking about some of our reporting methods and how we actually use the data.

Liz Kollmann:

Yes, yes. If you look at slides three in our Google presentation, the previous two webinars in this series spent some time talking about questions. How do you form questions and think about the kinds of goals you have for an evaluation, as well as how do you collect data and what does that look like and choose methods. But as Sheila said, today what she and I are going to focus on talking with you about is how do you analyze your data and how do you make changes and report on that data in order to use the data as you are completing an evaluation. We can definitely reshare the Google slides.

If you look at the agenda, we’ll go through a number of different pieces today in order to help you think about analyzing and reporting your data. On slide four, you see, we’ll spend some time talking about how do you prepare data for analysis. Then we’re going to talk a little bit about the analysis process itself, visualizing data once you’ve done the analysis, reporting it, using it for action. And then some resources to do this work on your own. As Ann said, please feel free to put questions in the chat. We’re hoping that we can kind of answer questions that you might have or hear from you along the way.

Sheila Brommel:

Oh, we have introduced ourselves, but we wanted to hear from the audience.

Liz Kollmann:

Yes, please.

Sheila Brommel:

We have a poll.

Liz Kollmann:

Thank you, Sheila. Yes. Can you put up the poll, Ann, so we can see a little bit about who’s in the room with us today? We’ll just take a few minutes, if you can answer this poll about describing your role in evaluation reporting. It looks like maybe most folks have answered it at this point. Hopefully you all can see the results. It looks like about half of you say you both create reports and use them about a quarter saying that you are primarily users of evaluation reports. Then a few folks say you primarily write or create reports. And some of you say you don’t do any of those things. That’s helpful.

Sheila Brommel:

I’m curious, those who neither write nor create reports, or use the reports, is that because they don’t have access to reports? Is that why they don’t use them? I’d love to hear a little bit more from non-users.

Liz Kollmann:

Yeah. We’d love to hear a role in the chat.

Sheila Brommel:

Yeah.

Liz Kollmann:

We’ll just… Should we keep moving on?

Sheila Brommel:

Yep. We’ll keep on moving. All right.

Liz Kollmann:

But this is helpful. It sounds like a lot of you have done this before. Most of you have done this before. Hopefully we can give you some new tips and tricks and share some examples of things we’ve done that might spur some ideas for you. If you look at slide five on our Google presentation, we’ll start talking about the data analysis process first and just, as you’re going into that process, you have your data. This is great. What do you do with it now that you’ve got it? Sheila and I just put in some questions to kind of think about as you’re getting ready to analyze your data. Some things to take into consideration is how should the data be entered so that you can easily analyze it? What kinds of analyses can you do with the different types of data that you have and what evaluation questions can you answer with your different pieces of data? Sheila, and I’ll share some thoughts about each of those pieces as we go through the next few slides, but please, like we said, ask any questions as we go through these sections.

If you look at slide six, I just wanted to share a quick look at what a data entry sheet might look like. You can do this in lots of different ways. You can enter data in lots of different ways. Sheila, I’d love to love to hear from you how you might do this differently, or how you feel about this. I think for me, no matter what kind of data you have as you prepare to analyze it, the first thing you want to do is make sure that it’s organized. You want to make sure that… For me, that often means that I need to put it in some kind of an electronic form so that I can kind of understand what is the data for each individual participant and look at it in all one place and be able to easily analyze it. This is just one kind of way you might do that.

You can see on here, in that column A, in some ways that’s my identifier for each participant. There’s actually a better identifier that I had as well. You want to make sure that you have some way to identify each subject so that you can keep people straight. You usually do not want to use names obviously, but you need some kind of another identifier there. For quantitative data, you want to think about what’s the best way to enter it. In this case, it was, as you can see in columns like S and T, those were Likert Scale questions with particular anchors. I entered it as text. That’s often probably not the way you might want to do it. In this case, because I had so few categories that I was looking at in the quantitative data, it was kind of easy for me to have it as text, but you might want to turn those into numbers to help you more easily analyze it in different ways.

Whether you decide that doing means or accounts are the easiest way to do the data analysis, you still might want to change that into numbers rather than text. Another thing is the quantitative data, you can see some of that, example of that in column U. That was just… It’s all in one place. We’ve removed, in this case, any spelling errors and made sure it’s clean. Again, we can kind of identify it with all the other data that person gave us. Sheila, would you add anything to that? What are some other tips that you use?

Sheila Brommel:

Yeah. Well, and I want to clarify that I am the sole evaluator at the Historical Society. So it’s up to me to do most, any, or all of this. I don’t get very fancy and I don’t run anything very sophisticated. My audiences are typically just interested in frequencies. We happen to use Alchemer as our online survey platform. I’ll typically… Alchemer, their ready made reports are fine if you just want to get a quick glance at what the data is saying. But they’re actually pretty cumbersome, so I typically export the data into an Excel spreadsheet. I may manipulate the spreadsheet to set it up in a way that’s usable for me, but again, I just rely on Excel. It does what I need it to do.

Liz Kollmann:

That’s a great transition, Sheila. Because the next slide, if you look at slide seven, this is an example of those same data I was just showing you as a spreadsheet where I was doing the analysis on the data. Similar to Sheila, most of the analysis I do is also kind of pretty simple and mostly counts and percentages. This just shows how I turn some of the quantitative data into just quick tables for myself with those counts and percentages, which let me in Excel do some easy graphs.

Sheila Brommel:

Your spreadsheet is much prettier than mine are. You are very organized, Liz

Liz Kollmann:

The pictures are covering other texts, so this is definitely not neat, but this is the way my brain works in doing this. I often use Excel to do some of my easy quantitative analysis too. It’s only if I know I’m going to do something more complex, like needing to do some influential statistics or do a lot of splits of my data, that I’ll move it into something like SPSS or R or Stata to do those analyses. Shouldn’t be intimidated to use tools that you may already have. R is free, but you have to know some programming language in order to use it well.

The next slide is just an example of what it might look like when you’re doing your qualitative analysis. This is actually a way of analyzing the data that I don’t use too often, but worked well with these data that I had. I kind of had short responses that I was trying to code. We already had somewhat fixed codes that we knew we were going to use. Well, we were looking for some codes to emerge, but we had some that were existing as we were analyzing the data. What I decided to do with my team in this case, was kind of put my codes across the top of the spreadsheet, and then keep my responses along the left side. Then, each time I saw a code within someone’s response, I put a one underneath that code. Tat helped me kind of be able to quantify how often these themes and these codes were coming up within the data, and also kind of just understand what happening within it more frequently. Sometimes I also split. I do a similar sheet to this, but instead of putting ones, I’ll split up a response so that I have the part of the response that fits that code within my spreadsheet. This was just this… There’s all kinds of different ways to do this. Sheila, are there other ways that you think about it?

Sheila Brommel:

No, this looks pretty much like my spreadsheet does from Alchemer. One of the features that I like about it is that it does allow for easy coding of data right within the software itself. It’s just clicking boxes, anytime a response fits a particular code. Then again, I export it into an Excel spreadsheet and it’ll look almost exactly like this and I can manipulate, do what with it what I want from there.

Liz Kollmann:

I think what we wanted to move into next was starting to talk about data visualization. But before Sheila and I move on, are there any questions that people have about kind of preparing data, or tips or tricks that you’d like to share with the group?

Ann Atwood:

I don’t see any questions right now, Liz, but one thing I sometimes do when initially looking through qualitative data, is I go real old school and might print it out and use highlighters to start thinking about what kind of patterns I’m seeing come out.

Sheila Brommel:

Yeah. I was just about going to mention that. That is also a fun option, is to go more old school and color code with markers and highlighters and such.

Liz Kollmann:

I’ve also cut up responses and use that to physically move them around to, again, look for themes. There’s all kinds of ways you can do that kind of analysis. Okay. We’ll keep going. After you’ve kind of set your data sheet up, it’s time to start thinking about how you’re going to visualize the data that you’ve analyzed. Some things to think about as you get into this stage are just what kinds of data do you have? What is the story you’re trying to tell, or what is the question you’re trying to answer with the data that might impact how you visualize it? And what kinds of visualizations will be the easiest or the best for your audience to understand? What will work best for them? Sheila, I think now the next few slides are for you to talk about.

Sheila Brommel:

Yep. In the next few slides, I’m showing some examples of some of our charts and graphs that we’ve used. This first one on slide 10, is just basic visualizing quantitative data. Something that I want to do when whenever I can, is to try to give some context to the data. I will try to make comparisons, either with the current year and the previous year or in this case, I’m looking at the current year and I’m looking at 2019, so pre-COVID pre-pandemic. We went into shutdown for two years, so are patterns changing? In this case, we were looking at reasons for visiting, and we did see some distinct changes in why people were visiting. By giving that just a little bit extra context that lets people see, are the results changing? Are things getting better? Are they getting worse? Are they same?

Then also when I’m preparing a report, that the title of each of the charts or graphs, I try to make descriptive. In this case, I highlighted 71% said their reason for visiting was something new. I could have been even more descriptive and added that this is a change from pre-COVID. It was an increase of so many percent, but I try to, most of the time give people a clue as to what they’re looking at. Because not everybody is used to looking at charts and graphs all the time, and it can be overwhelming. They just, “Wow, what is this?” It’s nice when you can give a very descriptive title to your particular chart. I did put a link in here, so you can see the full report/presentation.

That’s one caveat that I would make. We have a history of making the nice long, 30 to 60 page reports that rarely got read. From there, I went to a streamlined version of of a paper dashboard report. Those were more like four or five pages, sometimes six pages. But after the pandemic and we had layoffs, I just didn’t have the capacity to keep producing those reports. It’s been easier for me the past two years to just create charts and graphs and put it into a PowerPoint. I will meet with my teams and we review it as a presentation, and I share the presentation with them and that kind of serves as their report. You can click on this and see what the full report looks like.

In slide 11, I have a screenshot of our Tableau dashboard. We’ve tried to identify some common indicators we can use across the institution and put those into Tableau. The reason for using this is because it’s really easy to be able to filter by different things like in our case, historic site. I will have 12 to 14 historic sites listed. We can filter by fiscal year. Various demographics, we can also filter by. We’ve had this for probably six, seven years now, so people can see trends. If they have more than three years worth of data, you can, you can start to see movement in your data. Is it going up? Is it going down? Is it staying the same? Is that the direction we want it to go? The filtering option is very interesting.

One time a director, he just cut to the chase. He wanted to know who the detractors were in the net promoter score. Those are the people who are least likely to recommend a particular site. He just clicked on the lowest score, and by doing that, he could see the demographics. Here it was men visiting with young children were, were the least likely to recommend. So go figure, they were the least satisfied. That’s kind of an example of what we do in Tableau.

Another type of report that I’ve tried in slide 12 is a screenshot of a data place mat. This was actually done on, the extra large pieces of paper. What is that, 16 by 18 or something. Bigger than the legal size, but it’s the large pieces of paper. In this case, I added the graphs, but you’ll notice in the title I intentionally did not include a descriptive description for each graph, because that was part of the group process. I wanted the group to think about and talk about, and as a team determined for themselves what they were seeing in the data. So data placements are sometimes used at data parties, and that’s a way to get higher engagement with the groups that you’re working with. That data placement was done right before COVID hit, and I haven’t had a chance to do anymore, but that was pretty well received. Slide 13 is one of your examples, Liz.

Liz Kollmann:

Yeah. Yeah. Sheila just showed a lot of nice examples of ways you can visualize quantitative data. But, with your qualitative data, there’s a bunch of different ways you can visualize it as well. This just shows that, as I was talking about earlier, when you’re coding, especially that qualitative data often, especially if you have a big enough sample, you can quantify it or you want to quantify it. You want to understand kind of how prevalent different themes are, how often they, they came up, and what you can understand about your data based on that. This just shows you an example of just one way we quantified it. In this case, we did turn it into bar graphs. Often with qualitative data, we’ll just have kind of a table with the code, the number of people or the percentage of people that kind of hit that code, and then example quotes.

With the qualitative data, really, those quotes and having that rich data is so important. So you definitely at least want to always have some of your quotes used in your reporting. Sometimes you might realize your data, it’s really not important to know how prevalent a theme was, but you just want to be able to explain the different kinds of themes, the range of themes within your data. In that case, you might not want to show the quantitative aspects as well at all. You might just want to have lots of rich examples and just explain what was in your data. This is just something to think about with qualitative data.

Sheila Brommel:

Slide 14 is another way to visualize qualitative data, kind of using a heat map. This was a survey that I did seeking public input on a potential name change for one of our historic sites. We had over 7,000 responses. Imagine a survey, 7,000 responses closed and open ended questions. I got to analyze all of that and turn that into a report that would make sense to others. One of the things I did as kind of a summary graph, was across the top, I have the nature of the suggested name that people were coming up with, and down the left hand side were the different reasons why they would suggest that particular name change. Then I have the numbers of mentions that appeared. Then all I did was highlight the different cells, so the darker colored cells are the more prevalent times something showed up. The blank cells are cases where it didn’t show up.

People could see at a glance how prevalent something was. In this particular report, I went on to actually give quotes for all of these different categories. A heat map is a nice way to add a glance, get a sense of what you’re looking at. Another type of heat map shows up in our Tableau dashboard. This is answering the basic question, where are visitors coming from? The heat map was applied to the counties in our state, darker colors, representing the counties that more of the visitors were from. We do this in conjunction with some bar charts, so that we could spell out the actual numbers by county, and then also grouping by region. Are they in state out of state, Metro area, whatever? Another way of looking at heat maps. 16 is your slide, Liz.

Liz Kollmann:

Yeah. The next two slides, I just wanted to have us all think about… You can have the same data and visualize it in very different ways. This is just one example of that. In this case, this is a study where we were trying to understand facilitation that happened as educators were doing some hands on activities. We had a bunch of different questions about their facilitation. This was a pretty intensive analysis process. We had videos that we were coding, so I don’t think you need to worry about that. I think what you can see here on slide 16 is that one of the things we were just trying to understand is on average, what percentage of time did the educator spend doing using different facilitation strategies? This was just taking all the codes we had done and just seeing how often they came up with the educators.

That told us one thing. That told us that they’re spending most of their time in that supporting exploration category, and spending a fair amount of time in deepening understanding, but less time in other places. But if you look at the following side on 17, that’s actually the same data, just visualized in a completely different way, that let us answer a different question. In that case we were kind of really also interested, not only in how often these different facilitation strategies occurred, but when are they occurring, and what other facilitation strategies are happening near each other. Being able to visualize it in this way, which was really in the end, just an Excel sheet where we color coded different lines, and the top is the beginning of an interaction with an educator, and the bottom is the end.

We could kind of see, “Okay, they’re doing invite participation at the beginning, but they’re also doing that at other points throughout the experience, and that support exploration and deepen understanding kind of happen to near each other, typically. They usually happen one after another.” Just thinking about, you may have data that’s really rich and you think you have to analyze it and visualize it one way. But if you think about other questions that you have, you may think of other ways to analyze it that help you answer other questions. Sheila, do you want to take people through the exercise of some maybe not so good visualizations?

Sheila Brommel:

Sure. Yeah. I’m moving on to slide 18 now. I just want to say that by no means do I think I am perfect or an expert on this. I am a work in progress. I try to always improve my slides and graphs. I pay attention to how my audience is receiving it. If they seem to be understanding, if it’s making sense, if it’s not. Then I try to tweak it again the next time. With that, I have pulled out a couple graphs that I think could be more effective. I’m just curious, post in the chat, please, if you were to see this graph, what’s not working for you? What is wrong with this particular graph? I can’t see the chat, but I just-

Liz Kollmann:

I can read it to you. Someone said too many categories of data, too much information, too many slices or options.

Sheila Brommel:

Yep. All of the above. The first thing for me, when I… This is a funders’ dashboard that we have to use. When I open up this dashboard page, my instinct is to kind of shudder and turn away, because the colors hurt my eyes. They are… Maybe it doesn’t come across as good right now, but in the actual dashboard, they are downright fluorescent, and they hurt my eyes to look at it. Yeah, too many categories for a pie chart. Even the idea that they use, the 3D effect, that’s not necessary, and that can be misleading. It can make the slices at the bottom look like they represent a bigger area, because they have that 3D edge. Pie charts make it difficult for our brains to easily view and register the proportions. It’s just a harder thing to calculate angles. That was kind of my reaction in that pie chart, or that vis. Here’s another one on slide 19. Okay. We get rid of the pie, we move to bars. What’s not working for you with this bar chart? Liz, do we have any comments?

Liz Kollmann:

No. Put them in order.

Sheila Brommel:

Yeah. Oh, yeah. Putting them in order. There unnecessary things on this graph. Like that background grid, that isn’t necessary. If they simply added a label to each of the bars, you wouldn’t need that grid in the background, or even the access on the left. Just add the number or the percent to the bars. Here again, I have issue with the color. At least it’s not fluorescent, it’s not hurting my eyes, but then I’m thinking, “Okay, if a person has color blindness, I’m not sure what these colors would do for them.” Especially the dark ones in the similar tones, if they would blur together or not. Okay. One more, just because I can’t resist. You’re going to get the impression that I don’t like pies. Actually I like pie. I just don’t care for pie charts. But what do you-

Liz Kollmann:

Lots of people agree with you about the colors of that last bar chart.

Sheila Brommel:

Okay. How about this visualization? How easy is this for you to make sense of the data?

Liz Kollmann:

Someone said some are more opaque than others. Can’t tell how many slices referred to. Way too busy.

Sheila Brommel:

Yep. Yeah. I don’t know if the opaqueness means something, and that some are translucent. Does that mean something else? Not all of the slices are labeled. Does that mean they just had a smaller percentage? But even… This gets back to the descriptor. Even the labels, the fact that they are using one decibel point, how necessary is that for your understanding of the data? Most times I’m going to round the numbers because it’s not necessary for us to get to that level of detail, and it’s just easier for people to comprehend a rounded number. If I was working in finance, maybe decimals would be more important to me there, or if I was doing a higher end statistical analysis. In our instances, it’s not usually necessary. In fact it makes it easier to understand if the numbers are rounded. Anyway-

Liz Kollmann:

We’ll take a pause here. Are there any… I didn’t see any questions, I don’t think. In this section, I did see a great example report from one of our participants with some COVES exit survey data. That’s in the chat. Does anyone have any questions kind of just around data visualization or thoughts? Okay. We’ll keep moving on, yeah.

Sheila Brommel:

Then I’ll let you go on to slide 21, Liz.

Liz Kollmann:

Sure. Now that you’ve analyzed your data, you’ve figured out the best ways to visualize it, it’s time for you to start thinking about how are you going to report it to your audience. Some issues to take into consideration here is who is the audience that the report is for? What format will make that data useful to the audience. And how can you facilitate the use of the data by your audience? I think that’s really… That last point is especially key. You want to make sure if you’re going through this whole process of collecting data and gathering it from your visitors, that it gets used and gets put to some action. There’s a whole variety of ways to report data. Sheila’s already showed us a bunch of different ways.

I couldn’t let us go through this session with just mentioning formal reports. There are times when that is the way to go. You can see kind of an example of one that I’ve worked on, on slide 22. But I think the issue with formal reports, these are good if you want to share findings with an audience that maybe does not know the context of your project as well. If they need the background, or if you need to preserve that understanding of the background and document it in some way, then formal reports can be really important. Sometimes you really just need those for your funder. Your funder really wants it. You want to make them as useful as possible, and there’s certainly a place for longer formal reports.

They can also be really good if you just need to have that information long term, for your institution and you want to make sure they understand exactly how the work was done and what you found and all the detail level of that. More often than not, that might not be the most useful way of presenting data for an audience. Slide 23 is just kind of another example of another way that you can do things. This is just a short visual executive summary. This is one that we did for a Nice Net project. We worked with a graphic designer. You can probably tell on this visual, there’s no way I could make such nice pictures and do such nice layout on my own. This also still let us give… It was actually ended up being a four page executive summary, in this case. We could give some of the context, some of the background information, but it really kind of let us pull out what are the key findings, and show them in a hopefully more visually appealing, easy to digest way. So that if people just need to get takeaways really quick, this is a way to kind of present it for them to hopefully get those quick and easy takeaways. Sheila, I think the next ones are for you.

Sheila Brommel:

Yep. Again, I’m going back to a clip of our Tableau dashboard. I’m using this as an example of giving the audience what they want. We do a basic general visitor survey across all of our historic sites. Each of the sites are very unique in what they do, what they have to offer. It never occurred to me to compare them side by side, but our site managers seemed to be a competitive bunch and they were always asking, “Well, how did I do compared to the other site?” It’s a friendly competition, but they wanted that information. So we added that in our dashboard, and our dashboard is set up so that we can download and we can print these off as PDF reports.

Here, we list the sites. The vertical line represents the institutional average. I guess this has turned into kind of internal benchmarking, if you will. That’s a popular report within the dashboard. Slide 25 is another kind of… It’s a one pager answering one specific question, but what we did was take the data that we had entered into the funders’ dashboard, and we turned that into a visual depiction along with written explanation, to answer their specific question of how is cultural knowledge transferred between MHS artists and their communities. We were trying to show… We had counts of things like the number of artists, the number of programs, the number of arts represented, but we tried to then depict the relationship among all of those. The funder was very tickled with this one page report that reflected the data that we had provided for them.

Oh, another kind of report. Oh, here’s one. That last report was actually put together by our research consultant, as is this report. We had done several different focus groups over the years, unrelated programs. An underlying question among all of them was, “How can we increase or improve diversity and inclusion?” It occurred to us that, “Gee, we should synthesize these learnings across all these different focus groups. What would that look like?” What they did was, it’s basically qualitative information, but indicated, which particular focus group and project that theme happened to turn up in. Again, that ends up looking like a matrix, and they just indicate where the themes occurred across the different types of focus groups. So another style of report getting at more qualitative information over a number of projects. With that, any questions or thoughts or comments on reports, the actual physical reports or types of reports? Or any other tips and tricks that you might have to share?

Liz Kollmann:

There is a question in there that’s been answered. That was about, for visualizations in particular, resources for what colors or more appropriate for low vision or blind color blind viewers. Kate put in a nice resource of a website that you can use to generate a color palette to make sure it works with color blindness. I think in terms of blind, low vision, you want to make sure there’s high contrast. I think there’s lots of websites and ways you can kind of places you can put your own graphs in and see if it fits these different issues that you’re worried about. Whether it be low vision, color blind, there’s lots of free resources on the web for that. Sheila, do you have any in particular that you use? I know we’ve used something like what Kate put in our chat.

Sheila Brommel:

Yeah. I don’t remember the websites off the top of my head, but yeah, there are several that are out there. =I usually just Google and then… I wish I could remember the name. I don’t, I’m sorry.

Liz Kollmann:

Sometimes just using black and white and different patterns within your graphs is the best way to avoid some of these issues as well.

Sheila Brommel:

Yeah. Yeah. I’m going to go on to talk about how to get people to actually use the data that you have provided in these reports. You’ve spent all this time analyzing your data. You have created meaningful graphs that should be easy to interpret and understand. You’ve gone to the trouble of creating whatever level of reporting mechanism. How do you get people to use it, and make changes because of it? We really fell into using the team based inquiry process. If people weren’t at the last session or aren’t familiar with it, TBI is something that was created by a network of science museums across the country, Liz being key to that effort. But it was part of a large federal grant they had, and they developed an evaluation method for non-evaluators, essentially. You’re still doing evaluation, it’s just… They made it more user friendly, in my terms.

In the reflection phase, what I will do is bring the group, my team together, and I create a Google doc ahead of time. I’ve sent them the report ahead of time. When we meet, we go through these five different elements. First of all, I will ask them, “What findings from the report did you see that were particularly successful? What are you most proud of? What worked well?” I start with that because I firmly believe that they need to celebrate their successes, and I will cheer them on. If they aren’t identifying things, I will jump in and identify it for them and say, “This, this number is really high. You should be proud of that. That’s really good.”

Then we move to kind of the flip side of that coin. “Which of these findings do you think might need improvement? What numbers are you not happy with? Is there a trend that you see that’s not going in the direction that you want it to be?” We have them identify areas for possible improvement. Then, the third step I ask them to just brainstorm. “What are actions or improvements or changes that you can make to the program based on what the data is telling you?” I want their ideas to be grounded in the data, not something that’s just anecdotal, or something that they were thinking. I want to make the connection to the data and have them brainstorm a list. The fourth step, I have them prioritize those ideas by cost and impact. The cost could be not only financial, but it could be in terms of human resources, timing, maybe something is just too costly for whatever reason, but then what kind of impact would it have if we did implement that change.

Then we go into creating an action plan. I suggest that they start with the low cost high impact. That’s the low hanging fruit. Those should be the things that would be easy to implement. Not going to be a lot of cost, and you’re going to generate more impact because you made those changes. It’s not to say that the high cost high impact things aren’t important. It just means that maybe that’s going to take more planning. Maybe we need to go to development to try to raise funds to make a particular change for something. It’s just going to take more time to figure out how to make that happen. Then we try to fill out this action plan, assigning a person to be the lead on a particular action, and assign a date by when you think it can be accomplished. And then to come back and check in 30, 60, and 90 days.

That’s the ideal world, and we did that initially. It was helpful. It was successful. We’ve gotten now to the point where we work in more shorthand, and we may not always go through the full process, especially if it’s a team that’s been used to doing this. I may do it, the full process, with a new team or a new project, but people who are used to it, they know the questions that I’m going to ask, and so they’re kind of prepared with their answers.

I have linked to an actual TBI document that we use. The other reason for recording this in a document, you can have it, then… It becomes part of your institutional knowledge. This is what we learned. This is what we tried to do because of it. The next year you can come back and evaluate again to see if your changes made a difference. The beauty of TBI, I think, is that the resources are available free of charge, and they are adaptable. You can change them to suit your needs. That’s my commercial for TBI, Liz. The science museums did not pay me to say this. I came up with this on my own. I really do appreciate the process. So thank you, Liz, for your role in creating that.

Liz Kollmann:

Oh, but let me say Sheila. Yeah I love this process that you have here too. Because I think you’ve adapted some of this from TBI. I’ve certainly used that idea of prioritization in a slightly different way with some of my teams, even my evaluation teams to say, “What is our high priority right now? Is it going to take a lot of work or not?” I think that’s really great. I think in general with evaluation, the key is whatever kind of report you write, you don’t want it to just hand it off to someone and let it sit there and gather dust. I know for me, I’d love to hear from you Sheila, having those setting up those meetings with the the key folks who are the people who need to pay attention to the evaluation, having conversations with them, sharing the data with them, understanding what’s resonating with them, talking with them about what changes or additional analyses maybe you can make, and talking through these changes is really key. I don’t want to just email a report to a team. I want to make sure we’re having conversations with them about the data as well.

Sheila Brommel:

Right. If I’m just sending a report out, I’m assuming that it’s the likelihood of it even being read is small. The likelihood of anything being done because of the report results are slim to none. So I really push to have this meeting this, this time for group reflection.

Liz Kollmann:

Yeah, that’s great. I totally agree with you.

Sheila Brommel:

On slide 28, we’ve added a couple resources that are out there. Chris Lysy is an evaluator who’s gone into data visualization and report writing. He has a whole blog on resources for report writing. Stephanie Evergreen is kind of the go-to person in the evaluation field for data visualization. She has a couple of great books, but even her blog has small bites of useful information that you can implement as needed. The NISE Network for Team-Based Inquiry. And the CDC has an evaluation reporting guide. Those are a few of our go-to resources. With that, do you have a summary, Liz? Concluding comments?

Liz Kollmann:

Sure. I think I just wanted to… On the last slide, it’s just reminding you of all the different presentations that we’ve had within this grouping of webinars. The therapy webinar on crafting evaluation questions was from March 30th. I don’t believe any of these are posted quite yet. Is that right, Anne? Just keep an eye out if you want to look at the full cycle, but there was one already, as we said about crafting evaluation questions, discussing data collection methods. Then today, what Sheila and I were doing was talking about analyzing data for action. Hopefully this gave you some new ideas of ways to think about data analysis and visualization and reporting that maybe you didn’t know before, or reinforced ideas that you already had. We’d love to know. I know we have a few more minutes left. If there’s any final questions for Sheila or I, I believe we can take them right now.

Ann Atwood:

If you have any questions, please feel free to enter them into the chat. Or if you’d prefer to go ahead, turn on your video, unmute yourself, and ask Sheila and Liz directly. Give everyone a minute.

Sheila Brommel:

If you can’t think of questions now, that’s fine. If you’re like me, you might think of it in the middle of the night, a week from now, please don’t hesitate to contact us. We shared our email contacts at the beginning. We love to talk evaluation.

Liz Kollmann:

Yes, yes, yes. Please feel free to continue accessing these slides. Our contact information is in the slides. Links to, like I said earlier, a bunch of reports that kind of we gave you examples from today are in there. And there’s links to those resources that we’ve been mentioning as well. Yes, there’s definitely no one size fits all for this work. You really have to think about your audience, and your needs, and your questions, and your data.

Sheila Brommel:

Well, I appreciate you all taking the time to spend with us today.

Liz Kollmann:

Yes, thank you.

Ann Atwood:

Yes. Thank you all. Thank you, Sheila and Liz for sharing all of your insights and expertise on thinking about how to analyze data and how to share it in various ways with audiences. I want to thank everybody for joining the webinar, and hope to see you at our future therapy session. Thanks everyone.