How do we know what we know?

Posted on August 5th, 2019 by

This post was written by JMM School Program Coordinator Paige Woodhouse. To read more posts from Paige, click here!


How do we know what we know?

This is a big question. It’s bigger than big. It’s enormous.

So how do you tackle a question like that? A conference seems like a logical place to start. In July, the Visitor Studies Association (VSA) hosted its annual conference in Detroit Michigan titled “Ways of Knowing.” This question, “How do we know what we know?” was the question that the keynote speaker, Dr. Katrina Bledsoe, opened the conference with before diving into contemporary thoughts in the field of evaluation.

Conferences are exciting places. They harvest intriguing questions and ignite new ideas. They are a place to share success stories and struggles that happened along the way. They are places for learning. While I couldn’t attend The VSA conference in person, thanks to a new green initiative by the VSA, I attended their first-ever virtual conference. Like all conferences, there was more discussed than could possibly be written about in one blog post.

What is the Visitor Studies Association? As described on their website, VSA is “a membership organization dedicated to understanding and enhancing learning experiences in informal settings through research, evaluation, and dialogue.” So, what are informal learning settings? That’s us, JMM. Along with other museums, nature centers, historic sites, visitor centers, and zoos.

To learn from other organizations about their applications of evaluation, you have to learn about the projects they evaluated. I heard from lots of organizations that have recently undertaken interesting projects (Along with the great ways they are using evaluation to learn from them). Here’s are two examples:

“Studying Touch as a Way of Knowing in the Art Exhibition” researched how touch can be a method of interpretation for visitors interacting with artwork, specifically sculpture. The project monitored visitors’ encounters (through video recording) with artwork in the exhibit Evighetens Form (Eternity’s Form) by the National Museum of Art, Architecture and Design in Oslo, Norway (2016 – 2019). This project added to previous research on multi-sensory meaning-making processes. Listening to the findings of their study, I particularly enjoyed an unexpected outcome they had – visitors would going beyond gentle touching of the sculptures at times and knock on the sculptures for solidity, determining the material, and the sound that was produced.

Dr. Navaz DBhavnagri from Wayne State University spoke about “Using Museums to Promote Cultural Identity Among Yemeni Students.” This project explored how museums are places that can promote and enhance cultural identity. Working with Muslim Yemeni immigrant and refugee students across multiple visits to the Detroit Institute of Arts, as well as pre-visit and post-visit sessions, students were encouraged to make connections and express their cultural identity. Before visiting the Museum, students did a self-report on what they know and what they wanted to know. Visits to the Museum progressed with different activities. First, they went and took pictures throughout the exhibit writing comments back in the classroom about their photos and how they connected to what they took pictures of. This visit introduced students to the museum environment. Their second visit was a scavenger hunt to encourage more focused engagement with the exhibit. Their third visit encouraged them to select an object to sketch. This object needed to relate to their cultural identity. They needed to think about why they chose that object. What special meaning did it have? How was it connected to their cultural identity? This resulted in a more complex reflection and the students creating an intersection between their personal life and their cultural identity. After each visit, students would debrief their experience. They created art projects that integrated their knowledge. They considered what they learned and what new questions they have now. During this year-long projects, these students were also learning English, so translators were critical to assisting the project. At the end of the project, students presented their object and story to other students and teachers outside of their class – sharing not only their cultural identity, but their new language skills. All the materials produced through the project (photos taken with comments, collages, sketches, reflections, and presentations) were used to evaluate the project. Students moved beyond seeing the objects as “just old” to how they overlap with their own lives.

The Detroit Institute of Art has a strong collection of Arts of the Ancient Middle East and Arts of the Islamic World that students explored during their visits.

While each speaker highlighted a specific project, throughout the entire conference the theme of equity was present. How do we promote equitable evaluation? Equity, in the simplest of definitions, means fair access. Each person has access regardless of economic resources or personal traits. Every person has the right to be given equal treatment by the system.

Evaluation is often thought of as being objective. But we need to consider the ways our methodologies are shaped by underlying values. We need to consider different cultural and historical views. We need to make our research findings accessible. While measuring if the goals of a project are being met, we need to consider if the project developed in a culturally responsive way? Whose reality are we representing? Whose voice? Whose experiences?

The Detroit Zoo wanted to engage with audiences that weren’t coming to the zoo (even when offered free admission). They wanted to work with individuals who found themselves homeless and therefore needed to think about the barriers that were preventing people from visiting. When speaking about their project and evaluation, they said that evaluation for their team at the Detroit Zoo means continuously asking, “are we valuable? What is valuable about what we are doing?” The team constantly looks at communities in their neighborhood and asking who are the voices that they don’t reach and what do those communities need?

So, how do we know what we know?

We evaluate. And this takes many forms at JMM. Evaluation is not a one-size-fits-all tool. Especially when thinking about how to be equitable during the process. For the big picture, we want to make sure that what we are doing is valuable. That all our exhibits and programs reflect our mission. We seek to learn about our impact and the quality of experience we offer.

Evaluation can come to us informally through conversations, emails, and phone calls. For projects, (whether it is a public program, school group, or exhibit) we try to make evaluation part of the process.

Intern Hannah spoke with visitors about their experiences in Fashion Statement recently.

Recently, JMM’s Visitor Services Coordinator, Talia, shared how our FY2019 visitor numbers are one way that we evaluate our success. We also evaluate using surveys after public programs, or post-it notes with school groups and by observation. This summer our interns have been conducting surveys for our Fashion Statement exhibit. We are interested to see if visitors are making connections to the learning objectives we set out for the exhibit. Or, as I mentioned previously, what unexpected outcomes we may find.

So when our interns and staff are in the orientation space with clipboards asking if you would take a few minutes to fill out a form, or chat with them about your experiences, it is not just to collect data that will sit on a shelf with a checkmark beside it. It is because we genuinely want to know about you, what brought you here, how you did (or did not) connect with our exhibits. Your answers inform our decisions. We learn from them. They help us find ways so that you, the visitor, can “find yourself here.”

Conferences are inspiring. I am positive that the things I learned will trickle into projects at JMM. You can read the abstracts from all the VSA conference sessions here.


 

Posted in jewish museum of maryland




Intern Weekly Response: Evaluating Success

Posted on June 27th, 2019 by

Every week we’re asking our summer interns to share some thoughts and responses to various experiences and readings. This week, following a professional workshop on museum evaluation, led by JMM School Program Manager Paige Woodhouse we asked them to find an additional recent article on the subject to suipplement their workshop materials and relate what they have learned to the article. To read more posts from JMM interns, past and present, click here.


~Intern Hannah Balik:

Last week, us interns had the opportunity to learn about the different types of museum evaluations from JMM School Program Coordinator Paige Woodhouse. There are four types of evaluation that a museum do, in the four different stages of putting up an exhibit or executing a program. First, there is front-end evaluation, which takes place in the beginning stages of planning, and serves to evaluate the labor and monetary investment needed to complete the project. Through the use of focus groups and street interviews, museum professionals can get an understanding on visitor interest in a certain exhibit. The second type of evaluation, formative, takes place during the design and development of the project. Often in this stage, museums will prototype information panels or interactive learning objects in existing exhibits or neutral spaces. Users can then give feedback on their experience, informing the museum on the general reaction to the exhibit. Remedial evaluations happen after exhibits have opened. They give staff an opportunity to critically look at recently opened exhibits and change any small mistakes or other things that are not working for the exhibit. This can include spelling errors on information panels and other small errors. Visitor tracking is helpful in this stage, to help staff understand what parts of the exhibit need re-vamping, and which parts are working great. The final type of evaluation is summative, which takes place at the end of an exhibit, once it closes. This is the time for museum staff to look at the project as a whole and decide what worked, what didn’t, and discuss the outcomes of the project, expected and unexpected. This type of evaluation serves as an accumulation of all the data in the previous evaluations, while looking to the future and how to improve in the next exhibit. It surprised me how many types of evaluation there are, as well as all the different ways you can record and evaluate visitor experience.

Formative Evaluation at work. Visitor responses are very important when it comes to creating new exhibits or updating current ones. Source.

I decided to read the article “Prototyping AR in a University Museum: How User Tests Informed an Accessibility Plan Including and Beyond the Museum” by Max Evjen. This article discusses how Michigan State University Museum in 2017 decided to launch an augmented reality pilot addition to one of their exhibits on animal diversity in Michigan. Users were able to augment the existing exhibit by seeing animated animals and text appear on tablets when pointed to a mural the museum had in the exhibit. According to Evjen, who works at the Michigan State University Museum, “The Hall of Animal Diversity is a gallery with taxidermied birds, mammals, and insects arranged by theme, for instance, animals of mid-Michigan, camouflage, etc. There is a large brown bear … a couple of simple activity stations, an interactive video of different bird sounds, and the Michigan wetland mural, created by a local artist, that includes a key at the bottom for identifying animals featured in the wetland scene.” This mural was a cold spot, meaning visitors often walked right past it, or didn’t engage with it as heavily as they did with other parts of the exhibit. In order to increase interaction with this area of the museum, museum staff decided to add AR, in the form of 10 Samsung Nexus tablets. Looking back on evaluation data from other institutions, they decided to use tablets instead of an app users can download on their own devices, as they thought users would be more likely to use an already set up tablet than wait to download an app, that they know visitors will likely never use again once exiting the museum.

An example of the new Augmented Reality at work at the Michigan State University Museum. Source.

This endeavor shows examples of remedial and formative evaluations. Some of the museum’s galleries date back to the 1960s and have few, if any, interactive elements. Through visitor evaluations and watching what visitors interacted with most, the museum found that this mural in the Hall of Animal Diversity was rarely viewed by visitors. The use of AR through tablets would allow for an update to the exhibit without changing the exhibit itself and encouraged close looking at an often-overlooked part of their museum. Also, through the use of evaluations from other museums who created interactives that needed a downloadable app, they realized that would not be the best option for them, because most people did not download the app or ever open it again after the visit. They instead decided to use tablets ready for visitor use.

After testing the AR, the museum already has ideas for its improvement. Through the use of questionnaires and behavior sampling, museum staff were able to get a better idea of what visitors thought about the Visitors said they would like to see more audio and interactive capabilities in the software, which the MSU Museum staff have taken note of.  This project shows the power of evaluation – finding the shortfalls in current exhibits, and allowing for new ones to take shape, in a controlled, visitor approved manner.


~Intern Elana Neher

Before the workshop we had last Wednesday, I had only vaguely heard of museum evaluation. I had seen survey areas and visitor satisfaction boxes at the end of exhibits in museums before, but I had not thought about how important they are to museums. After our workshop, I came to understand that if no evaluation is done, there is no way to know whether the exhibits, programs, or any aspect of the museum is achieving its goals effectively and there would be no way to know how to change in order to achieve these goals better.

As museums move into the future and integrate technologies such as virtual reality, apps, and touch screens into their exhibits, it is important to know how effective these new interactives are in engaging visitors and achieving learning goals. Recently, the Cleveland Museum of Art released the results of a two-year study into whether their ARTLENS gallery, an experiential space that allows visitors to engage with artworks in a variety of ways through technology, is actually effective or just a gimmick.

With these new forms of technology, not only did the questions asked in the evaluation have to change, but the evaluation method itself had to change. They completed a variety of surveys to look at different age groups, visitor experience enhancement, and engagement with other parts of the museum. Surveying like this will become crucial as museums add more technological interactives to see if they are truly worth the expense.

Like any other kind of surveying, prototyping, or analysis, museum evaluation has its issues. The limited amount of questions that one can ask in a survey can limit the amount of information the museum gains and can leave valuable information out. Additionally, no matter how much time or money is spent on evaluation, if the museum does not use the results of the evaluation to change, then completing evaluation is useless. These are only two of the many issues that can come up with museum evaluation. Despite these difficulties, evaluation is crucial to museums in understanding how effectively they are achieving their goals and how they can change in the future to better achieve these goals.

Additional article: Why Evaluation Doesn’t Measure Up.


~Intern Megan Orbach

Last week, along with the other interns, I attended an evaluation workshop with Paige Woodhouse. As someone who has not been previously introduced to museum evaluation methods and after reading another article about it online, I was surprised to learn that most museums employ similar guidelines for evaluating their exhibits. The commonalities between the guidelines used seem to be four main steps: front end evaluation, formative evaluation remedial evaluation and summative evaluation.

One person cannot evaluate by themselves; it takes a lot of people together! Source.

We were able to learn, in this workshop, just how important evaluations of exhibits are for future applications. Firstly, some of the statistics involved in evaluation show who the visitors are, what they thought about the exhibit, feedback, general visitor experience and more. This kind of information is useful because it can help to improve the existing exhibit, or it can give pointers for the next one. It is also helpful because it can show how much visitors learned from visiting the museum and if there are better or different ways to present information.

Some challenges I envision when it comes to museum evaluations are what methods to use in order to gain the most information on the visitor experience. For example, some questions on surveys work better than others and this is sometimes not found out until after both types are tested. Other times, I would think that feedback is not always constructive and the path to improvement is not always clear.

Evaluations should include constructive feedback and employ thoughtful methods! Source.

The article I read discussed the four different types of evaluation, just as Paige did in the workshop. It outlined to me, as mentioned above, that many different types of museums can use similar evaluating guidelines.

Overall, I was very interested in the workshop and I feel that I learned a lot about project evaluation in general. A lot of the methods used can be used outside of museums as well and as a way to analyze various types of work. It emphasizes the importance of constructive feedback which is extremely important for growth and progress. The workshop and article also taught me that it is important to consider who the intended audience is before beginning a project; the work should and can be completely different depending on who is supposed to be viewing it. Lastly, I learned that it takes a decent number of people or visitors to form a good evaluation.


~Intern Ariella Shua

Last week, we learned about different methods of museum evaluation. One of the main ways evaluation takes place is by checking to make sure that the museum actually accomplished the tasks at hand. One of these factors is keeping visitors interested and pleased with their experiences. Visitors are often quick to point out if they’re unsatisfied. If something is broken, or if they simply aren’t impressed, they’ll remember that they were unhappy.

As we learned in our workshop, museums want this kind of feedback. Without it, they wouldn’t be able to plan future exhibits. And they wouldn’t be able to fix the exhibits that are already standing. Because at the end of the day, museums want to entertain and educate their visitors. That doesn’t happen effectively if the exhibits aren’t up to par. Museums that want to impress their visitors, and draw new ones, are willing to make changes. Fortunately for locals, the Baltimore Museum of Art (BMA) is one of these museums.

The Baltimore Museum of Art, located in the Charles Village neighborhood.

It’s not often that museums decide to put food and art right near each other. I can’t remember the last time I was able to bring anything edible into a gallery. It’s basically a museum taboo. But the BMA, in an announcement on Tuesday, revealed that they’re hoping to engage new visitors by appealing to those who prioritize food. They’re opening a new exhibit space downtown, at Lexington Market. Lexington Market is the oldest American marketplace, and it’s still open. The crab cakes there are rumored to be delicious, though I can’t back that claim up myself.

But art? That’s not what most people head to Lexington Market for.

Now, the BMA hopes to change that. With the opening of BMA Lexington Market, they’re aiming to draw new audiences. The first exhibit in the new space will feature photography, perfect to casually stroll by while eating a marketplace meal. The opening is a clever one. Customers at Lexington Market don’t anticipate seeing art. But if they like the exhibit, they may be intrigued to visit the main BMA. It’s been done before- the BMA has tested five other satellite locations over the years. In this case, it’s a change, but also a reliance on what has worked before. Almost a two-pronged evaluation: taking an effective method, and repurposing it for future visitors.

Some of the photographs on display at Lexington Market, part of the Baltimore Museum of Art’s new expansion.

As Dave Eassa, manager of community engagement at the BMA explained in an email to the Baltimore Sun, “It’s for people who have never set foot in the BMA and folks who come to the BMA all the time to meet, create, and share.” Hopefully, the prediction is right. I, at least, may soon use the pretense of viewing photography to arrange a trip to score a good meal.


~Intern Mallory Connaughton

Last week we had a workshop where we discussed and learned about museum evaluation.

Personally, I didn’t know much before going into the workshop. I had always connected evaluations to the end of a project. But we also discussed other types of evaluations, ones that can be done at the start of planning and during. And we discussed the various methods of conducting evaluations.

One thing that caught my attention during the workshop was when we talked about journey maps, which are maps drawn by the conductor of the evaluation. These maps catalogue a visitor’s trip through the exhibits, tracking time and attention. This is used during the exhibit and can show what points attract attention and the retention time at the various sections. It not only provides an insight into the visitor’s interests and wants from the various points of the exhibit but can also be used to evaluate the set up of the exhibit, to see if the flow of the floor is good.

Looking more into journey maps I found an article, “Visitor Journey Mapping in Museums”. This article discusses journey maps, specifically the exploration of uses and conduct in a workshop run by Allegra Burnette at the Pratt Institute School of Science. The article discusses that the journey maps can be used to look at the different personas that may be visiting the museum and see how the exhibit impacts then directly. The article explains that detailed journey maps can track not only visitor’s movements and retention times but also their emotions as they move through the exhibit, any questions they may pose, and so on.

These journey maps, both the heavily detailed and the simpler ones, can both be implemented to track retention time which can aide in exhibit set-up. This information can be used to directly change a current exhibit or can be used to positively impact the following exhibit. Before this, I hadn’t known about journey maps, and I found these to be very interesting. It is a very dynamic way to evaluate museum exhibits.

Outside of the more hands-on evaluation that journey maps are we also discussed direct communication with guests; surveys, interviews and interactive exhibits. We also briefly talked about newer technology-based methods of conducting evaluations. From a quick search there are cites which comp0ile lists of evaluation methods and research, one of these being connected to the Institute of Museum and Library Services.

Evaluations can be in the form of a short survey, first collection basic demographic information then asking two to three questions to gauge how an exhibit sites with the various demographics visiting the museum. Or there could be an interactive within the museum that guests can partake in, which the data can be interpreted for an evaluation, like walls that guests can write on (giant sticky pads that are strategically placed to draw attention and that ask for guest feedback).

These ways of interacting with customers, either active or passive, are vital in preforming museum evaluations. They can be seen in some form at all museums and are imperative to exhibit creation. I think that, especially with the modern uses of technology, there will be more evaluation methods coming out surrounding technology, and direct interactives for the guests to partake in.


 

Posted in jewish museum of maryland