Do You Fear Innovation Will Threaten Your Effectiveness Metrics?

Over the last few years, I have frequently written about the problem with using metrics as a measure of value and performance.  As long as we continue to be told that use of quantitative measures are important, I am gonna keep pushing back and reminding you it ain’t the be all and end all of evaluation.

Carter Gillies is actually more adamant and eloquent on this topic than I am so when I saw a piece on Aeon that started out sounding almost verbatim like Carter, I did a double take to check who the author was.

The author, Jerry Z. Muller, points out that performance metrics often incentive a gaming of the system in a manner which often runs counter to the purpose of the organization.

Or take the case of surgeons. When the metrics of success and failure are made public – affecting their reputation and income – some surgeons will improve their metric scores by refusing to operate on patients with more complex problems, whose surgical outcomes are more likely to be negative. Who suffers? The patients who don’t get operated upon.

One of the other issues is all too familiar to non-profit organizations come grant report time:

To the debit side of the ledger must also be added the transactional costs of metrics: the expenditure of employee time by those tasked with compiling and processing the metrics in the first place – not to mention the time required to actually read them. As the heterodox management consultants Yves Morieux and Peter Tollman note in Six Simple Rules (2014), employees end up working longer and harder at activities that add little to the real productiveness of their organisation, while sapping their enthusiasm.

Non-profit organizations are well acquainted with implications of metrics. Organizations are often restricted to what government entities, foundations and donors are willing to fund. It can be difficult to innovate or address needs if your funding source has different priorities or restricts how funding can be used.  I have discussed before that there can be a tendency to report that everything you did met or exceeded the plans laid out in your grant proposal.  The fear of losing funding for not being successful enough disincentivizes being honest about challenges the organization faced.

While there have been plenty of embezzlement scandals at non-profits to leave funders concerned about whether their money is being used responsibly, metrics provide faulty assurances because they are so easily falsified.

So what should be used instead of performance metrics? Well, Muller really doesn’t say.  Doing a good job and having good outcomes might be a start. You’ll want to examine numbers to assist in the process of reducing needless waste. But trying to squeeze an extra percentage out so you can improve your efficiency score over last year when you squeezed an extra percentage over the previous year is not constructive.

Ultimately, the truth is that evaluation is hard. Even if we were to urge funders to invest more time in investigating outcomes directly rather than relying on numbers, the tendency to have positive associations for feel good stories will benefit some organizations over those that do unsexy, but impactful work. Then we will be back to rallying removing the emotional element by employing cold, hard numbers for evaluation.

The Road To Creative Enlightenment Is Paved With Cardboard

I have written before about the visual arts fair I started about two years ago to provide an opportunity for students and artists in the community to sell their work and get experiencing talking about it with people who don’t share their vocabulary.

Two weeks ago we had the fifth iteration of the event and we think it was easily the best one we have had thus far. We have experimented with the time and date a little bit. It appears that Spring is more popular than Fall in terms of artists having the time and material to participate.

We had so many applicants, that we decided to expand into a new area of the building. Previously, we had been concerns that if one or two people were placed into the overflow area that was slightly apart from the main area, they would feel slighted. Having reached a certain critical mass, we had the numbers to better populate that area. Additionally, a recent re-lamping project provided much better illumination.

A year ago we started placing art works in area businesses prior to the art fair event. We posting pictures daily on social media so people could find the artworks and at the very least generate good will for the business.  This year we saw an increase in participation by both artists and businesses. In one case, I ended up placing a work in a business on the other side of the county 45 minutes away.

Based on this alone, I would feel like we were making progress toward a goal of helping people recognize their capacity to be creative in line with the effort to build public will for arts and culture with which I am involved.

However…once again I partnered with my frequent collaborators, The Creative Cult who designed a “creative journey” visitors to the arts fair could embark upon. The journey took people along the fringes of the art fair and across three floors of the building in an attempt to find the guru who would provide the answer to creativity.

Here is a map of the odyssey

Participants were placed in the role of subjects of a tyrant who suppressed creativity. In order to escape, each party had to construct an item from a pile of supplies that would help them escape the walls of their prison. People made everything from drills to ray guns to bombs.

The next station was a field of strange flowers. When people touched the flower, they were overcome with the image of a monster. They had to draw the monster which was preventing them from being creative that day (which could be anything from lack of confidence to obligations) and a weapon with which they would slay the monster.

Next they ascended to a chasm guarded by a troll who asked riddles. The bridge was made of broken planks–but the only safe path was to step in the empty spaces rather than on the actual planks.

On the other side, they met “Steve” a guy playing video games and surrounded by half finished drawings. He never completes anything due to lack of confidence and commitment. There is always a last touch that needs to be added. There is a puzzle that needs to be solved to open the door at the top of the next level where they meet the guru face to face.

As to what the guru tells our intrepid questors, well that is for them to know.

Among the benefits I saw in this whole endeavor was that attendees were offered an alternative hands on creative activity in which they could participate at the visual art fair. If you were feeling uncertain about how to react to what you saw on display on the artists’ tables or how to interact with the creators, you could always run over and check out the crazy guys leading people around the building.

Days later when I had a little follow up conversation with one of the Creative Cult members, he remarked on how freeing the act of roleplaying was. A particularly shy member of their collaborative had fearless jumped in with both feet because he equated the whole activity as playing a Dungeons and Dragons character rather than himself leading a group of strangers.

As someone in the performing arts, this benefit of roleplay has long been apparent to me, but for people who identify themselves primary as visual artists, this was something of a revelation for them.

As they got excited about the prospect of adding roleplay to their toolbox, I started to consider how the resurgent popularity of games like Dungeons and Dragons might be employed to the benefit of arts and culture organizations.

The Creative Cult guys made a recap video of the experience that can be viewed on Facebook. I am having some problems getting it to embed successfully.


There is a conspiracy to keep your creativity in a box Photo credit: Carla Bentley

Art Museum, The Game

Intriguing video came across the old social media feed today in the form of a review calling the visual art oriented MMO Occupy White Walls as “The Weirdest MMO I Have Every Played.” The game is in alpha testing, but according to the reviewer, Fevir, that doesn’t detract from the experience.

Part of the motivation behind the game, according to Yarden Yaroshevski Founder and CEO of StikiPixels which is creating the game, is to provide a place to display the large segment of museums’ collections that are kept in storage and rarely seen.

The game allows you to create your own art gallery or museum and display different works in it. Other players can come by and browse. The game will also send non-player characters (NPCs) to visit and potentially pay admission fees. The in game currency can be used to expand and further customize your space–including “purchasing” art to display.

The game’s Artificial Intelligence (AI) learns what type of art you like as you browse, like or buy. My first thought, given the social media climate, is that this information could be sold to businesses who would then start soliciting you to buy art in real life. Not something I would be too keen on.

However, since people can upload their own images, the AI will serve the valuable function of keeping you from being inundated with pornographic images and memes. The company doesn’t want to be in the position of declaring what is and is not art. Ultimately, they may end up needing to limit or ban some content. If you are concerned about being subjected to dick-pix it would probably be prudent to wait until later stages of game development when the AI algorithm has learned to effectively filter them out.

Currency generated from player and NPC traffic appears to be the only competitive element to the game right now. For the most part, the game is largely focused on providing a platform to present, learn and discuss a wider range of visual art works.

I think it would be great if there was a way to pilot promotional efforts and test responses virtually. If the Artificial Intelligence (AI) was sophisticated enough to cross reference human player responses to different programmatic and promotional efforts and then direct appropriate NPC efforts to create a feedback loop to attract the notice of more players, that might be helpful.

In the context of problems with social media AI algorithms being effective at filtering undesirable activity, I would be concerned that the AI could be easily gamed to garner the most response for the least amount of effort. On the other hand, a my suspicion is lot of social media algorithms are purposefully permissive so as to retain the largest number of users. This game wouldn’t necessarily need to make things so flexible.

Fevir describes himself as “art indifferent” based on his experience with physical museums, but said after a few hours with the game AI, he started to see stuff that appealed to him. Once that happened, he began to delve into the notes about the artist, media and historic period in which the work was made.

He seems to feel that the ability for people to curate exhibits focused on particular artists, mount temporary exhibits and have conversations about brush technique took this beyond the experience of websites like Art Station, Deviant Art and the Museum of Modern Art’s online collection.

While it admittedly may have niche appeal, the basic concept and direction combined with opportunities for virtual reality viewing have some potential. There might be opportunities to mount exhibitions virtual exhibits that parallel real life exhibits. Being able to view and learn about something privately may provide people with the confidence to visit in real life. If people understand that the virtual experience doesn’t compare with really standing before a work, it may drive more visits to museums.

The downside, of course, is that if people feel like a virtual facsimile is acceptable, there may be fewer objections to works being deaccessioned and sold to private entities.

But as with so many pieces of technology, there is always potential for explosive usage in a manner the creators didn’t anticipate. It may be worth keeping an eye on this game or similar software that may emerge.

Respect The Authority of The Resource

Hat tip to Nina Simon for calling attention to a post about how the Barnes Foundation is working toward changing their user experience. Even though the Barnes Foundation art collection was moved from a residential area to Philadelphia, efforts were made to replicate the close quarters environment of the original house. This complicates matters because it is easier for more people to access a place whose interior space hasn’t increased.

(There have been decades long conflicts related to Albert Barnes’  detailed instructions about the way the collection is displayed which informed the design of the new space.)

Shelley Bernstein, Deputy Director of Audience Engagement and Chief Experience Officer for the Barnes Foundation wrote about the poor impression people received before they even entered the door.

We had a no photo, no sketching, no bags, no coats, and “stay behind the line” policies. All of these things were well intentioned and all were in place in the name of collection safety — a very important thing — but, still, it was a lot of “no” to those we were trying to welcome. And we were trying to tell visitors about all these “nos” before they would enter the collection.


You can see this surface in online reviews, “… had to be told the exact etiquette before entering which makes it feel like they treat their visitors as if none of them has ever set foot in any other major museums/collections before(???)”

Bernstein set about trying to change how all these rules were communicated as well as exploring which policies could be changed. While she originally intended to scrap the no-photography policy, she realized with half the rooms only having 100 square feet that people could occupy, allowing free rein on photograph wasn’t going to work. So they piloted different policies over the course of a year to see which worked best in their environment.

Perhaps most importantly, they created staff training materials for interacting with visitors based on a process known as Authority of the Resource (ART), that the National Park Service uses in their training. This approach makes citing the rules the last step rather than the first. Even better, Bernstein has shared those materials, which includes signage and the National Park Service discussion of the approach, for others to reference.

Authority of the Resource (ART) shifts the focus away from the concept of enforcement power and toward the requirements to preserve the resource. There are obviously going to be people who don’t give a damn and want to do what they desire, but it does shift the initial interaction away from “because I said so and I am the authority figure.”

From the Park Service’s article.

“…the visitor ends up thinking about laws, regulations, badges and the ranger’s presence rather than focusing on the natural authority inherent in the requirements of a healthy ecosystem.


“…the AR technique goes one step further and asks the ranger/manager to subtly de-emphasize the regulation and transfer part of the expectation back to the visitor by interpreting nature’s requirement.”

Here the the Barnes Foundation training pamphlet. Clicking on either image will take you Bernstein’s Dropbox copy of the document.

Send this to a friend