If The Metric Is Valued, Someone Is Probably Trying To Game The System

Okay, so I promise I am not seeking out articles that discuss the problems with depending on quantitative metrics to determine effectiveness and value. They just keep falling into my lap. This one is via Dan Pink and is kinda fun to read thanks to some animations.

The piece in The Hustle has us follow the “career” of  Otis has he moves from being a cashier to sales to online advertising to programming to surgery in order to illustrate how the use of quotas and efficiency metrics permeates every industry and every profession to incentivize gaming the system in order to generate the best appearance.

But Otis came to learn that metrics weren’t inherently bad — his bosses had just failed to grasp two important economic principles:

  • Goodhart’s Law: “When a measure becomes a target, it ceases to be a good measure,” and
  • Campbell’s Law: The more a metric is used, the more likely it is to “corrupt the process it is intended to monitor.”

He realized that when his performance was measured with a specific metric, he optimized everything to hit it, regardless of the consequences that arose. As a visiting professor at the London School of Economics told him, improper targets could:

  • Encourage “gaming” the system (e.g., bagging free groceries)
  • Incentivize the wrong aspects of work (e.g., writing trivial code)
  • Erode morale (e.g., writing clickbait)
  • Harm customers (e.g., turning away critical surgery patients)

And so, Otis decided to start his own company — a company where metrics would serve their true purpose: To motivate and align. Efficiency, Otis finally realized, isn’t just output; it is the value of what is produced.

If you think about the measures being applied to non-profit arts and cultural organizations like overhead ratio, economic impact, test scores, etc and pay attention to what organizations are doing in order to meet those metrics, you will probably start to see behaviors that conform to those listed above.

It could manifest as massaging numbers in financials and research; chasing funding that doesn’t align with mission and strains capacity; superficial efforts that check desired boxes; pursuit of a narrow segment of community rather than a focus on broader inclusion. I am sure readers can think of many examples from their own experiences.

Unexamined Initiatives Are Not Worth Implementing

It is no news flash to even casual readers of the blog that I am involved with Arts Midwest’s Creating Connection program to build public will for arts and culture. Last week, they ran a webinar just to present the basic research and program. In recent months they have been featuring two case studies where people talk about how their organizations are putting the research and messaging into practice. This session was aimed at giving people more complete information about the program.

As much as I have been a fan boy cheer leading the program, what I really appreciated about the webinar last week was the number and type of questions people were asking of the presenters.

It was an indication of just how serious people were thinking about implementing the research that webinar attendees were questioning the research methodology. I think people in arts and culture field are wise to scrutinize whether a new approach to doing business is a popular fad soon to fade or has some rigorous thought behind it. They have little enough time and resources as it is and don’t want to waste it on initiatives lacking substance.

What I really appreciated was when one person, identified as Zi Li, asked about case studies on failed programs because they were interested to learn why those program failed.  My friend Carter Gillies often mentions the problem of survivorship bias  where you only study the successful cases rather than gaining insight from those that failed.

The music on the Awesome 80s radio station is always going to be better than the music today because you are comparing the cream that rose to the top and endured the last 30 years to all the music being performed today, both good and bad.

If you are new to the concept of Creating Connection or just want a refresher, take a look at the video from the webinar which includes all the questions and comments made that day.

Non Profits & Buying Locally – Good For The Community vs Bad For Overhead Ratio

Back in September, Non-Profit Quarterly (NPQ) pointed to a new research study which has found overheard ratio is not a valid measure of organizational effectiveness. In fact, it there is a slight negative correlation between overheard ratio and commonly used measures of efficiency.

“…but our work is the first to approach it using efficiency theory—and we were able to demonstrate the problem using real-world data.”

….“In short,” Coupet says, “this demonstrates that not only is the overhead ratio bad at assessing efficiency, but also that using it to assess efficiency may actively mislead donors. We argue that nonprofit scholars, managers, and donors should move away from concepts and measures of efficiency based on financial ratios, and toward ones that embrace maximizing what nonprofits are able to make and do.”

NPQ says according to the study, overhead ratio is a poor measure of effectiveness because it doesn’t reflect what organizations are doing with their resources or what they “are accomplishing with their non-overhead spending.”

In other words, like I have written so often before, the value of what a non-profit does is not reflected by transactional data, economic impact numbers or test scores.

This being said, another part of the article raises the intriguing idea that if a non-profit is supposed to be working for the benefit of their community, shouldn’t they be focusing on buying locally rather at chain stores or wholesale warehouses? If so, the higher cost of buying locally would raise their costs a bit and impact their overhead ratio. But it may be worthwhile to do so.

Should we stop looking for cost savings that benefit our bottom line but lead to purchasing that harms the greater community? In other words, should nonprofits be considering (and be supported to pursue) their own “buy local” policies?

‘Nonprofits should be shouting about how much of their spending happens at locally owned, minority-owned, women-owned, veteran-owned or disabled-owned businesses. There is a multiplier effect in spending locally that shows that for every $100 spent at a locally owned business, $45 of that is re-spent locally, while national chains only spend $14 of that sale locally.’

This is an intriguing idea that has this author (a nonprofit executive who manages purchasing) feeling the financial pinch of a cogent ethical argument: If buying local supports healthy communities, and the mission and values of my organizations are tied to relevant healthy community outcomes, why am I doing my shopping at big box (including online) retailers?

This broadens the scope of what it means to be a non-profit in service to the community. Touting how much is being spent at locally owned business won’t necessarily smother the use of overhead ratio as a standard, but it has the potential to blunt the ratio’s use in an argument of a non-profit’s worthiness.

Are You Persistent or Consistent In Your Pursuit of Excellence?

Seth Godin made a post earlier this week comparing Persistence with Consistent wherein he starts with the statement “Persistence is sort of annoying.”

He goes on to talk about the way in which consistent is the desirable opposite side of the coin,

Consistent with your statements, consistent in the content you create, consistent in the way you chip away at the problem you’re seeking to solve.

Persistence can be selfish, but consistency is generous.

And the best thing is that you only have to make the choice to be consistent once. After that, it’s simply a matter of keeping your promise.

In this context,  persistence seems to be about performance of a specific action whereas consistent is policy. In this sense persistence is approaching a challenge in the same way until it is worn down to the point you can pass. Whereas consistent is more about dedication to finding a way past that obstruction.

While both approaches never falter in achieving a singular goal, the latter entertains options regarding the methods by which this can be accomplished. In fact, consistent may be better equipped to recognize that surmounting the barrier isn’t the goal but rather getting to the place beyond the barrier and therefore there may be no reason to engage with this particular barrier at all.

What actually drew my attention to Godin’s post was that last line about keeping a promise. Working in the non-profit arts sector is often such a struggle that we feel like we can only survive with dogged persistence. Perhaps what is really needed is a focus on consistency.

If our promise to the community we serve is to provide a certain experience, a persistent approach may keep you locked into executing an approach and methods which have decreasing relevance. A determination to offer a consistently valuable experience can lead you to place more importance on needs of those you intend to serve rather place importance on the methods by which you accomplish it.

Think about it this way. If you want to keep a promise to provide excellent customer service do you do the same thing today as you did five years ago? Do you use the same approach for small groups as large? Kids as for elderly? Film audiences as for Broadway musical audiences? 2500 seat theater as for 150 seat theater?

Sure you will still make bad choices, but a consistent approach to great customer service is likely better able to take the differences of time, place, environment and expectations into account than a persistent approach.

If you are like me, Emerson’s line, “a foolish consistency is the hobgoblin of little minds,” might have come to mind when you first saw the term. In that context consistency has a negative connotation. After reading and pondering Godin’s post I wondered if it might have been better said as “a foolish persistence.” Though I learn toward consistency is a better word choice.

It should also be noted, Emerson never mentions what the characteristics of wise or non-foolish consistency are. Consistency is not necessarily negative in and of itself. Persistence isn’t either, but it does have implications of a single-mindedness that can quickly become problematic.

You may have noticed I didn’t make definitive claims about persistent being one thing and consistent being another. Ultimately, of course it isn’t about what word you use as much as what practice you embody.

Send this to a friend