surveying

Reaching The Unknown Non-Participant

by

I almost threw money away, literally.

I received piece a mail last week bearing the logo of a state university indicating a survey was enclosed. I was just about to throw it away when I became curious to see if they were doing a market survey to gauge the educational needs of the community.

When I opened it, I saw it was a public health survey. Just as I was about to throw that away, a $2 bill fluttered out on to my coffee table. I understand why they may have chosen to include the money as an inducement to complete the survey, but I wonder how much money has literally been thrown away in unopened envelopes around the state this week.

I feel some empathy for those administering the survey because this illustrates the effort and expense you have to go in order to reach people with whom you don’t have a relationship.

One of the challenges the arts have is to have a conversation with those who aren’t participating in order to learn why they aren’t. There will always be people who will never participate in an activity, regardless of what you do, but there are many who will be willing but do not participate for various reasons.

These reasons might be lack awareness; currently lacking the time due to child rearing obligations which will abate in a few years; perceive they lack the time, knowledge, etc, for whom opportunities exist and can be communicated or created.

Reaching these unknown people to survey them or even communicate with them is even more difficult now than a decade or so ago. So many people lack landlines that there is no destination for even an annoying robocall to reach.

Thanks to the option to receive financial statements, bills and other communications online, it is much easier to identify the few pieces of postal mail that is relevant to you without even opening them. (Those emails about winning $5 million in the Irish-Nigerian lottery that was liberated after a palace coup dispossessed the prince of an oil rich nation still almost fool me.)

As we can see, even trying to bribe people to participate is fraught with its own challenges.

My feeling is that an effective campaign to reach the unreached is going to require much more personal interactions either with organizations reaching out to individuals in an involved, relationship building effort or calling upon people with whom they already have a relationship to reach out to people and encourage them to at least respond to a request for information.

Certainly, technology can facilitate this by allowing people to spread the word quickly over social media. A degree of trust in all involved parties needs to be there. I am thinking, in part, about the post I made last week about a greater lack of willingness to trade privacy for benefits than we have been lead to believe.

It would be great if someone could figure out how to create survey that engaged as many people as asking about what color lightsaber you would wield while yielding, effective, meaningful, actionable results. (Which of Yo Yo Ma’s cellos are you?)

Until then, it may take a lot of asking.

Careful What You Ask

by

One of the primary rules of surveying people is to avoid asking questions about things that you have no intention or ability to follow through on. A corollary to that is; and don’t do it on the internet.

The White House is probably learning this lesson after promising to respond to any petition receiving more than 25,000 signatures on their We The People website. If you haven’t heard about this already, people from every state in the union have petitioned to be allowed to secede.

Puerto Rico on the other hand just had a non-binding resolution to become a state.

As of this writing, Texas has over 100,000 signatures on their petition and a number of other states like Georgia, Alabama, North Carolina and Tennessee have exceeded 25,000 signatures.

Austin, El Paso and Atlanta have counter petitions to secede from their respective states to remain part of the U.S.

There are various other counter secession petitions listed as well including everything from exiling people who signed the secession petitions, making seceding states reimburse the U.S. for taxes and forcing Papa Johns and Pizza Hut to give everyone who remains in the U.S. free pizza if Missouri secedes.

There are actually some petitions that probably approach the intention of the website: asking the US to use its influence to gain the freedom of political prisoners, creating unified standards for redistricting, and justice for populations displaced by the U.S. government.

These are a little harder to find amidst all the demands for dissolution, impeachment and vote recounts.

I have read some articles that suggest that while the White House promises a response, it may resemble the form letter you get from a company after complaining: “Your concerns are important to us and we appreciate the feedback of our citizens. However, at this time the administration has no plans to dissolve the country. Be assured, we will keep your suggestions on file against the time that we might make such a decision and will contact you for your input into the process.”

I usually keep away from general politics for the simple reason that as the last election showed, there isn’t all that far to fall down the slippery slope before reaching the mud.

However, I think this is a great example, writ large, of the dangers inherent to soliciting feedback over the internet. It doesn’t take much thought about the consequences to jump on a petition signing bandwagon. Often you will get suggestions from people who have no concept of your business model, audience and organizational history.

However, if something in the input solicitation process leads them to believe there is a good chance of some action being taken and it isn’t, a simple “what do you think?” can turn someone who was generally positively inclined toward your organization into someone who has a bad impression. Better not to have asked in the first place.

The people who signed those petitions are probably expecting the White House to at least acknowledge their requests and start the process of exploring whether their state should be allowed to leave the Union. If nothing of the kind happens, people who merely grumbled “we should show them and leave the US” over a beer before may perceive the whole website as an empty sham and become even more resentful.

And sure, lets face it, that is pretty much what we have come to expect from promises politicians make. This will just be another one of those cases and nothing significant will likely come of it.

However, we do expect more from the businesses with which we interact. This petition system is just a really clear example of how you should watch what you ask, how you ask it and what expectations you create about your response.

Stuff To Ponder: Surveying The Whole Person

by

Two thought provoking articles about surveying popped up on my computer today. While you may not think surveying is terribly exciting, I encourage you to read on. I promise there is no talk of statistical analysis.

The first I found on the Createquity blog where Crystal Wallis recounts how the North Carolina Arts Council turned to folklorists from the North Carolina Folklife Institute to help establish an arts council in one of the counties. Once Wallis explained the reason the state arts council tapped the folklorists, it made perfect sense to include them. Then I started wondering why more surveys don’t involve folklorists.

Folklorists, as it happens, are some of the best trained interviewers out there. They also have a particular advantage when it comes to arts research: folklorists are trained to seek out and recognize creativity in all forms, especially that which comes from people who don’t consider themselves “artists.”

From all accounts, it looks like the folklorists achieved excellent penetration into all corners of the community, including many niche populations that revealed the diverse historic and present influences in daily life. They didn’t just identify these elements in the community, but spoke with them as well.

Wayne Martin, Senior Program Director for Community Arts Development at the North Carolina Arts Council, explains the benefits that came from using folklorists in this project.

* Authenticity

“By having folklorists trained in interviewing, we got some really eloquent statements that we were able to quote exactly. The results of the research were in the words of residents, which was a different tone than when other consultants would come in and write about a place. We were confident that the assets they reported on were valued by those in the community, lending an air of authenticity and connection we hadn’t had from other reports.”

Martin’s words came back to me when I read the next article on Asking Audiences blog. Peter Linett talks about a New York Times piece criticizing a Brooklyn Museum exhibit on Plains Indian tipis for being bland, blaming the use of focus groups and visitor surveys in the planning process.

Linett addresses the problem most arts organizations face when asking audiences about future programming. Programming per popular acclamation of committee results in something that is uninspiring to everyone. Foregoing feedback entirely risks appearing highbrow and elitist. Because people are often at a loss to offer suggestions and questions on topics they know nothing about, the best intentions to avoid confusing complexity and condescending simplicity result in a middle of the road product in which “you can sense the oversimplifications even if you don’t know enough to say exactly what they are, and you can feel the flat, pedantic tone.” While Linett makes this observation in term of museum exhibitions, I am sure you can think of similar examples in other disciplines.

Linett identifies a likely source of the problem. (emphasis his)

But that’s because we’re starting with a narrowly cognitive, educative purpose in mind. We’re interested in what visitors know about tipis rather than (for example) what they feel, what they wish, what they fear, what they find beautiful, what they find sad. We’re looking at a single, isolated aspect of human connection to the material. It’s not necessarily the most interesting aspect, but it’s the one that museums, as Enlightenment institutions, have traditionally cared about most.

What kinds of questions would we ask if we cared just as much about emotional, spiritual, social, ethical, imaginative, and physical connections to that material? How would we start a conversation with our audiences about those kinds of engagement…

Upon reading this last bit, I was struck that this was what the North Carolina folklorists were asking of those they surveyed — or at least these elements were present within the answers they were recording. The greater degree of authenticity Wayne Martin observed in the survey results was likely due in part to answers that reflected these aspects of the interviewees’ connections with arts and the idea of a county arts council.

Surveying on an emotional rather than an intellectual level makes a lot of sense. People react to art and even the idea of the arts on a visceral level that they can have difficulty verbalizing. Surveying factual information isn’t going to help elicit a truly valuable response because people often don’t know why they do or don’t like art.

At least once a day when I am reading about arts topics in a newspaper article or a blog, there will be a comment that says “as long as no tax money is used for it…” and/or “art(ists) should support themselves.” I suspect these phrases are just convenient ways for people to get past the fact they don’t really know how to discuss how they feel about the arts. Certainly this inability is shared by those who want to offer praise as well. Asking Linett’s questions about what people felt, feared, admired and pitied might bring more sophisticated answers and avoid that question all performing artists fear–“How did they memorize all those words/steps/notes.”

Lies, Damn Lies and Are We Ever Going To Use These Survey Results?

by

My Google Alerts informed me that the phrase butts in the seats was used recently in what turned out to be a reprint of a 2008 article on MinnesotaPlaylist. The topic of the December 2008 issue was “Know Your Audience.” In addition to the original piece that brought me there, a couple others caught my attention as well.

The first article, Joseph Scrimshaw’s “Humans With Pulses,” addresses the idea that everyone will enjoy a quality artistic product if only they are aware of it.

“I’m beginning to think that failing to be specific about who you want in the audience presents a risk to both the profit and enjoyment of theater. There is a tendency for artists to believe that any cross section of people will enjoy their work. After all, theater is good. Theater reaches out to people. It’s easy to reason, “This piece of theater I have created is good, so why shouldn’t any human with a pulse enjoy it?” Music is good, too. When I was fifteen years old my two favorite musical acts were Frank Sinatra and Guns N’ Roses. I chatted with my Grandma about one and with my friends about the other. Unfortunately, there was no chance in hell of any crossover.”

Scrimshaw talks about being aware of the specific segments of the community who might be interested in a performance and why. Granted, the venues he seems to work in sound like they are at the smaller end of the spectrum and thus can serve very narrow segments of the population–like knitters. His description of the audience at his knitting-based, murder mystery comedy sounds like the diverse group everyone yearns for-

“We had knitters from 18 to 82, multiple ethnicities, sexual orientations, and genders. The knitting demographic trumped all because they had an investment in the show: They wanted to see if this Scrimshaw guy (who’s supposed to be funny) had anything interesting to say about this craft that is their hobby and their passion.”

Using the topic of demographics as a segue, John Middleton provides an amusing look at the whole idea of scrutinizing demographics in “They, the People”

She and her husband—let’s call him David—went to Mexico last year and have been thinking about a trip to Europe. However, they’ve both lost a lot of money from their 401(k) accounts and what with Nancy’s allergies and recent weight gain and David’s high cholesterol and occasional erectile dysfunction, that romantic trip to the moors of England might have to wait.

“Whoa!” you’re saying. “This is creepy. Does he really know this stuff or is he just making it up?”

I know this stuff. Visit the raw data (PDF) and draw your own conclusions. Remember, demographers don’t lie.

I actually checked out the raw data, and believe it or not the survey actually asked about all these aspects of the participants’ lives. Here is the real stuff to remember:

It’s true demographers don’t lie. But remember, these numbers are simply a tool. They are descriptive. They do not tell us what plays to do or how to do them. They simply allow us to think about Nancy and her life. They remind us that we’re not just asking Nancy to plunk down twenty bucks to see a show; we’re asking for an investment of her time. She has to find out about our production, decide to go, come up with a night she and David are both free, leave work in time to dress, figure out where our theater is, find parking, and so on. Are we making this process as easy for Nancy as possible?

You see, demographers are not soulless, art-killing philistines. We’re here to help.

However, if you still have doubts about the usefulness of demography, let me tell you something: You are not alone. There is a tiny checkbox labeled “terror” in the heart of every demographer. The system of gathering respondent data is filled with imperfections from start to finish. One flawed remembrance here, one inflated self-aggrandizement there—each insignificant on its own, but they start to add up. Then as we extrapolate the data, every imperfection becomes multiplied many times over until we have nothing left but a spider’s web of half-truths and sweat-soaked guesses. We crush this seething mess into solid-seeming charts, tables and graphs in order to give it the look of Truth, but we know: Demographers lie! And if you think this is only true of demographics, you’re kidding yourself.

Filling out the theme of surveying and data is Sara Stevenson Scrimshaw piece, “Doing Data.” What was immediately interesting to me was her story about studying in London and wanting to do her Master’s thesis on whether theatre and dance organizations used the data they collected. Her advisor couldn’t understand why.

“His response was of course all organizations use data, that wasn’t interesting, I should focus on what they were doing with the data or whether they were satisfied instead. He had worked for years with Theatrical Management Association, a service organization for theaters in the UK, and he had started an audience data collection and dissemination program in 1990—fifteen years before I was coming to him with this topic. So he thought my concerns were old news.”

But Sara knew this wasn’t how it was in the US where data was collected, but rarely acted upon. Part of the reason she felt was because people don’t know how to effectively use the information that passes through their hands. This is what she found in the course of her research:

“I was also surprised that less than 50 percent of organizations collected demographic information, as that is frequently needed for grant reporting. The results also showed that over 50 percent of organizations did not consistently have access to their own box office data—meaning that they had to rely on other methods to collect data about their audiences.

Do I have any recommendations? I think using data is simple and complex at once. The most important things are fairly easy: looking at the data, analyzing it for trends, inspecting to see if there are any obvious gaps. However, the key is really using the data to understand your audience—asking who they are, where they are, why they come to you, then using that data to help inform your future decisions. Do you want to compare yourself to other organizations? Do you want to reach people you currently aren’t reaching? Is your audience who you thought they were? Audience data can help to answer all of these questions more honestly.

[…]

…I don’t believe that there is a magic formula or a correct answer. Instead, I think it’s a process of using little pieces of information to help create an overall picture.”

I have to agree that it is both simple and complex to use data. For me it is as much about the suspected dearth of information from certain segments of our audience as trying to accurately process the information I do have. I know our audience surveys aren’t being filled out by a representative portion of our audience because we barely get any completed by people in the 20s and 30s, but we see a lot of them passing through the doors.

Likewise, so many people purchase tickets at the door where we don’t have the time to collect information not directly related to the ticket purchase that the only data I have is from the lesser percentage that plan ahead. I could send reminder postcards to people who make their decisions last minute and perhaps improve my relationship with them. My software tells me exactly what time they made their purchase–if only I knew where they live or who they are!

I don’t want to be making decisions about how to serve all of my audience based on what I know about 20% of them. I figure that is a formula for retaining slightly more than that 20% but that is all I have to work with at the moment. Though that isn’t to say we can’t ask audience members to help us serve them by making note of their account numbers so we can better track simple things like attendance preferences. We may still miss a large segment of the audience, but we will narrow the gap a little and let people know their presence is important to us.