Seth Godin offers a very relatable example of why more data isn’t always better by emphasizing the need for vigilance when setting an alarm clock in a hotel room. If you set the alarm for 7 am before going to bed at 10 pm but don’t notice that the clock currently reads 10 am, you aren’t going to be woken by the alarm clock the next morning. (I am sure we have all done this at least once.) He suggests the am/pm setting is an extraneous bit of data serving as an impediment to the clock fulfilling its purpose.
This is a very simple illustration of a point I bring up often on the blog — just because you can measure something doesn’t mean the data is useful for your goals and, in fact, may be an obstacle to understanding the relevant data. Just because you can measure the economic impact of the arts doesn’t mean economic impact is a valid measure of the value of arts and culture.
This concept also has relevance in terms of the regular practice of surveying audiences/attendees. Just because you can ask a question doesn’t mean you should or that what you learn will be useful.
As Godin writes,
The metaphor is pretty clear: more data isn’t always better. In fact, in many cases, it’s a costly distraction or even a chance to get the important stuff wrong.
Here are the three principles:
First, don’t collect data unless it has a non-zero chance of changing your actions.
Second, before you seek to collect data, consider the costs of processing that data.
Third, acknowledge that data collected isn’t always accurate, and consider the costs of acting on data that’s incorrect.
All this being said, my staff usually starts out surveys asking a question for which the answers will be useless as data points, but for which the goal is to establish a connection and willingness to respond in the survey taker. Basically, we figure people are more apt to answer 4-5 questions if the first one is a fun question about themselves. So for example, if we are doing Man of La Mancha, the question might be, “what is the impossible dream you dream?”
There are times when it is okay to collect data when it won’t shape your decision making and there might be a cost to collecting and processing it if doing so advances goals in other areas.