They blinded me with science.

violin

Last week there was an enormous amount of press pertaining to a recently published paper that examined whether a group of professional violinists could tell the difference between fine old Italian instruments (such as Stradivari or Guarneri) and more modern (and much less expensive) violins. The researchers concluded that the participants were not able to distinguish the differences with any degree of accuracy, with various implications. I was struck by a number of factors in the press accounts of this study, so I decided to read the actual paper in its entirety and draw my own conclusions.

This analysis was conducted in Indianapolis in September 2010 during the International Violin Competition of Indianapolis. The competition attracted quite a crowd of violinists, violins, and violin makers, so the researchers viewed this as an ideal location and time. There were 21 participants in the study, including contestants from the competition (four), jury members (two), and members of the Indianapolis Symphony. Two participants went on to be competition laureates. At least some of the group owned fine old Italian instruments. So one can assume these participants were mostly seasoned musicians with material knowledge of how violins are typically compared to one another.

Six instruments were collected, three new and three old. The newer ones were between several days and several years old and presumably were superior examples of a “modern” instrument, although none were specifically identified in the paper. The others were two Stradivaris (one from 1700, one from 1715) and one by Guarneri del Gesu, from 1740. The participants wore modified welder’s goggles that prevented them from identifying the violins. Regarding the older ones:

These instruments were loaned with the stipulation that they remain in the condition in which we received them (precluding any tonal adjustments or even changing the strings), and that their identities remain confidential. All strings appeared to be in good condition.”

There are countless factors that can shape perceptions while comparing violins (even in a double-blind study), and this was the first genuine red flag for me. The setup of a fine violin is critical and highly subjective-strings, placement of the bridge, etc. The soundpost can move a fraction of an inch and completely change the way a violin responds and sounds, and this is particularly true for notoriously finicky Strads. It’s not uncommon for a spectacular instrument to seem inferior just because of an unusual setup or old strings (or getting bumped around on an airplane). It is unclear who set up the newer instruments (or when). I enjoyed the confident visual assessment of the strings- for the record, I regularly play on a 1715 Stradivari and use Vision Solo strings. They completely burn out after about a month and the instrument sounds totally different, but they look fine.

“Most violinists prefer to try out violins in a room with relatively dry acoustics…..sessions were therefore held in a hotel room whose acoustics seemed well-suited to the task”.

This is misleading and possibly indicative of preconceived notions by the researchers. It is true that many violinists try instruments in a “dead” room as a first step. No professional I know of would make any serious determination of an instrument’s quality until they played it in at least one hall (and had trusted colleagues play it while they listened). Even with great instruments, the sound “under the ear” is often deceptive- a peculiar (and sublime) aspect of great old Italian instruments is that the sound somehow expands and gains more complexity from a distance, especially in a concert hall. With many “modern” instruments, it is common that in a confined space (under the ear), the sound may also be huge and seem quite promising. But frequent criticism of some newer instruments (not all) is that in a hall it sometimes doesn’t carry past the 6th row, and the sound is often bland or one-dimensional. This entire process took place in a hotel room.

Prior to their arrival at the hotel, the violinists were informed they would be playing several fine violins, at least one of which was a Stradivari. In the main part of the study, each person was given 20 minutes (total) to try the six instruments and then rank them in order of preference using several criteria. To anyone with comprehensive experience in this regard, this is absurd; it’s nowhere near enough time to make an accurate determination, especially in a hotel room. The best one could hope for is a preliminary opinion, which could absolutely change in a different room with a different setup and more time to get used to the instrument.

The second part of the study was even more ridiculous, a kind of “speed date” approach that allowed for one minute of playing on each of two violins (one old, one newer, 10 pairs total), then offer some sort of opinion.

I could go on, but some of you may get the idea by now. The conclusions of the study seem predetermined to a degree: statistically, most of the participants couldn’t tell new from old, and everyone’s perceptions were somewhat altered by the knowledge that some of the instruments were Strads. Is this a revelation, given that actual humans were involved, under conditions that heavily favored the newer instruments? I can guarantee that the results would be completely different if the double-blind study had a third step in a concert hall (or two), with a luthier on hand and some of the participants listening to the instruments as well as playing them.

With all due respect, and despite the rigorous science and controls applied, my sense is that the researchers started with a premise and set out to prove it. The methodology of the study virtually guaranteed that in a momentary, cursory assessment (in a hotel room) the modern instruments would be favorably compared to the older, much more valuable specimens. To be fair, the paper states that the study was “designed not to test the objective qualities of the instruments, but the subjective preferences of the subjects under a specific set of conditions”. That was achieved, and “subjective” is the operative word. It also demonstrates why most professionals won’t form any serious opinion regarding quality until a) they get into a concert hall (or two) and b) they have the violin for an adequate amount of time under various conditions. I’ve played lots of outstanding modern instruments, although none yet that reliably compared to a top-level Strad or del Gesu (just my opinion); this study seems to confirm what’s already obvious to many of my professional colleagues.

Unsurprisingly, the paper itself feels like it was written by, well, a bunch of scientists who are unfamiliar with how professionals seriously assess string instruments. The team seems slightly fixated on instrument valuations and (bizarrely) another research paper from 1993 that I’ve never heard of. Despite the requisite graphs, statistics, fancy charts, and scientific doublespeak, I think it’s an entertaining read for anyone interested. You can get a copy (for a fee) at the unfortunately-acronymed Proceedings of the National Academy of Sciences.

Apologies to Thomas Dolby for my title. But look for the f-holes.

UPDATE Jan. 18 – Please read the followup article in response to the head researcher, Claudia Fritz.

 

19 thoughts on “They blinded me with science.”

  1. Frank,
    Many thanks for enlightening me on the study. It makes me realize how often we never get the whole story on issue. Do you ever sleep? MSO, FM, blogs, family…

    Reply
  2. According to Laurie Niles in violinist.com, the organizers of the study also asked the audience in the competition hall to evaluate four pairs of violins played by the CM of the Indianapolis Symphony, with one old and one new in each pair. It isn’t known whether any were the same violins used in the hotel room study. Ultimately the audience chose three Strads, from 1699, 1714, and 1715, and one modern violin.The favorite was a 1715 Strad.

    Reply
    • Hi Char,

      Sorry for the delay; this somehow got stuck in the spam filter.
      What you refer to was an informal demonstration unconnected with the study and not part of the research paper, and I don’t think the researchers were involved with it at all.

      Reply
  3. Frank,

    For those of us whose experience is limited to low end instruments your comments were a welcome and articulate discussion of issues and considerations involved in the assessment of string instruments. Thank you.

    There are several people around here who would be interested in reading your thoughts. I’m hoping you won’t mind my passing this on.

    Reply
  4. “You can get a copy (for a fee) at the unfortunately-acronymed Proceedings of the National Academy of Sciences.”

    Almost spit my coffee over the computer. Seriously.

    Enjoyed your perspective. Thanks.

    Reply
  5. Frank,
    Thank you very much for your reactions, based not on what you have read in the press (often inaccurate, with little details about the experiment and with many extrapolations) but on the PNAS paper itself. After having read your first few paragraphs, I was impressed: you had read our paper carefully, and your report was accurate. But, unfortunately, not entirely. You seem to have forgotten to read the list of the authors. Otherwise, how could you have considered Joseph Curtin, one of the most renowned makers as “a scientist who is unfamiliar with how professionals seriously assess string instruments”?
    If I may, some of your criticisms are thus unfair: with Joseph on board, we did have “a luthier on hand” and regarding your comment ” I enjoyed the confident visual assessment of the strings”, I can tell you that the assessment was much more thorough than this. Joseph Curtin and Fan-Chia Tao (an engineer for the string company D’Addario (!) as well as a good player) are well qualified to judge whether the strings were good.
    Now, may I ask you a few questions?
    How can you write ” It is true that many violinists try instruments in a “dead” room as a first step ” and then write that the experiment was not valid because it was run in a dead room? Remember, the question we asked was “Which violin would you like to take home” (meaning so you can try it further), not ‘Which one would you like to buy” as we know that violinist can’t buy a violin without having played it for a while, and in different venues (in particular in a concert hall). Yes, maybe the results would be different in a large hall (we’re hoping to test this in a near future), we totally agree with you on this, but this doesn’t change the results of our study, nor makes it invalid. Our conclusions were clearly stated “under the ear, in a dead room”. We are not extrapolating (though, unfortunately some people are) about what would happen in a large hall.
    My second question is: why do you think that “the methodology of the study virtually guaranteed that the modern instruments would be favorably compared to the older”? It was the exact same conditions for all violins! Isn’t it biased to say that a hotel room and a small amount of time is fine for new violins but a concert hall and a lot of time is needed for old violins? And who is going to take a violin home and/or to a concert hall if he doesn’t like it after a relatively short trial in a maker’s shop (usually quite dry acoustically)? Nobody … at least if it’s a modern violin. Now tell them it’s a Strad … and they will all want to take it!
    Looking forward to hearing from you. All comments are welcome to design another study to cover the limitations of this one. Research with humans works like this; you can never test everything in one experiment, for practical reasons: humans get tired so you usually can’t involve them in an experiment which is longer than 2 hours, and are busy with their own life so it’s hard to have them come for many sessions. So you need to go step by step, and although this can be frustrating sometimes, it’s how it works!
    Best wishes.

    Reply
  6. I don’t understand. I left a long and very detailed comment yesterday (it took me at least 30 min to write it!), and it’s now gone! What has happened?
    Best wishes.

    Reply
    • Hi Claudia,
      First, many thanks for your comments. As a matter of policy, all comments are moderated before they appear. Sometimes takes a day or so depending on my schedule, usually much faster.
      I’m posting another article later on that will hopefully address some of your points, and of course will include your original comment. Thanks again for reading and responding.
      Regards, FA

      Reply
  7. Hi Frank,

    It seems clear that you feel this wasn’t a truly fair test. I was particularly struck by an assertion where you speak of the tests occurring “…under conditions that heavily favored the newer instruments”. I’m interested in teasing out exactly which conditions you believe contribute to this.

    For example, in your piece above, you drew attention to the importance of instrument setup. My specific question is: Do you actually have reason to believe that there was a bias in the setup of the instruments? Whether towards/against the old Italians? Or indeed towards/against the new instruments?

    I ask because although you allude to the importance of instrument setup, I don’t see you reporting a clear case of this being skewed one way or another.

    Thanks,

    Ed Barrett

    Reply
    • Hi Ed,

      Thanks for your comment. I’m not sure “fair” is applicable here, maybe “incomplete” is more accurate. I’m trying to get an article up a little later that will expand on the points in your comment, as well as those of Ms. Fritz. Suffice to say that since I wasn’t there, I cannot offer any authentic case of something being “skewed”, just my own opinions based on experience and reading the actual paper. Stay tuned, and thanks for reading.

      Reply
  8. Well written article Frank.

    If you neglect a plant, it will wilt. A flower will droop without water and sunshine. A Stradivarius dies behind glass. A Stradivarius dies without attention to the minute details needed for it to bloom. Old strings, a soundpost pinch to the right or left (the late Rene Morel most likely would have cut a new soundpost if need be) even the weather can make a great deal of difference whether a Stradivarius sounds its best. Not to mention the player’s ability knowing how to play this kind of instrument. Certainly first-class bow control is required. Hotel rooms are certainly not ideal for this kind of experiment and am surprised to learn that is where it was conducted rendering this exercise pointless. A publicity stunt I dare say.

    Reply
  9. Claudia Fritz wrote “How can you write ” It is true that many violinists try instruments in a “dead” room as a first step ” and then write that the experiment was not valid because it was run in a dead room? Remember, the question we asked was “Which violin would you like to take home” (meaning so you can try it further), not ‘Which one would you like to buy” as we know that violinist can’t buy a violin without having played it for a while, and in different venues (in particular in a concert hall).” I must say that I took “which one would you want to take home” to mean which one would you want to keep not which one would you want to further audition. Seems to me the methodologies for auditioning violins needs to be more thoroughly addressed here. Especially with regards to Strads and the like. Obviously this is not something ever done blind nor ever done casually. No doubt bias plays a part of any such audition. at the same time many of the claims about the alleged unique properties of a Strad in concert halls are not addressed by this test at all. I am kind of surprised the test was even done if the question on tap really was which violin would you want to take home to audition further. Clearly this does not come into play with Strads. Do Srads really go through a preliminary auditon by the violinist in a dry acoustic before the violinist decides to give it further consideration? I thought the question, the real question, was do Strads really have a unique and superior quality to their sound. Although I do appreciate the attempt to remove bias the test itself is at best incomplete if it was simply a test of violin audition methodologies. IMO If that is what was meant to be tested the authors needed to start with an in depth investigation into those methodologies. Do violinists chosing between two violins to further audition ever just limit themselves to one minute with each instrument in an acoustic space such as a hotel room? If it was an attempt to test the legitimacy of the alleged legendary sound qualities of the Strad it seems to me to have missed the mark.

    Reply

Leave a Reply to Claudia Fritz Cancel reply