What Does it Mean to be "Significant"?

Research Series: We like to talk about music therapy research on this blog, and for good reason. Music therapy is a field that's based in evidence-based research, meaning that the techniques we use are supported by scientific experiments conducted by music therapists and other researchers throughout the world. Maintaining this level of research is not easy or cheap, but as a field it's vital we maintain this dedication to evidence-based interventions. Otherwise, we cannot ethically say that what we do...works.

As a result of this dedication, music therapy students get a heavy dose of research courses in both undergrad and graduate programs. But all this research talk comes with a lot of jargon that can get confusing. So what are we talking about when we say that a music therapy technique "significantly" improved something? Let's detangle some of this research jargon! Here's part 1 of our series on research terminology, statistical significance.


Statistical Significance

The word significant has entered our vernacular to mean large, massive, great, etc. But what does that word mean when discussing research? It actually has a very specific, important meaning. It's not just something we throw around. When discussing a research study, we often say that a variable "significantly" affected another variable. Let's take this study from the Journal of Music Therapy on the long-term effect of once-a-week group music therapy sessions for elderly adults with moderate to severe dementia.

In this study, the researchers measured systolic blood pressure levels of the participants 1 and 2 years after the start of the music therapy group. The control group (the group that did not receive music therapy services) experienced significantly higher blood pressure levels. Blood pressure naturally increases as we age, so the fact that the music therapy group did not experience this increase is a big deal! All from once-a-week group music therapy sessions.

But what does it mean that the difference was "significant?" In order to scientifically prove something, we need to be sure that it didn't happen by random chance. Think of fortune cookies. I'm sure at some point, we've all had the experience of getting a fortune cookie that seemed to perfectly match our current life situation. We might stick this fortune in our wallet for safekeeping. But does that mean that fortune cookies are accurate predictors of our future? How many fortune cookies do you get, read the fortune, and toss in the trash? Probably a good amount. The fact that one of them matched our current situation was a coincidence. A random happening. Like one of my favorite sayings goes: "Even a broken clock is right two times a day."

So to ensure the result of a scientific experiment are accurate, we need a way to prove that the results didn't happen by random chance. So we do this with a degree of confidence. Once we've obtained our data (in our example, blood pressure), we can run some mathematical tests (for example, a chi-squared analysis) to see what the likelihood is that this event happened by random chance.

Every time we flip a coin, there's a 50% chance that it will land on heads. It's random. If after we ran our test on the blood pressure numbers, we saw that there was a 50% chance these levels happened by chance, we cannot say with scientific certainty that music therapy had any hand in that result. However, if our test reveals that there was only a 4% chance the blood pressure changes were a random event, then we CAN say that music therapy likely had an effect. We call this a significant difference.

So when we say that a variable had a significant effect, we mean there's a very small chance it happened by random, and we can be pretty confident that the results happened because of what we did. Our threshold for significance is typically between 5 - 10%. This number is sometimes referred to as the "alpha." The lower the number, the more confidence we can put behind the result.

Alright, that was quite a lecture, let's recap:

  • If something is significant, there's a VERY little (5%) chance it happened at random
  • We use mathematical tests to determine this percentage
  • If the chance that our result was random is OVER 5-10%, we cannot say that our intervention had anything to do with the result.

When it comes to music therapy, what does all this research mean to YOU? Let's talk about how music therapy can serve your needs!


Image credit: http://memegenerator.net/instance/34457778?urlName=Chemistry-Cat&browsingOrder=Popular&browsingTimeSpan=AllTime