Monday, May 10, 2010

How not to review outcome studies

Bert will probably hate me for undermining his review, but that's what a scientific debate is. On the mailing list STUTT-L, Bert Bast writes (in the name of the Dutch Stuttering Federation?):
I would like to jump in on the '80% spontaneous recovery' especially as a measure for the (lack of?) success of early intervention - e.g. by the Lidcombe approach.
... I got these data from literature:

· spontaneous recovery 50-74%;



· success of early intervention in various studies (a.o. Lidcombe):
82%, 100%, 91%, 94%, 99%, 84%, 98%, 94%, 90% with two ‘outstanding studies:

o a large one (Onslow, 2000): 250/261 children stutter free after therapy

o a long one (Jones 2006): 84% stutter free after 5 years

So, when adopting these rough estimates for toddler stuttering (4%) and persistent stuttering (1-1.5%), the (as well roughly measured) difference between therapeutically helped recovery (92%) and spontaneous recovery (62%) might mean that 30% from that 4% = ±1% less persistent stutterers might emerge after wide application of early intervention. One might bicker as to the preciseness of this evaluation, but the message is still clear.

We in our generation (as I do myself) still suffer from the ‘wait, it will go over’ approach, but we wouldn’t like to prevent our children to benefit from these new insights, won’t we?

Bert Bast, Chairman Dutch Federation on Stuttering (NFS)

My response:
I need to respond to Bert's discussion on outcome numbers, which is misleading and naive.

First of all, an outcome study needs to full-fill certain minimum criteria:

1) LONG-term outcome. Not after therapy, where everyone even adults are 100% fluent, but more than a year later. For kids, several years later.

2) You need to have a VERY LARGE sample for kids at the very least 100 kids to reduce statistical error (which is much greater due to natural recovery)

3) You need a CONTROL GROUP of no treatment to have the natural recovery rate captured.

So what is the number of outcome studies on Lidcombe satisfying all three? ZERO!

So let's drop the requirement for a control group and a +100 sample. Then there is ONE study (Jones 2006).

Bert says: "84% stutter free after 5 years"

Bert, if you actually read the study, you will find out that a large proportion of the children could not contacted anymore. So we have a large scope for a bias, a small sample, and no control group. And on top the 84% is not that convincing given large stat errors.

On top, the study does not say anything about Lidcombe-specific methods, but just looked at intervention in general. This is a very important aspect that is neglected. Maybe any treatment would get the same effect, if there is an effect.

And lastly, where is the theoretical basis? If there is a brain issue or a mutation, it will NOT go away by any behavioural therapy.

No comments: