Saturday, May 26, 2007

Mark Onslow and Lidcombe at ISA


At ISA, I attended Mark Onslow's workshop on the Lidcombe program. He didnt seem to be very pleased when I told him during the workshop that his random control trial is flawed. The day before, I had given a talk (which Mark did not attend, I think) on why the random control trial did not show efficacy despite their claims. Sue Block was not too pleased, and said that random control trials are widely used in all kinds of areas. Of course, this is the typical response without addressing my arguments. Why can no-one say: Tom, I have listened to your arguments. I don't agree with your argument on slide 10, because of X and Y. That's why you are wrong. No, everyone says but RCTs are widely used, or the RCT's statistics have been computed by a statistician. Which kind of scientific debate is this??? Anyway, I met Sue Block later at the B&withnoB, and we had a more constructive debate.

Several people approached me regarding my talk or even congratulated me for standing up against Lidcombe. I just want to emphasise that I am not really against Lidcombe, just against their claim that they have shown efficacy with an RCT. I am not an ideologue who feels a strong urge to defend his/her view of the world. Here are my thoughts:

1) I have been to the workshop, and overall the program seems to make sense and I dont think it will harm the children at the very least. In fact, it does encourage good parent-child interaction, which should be beneficial anyway. So it might well prevent greater behavioural, social and psychological effects from developing.

2) What I do criticise is that Mark Onslow claims that research has shown efficacy. However, as I said before, they have proven anything. In fact, during the workshop he himself showed some long-term data that suggests that only 84% of the children recover, which is close to natural recovery.

3) My guess is that Lidcombe might well help kids to recover faster that would have recovered anyway, might well ease tension between kid and parents (the therapist acting as supper-nanny), and for the non-recovers reduce some secondary effects, BUT I doubt very much that it makes the kids fluent that would not have recovered anyway. To do this, the program is not interventionist enough and one needs to do more active speech work.

4) If I had a stuttering child and the choice of doing nothing or doing Lidcombe, I would certainly do Lidcombe with them.

5) I get the feeling Mark does not really know how to handle my statistical arguments, but his statistician should have: Mark Jones. He has never properly addressed my issues I have raised by replying to my email or my rapid response in the British Medical Journal.

6) Lidcombe might well have an effect but on a behavioural (less severe secondary effects like grimaces or strong blocks) and psychological (avoidance and shame) level, which is much more difficult (if not impossible) to capture in a trial.

7 comments:

Ora McCreary said...

Tom - Your link "a talk" points to a July 2006 blog entry of yours, describing some thoughts on what an RCT is. Is that the link you intended? The July 2006 entry does not say much about your ideas on RCTs, and nothing about Mark Onslow or Lidcombe.

It's unclear from today's blog entry whether your recent talk at ISA casts doubt on the use of RCTs in general, or whether it was directed more specifically at the misuse of the RCT technique (or misinterpretation of results) by Mark Onslow in analysis of the Lidcombe approach.

Do you have any more specific info concerning your thoughts on RCTs? Can you provide a copy of your talk or a PowerPoint or something? Or at least a blog entry describing your ideas?

Thanks.

Ora McCreary said...

P.S. I've just read your September 2005 letter to the BMJ "The statistical fluctuation in the natural recovery rate between control and treatment group dilutes their resultsThe statistical fluctuation in the natural recovery rate between control and treatment group dilutes their results"
(http://www.bmj.com/cgi/eletters/331/7518/659#115912)

In that letter, you wrote: I find a 10% probability that a "minimum worthwhile difference between the two arms set at 1.0% syllables stuttered" occurs due to statistical fluctuation and not due to treatment effect.I find a 10% probability that a "minimum worthwhile difference between the two arms set at 1.0% syllables stuttered" occurs due to statistical fluctuation and not due to treatment effect.

I'm no statistician and I may have this wrong... My understanding is that the standard RCT aims for a less than 5% chance that the observed results occur by chance(usually expressed as p<.05), whereas you're observing that this trial achieves only 10% (p<.10). Is that what you're saying? If so, 90% is obviously not as powerful a result as 95%, but it's not worthless either.

Possibly instead of choosing an effect size (1.0% difference in syllables stuttered) that allowed them to claim p<.10, they should have claimed a smaller effect size (for example, 0.5% difference in syllables stuttered) that would have supported the stronger claim of p<.05.

Though I don't fully understand your argument in your Sept 2005 letter, I would agree if you're merely saying that their results may not be as strong as they're suggesting. But on the other hand it seems to me that your claim that "the random control trial did not show efficacy despite their claims" is also too strong; 90% may not be 95%, but a 90% number is still strongly suggestive.

As I say, I'm not a statistician. I may have misinterpreted your letter. Also, I can't tell from your blog entry whether your recent talk at ISA responds to the same research and claimed results that Jones and Onlow presented in their article in BMJ in 2005. I'd be pleased to see a copy of your recent presentation, and/or a more complete of discussion of your ideas.

Thanks.

Ora McCreary said...

PPS.

I don't mean to monopolize the discussion here...

In case you didn't notice it, I just want to bring to your attention that the Bothe article that I emailed you on 29 April (the same article which you referenced in your 8 Feb 2007 blog entry) makes several references to your Sept 2005 letter to BMJ and the issue you've raised (untreated recovery).

Tom Weidig said...

Hi Ora,

thx for your comments and probing questions! Finally someone who has actually read, digested, and questioned what I write! :-)

1a) It was directed more specifically at the misuse of the RCT technique (or misinterpretation of results) by Mark Onslow in analysis of the Lidcombe approach.

OR 1b) casts doubt on the use of RCTs for children due to the natural recovery rate.

2) I have changed the link, which now points to a more specific post.

3) I will reply in a post to your questions in your P.S.

4) Yes, I know that Bothe has cited my rapid response.

Ora McCreary said...

Hey Tom, ich interessiere mich besonders fuer dein (1a) - I'll be interested to read a more complete description of your objections to Onslow etc.

As you may recall from my email to you, my background is in math and computer science, but I'm definitely not up-to-date in statistical/analytical research techniques. So I'm still not sure whether my questions to you are based on my own weak understanding of statistics, which you will be able to put aside with a simple counter-thrust or two...or whether I am posing serious questions which merit a serious response.

Since the time that I posted my original messages yesterday or the day before, I've read the original BMJ article by Onslow and his statistician Jones?. I still can't see the flaw in the original article. My initial, simple reaction is that your statistical objections are probably already incorporated into the original statistics from the BMJ article. I'm not suggesting that your objections are not valid, but rather that the original BMJ analysis incorporates and supersedes them.

But I look forward to your further explanations.

Best regards, Ora

adrian said...

Tom and Ora are far better qualified then I to debate the statistical portion of the RCT, so I will stay out of that debate. Although it does make me think of a saying I once heard, "statistics are like prisoners, if you torture them long enough they'll tell you exactly what you want to know."

But I do want to point out another possible flaw in the RCT. The study states:

"Main outcome measures Frequency of stuttering was measured as the proportion of syllables stuttered, from audiotaped recordings of participants' conversational speech outside the clinic. Parents in both arms of the trial collected speech samples in three different speaking situations before randomisation and at three, six, and nine months after randomisation."

I am not convinced that three fluent speech samples is enough to support the idea that early intervention works. Stuttering is a very strange disorder. Many of us tend to be more fluent when we know we are being observed and are better able to use any techniques we have learned. As a child I became completely fluent when in the company of a speech therapist. This was the case in her office or if she took me out to lunch and asked me to order my food. However, if ordering food with my parents the same night I would stutter horribly. This false fluency effect is not uncommon in children or adults. Considering the parents are the therapist in Lidcombe, are these speech samples really representative of the child's speech in the outside world with friends or in school? Another point, considering Lidcombe praises clients for stutter free speech and uses a gentle correction for stuttered speech, isn't it possible some of the children have begun using word avoidance and substitution. If so they will appear more fluent, but may simply be masking the stutter.

Like Tom, I am not against Lidcombe. It may be a fine program which actually helps these children. However, I do applaud Tom for questioning the RCT. Other programs such as Hollins have used similar tactics in order to boast high success rates and increase their enrollment and profitability. This may or may not be the case with Lidcombe, but these questions should be asked.

Tom Weidig said...

Adrian,

I agree that the children might be more conditioned on average on fluent speech.

Unlike the control group, they know the clinical environment better and might be more fluent.