Saturday, March 20, 2010

Pagoclone: Data locked & results known?

A reader writes:
Had my two month open label appointment with the clinic conducting the trial. The doctor said that the database with the results from Phase IIb was "locked" two weeks ago. He felt sure that they had the analyzed results that same night and that the statistician is now making sense of it all. He thinks we should have something in the next two months or so.
So expect the results within the next two months...

I get chills when I read stuff like "the statistician". Effectively, one person is analyzing the data, and everyone else is kind of accepting the verdict of that person. I have seen so many experts that are incompetent (like for the Lidcombe RCT), make a stupid mistake, are sloppy because it's Friday afternoon and they want to go home, are ideologically driven, misunderstand the nature of some data measures, fill in missing data, or on purpose don't do consistency checks because they like the result as it is. I must know, because I have probably been guilty of all but not at the same time though! ;-)



It is of utmost importance to have more than one statistician look at this, and not just look at the work of the
colleagues but actually doing the analysis independently without having met him/her before! You would be surprised at the different interpretations! And it is important that the co-authors are actually co-authors in the sense that they understand every single dot and letter of the article. Anything else is unprofessional.

Imagine the following scenario. The expert make a coding mistake (which happens to the best), but the results are positive, his approach is perfect, and everyone else is happy with the approach. People now start to create theories on why it works, and the compound is being marketed. But at the end of the day, the little coding mistake changed the world, let people believe stuff, just because everyone is parroting rather than thinking for themselves.

Now imagine the opposite scenario. He makes a mistake, and the results are really bad i.e. no effect. What happens? Everyone starts saying: Oh, that's strange because we expect an effect. Just re-do the calculations. Or the company CEO says: we need to have this, hire another expert. And then someone says but you do that because you don't like the first one, and then the CEO rightly says: Oh no, I am making sure that quality is ensured with an independent third party review... And they will find the mistake. See what's happening... the CEO would never have re-done the calculations for positive results.

And here is another twist. If the results had been way too positive, they might have re-done. So the results need to be good but not exceptionally good, then they sail through and there is no checking...

Again, that's what I have seen time and time again with people who are far cleverer than most statisticians or medical researchers... it is hard to counter our flaws...

I am not saying that is going to happen here. The FDA has their own experts, but I have seen it in financial regulators. They look at the approach but they very rarely re-do themselves. I mean why do you think the big investment banks have model validation teams, precisely because they know that another team of experts should take an independent look without actually taking to the modeling team. And they still mess up from time to time...

11 comments:

jhnhth said...

I'm in the study and haven't heard anything about the "lock" your reader mentions. I will check with my research center next week and will advise.

Regarding your concerns, I doubt there is "one" statistician analyzing the data. Rather, I am inclined to believe there are many professionals working on this. My knowledge of the program is limited, but I haven't seen anything to lead me to believe the research program has been anything less than first rate.

I've been blogging about my experiences with the study here:

http://pagoclone.blogspot.com/

Tom Weidig said...

"Regarding your concerns, I doubt there is "one" statistician analyzing the data. Rather, I am inclined to believe there are many professionals working on this."

They might have 2-3 statistician, but the vast majority have no clue about statistics, measurement biases, and mathematics. They are passive consumers, and won't detect subtle errors. And there are those statisticians which are good and those who just do procedures. Given the nature of stuttering, subtleties may distort results.

"My knowledge of the program is limited, but I haven't seen anything to lead me to believe the research program has been anything less than first rate."

Adhering to procedure does not make the result sound, this is ESPECIALLY true because measuring stuttering is so much more tricky than standard disorders like measuring the weight of someone...

My rule is the slicker it looks the worse it is! Because real scientists really care about image...

Dave Rowley said...

I think there is a broader point to be made here too, which is that most readers of journal articles don't really understand the statistics which have been carried out. Hand on heart, how many readers really understand the tests used in most journal articles? Is enough information given about the data, design, etc. to even be able to make a judgement? More worrying do academic reviewers actually understand the tests used in article the editor has sent them for review?

Tom Weidig said...

You can only truly understand a paper if you have done the calculations yourself independently without looking at how they did it. That's the plain truth.

Though you can point out weaknesses, suggestions and flaws without re-doing the work, but you will not find all weaknesses and flaws...

They should release the underlying data so that everyone can make up their mind and some of us re-do the statistics!

Anonymous said...

Yes it is a real concern that there is more pressure on researchers to achieve the results that they were aiming to achieve than to resolve the overall problem. It is a concern for people who stutter who want real results but I guess if I was in the researchers position I might use a few of the loop holes available to keep the gravy train moving.

Anonymous said...

I work in science. My view is that statistics is valuable and important but at the same time it is overvalued and can be misused. From what I have observed most scientists don’t understand statistics very deeply and can be seduced by fancy statistical jargon without really understanding what it means. Statisticians, even ones who know science, have a difficult time seeing the big picture and don’t fully understand the nuances of scientific data. I am of the opinion that actually looking at the data without fancy statistical analysis can be very informative but I seem to be in a minority these days.

Harry said...

Tom writes.... "They might have 2-3 statistician, but the vast majority have no clue about statistics, measurement biases, and mathematics."
Firstly, if a statistician doesn't have a clue about statistics, then god help us!
Secondly, it's like someone saying to you "A vast majority of scientists don't have a clue about science"
Thirdly, you should look at some of the generalisations and non-sense you write.

Tom Weidig said...

To Harry,

it's my experience:

1) Statisticians get things wrong, and apply stats blindly and surprisingly often with passive understanding.

2) A vast majority of scientists has not clue about statistics.

3) My generalizations are generalizations in that they on average are right.

jhnhth said...

About the info from your reader, the study is definitely still going and any conclusions are a long way off!

Read all about it:

http://pagoclone.blogspot.com/

Anonymous said...

I want to participate to the Pagoclone trial study. I am living in Quebec Canada. Is there any trial taking place in Canada? If not, can I attend to any trial in US? Thanks a lot for your help.
Belinda.

Anonymous said...

I want to participate to the Pagoclone trial study. I am living in Quebec Canada. Is there any trial taking place in Canada? If not, can I attend to any trial in US? Thanks a lot for your help.
Belinda.
email: barbaresques@hotmail.com