Ben Goldacre's Review of the Year

Moderators: Debbie Hepplewhite, maizie, Lesley Drake, Susan Godsland

Post Reply
g.carter
Posts: 1859
Joined: Wed Nov 05, 2003 7:41 pm

Ben Goldacre's Review of the Year

Post by g.carter » Sun Dec 19, 2010 1:28 pm

There will doubtless be an avalanche of informational misuse via test ‘analysis’ (a classic example being the Reading Recovery use of the BAS tests which showed how RR magicked years and years of progress out of the rabbit’s hat ). Surely the BAS test provides useful information only when implemented by a educational psychologist with NO stake in the results?.

Then there will be the meaningless trumpeting of standardised gains in years – an endless source of dodgy statistics in that they obscure the information on those children who are drowning. Thank goodness for the BBC’s Education Correspondent who actually went into Secondary Schools and began to piece together the information on these children who haven’t got a cat’s chance of accessing the secondary curriculum. ‘Standardized reading tests’ tend to obscure this information and do little to throw light on instruction – vital information for teachers, parents, children themselves. Perhaps when when there is a complete overhaul, teachers will begin to understand that it’s not the children/parents who are at fault but instructional methods they use and how these are implemented.

Ben Goldacre’s bad science review of the year is well worth reading in full:
http://www.guardian.co.uk/commentisfree ... f-the-year

- the exacts below are particularly interesting to RRF readers:
There were dodgy government figures on how many children were "saved from abuse" by some new rules, while the London mayor, Boris Johnson, announced that instead of a simple, well-conducted randomised trial to find out which of two teaching methods is best, he would answer the question with a public competition between schools that choose each method…

More chillingly, if a piece of information which reinforced your prejudices is corrected, this only reinforces your prejudices; and we think crimes are less serious, when they have more victims.

User avatar
Susan Godsland
Administrator
Posts: 4973
Joined: Thu Oct 30, 2003 11:10 pm
Location: Exeter UK
Contact:

Re: Ben Goldacre's Review of the Year

Post by Susan Godsland » Sun Dec 19, 2010 11:28 pm

Ben Goldacre wrote:
More chillingly, if a piece of information which reinforced your prejudices is corrected, this only reinforces your prejudices
Yes, I read that too Geraldine, and my heart dropped a little :sad:

Goldacre also said this in an interview:

Q. How do we get people to pay more attention to the research evidence, rather than making choices based on anecdote or authority?

A. I don't think you can. You set yourself up for a heart-dropping lifetime of misery and feeling like a failure if you say my objective is to make people, who don't want to pay attention to evidence, pay attention to evidence. To me, what I'm interested in is telling people about how this stuff works

hhttp://healthland.time.com/2010/12/17/mind-reading-dr-ben-goldacre-wants-his-book-to-ruin-your-christmas/

chew8
Posts: 4162
Joined: Sun Nov 02, 2003 6:26 pm

Re: Ben Goldacre's Review of the Year

Post by chew8 » Mon Dec 20, 2010 11:54 am

Geraldine wrote:There will doubtless be an avalanche of informational misuse via test ‘analysis’ (a classic example being the Reading Recovery use of the BAS tests which showed how RR magicked years and years of progress out of the rabbit’s hat ). Surely the BAS test provides useful information only when implemented by a educational psychologist with NO stake in the results?
The trouble is that this reasoning calls into question not only the RR findings but also the Clackmannanshire findings. The Clack. study also used the BAS (though an earlier version), and the testing was done by people who had a stake in the results. Test results from other synthetic phonics programmes (e.g. SRS?) have probably also been produced by people with a stake in the results. I don't think we can assume that the RR use of the BAS was dodgy without more evidence than we have - or do you already know of such evidence, Geraldine?

Jenny C.

g.carter
Posts: 1859
Joined: Wed Nov 05, 2003 7:41 pm

Re: Ben Goldacre's Review of the Year

Post by g.carter » Tue Dec 21, 2010 1:48 am

I know that very reasonably, Jenny, you don't like emotive words like 'dodgy' unless there is 'evidence' that one can provide root and branch evidence.

But I work on observation, reading of hundreds of comments on fora, newspaper articles, papers, video evidence, and the words of Marie Clay herself and proponents like Brian Cambourne (sp?) and also draw conclusions from the lack of any comparative studies. Why on earth wouldn't something so heavily funded set in place comparative studies - one-to-one daily SP tutoring for 20 weeks and see if those children could read words like :
in at cat ball fox red the her door shop
('read' 8 out of the 10 words correctly and hey presto bring out a press release to say that ie Reading Recovery produces amazing 'gains' - then get Gordon Brown's henchmen to conduct "research" to show that it's worth spending £5,000 per child on the thousands of children labelled as defective ... ).
Some things are beyond belief and while it is reasonable to believe that BAS can be used responsibly, it is also reasonable, given the words of Clay and Cambourne and the history of Reading Recovery, to make the accusation of 'dodgy'.

But this is the point that I have been trying to make - you need to look at the instruction and how the child performs, rather than at a battery of tests that, if they are done 'in house' ,are susceptible to over-eager interpretation, and if conducted by 3rd party can be subject to candidate stress.

When there is universal recognition that some children need a massive amount of appropriate text practice to acquire basic skills and that this simply requires a Teaching Assistant, or volunteer parent (and direction and understanding of SP from the Head down), there is no need for this tragic tail of illiterate children nor the obscene sums of money spent.

JIM CURRAN
Posts: 3129
Joined: Fri Oct 31, 2003 7:18 am

Re: Ben Goldacre's Review of the Year

Post by JIM CURRAN » Tue Dec 21, 2010 9:50 am

Geraldine said: "When there is universal recognition that some children need a massive amount of appropriate text practice to acquire basic skills and that this simply requires a Teaching Assistant, or volunteer parent (and direction and understanding of SP from the Head down), there is no need for this tragic tail of illiterate children nor the obscene sums of money spent."


I agree Geraldine, but there needs to be a real will to make this happen. In thirty six years teaching I’ve not felt that will except for a few individuals like yourself. The powers that be seem content to do nothing more than go through the motions as if all these children mattered, when that’s not really how they feel.

chew8
Posts: 4162
Joined: Sun Nov 02, 2003 6:26 pm

Re: Ben Goldacre's Review of the Year

Post by chew8 » Tue Dec 21, 2010 11:45 am

What I was trying to say, Geraldine, was that we could be accused of inconsistency on a particular point: the point about testing being done 'in house' by people with a 'stake in the results'.
You wrote:Surely the BAS test provides useful information only when implemented by a educational psychologist with NO stake in the results?

If we apply this argument rigorously to RR research, then we could look inconsistent if we don't also apply it rigorously to synthetic phonics research, the Clackmannanshire study being a key example. In fact, however, I think there's probably quite a lot of perfectly reputable research where the testing is done by people who have a stake in the results.

Jenny C.

JIM CURRAN
Posts: 3129
Joined: Fri Oct 31, 2003 7:18 am

Re: Ben Goldacre's Review of the Year

Post by JIM CURRAN » Tue Dec 21, 2010 1:32 pm

“In fact, however, I think there's probably quite a lot of perfectly reputable research where the testing is done by people who have a stake in the results.”

I agree in principle with what you say Jenny but unfortunately we know that when the stakes are high people tend to cheat in test situations. The higher the stakes the higher the probability that cheating will occur. There’s actually a law to describe this type of behaviour but unfortunately I can’t recall it. I have referred to this phenomenon on the board before.

chew8
Posts: 4162
Joined: Sun Nov 02, 2003 6:26 pm

Re: Ben Goldacre's Review of the Year

Post by chew8 » Tue Dec 21, 2010 3:14 pm

Yes, Jim, but my point is that if cheating is a possibility when testing is done by people with a stake in the results, then it's a possibility in synthetic phonics studies as well as in RR studies. We need to beware of opening ourselves to the charge of the pot calling the kettle black on this specific point.

Jenny C.

User avatar
Debbie Hepplewhite
Administrator
Posts: 3653
Joined: Mon Aug 29, 2005 4:13 pm
Location: Berkshire
Contact:

Re: Ben Goldacre's Review of the Year

Post by Debbie Hepplewhite » Tue Dec 21, 2010 4:37 pm

Is Clackmannanshire a direct comparison with regard to in-house testing - in that Reading Recovery is a specific, existing programme with a commercial interest?

Was there a commercial interest as a 'programme' at the time of the Clackmannanshire testing?

I do agree, however, with the principle of the point you are making, Jenny.

Is there resistance to test and compare different programmes in the UK I wonder?

It is one thing to test for results of 'a' programme or method and it is another to set up a direct comparison with other methods or programmes - and then test.

I know of personnel in one university that seemed to want to avoid a study of one particular SP programme because the university had 'its own programme'.

It is not so easy to break into the field of transparent testing - it takes a lot of money, professional support frpm honest and knowledgeable people, a lot of time, available schools, - in other words, the 'will' to do it by more than just the programme writer!

Perhaps the SP programmes would stand a better chance if they had the kind of business and government financial support of which some organisations have been the recipients.

yvonne meyer
Posts: 736
Joined: Tue Oct 21, 2008 4:58 am
Location: Melbourne Australia

Re: Ben Goldacre's Review of the Year

Post by yvonne meyer » Tue Dec 21, 2010 9:55 pm

As I understand evidence based research, no single trial, no matter how well-conceived, can be flawless. However, when multiple trials all come to the same conclusion, then the evidence can be considered 'overwhelming'.

The following quote is Reid Lyon, Children of the Code. The "four questions" are 1) What does it take to learn to read? 2) What goes wrong when you don’t? 3) How do you prevent it? 4) How do you remediate it?
What we have built is a research network composed of forty-four sites in North America, Europe and Asia. All of the sites go after these four questions because we need to replicate. Science, as you know, builds, and when you’re doing science in very complex situations, social and behavioral science versus petri dishes, no one study will ever illuminate fully what we’re trying to understand. So, you have to have multiple converging studies with converging evidence bearing on these questions so that we can trust the outcomes. It’s a long process. It takes tremendous amounts of money; it takes tremendous amounts of talent and so on.

We have at each of these sites multi-disciplinary teams. What I mean by that is there’s no way to answer those four questions unless you have neurologists on board, pediatricians on board, neuroscientists on board, educators on board, cognitive neuroscientists on board and so forth and so on. All of the teams at the sites are going after these four questions. They’re replicating each other, but then they also extend at their own sites their own particular looks at specific problems they’re interested in.
I see 'Clacks' as a continuation of the above rather than as a stand-alone study. On the other hand, the Reading Recovery research is based on flawed design and the results are only replicated by trials using the same flawed design. None of the trials using 'Gold Standard', (or even silver/bronze/tin standard) criteria support the RR findings.

chew8
Posts: 4162
Joined: Sun Nov 02, 2003 6:26 pm

Re: Ben Goldacre's Review of the Year

Post by chew8 » Wed Dec 22, 2010 9:21 am

I agree about research on RR etc. in general, Yvonne, but what I've been trying to focus on in this thread is a very particular point raised by Geraldine in connection (I think) with a specific UK RR study: the administration of the British Ability Scales (BAS) reading test by people who presumably have a stake in the results.

Jenny C.

yvonne meyer
Posts: 736
Joined: Tue Oct 21, 2008 4:58 am
Location: Melbourne Australia

Re: Ben Goldacre's Review of the Year

Post by yvonne meyer » Wed Dec 22, 2010 10:26 pm

Jenny, I take your point which is that we can all be accused of having a stake in the results. I am happy to admit that I am convinced that systematic, synthetic phonics is the most effective method of teaching beginning reading and I take that bias with me when I view other methods.

I assume the research that you & Geraldine refer to is;
Early Intervention in Children with Reading Difficulties: An Evaluation of Reading Recovery and a Phonological Training
Sylva, K, & Hurry, J. (1996). Literacy, Teaching and Learning: An International Journal of Early Literacy, 2(2), 49–73.
The purpose of this study was to evaluate the effectiveness of two different interventions, Reading Recovery and Phonological Intervention. Their study included almost 400 children from seven English local authorities. Although the sample was diverse, inner city children were over represented. The schools included 22 Reading Recovery schools, 23 Phonological Intervention schools, and 18 control schools. The measures used included the British Ability Scale Word Reading, Neale Analysis of Reading, Clay’s Diagnostic Survey (five tasks), Assessment of Phonological Awareness, British Ability Scale Spelling, and background information on each child.
Regardless of my own bias, this study did, in fact, find that children made more progress with RR than they did with the 'Phonological Intervention' and no intervention at all.

The 'Phonological Intervention' appears to have been brief 1:1 training in (listening & speaking) phonemic awareness but does not appear to include phoneme/grapheme instruction.

The authors of the study have used the various tests to prove what they wanted to prove; that RR is more effective than either 'Phonological Intervention' or nothing at all. Of course, what they did not do was compare groups of students who had received initial synthetic phonics instruction in the first place to see how many of them needed remediation. Of the children who did need remediation, they did not compare RR with more intensive synthetic phonics instruction.

As scientists like Wheldall & Tunmer have informed us, when children receive ineffective initial instruction, many of them will require remediation, and remediation with RR is better than nothing.

I think the point that Geraldine was, perhaps (?), trying to make is that 'The Blob' accepts research unquestioningly if the research findings agree with the preferred philosophy, and ignores research that brings into doubt what 'The Blob' believes to be 'true'.

yvonne meyer
Posts: 736
Joined: Tue Oct 21, 2008 4:58 am
Location: Melbourne Australia

Re: Ben Goldacre's Review of the Year

Post by yvonne meyer » Wed Dec 22, 2010 10:34 pm

Slightly off the point, but for someone like myself with no formal training, I have found the DDOLL review & critique of various tests very helpful. They reviewed tests commonly used in Australia and don't mention the BAS, but some RRF'ers may find some of the information helpful.

http://www.maccs.mq.edu.au/ddoll/r_tests.htm

Example:
Burt Word Reading Test has a long history and is a measure of single word recognition. The version we employ is based on a standardisation carried out in the early 1980s in New Zealand by the New Zealand Council for Educational Research. While doubts may be expressed over the utility of reading single words in isolation, it remains a robust test especially when used as part of a battery of reading tests. Our experience with this test suggests that it now frequently overestimates reading age. This needs to be remembered when interpreting the results obtained with this test but it still provides a good reliable measure of relative reading gain over time. Note that the maximum reading age possible on the Burt is about 13 years.

Cambridge Contextual Reading Test, is a poor measure of reading ability and rather serves as a predictor of IQ, indeed it was developed as a measure of premorbid IQ.

Castles and Coltheart reading lists are good because they separate different types of reading, and they have some norms. They are good because they are free. They are not-so-good because the norms have been done by researchers and are published in journals, rather than the test being normed commercially. Also, they have no discontinue rule so a child has to suffer all items.

chew8
Posts: 4162
Joined: Sun Nov 02, 2003 6:26 pm

Re: Ben Goldacre's Review of the Year

Post by chew8 » Thu Dec 23, 2010 10:13 am

Hi Yvonne -

The study that I think Geraldine had in mind is much more recent than the 1996 Sylva and Hurry one that you mention. It was by Sue Burroughs-Lange and was published in 2006, I think. I've had a quick look for it but can't immediately find the reference. The nearest I can get is

http://www.ioe.ac.uk/Evaluation_of_Read ... 5-2006.pdf

There was also a follow-up study by Sue Burroughs-Lange a year later.

Jenny C.

Post Reply

Who is online

Users browsing this forum: No registered users and 5 guests